I'm working on an ETL process with Pentaho Data Integration (Spoon, before Kettle).
In the Modified Javascript step of Pentaho you can set a start, end and transform script. In the transform script you can write code that it will be executed only for each row, and from here I don't know how to access to data of the previous row (if it's possible).
I need access to the previous row because all rows are ordered by product, store and date (respectively), and the goal is to get the quantity on hand from the previous row and add the quantity sell or received on the current row (this would be the same product, same store but different date). I also need accessing to the previous row to compare the product and store of the current row with the previous row, because if someone of them changes I must to restart the field quantity_on_hand (I do it with a field of all columns named initial_stock).
On pseudocode would be something like this (if I hadn't the restriction of that the code written on the step is executed only for each row):
while(all_rows_processed()){
current_row.quantity_on_hand = current_row.initial_stock;
while(id_product_current_row == id_product_previous_row && id_store_current_row == id_store_previous_row){
current_row.quantity_on_hand = previous_row.quantity_on_hand + current_row.stock_variation;
}
}
This question related couldn't help me.
Any ideas to solve my problem would be appreciated.
May I ask you to reconsider Group By step? It seems suitable for your scenario.
If you sort the stream accordingly to your combination date/store/article, you can calculate cumulative sum for sell/received quantity. This way you can have a running total of inventory variation that would be reset on a group basis.
Also give a look both at this blog post and at the forum post it quotes.
I doubt you need to go to JavaScript for this. Check out the Analytic query step. That will allow you to bring a value from the previous row into the current.
The JavaScript step gives you tremendous flexibility, but if you can do it with the regular transform steps, it will typically be much faster.
use Analytic Query. By Using this Step u can access the previous / next record. Actually, not only prev and next record that you can read, but you can read N Rows Fordward or N Rows Back Wards.
Check the following URL for clearer expalanation :
http://wiki.pentaho.com/display/EAI/Analytic+Query
http://www.nicholasgoodman.com/bt/blog/2009/01/30/the-death-of-prevrow-rowclone/
Thanks for all, I've got the solution to my problem.
I've combined all your suggestions and I've used the Analytic Query, Modified Javascript and Group by steps.
Although the question wasn't very well formulated, the problem I had was to calculate the stock level on each row (there was one row for each product, date and store combination).
First (obviously later than sort rows by product_id, store_id and date ascending), I used the Analytic Query step to group by product_id and store_id, because with this step I've got a new field previous_date to identify the first row of each group (previous_date=null on the row of the group where date was the oldest).
Then I needed to calculate the quantity_on_hand of each group [product,store] at first row (first date of each group because it's sorted by date) because the initial_stock is different for each group. This is because of (sum(quantity_received) - sum(quantity sold)) != quantity_on_hand.
Finally (and the key was here), I used the Group by step like #andtorg suggested and do it as the next image shows.
This link that #andtorg suggested was very useful. It includes even two .ktr example files.
Thank you so much for help!
Related
This is the post table
And this is the comments table
There is a way to encrease the value "commentsNumber" for each comments that has "postId" = post to increase the value?
I hope I made myself understood in some way
The idea was to increase the value of "commentsNumber" and then fetch the data so as to show the number of comments on the post. Of course, if there's a better way to handle this, suggestions are welcome.
I specify that initially the "commentsNumber" column was not present, if someone knows how to do it through a query and you think it's better, tell me
When including sample data in your question, do not insert images! Instead you should include a markdown table of data (tableconvert.com makes this very easy) and/or the CREATE TABLE statements and INSERTS so we can quickly and easily reproduce your example/problem.
In the vast majority of cases, storing the redundant count of child records is considered premature optimisation (at best). RDBMSes can perform these simple tasks incredibly quickly and efficiently, as long as the data is indexed appropriately.
It should be calculated on the fly -
SELECT p.*, COUNT(c.id) AS commentsNumber
FROM posts p
LEFT JOIN comments c
ON p.id = c.postId
GROUP BY p.id
It is important that there is an index on comments.postId but it should already be there as it is a foreign key.
After some research i find this post what have an example about how to manage a complex array in Vue.js
https://forum.vuejs.org/t/vuex-best-practices-for-complex-objects/10143/2
I have create this jsbin (https://jsbin.com/jocezud/5/edit?html,js,output) to make some test but i´m stuck at this point. I want to make some totals based in the checked topics:
A) When the checkbox of each Topic is checked the price value must be copied to the subTotal value.
B) Also every user must have the userTotal updated with the sum of the subTotal of the checked Topics.
Here is a capture about my goal.
Thanks in advance
Finally i get it working with also a Final total and a Reset feature.
Here is the actual working version (https://jsbin.com/jocezud/12/edit?html,js,output)
Anyway i think there must be a better way to, in this case run some calculations of data in arrays when a array value change happens, right?
I am using Symfony2 to build up my page.
When I try to update a collection of forms (like described in the cookbook entry "How to Embed a Collection of Forms"), i get a collision of the indexes of the frontend and the indexes of the ArrayCollection in the backend.
I've got the relation User <-> Address (OneToMany). A user wants to create/update/delete his addresses, therefore he can add / delete in the frontend with the help of the javascript part new address elements. He does the following:
(1) Adds new address (has index: 0)
(2) Adds new address (has index: 1) and instantly removes this address again
(3) Adds new address (has index: 2).
When he clicks on save button, the following code saves/updates the user (and its addresses):
$this->em->persist($user);
$this->em->flush();
New addresses for example are then correctly persisted to the database.
Now the user wants to update the address e.g. with index 0.
When he now clicks on the save button, it updates the adress with "index 0", but at the same time, it adds again the address with "index 2" to the database (object).
To better understand the problem, i've drawn a small illustration (handmade, sorry for my bad art skills):
Now , i've got two times the address with "index 1" within my object / database.
I know why this happens, it's because the first "index 1" address gets mapped to the ArrayCollection element "number 1", and the second gets mapped to "number 2 "(because of the frontend name "index 2").
You can say: "it just fills up the addresses, until it reaches the frontend index in the backend"..
But how can I fix this behaviour ?
Site note:
This behaviour occurs using ajax requests, because if you would reload the page after clicking "save button", it would reindex the addresses in the frontend correctly with the indexes in the backend.
My suggestion to handle that situation:
Reindexing the frontend indexes after clicking save with the server side
indexes. Is this a clear / the only solution for my problem?
Yes, this is problem of Symfony form collection and it has no easy solution imho. But I have to ask why don't you do exactly the same thing what page refresh does? You can refresh only html snippet with collection. HTML code for snippet can come from server-side. Back to your question - yes, reindexing is good solution until you do not want to try write custom collection type on your own.
symfony/symfony/issues/7828
There is similar problem with validating in collection - symfony/symfony/issues/7468.
Well I think default collection type and the tutorial in Symfony docs has the some drawbacks. Hope that's help.
I have come round this issue on the client side by modifying the Javascript/Jquery code given in the Symfony Documentation.
Instead of numbering the new elements by counting the sub-elements, I am looking at the last element's id and extracting its index with a regular expression.
When adding an element, I am incrementing the last index by 1. That way, I never use the same index.
Here is my code :
// Initializing default index at 0
var index = 0;
// Looking for collection fields in the form
var $findinput = $container.find(':input');
// If fields found then looking for last existing index
if ( $findinput.length > 0 ) {
// Reading id of last field
var myString = $findinput.last().attr('id')
// Setting regular expression to extract number from id containing letters, hyphens and underscores
var myRegex = /^[-_A-Za-z]+([0-9]+)[-_A-Za-z]*$/
// Executing regular expression on last collection field id
var test = myRegex.exec(myString);
// Extracting last index and incrementing by 1
if (test.length > 0) index = parseInt(test[1]) + 1;
}
I ran into this problem a couple of times during the past two years. Usually, following the Symfony tutorial How to Embed a Collection of Forms does the job just fine. You need to do a little bit javascript coding to add the "edit/update" functionality, but other than that - you should be just fine using this approach.
If, on the other hand, you have a really complex form which uses AJAX to validate/save/calculation/business logic/etc, I've found it's usually a better to store the final data into an array in the session. After submitting the form, inside the if($form->isValid()){...} block, you would have
$collection = new ArrayCollection($mySessionPlainArray);
$user->setAddress($collection);
I would like to warn you to be careful with the serialization of your data - you might get some awkward exceptions or misbehavior if you're using entities (see my question).
I'm sorry I can't provide more code, but the solution to this problem sometimes is quite complex.
I am making a survey in Qualtrics. This survey has a repeating question with six answer choices. The six choices are randomized (in the standard way, no javascript). The question is being repeated using loop&merge, which works great because it's the same question structure over and over (36 times), but I can use the field function to adjust the question and answers for every iteration.
However, one problem I am running into is that Qualtrics does not (as standard) support the recording of the randomization data in the results - i.e. how it has randomized the six answer choices in each iteration. When I use the 'Export Randomized Viewing Order data' function when downloading results, it only shows the answer order of the last time it asked the question. So it seems that this is a value that gets overwritten after each iteration.
So now I'm looking to record the answer order for each iteration through javascript. However, I haven't found a function that gives the order answer (after randomization). I have consulted the Qualtrics javascript API and found some functions that seem promising, such as getChoices (), but when I try this all I get back is the order of answers without randomization (i.e. just 1,2,3,4,5,6).
Does anyone know a way to record the randomized choice order for each iteration, using javascript or otherwise?
I found a different way to record the loop and merge randomization order.
Create an embedded data field in survey flow. Here we will record the randomization order. I will call the field rand_order.
Add a loop and merge field with a unique number to identify each loop (e.g. 1, 2, 3, 4, 5, ..., n).
Then add the next javascript to any page of the looped block.
//*Place Your Javascript Below This Line*/
var questionText = "${lm://Field/1}"; // "${lm://Field/1}" will actually evaluate to
//whatever is Field 1 in the current Loop & Merge loop.
// You can do this with embedded data too, as seen in the next line
var order = "${e://Field/rand_order}" + "|" + questionText; // gets the value of the embedded data
// field "rand_order", concatenates the current loop's identifier to it,
//and stores that as a variable
Qualtrics.SurveyEngine.setEmbeddedData('rand_order', order); // updates the
//embeddeddata field "rand_order" to be our order variable, which has the current loop's
//identifier attached, effectively constructing a string of numbers representing the order
You will get a column with the name "rand_order" filled with "1|5|23|2...|n". You can change the separator to make more compatible with whatever script you are using to manipulate data.
Qualtrics already records this information for you. It's just a matter of explicitly asking for it when you download your data. Number 5 on this page has more info, but I'll recount the important bits:
In the “Data & Analysis” tab, click “Export & Import” and then “Export Data”.
In the “Download Data Table” window click “More Options”.
Check the box for “Export viewing order data for randomized surveys”.
I think the thing here is to look at the order of choices in the DOM. Qualtrics provides the getChoiceContainer() method to get the div containing the choices. Here's a snippet I wrote and minimally tested:
//get the div containing the choices, then get all input child elements of that div
var choices = this.getChoiceContainer().getElementsByTagName("input");
//initialize an array for the IDs of the choices
var choiceIDs = []
//add the ID of each choice to the array
for (var i=0; i < choices.length; i++) {
choiceIDs.push(choices[i].id);
}
//get the current choice order from embedded data and add this loop to it.
//Add a | to distinguish between loops.
var choiceOrder = "${e://field/choiceorder}" + choiceIDs.toString() + "|";
//set the embedded data with the new value
Qualtrics.SurveyEngine.setEmbeddedData("choiceorder", choiceOrder);
A couple of notes/caveats:
I only tested this on a basic multiple choice question with radio buttons. It may need to be adjusted for different question types.
I also just got the IDs of the question choices. You could probably modify it pretty easily to get other information, like the label of the choice, or the numeric value it corresponds to.
I have an ajax function which call a servlet to get list of products from various webservices, the number of products can go up to 100,000. I need to show this list in a html table.
I am trying to provide users an interface to filter this list based on several criteria. Currently I am using a simple jQuery plugin to achieve this, but I found it to hog memory and time.
The Javascript that I use basically uses regex to search and filter rows matching the filtering criteria.
I was thinking of an alternate solution wherein I filter the JSON array returned by my servlet and bind the html table to it. Is there a way to achieve this, if there is, then is it more efficient than the regex approach.
Going through up to 100,000 items and checking if they meet your criteria is going to take a while, especially if the criteria might be complex (must be CONDO with 2 OR 3 bedrooms NOT in zip code 12345 and FIREPLACE but not JACUZZI).
Perhaps your servlet could cache the data for the 100,000 items and it could do the filtering, based on criteria posted by the user's browser. It could return, say, "items 1-50 of 12,456 selected from 100,000" and let the user page forward to the next 50 or so, and even select how many items to get back (25, 50, all).
If they select "all" before narrowing down the number very far, then a halfway observant user will expect it to take a while to load.
In other words, don't even TRY to manage the 100,000 items in the browser, let the server do it.
User enters filter and hits
search.
Ajax call to database, database has indexes on appropriate
columns and the database does the filtering.
Database returns result
Show result in table. (Probably want it to be paged to
only show 100-1000 rows at a time
because 100,000 rows in a table can
really slow down your browser.
Edit: Since you don't have a database, the best you're going to be able to do is run the regex over the JSON dataset and add results that match to the table. You'll want to save the JSON dataset in a variable in case they change the search. (I'm assuming that right now you're adding everything to the table and then using the jquery table plugin to filter it)
I'm assuming that by filtering you mean only displaying a subset of the data; and not sorting.
As you are populating the data into the table add classes to each row for everything in that row you want to filter by. e.g.:
<tr class="filter1 filter2 filter3">....
<tr class="filter1 filter3">....
<tr class="filter2">....
<tr class="filter3">....
Then when you want to apply a filter you can do something like:
$('TR:not(.filter1)').hide();
I agree with Berry that 100000 rows in the browser is bit of a stretch, but if there's anything that comes close to handling something of that magnitude then it's jOrder. http://github.com/danstocker/jorder
Create a jOrder table based on your JSON, and add the most necessary indexes. I mean the ones that you must at all cost filter by.
E.g. you have a "Name" field with people's names.
var table = jOrder(json)
.index('name', ['Name'], { sorted: true, ordered: true });
Then, for instance, this is how you select the records where the Name field starts with "John":
var filtered = table.where([{ Name: 'John' }], { mode: jOrder.startof, renumber: true });
Later, if you need paging in your table, just feed the table builder a filtered.slice(...).
If you're getting back xml, you could just use jQuery selection
$('.class', context) where context is your xml response.
From this selection, you could just write the xml to the page and use CSS to style it. That's where I'd start at first, at least. I'm doing something similar in one of my applications, but my dataset is smaller.
I don't know what you mean by "bind"? You can parse JSON and then use for loop (or $.each()) to populate ether straight HTML or by using grid plugin's insert/add