Develop Reference

Weirdness when modifying massive json file/array in Node.js

I am working with a massive json file (almost 60 MB) that i am trying to remove all entries in it where volume = 0. The format of the array is
{
"date": 1424373000,
"high": 0.33,
"low": 225,
"open": 0.33,
"close": 225,
"volume": 0.999999,
"quoteVolume": 0.00444444,
"weightedAverage": 225
}
To do this I am using this code.
fs.readFile('JSONFiles/poloniexBTCDataFeb19|2015-July2|2018.json', function read(err, data) {
if (err) {
throw err;
}
rawdata = JSON.parse(data);
rawdata.forEach(function(val, index, array) {
if (rawdata[index].volume == 0) {
rawdata.splice(index, 1)
}
})
});
The problem with this is that it only deletes about half of the entries with this characteristic (60k/108k). The way I fixed this was to use a for loop which runs the code 9 times, which deletes them all, but this causes the code to take significantly longer cause the whole json file has about 360k entries and it has to check each one with that if statement. I was wondering if there was any way to do this where it actually deletes them all without having to use a for loop in this manner?
EDIT: I have realized that I do not need this code in the first place so nevermind but thanks for all the answers. I hope this helps someone else when they hit a simmilar issue.

The problem is, you are mutating the array rawdata. Let's take an example array [e1, e2, e3, e4] and the code,
var arr = ['e1', 'e2', 'e3', 'e4']
arr.forEach(function(elem, idx){
console.log('checking elem', elem);
if (elem === 'e2'){
arr.splice(idx, 1)
}
});
console.log('\nAfter iteration', arr);
As you can see, I am removing e2 when I encounter it. This affects the actual array and the element which is getting replaced for it, will not be inspected (as forEach iteration already visited element at that index). In the above code, e3 did not get checked. So, it is advised to not mutate array in forEach iteration.
You can do like,
rawdata.slice().forEach(function (val, index, array) {
if (rawdata[index].volume == 0) {
rawdata.splice(index, 1)
}
});
Here slice() will make a new array and mutating your original rawdata will not affect the iteration.

You are splicing the records, that might be taking time.
Instead of the forEach, try this:
var filteredData = rawdata.filter(function (val) {
return val.volume != 0
})

Your code is buggy and the error is quite common (iterating over an array while mutating it). The code is also very inefficient because for each element to delete will move all other elements by one place (the fact that you use splice doesn't mean the a loop is not done... there is still a loop behind the scenes to implement that function).
If you need to remove elements from an array in-place (i.e. you don't want to get a copy) a simple approach is using what I normally call a read-skip-write loop:
let wp = 0; // the "write pointer"
for (let x of data) {
if (keep(x)) data[wp++] = x;
}
data.length = wp; // trim unused space
PS: as a side note try to change your mindset about programming. If your first thought is that node is buggy then you're not going to get very far into coding. The reality is that bug is 99.99% of the times in your code... looking somewhere else won't make you a better programmer.

Programming optional ignorance

In Javascript what is the best way to handle scenarios when you have a set of arrays to perform tasks on sets of data and sometimes you do not want to include all of the arrays but instead a combination.
My arrays are labeled in this small snippet L,C,H,V,B,A,S and to put things into perspective the code is around 2500 lines like this. (I have removed code notes from this post)
if(C[0].length>0){
L=L[1].concat(+(MIN.apply(this,L[0])).toFixed(7));
C=C[1].concat(C[0][0]);
H=H[1].concat(+(MAX.apply(this,H[0])).toFixed(7));
V=V[1].concat((V[0].reduce(function(a,b){return a+b}))/(V[0].length));
B=B[1].concat((MAX.apply(this,B[0])-MIN.apply(this,B[0]))/2);
A=A[1].concat((MAX.apply(this,A[0])-MIN.apply(this,A[0]))/2);
D=D[1].concat((D[0].reduce(function(a,b){return a+b}))/(D[0].length));
S=S[1].concat((S[0].reduce(function(a,b){return a+b}))/(S[0].length));
}
It would seem counter-productive in this case to litter the code with tones of bool conditions asking on each loop or code section if an array was included in the task and even more silly to ask inside each loop iteration with say an inline condition as these would also slow down the processing and also make the code look like a maze or rabbit hole.
Is there a logical method / library to ignore instruction or skip if an option was set to false
All I have come up with so far is kind of pointless inline thing
var op=[0,1,1,0,0,0,0,0]; //options
var L=[],C=[],H=[],V=[],B=[],A=[],D=[],S=[];
op[0]&&[L[0]=1];
op[1]&&[C[0]=1,console.log('test, do more than one thing')];
op[2]&&[H[0]=1];
op[3]&&[V[0]=1];
op[4]&&[B[0]=1];
op[5]&&[A[0]=1];
op[6]&&[A[0]=1];
It works in that it sets only C[0] and H[0] to 1 as the options require, but it fails as it needs to ask seven questions per iteration of a loop as it may be done inside a loop. Rather than make seven versions of the the loop or code section, and rather than asking questions inside each loop is there another style / method?
I have also noticed that if I create an array then at some point make it equal to NaN rather than undefined or null the console does not complain
var L=[],C=[],H=[],V=[],B=[],A=[],D=[],S=[];
L=NaN;
L[0]=1;
//1
console.log(L); //NaN
L=undefined;
L[0]=1
//TypeError: Cannot set property '0' of undefined
L=null
L[0]=1
//TypeError: Cannot set property '0' of null
Am I getting warmer? I would assume that if I performed some math on L[0] when isNaN(L)===true that the math is being done but not stored so the line isn't being ignored really..

If I understand what you want I would do something like this.
var op = [...],
opchoice = {
//these can return nothing, no operation, or a new value.
'true': function(val){ /*operation do if true*/ },
'false': function(val){ /*operation do if false*/ },
//add more operations here.
//keys must be strings, or transformed into strings with operation method.
operation: function(val){
//make the boolean a string key.
return this[''+(val == 'something')](val);
}
};
var endop = [];//need this to prevent infinite recursion(loop).
var val;
while(val = op.shift()){
//a queue operation.
endop.push(opchoice.operation(val));
}
I'm sure this is not exactly what you want, but it's close to fulfilling the want of not having a ton of conditions every where.
Your other option is on every line do this.
A = isNaN(A) ? A.concat(...) : A;
Personally I prefer the other method.

It looks like you repeat many of the operations. These operations should be functions so at least you do not redefine the same function over and over again (it is also an optimization to do so).
function get_min(x)
{
return +(MIN.apply(this, a[0])).toFixed(7);
}
function get_max(x)
{
return +(MAX.apply(this, a[0])).toFixed(7);
}
function get_average(x)
{
return (x[0].reduce(function(a, b) {return a + b})) / (x[0].length);
}
function get_mean(x)
{
return (MAX.apply(this, x[0]) - MIN.apply(this, x[0])) / 2;
}
if(C[0].length > 0)
{
L = L[1].concat(get_min(L));
C = C[1].concat(C[0][0]);
H = H[1].concat(get_max(H));
V = V[1].concat(get_average(V));
B = B[1].concat(get_mean(B));
A = A[1].concat(get_mean(A);
D = D[1].concat(get_average(D));
S = S[1].concat(get_average(S));
}
You could also define an object with prototype functions, but it is not clear whether it would be useful (outside of putting those functions in a namespace).
In regard to the idea/concept of having a test, what you've found is probably the best way in JavaScript.
op[0] && S = S[1].concat(get_average(S));
And if you want to apply multiple operators when op[0] is true, use parenthesis and commas:
op[3] && (V = V[1].concat(get_average(V)),
B = B[1].concat(get_mean(B)),
A = A[1].concat(get_mean(A));
op[0] && (D = D[1].concat(get_average(D)),
S = S[1].concat(get_average(S)));
However, this is not any clearer, to a programmer, than an if() block as shown in your question. (Actually, many programmers may have to read it 2 or 3 times before getting it.)
Yet, there is another solution which is to use another function layer. In that last example, you would do something like this:
function VBA()
{
V = V[1].concat(get_average(V));
B = B[1].concat(get_mean(B));
A = A[1].concat(get_mean(A));
}
function DS()
{
D = D[1].concat(get_average(D));
S = S[1].concat(get_average(S));
}
op = [DS,null,null,VBA,null,null,...];
for(key in op)
{
// optional: if(op[key].hasOwnProperty(key)) ... -- verify that we defined that key
if(op[key])
{
op[key](); // call function
}
}
So in other words you have an array of functions and can use a for() loop to go through the various items and if defined, call the function.
All of that will very much depend on the number of combinations you have. You mentioned 2,500 lines of code, but the number of permutations may be such that writing it one way or the other will possibly not reduce the total number of lines, but it will make it easier to maintain because many lines are moved to much smaller code snippet making the overall program easier to understand.
P.S. To make it easier to read and debug later, I strongly suggest you put more spaces everywhere, as shown above. If you want to save space, use a compressor (minimizer), Google or Yahoo! both have one that do a really good job. No need to write your code pre-compressed.

MapReduce or Normal Queries? (Several emits per Map)

I have some different Map/Reduces functions that I use in my project. But one is a lot different than the others since it requires a loop in the map functionality. And for each count in the loop, I send an emit.
What I have is this scenario (in the user collection):
"channels" : [
"Channel 1",
"Channel 2",
],
What I want to do is to count how many users each channel has. So for that I could use db.users.find({channels: "Channel 1"}).count() but unfortunately channels are dynamic which means I don't know all the possible channel names and it may well change in the future.
So I thought that a Map/Reduce job would sit just perfect. But the problem is that the first Reduce job I wrote calculated wrong. And the other where I used a query for each emit, would come to take forever (more than 3 hours before the ssh session shut down).
So now I'm stuck and I need help, preferably I would want to have a Map/Reduce job since it's more nice than a bunch of queries which is kind of slow to run in real time.
This is the latest Map and Reduce functions I wrote:
var map = function() {
if(this.channels) {
for(var i = 0, imax = this.channels.length; i<imax; i++) {
emit(this.channels[i], 1);
}
}
}
var reduce = function (key, values) {
var result = 0;
values.forEach(function (value) {
// had this before: result += 1;
result = db.users.find({'channels' : key}).count();
});
return result;
}
I knew that the reduce function was horrific but I just tried the best I could think of. I think my logic may seem wrong but I can't find a good solution. Now I'm thinking of just doing a bunch of queries on every page load, but it will be slow as hell.
Please help! :)

In your scenario the reduce function should look like this:
var reduce = function (key, values) {
var result = 0;
values.forEach(function (value) {
result += value;
});
return result;
}
Let me know if it is still not working and if it does please give an example of input and (incorrect) output.

MR is sometimes a bit slow. So you might want to check out the new aggregation framework coming with 2.2 (which i think is s currently in release phase).
See: http://docs.mongodb.org/manual/applications/aggregation/
Additionally you might need to speed up the queries via using proper indices. Or adding a user count to the channels and increasing/decreasing when the user joining/leaving a channel. Depends on your app's use case of course.

Delete certain properties from an object without a loop

I'm trying to find the most efficient way of deleting properties from an object whose properties of commentCount and likeCount are both equal to 0. In the following example, Activity.3 would be removed. I don't want loop over them with a $.each() as that seems like it would take more time than necessary.
Activity = {
0 : {
'commentCount' : 10,
'likeCount' : 20
},
1 : {
'commentCount' : 0,
'likeCount' : 20
},
2 : {
'commentCount' : 10,
'likeCount' : 0
},
3 : {
'commentCount' : 0,
'likeCount' : 0
}
}
UPDATE
The circumstances of the creation of this object have come into question. To clarify, the Activity object can have up to 3 million properties inside of it. It's generated server side as an AJAX JSON response which is saved into memory. It includes more than just commentCount and likeCount that are used elsewhere, so I can't just not have the server not respond with things that have a 0 for both commentCount and likeCount.

Ah, the smell of premature optimization ^_^
How many of these objects do you have? How many do you need to clean them? If the answers are "less than 1 million" and "once or rarely", it's probably not worth to bother.
If you need a quick and optimal way, here is an idea: Create a new data structure and setters for the properties. Every time they are set, check whether they are both 0 and put them into a "kill" list.
That way, you just have to iterate over the kill list.
[EDIT] With several million objects and the need for a quick cleanup, a kill list is the way to go, especially when the condition is rare (just a few objects match).
Just write a function that updates these properties and make sure all code goes through it to update them. Then, you can manage the kill list in there.
Or you can simply delete the object as soon as the function is called to set both or the second property to 0.

I'm adding a second answer, because this solution comes from a completely different angle. In this solution, I attempt to find the fastest way to remove unwanted entries. I'm not aware of any way of doing this without loops, but I can think of several ways to do it with loops using jQuery as well as just raw javascript.
This jsperf shows all of the test cases side-by-side.
I'll explain each test and the caveats associated with each.
Raw JS: Slowest option. It looks like jQuery knows what they are doing with their $.each and $.map loops.
var obj;
for (var field in Activity) {
if (Activity.hasOwnProperty(field)) {
obj = Activity[field];
if (obj.commentCount === 0 && obj.likeCount === 0) {
delete Activity[field];
}
}
}
$.each: Tied for 2nd place. Cleaner syntax and faster than raw js loop above.
$.each(Activity, function(key, val){
if (val.commentCount === 0 && val.likeCount === 0) {
delete Activity[key];
}
});
$.map (Object version): Tied for 2nd place. Caveat: only supported in jQuery >= 1.6.
Activity = $.map(Activity, function(val, key){
if (val.commentCount === 0 && val.likeCount === 0) {
return null;
}
});
$.map (Array version): Fastest option. Caveat: You must use the $.makeArray function to convert your object to an array. I'm not sure if this is suitable for your needs.
var arrActivity = $.makeArray(Activity);
Activity = $.map(arrActivity, function(val, key){
if (val.commentCount === 0 && val.likeCount === 0) {
return null;
}
});
Conclusion
It looks like $.map is the fastest if you convert your object to an array using $.makeArray first.

This is just a starting-point, but how about something like this? Basically, it puts the Activities into buckets based on the sum of their likeCount and commentCount. It makes it easy to kill all of the Activities with no likes or comments, but I would assume there is a trade-off. I'm not sure how you are inserting these things and reading them. So, you'll have to decide if this is worth it.
var ActivityMgr = function(){
if(!(this instanceof ActivityMgr)){
return new ActivityMgr();
}
this.activities = {};
};
ActivityMgr.prototype.add = function(activity){
var bucket = parseInt(activity.commentCount, 10) + parseInt(activity.likeCount, 10);
if (this.activities[bucket] === undefined) {
this.activities[bucket] = [activity];
}
else {
this.activities[bucket].push(activity);
}
this.cleanse();
};
ActivityMgr.prototype.cleanse = function(){
this.activities[0] = [];
};
//Usage:
var activityMgr = new ActivityMgr();
activityMgr.add({
likeCount: 0,
commentCount: 10
});
EDIT:
After posting this, it becomes incredibly apparent that if you are adding items in this manner, you could just not add them if they have no likes or comments. My guess is that things aren't that simple, so please provide some detail as to how things are added and updated.

Iterate Through Nested JavaScript Objects - Dirty?

I have some JavaScript that I wrote in a pinch, but I think it could be optimized greatly by someone smarter than me. This code runs on relatively small objects, but it runs a fair amount of times, so its worth getting right:
/**
* Determine the maximum quantity we can show (ever) for these size/color combos
*
* #return int=settings.limitedStockThreshold
*/
function getMaxDefaultQuantity() {
var max_default_quantity = 1;
if (inventory && inventory.sizes) {
sizecolor_combo_loop:
for (var key in inventory.sizes) {
if (inventory.sizes[key].combos) {
for (var key2 in inventory.sizes[key].combos) {
var sizecolor_combo = inventory.sizes[key].combos[key2];
if (isBackorderable(sizecolor_combo)) {
//if even one is backorderable, we can break out
max_default_quantity = settings.limitedStockThreshold;
break sizecolor_combo_loop;
} else {
//not backorderable, get largest quantity (sizecolor_combo or max_default_quantity)
var qoh = parseInt(sizecolor_combo.quantityOnHand || 1);
if (qoh > max_default_quantity) {
max_default_quantity = qoh;
};
};
};
};
};
};
return Math.min(max_default_quantity, settings.limitedStockThreshold);
};
First, inventory is a object returned via JSON. It has a property inventory.sizes that contain all of the available sizes for a product. Each size has a property inventory.sizes.combos which maps to all of the available colors for a size. Each combo also has a property quantityOnHand that tells the quantity available for that specific combo. (the JSON structure returned cannot be modified)
What the code does is loop through each size, then each size's combos. It then checks if the size-color combo is backorderable (via another method). If it any combo is backorderable, we can stop because the default quantity is defined elsewhere. If the combo isn't backorderable, the max_default_quantity is the largest quantityOnHand we find (with a maximum of settings.limitedStockThreshold).
I really don't like the nested for loops and my handling of the math and default values feels overly complicated.
Also, this whole function is wrapped in a much larger jQuery object if that helps clean it up.

Have you considered using map-reduce? See a live example of a functional approach.
This particular example uses underscore.js so we can keep it on a elegant level without having to implement the details.
function doStuff(inventory) {
var max = settings.limitedStockThreshold;
if (!(inventory && inventory.sizes)) return;
var quantity = _(inventory.sizes).chain()
.filter(function(value) {
return value.combos;
})
.map(function(value) {
return _(value.combos).chain()
.map(function(value) {
return isBackorderable(value) ? max : value.quantityOnHand;
})
.max().value();
})
.max().value();
return Math.min(quantity, max);
}
As for an explanation:
We take the inventory.sizes set and remove any that don't contain combos. We then map each size to the maximum quantity of it's colour. We do this mapping each combo to either its quantity or the maximum quantity if backordable. We then take a max of that set.
Finally we take a max of set of maxQuantities per size.
We're still effectily doing a double for loop since we take two .max on the set but it doesn't look as dirty.
There are also a couple of if checks that you had in place that are still there.
[Edit]
I'm pretty sure the above code can be optimized a lot more. but it's a different way of looking at it.

Unfortunately, JavaScript doesn't have much in the way of elegant collection processing capabilities if you have to support older browsers, so without the help of additional libraries, a nested loop like the one you've written is the way to go. You could consider having the values precomputed server-side instead, perhaps cached, and including it in the JSON to avoid having to run the same computations again and again.