I am trying to create an array, that will be MAASSSIVVEE...i read somewhere that javascript can create an array of up to 4.xx Billion or so. The array i am trying to create will likely be in the quadrillions or higher. I don't even know where to go from here. I am going to assume that JS is not the proper solution for this, but i'd like to give it a try...it is for client side, and i would prefer not to bog down my server with this if there are multiple people using it at once. Also, not looking to learn a new language as i am just getting into JS, and code in general.
could i possibly use setTimeout(),0 breaks in the totalcombos function? Time is not really an issue, i wouldn't mind if it took a few minutes to calculate, but right now it just crashes.
i have tried this using a dedicated worker, but it still crashes the host. the worker code is what i am posting, as the host code is irrelevant to this question (it only compiles the original objects and posts them, then receives the messages back).
The code: (sorry for the mess...im coding noob and just an enthusiast)
onmessage = function(event){
//this has been tested on the very small sample size below, and still runs out of memory
//all the objects in these first arrays are formatted as follows.
// {"pid":"21939","name":"John Smith","position":"QB","salary":"9700","fppg":"23"}
// "PID" is unique to each object, everything else could appear in another object.
// There are no repeated objects.
var qbs = **group of 10 objects like above**
var rbs = **group of 10 objects like above**
var wrs = **group of 10 objects like above**
var tes = **group of 10 objects like above**
var ks = **group of 10 objects like above**
var ds = **group of 10 objects like above**
//This code works great and fast with small sets. ie (qbs, rbs, wrs)
function totalcombos() {
var r = [], arg = arguments, max = arg.length-1;
function helper(arr, i) {
for (var j=0; j<arg[i].length; j++) {
var a = arr.slice(0); // clone arr
if(a.indexOf(arg[i][j]) != -1){
j++;
} else
a.push(arg[i][j]);
if (i==max) {
r.push(a);
} else
helper(a, i+1);
}
}
helper([], 0);
return r;
};
//WAY TOO BIG...commented out so as not to crash when run
//var tCom = totalcombos(qbs, rbs, wrs, tes, ks, ds);
//postMessage(tCom.length);
}
When the sets get to be larger like 50 objects in each, it just crashes as it is out of memory. I reduce the set with other code but it will still be very large. How would i fix it?
I am trying to create all the possible combinations and then go through and reduce from there based on total salary of each group.
When working with data, regardless of language or platform, its usually best practice to only load the data that's otherwise you encounter errors or bottlenecks etc. as you are finding.
If your data is being stored somewhere like a Database, a JSON file, or a Web Service or an API etc. (anything basically), you'd be better of searching that set of data to retrieve only that which you need, or to at least reduce the size of the Array data your're trying to traverse.
As an analogy, if you're trying to load the whole internet into memory on a PC with only 2GB of RAM, you're going to have a really bad time. :)
I am trying to get the max value of only two teams, there are several teams, but In my app I am trying to return the max value for the teams that are playing. They are set in a data attribute that I can access via jQuery. Below is my max variable.
var max = this.collection.max(function(player){
return player.get('points');
});
I tried using the where function after this.collection and after player just to test it, and obviously it isn't valid, I feel like the answer is right in front of me, I've been through the docs, but I am not good enough with backbone.js yet to realize what I need to do here.
So what I am asking is how can I filter this max function to be limited to only two teams.
Lakers
Heat
Kings
The teams that are playing are Lakers and Heat set in the data-attribute, in the example below I am using a static string for the teams, I will substitute them for variables later, but I am more focused on the syntax. I am not sure how to do this, just some bad examples below.
//Bad example #1
var max = this.collection.where({teams: 'Lakers', teams: 'Heat'}).max(function(player) {
return player.get('points');
})
//Bad example #2
if(this.collection.where({teams: 'Lakers'}) || this.collection.where({teams: 'Heat'}))
{
var max = this.collection.max(function(player){
return player.get('points');
});
}
I don't think the examples above are valid or technically make sense to the browser, but hopefully they help you understand what I am going for?
Thanks, ask questions if you need to, this should be pretty easy.
Assuming teams is the property of player then try
var max = this.collection.max(function (player) {
return player.get('teams') == 'Lakers' || player.get('teams') == 'Heat' ? player.get('points') : 0;
});
Please forgive me if this is answered on SO somewhere already. I've searched, and it seems as though this is a fairly specific case.
Here's an example of the JSON (NOTE: this is very stripped down - this is dynamically loaded, and currently there are 126 records):
var layout = {
"2":[{"id":"40","attribute_id":"2","option_id":null,"design_attribute_id":"4","design_option_id":"131","width":"10","height":"10",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"},
{"id":"41","attribute_id":"2","option_id":"115","design_attribute_id":"4","design_option_id":"131","width":"2","height":"1",
"repeat":"0","top":"0","left":"0","bottom":"4","right":"2","use_right":"0","use_bottom":"0","apply_to_options":"0"},
{"id":"44","attribute_id":"2","option_id":"118","design_attribute_id":"4","design_option_id":"131","width":"10","height":"10",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"}],
"5":[{"id":"326","attribute_id":"5","option_id":null,"design_attribute_id":"4","design_option_id":"154","width":"5","height":"5",
"repeat":"0","top":"0","left":"0","bottom":"0","right":"0","use_right":"0","use_bottom":"0","apply_to_options":"0"}]
};
I need to match the right combination of values. Here's the function I currently use:
function drawOption(attid, optid) {
var attlayout = layout[attid];
$.each(attlayout, function(k, v) {
// d_opt_id and d_opt_id are global scoped variable set elsewhere
if (v.design_attribute_id == d_att_id
&& v.design_option_id == d_opt_id
&& v.attribute_id == attid
&& ((v.apply_to_options == 1 || (v.option_id === optid)))) {
// Do stuff here
}
});
}
The issue is that I might iterate through 10-15 layouts (unique attid's), and any given layout (attid) might have as many as 50 possibilities, which means that this loop is being run A LOT.
Given the multiple criteria that have to be matched, would an AJAX call work better? (This JSON is dynamically created via PHP, so I could craft a PHP function that could possibly do this more efficently),
or am I completely missing something about how to find items in a JSON object?
As always, any suggestions for improving the code are welcome!
EDIT:
I apologize for not making this clear, but the purpose of this question is to find a way to improve the performance. The page has a lot of javascript, and this is a location where I know that performance is lower than it could be.
First and foremost you should measure and only act if there is a real performance concern. You need exact numbers like 200ms or 80% time is spent there. "It runs a lot" doesn't mean anything. The browser can loop very fast.
You can improve constant factors as others mentioned, like using a native for loop instead of jQuery.each. Chaching global variables won't help you too much in this case.
If you really want to improve efficency you should find a better algorithm than O(n). Assuming that you only use this data for finding elements matching a certain criteria you can use JS objects as hashes to achive a O(1) performance.
Just an example for you specific case:
var layout = {
"2": { "2,,4,131,10,0": ["40", "93"], "2,115,4,131,0": ["41"] },
"4": { ... },
...
};
It's fairly easy to generate this output in php, and then you just use a lookup to find ids matching your particular criteria.
IMHO, a simple hashmap index will probably work best. This does require you to loop over the data ahead of time, but the index can be easily appended to and cached.
Once the index is generated, this should be O(1) for lookups, and will handle multiple entries per key.
var layout = {
"2":[[...], ...],
"5":[[...], ...]
};
var index = {};
function makeKey(data) {
return data.join('_');
}
for(var l in layout) {
var cur = layout[l];
for(var i in cur) {
var item = cur[i];
var key = makeKey([item.p1, item.p2, ...]);
index[key] = index[key] || [];
index[key].push(item);
}
}
function find(attid, optid) {
var key = makeKey([attid, optid, 1, d_att_id, ...]);
return index[key]; //this is an array of results
}
My first suggestion would be to stop using $.each if you want to squeeze out every bit of performance you can. jQuery.each does a bit more than a traditional loop. Take a look at this jsFiddle with your browser's debugger running (i.e. Safari's/Chrome's web developer tools) and step through the fiddle until execution fully returns from jQuery.
For reference, the fiddle's code is:
var myArr = [{"foo":"bar"},{"answer":42}];
debugger;
$.each(myArr, function(k, v) {
console.log(k + ': ');
console.dir(v);
});
Now run through the second version:
var myArr = [{"foo":"bar"},{"answer":42}],
i, j;
debugger;
for (i = 0, j = myArr.length; i < j; i += 1) {
console.log('myArr[' + i + ']: ');
console.dir(myArr[i]);
}
Notice that there are far fewer operations being executed in the second version. So that's a tiny bit of performance gain.
Second, eliminate as many lookups outside of the local scope as you can. Even if you cache a reference to your global variables (boo!) then you can save a lot of time given that this loop will be executed possibly hundreds of times. So instead of:
function foo(a, b) {
if (a === globalA && b === globalB) {
// Do stuff
}
}
You'd do:
function foo(a, b) {
var gA = globalA,
gB = globalB;
if (a === gA && b === gB) {
// Do stuff
}
}
As for pairing down the conditional based on the object members, I'm not seeing much else that could be improved. The object properties you are checking are top level, and you're looking at local instances of each object (so the scope chain lookups are short).
Without knowing more about how this is actually supposed to work, those are the best recommendations I can make. However, I can make the guess that your idea of starting with simpler JSON data would be a big improvement. If you know what the layout, and its constraints, is, then requesting the specific details from the server would mean you don't have to check so many conditions. You could simply ask the server for the details that you actually need to implement and loop through those (to update the DOM or whatever).
I see that the you are searching by
5 fields: v.design_attribute_id,v.design_option_id,v.attribute_id,v.apply_to_options,v.option_id.
What you could do is add an extra field to the objects called "key" that is a composite of the values in those fields.
Here's an example
{
"key": "4_131_2_0_0" //i picked 0 to represent null, but you can pick any number
"id": "40",
"attribute_id": "2",
"option_id": null,
"design_attribute_id": "4",
"design_option_id": "131",
"width": "10",
"height": "10",
"repeat": "0",
"top": "0",
"left": "0",
"bottom": "0",
"right": "0",
"use_right": "0",
"use_bottom": "0",
"apply_to_options": "0"
}
Note though that you must normalize the length of each value.
Meaning that if one objects optionId is 1 and another object optionID is 566 you must represent
the first optionId as 001 in the key string.
With this field you can then sort the array on the server side before returning it to the client.
Then you can use a binary search to find the values on the client.
Using the binary search implementation located here
http://www.nczonline.net/blog/2009/09/01/computer-science-in-javascript-binary-search/
Your search function would look something like
function drawOption(attid, optid) {
var attlayout = layout[attid];
var needle = d_att_id + "_" + d_opt_id + "_" + attid + "_" + optid; //remember to normalize length if you have to
var idx = binarySearch(attlayout,needle);
var item;
if(idx !== -1){
item = attlayout[idx];
//do something
}
}
Another method you can try using this composite key idea is to have the server return
the layout objects in one big object mapped by
attid,v.design_attribute_id,v.design_option_id,v.attribute_id,v.apply_to_options,v.option_id
Then you can look up in O(1) time.
It would look something like
function drawOption(attid, optid) {
var needle = attid + "_" + d_att_id + "_" + d_opt_id + "_" + attid + "_" + optid; //remember to normalize length if you have to
var item = layout[needle];
if(typeof item !== "undefined"){
//do something
}
}
When trying to improve your code, it is always better to check which functions are taking time using firebug profiling. You can either profile by clicking on profile button in firebug's console panel and then run your code or using firebug's profiling commands in your code
From the code that you have given, only a few improvement points can be given.
$.each is slow compared to native looping solutions. For the best
looping solutions, check out this JsPref test
It would be better to change the JSON to use arrays instead of object literals. It is said to be more faster to retrieve values.
I have experience to such issue before, my js array of objects consist of 8 thousands record and more.
My experience is not about the performance, but the readability, maintainability, scalable of the codes.
hence I developed an JS Object Query Library 2 years ago: JSOQL
http://code.google.com/p/jsoql/
It works like SQL to allow you query your js array of objects with syntax similar to SQL.
The example usage is something like this, I cant really remember, but you can download the example usage in the download tab.
new JSQOL().Select([field1, field2, field3 ...]).From({ ... }) .Where(fn) .Offset(int) .Limit(int) .Get();
Note:
{...} is your array of objects, or an object it self.
Hope it helps, you can send me message if you need more information.
It's not going to work everywhere but your problem sounds like something that can be done with webworkers
Another thing I would look at if you dont have webworkers is trying not to block the ui to long. If you can chunk it into bits of about 40ms and then setTimeout the next chunck for just a few ms later the user will have a more pleasant experience. This needs a little fiddling but users will start to notice stuff when something takes longer than somewhere between 50 and 100ms
Have you considered using the jQuery grep function?
jQuery grep
And jquery grep on json object array for an example.
Here is one technique that will probably yield better performance at the expense of using a bit more memory.
I'll leave my code examples simple just to illustrate the concept.
First, you'll want to pre-process your JSON data into some additional arrays that act as indexes. Here is an example of what the final arrays might look like after pre-processing:
var layouts_by_attribute = {
// attribute_id => array(layouts)
2: [40, 41, 44],
5: [326]
};
var layouts_by_design_attribute_id = {
// design_attribute_id => array(layouts)
4: [40, 41, 44, 326]
};
Finding a layout by attribute is now very quick:
function findByAttribute(attribute_id) {
return layouts = layouts_by_attribute[attribute_id];
}
function findByDesignAttribute(design_attribute_id) {
return layouts = layouts_by_design_attribute[design_attribute_id];
}
I have some JavaScript that I wrote in a pinch, but I think it could be optimized greatly by someone smarter than me. This code runs on relatively small objects, but it runs a fair amount of times, so its worth getting right:
/**
* Determine the maximum quantity we can show (ever) for these size/color combos
*
* #return int=settings.limitedStockThreshold
*/
function getMaxDefaultQuantity() {
var max_default_quantity = 1;
if (inventory && inventory.sizes) {
sizecolor_combo_loop:
for (var key in inventory.sizes) {
if (inventory.sizes[key].combos) {
for (var key2 in inventory.sizes[key].combos) {
var sizecolor_combo = inventory.sizes[key].combos[key2];
if (isBackorderable(sizecolor_combo)) {
//if even one is backorderable, we can break out
max_default_quantity = settings.limitedStockThreshold;
break sizecolor_combo_loop;
} else {
//not backorderable, get largest quantity (sizecolor_combo or max_default_quantity)
var qoh = parseInt(sizecolor_combo.quantityOnHand || 1);
if (qoh > max_default_quantity) {
max_default_quantity = qoh;
};
};
};
};
};
};
return Math.min(max_default_quantity, settings.limitedStockThreshold);
};
First, inventory is a object returned via JSON. It has a property inventory.sizes that contain all of the available sizes for a product. Each size has a property inventory.sizes.combos which maps to all of the available colors for a size. Each combo also has a property quantityOnHand that tells the quantity available for that specific combo. (the JSON structure returned cannot be modified)
What the code does is loop through each size, then each size's combos. It then checks if the size-color combo is backorderable (via another method). If it any combo is backorderable, we can stop because the default quantity is defined elsewhere. If the combo isn't backorderable, the max_default_quantity is the largest quantityOnHand we find (with a maximum of settings.limitedStockThreshold).
I really don't like the nested for loops and my handling of the math and default values feels overly complicated.
Also, this whole function is wrapped in a much larger jQuery object if that helps clean it up.
Have you considered using map-reduce? See a live example of a functional approach.
This particular example uses underscore.js so we can keep it on a elegant level without having to implement the details.
function doStuff(inventory) {
var max = settings.limitedStockThreshold;
if (!(inventory && inventory.sizes)) return;
var quantity = _(inventory.sizes).chain()
.filter(function(value) {
return value.combos;
})
.map(function(value) {
return _(value.combos).chain()
.map(function(value) {
return isBackorderable(value) ? max : value.quantityOnHand;
})
.max().value();
})
.max().value();
return Math.min(quantity, max);
}
As for an explanation:
We take the inventory.sizes set and remove any that don't contain combos. We then map each size to the maximum quantity of it's colour. We do this mapping each combo to either its quantity or the maximum quantity if backordable. We then take a max of that set.
Finally we take a max of set of maxQuantities per size.
We're still effectily doing a double for loop since we take two .max on the set but it doesn't look as dirty.
There are also a couple of if checks that you had in place that are still there.
[Edit]
I'm pretty sure the above code can be optimized a lot more. but it's a different way of looking at it.
Unfortunately, JavaScript doesn't have much in the way of elegant collection processing capabilities if you have to support older browsers, so without the help of additional libraries, a nested loop like the one you've written is the way to go. You could consider having the values precomputed server-side instead, perhaps cached, and including it in the JSON to avoid having to run the same computations again and again.
I have a cpu intensive task that I need to run on the client. Ideally, I'd like to be able to invoke the function and trigger progress events using jquery so I can update the UI.
I know javascript does not support threading, but I've seen a few promising articles trying to mimic threading using setTimeout.
What is the best approach to use for this? Thanks.
Basically, what you want to do is to divide the operation into pieces. So say you have 10 000 items you want to process, store them in a list and then process a small number of them with a small delay between each call. Here's a simple structure you could use:
function performTask(items, numToProcess, processItem) {
var pos = 0;
// This is run once for every numToProcess items.
function iteration() {
// Calculate last position.
var j = Math.min(pos + numToProcess, items.length);
// Start at current position and loop to last position.
for (var i = pos; i < j; i++) {
processItem(items, i);
}
// Increment current position.
pos += numToProcess;
// Only continue if there are more items to process.
if (pos < items.length)
setTimeout(iteration, 10); // Wait 10 ms to let the UI update.
}
iteration();
}
performTask(
// A set of items.
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o'],
// Process two items every iteration.
2,
// Function that will do stuff to the items. Called once for every item. Gets
// the array with items and the index of the current item (to prevent copying
// values around which is unnecessary.)
function (items, index) {
// Do stuff with items[index]
// This could also be inline in iteration for better performance.
});
Also note that Google Gears has support to do work on a separate thread. Firefox 3.5 also introduced its own workers that do the same thing (although they follow the W3 standard, while Google Gears uses its own methods.)
I had a similar problem to solve recently where i needed to keep my UI thread free while crunching some data to display.
I wrote a library Background.js to handle a few scenarios: a sequential background queue (based on the WorkerQueue library), a list of jobs where each is called on every timer, and an array iterator to help break up your work into smaller chunks. Examples and code here: https://github.com/kmalakoff/background
Enjoy!
If you can enforce the browser to be used, or you otherwise already know it to be a new version of Firefox, you could use the new WebWorkers from Mozilla. It allows you to spawn new threads.
Depending on what your requirements are, you may get off easily by using Gears. Gears supports threads, which could do what you want.
As you mentioned, setTimeout is the other option. Depending on the type of your task, you can hand off each iteration of a loop to a separate setTimeout call with some spacing in between, or you may need to separate pieces of your main algorithm into separate functions which can be called one by one in a same manner as you'd call each iteration.
Great answer Kevin! I wrote something similar a few years back, though less sophisticated. Source code is here if anyone wants it:
http://www.leapbeyond.com/ric/jsUtils/TaskQueue.js
Anything with a run() method can be queued as a task. Tasks can re-queue themselves to perform work in chunks. You can prioritize tasks, add/remove them at will, pause / resume the entire queue, etc. Works well with asynchronous operations - my original use for this was to manage several concurrent XMLHttpRequests.
Basic usage is pretty simple:
var taskQueue = new TaskQueue();
taskQueue.schedule("alert('hello there')");
The header comments in the .js file provide more advanced examples.
Here is my solution to the problem, in case someone wants simple copy-pasteable piece of code:
var iterate = function (from, to, action, complete) {
var i = from;
var impl = function () {
action(i);
i++;
if (i < to) setTimeout(impl, 1);
else complete();
};
impl();
};
This is a very basic example of how to create threads in JavaScript.
Note that is is up to you to interrupt your thread functions (yeld instruction). If you want, you can use a setTimeout instead of my while loop to run the scheduler periodically.
Note also that this example only works with JavaScript version 1.7+ (firefox 3+)
you can try it here: http://jslibs.googlecode.com/svn/trunk/jseval.html
//// thread definition
function Thread( name ) {
for ( var i = 0; i < 5; i++ ) {
Print(i+' ('+name+')');
yield;
}
}
//// thread management
var threads = [];
// thread creation
threads.push( new Thread('foo') );
threads.push( new Thread('bar') );
// scheduler
while (threads.length) {
var thread = threads.shift();
try {
thread.next();
threads.push(thread);
} catch(ex if ex instanceof StopIteration) { }
}
The output is:
0 (foo) 0 (bar) 1 (foo) 1 (bar) 2 (foo) 2 (bar) 3 (foo) 3 (bar) 4 (foo) 4 (bar)
My strongest recommendation is to display a simple loading.gif during the operation. User often accepts some duration if they have been "told" that it could take some time.
Ajaxload - Ajax loading gif generator
use requestIdleCallback().
Then you will use the browsers idle time to process your task.
Sample found in:
https://developer.mozilla.org/en-US/docs/Web/API/Background_Tasks_API
there is a feature (experimental when I am writing):
https://developer.mozilla.org/en-US/docs/Web/API/Window/requestIdleCallback
seems not real multithread, but execute the task with low priority than the other (UI) tasks
so the solution can be:
make your task asynchronous
start your task with this
requestIdleCallback
loop using a setTimeout() with microtimes in your task, like between some main loop and his inner logic
the UI can update frequently in these timeouts
Seems this problem has been solved in node itself.
require child_process which is included in nodejs 0.10.4