This question already has answers here:
Best way to iterate over an array without blocking the UI
(4 answers)
Closed 6 years ago.
In my application I have a very big array (arround 60k records). Using a for loop I am doing some operations on it as shown below.
var allPoints = [];
for (var i = 0, cLength = this._clusterData.length; i < cLength; i+=1) {
if (allPoints.indexOf(this._clusterData[i].attributes.PropertyAddress) == -1) {
allPoints.push(this._clusterData[i].attributes.PropertyAddress);
this._DistClusterData.push(this._clusterData[i])
}
}
When I run this loop the browser hangs as it is very big & in Firefox is shows popup saying "A script on this page may be busy, or it may have stopped responding. You can stop the script now, or you can continue to see if the script will complete". What can I do so that browser do not hang?
You need to return control back to the browser in order to keep it responsive. That means you need to use setTimeout to end your current processing and schedule it for resumption sometime later. E.g.:
function processData(i) {
var data = clusterData[i];
...
if (i < clusterData.length) {
setTimeout(processData, 0, i + 1);
}
}
processData(0);
This would be the simplest thing to do from where you currently are.
Alternatively, if it fits what you want to do, Web Workers would be a great solution, since they actually shunt the work into a separate thread.
Having said this, what you're currently doing is extremely inefficient. You push values into an array, and consequently keep checking the ever longer array over and over for the values it contains. You should be using object keys for the purpose of de-duplication instead:
var allPoints = {};
// for (...) ...
if (!allPoints[address]) { // you can even omit this entirely
allPoints[address] = true;
}
// later:
allPoints = allPoints.keys();
First of all, avoid the multiple this._clusterData[i] calls. Extract it to a variable like so:
var allPoints = [];
var current;
for (var i = 0, cLength = this._clusterData.length; i < cLength; i+=1) {
current = this._clusterData[i];
if (allPoints.indexOf(current.attributes.PropertyAddress) == -1) {
allPoints.push(current.attributes.PropertyAddress);
this._DistClusterData.push(current)
}
}
This should boost your performance quite a bit :-)
As others already pointed out, you can do this asynchronously, so the browser remains responsive.
It should be noted however that the indexOf operation you do can become very costly. It would be better if you would create a Map keyed by the PropertyAddress value. That will take care of the duplicates.
(function (clusterData, batchSize, done) {
var myMap = new Map();
var i = 0;
(function nextBatch() {
for (data of clusterData.slice(i, i+batchSize)) {
myMap.set(data.attributes.PropertyAddress, data);
}
i += batchSize;
if (i < clusterData.length) {
setTimeout(nextBatch, 0);
} else {
done(myMap);
}
})();
})(this._clusterData, 1000, function (result) {
// All done
this._DistClusterData = result;
// continue here with other stuff you want to do with it.
}.bind(this));
Try considering adding to the array asynchronously with a list, for a set of 1000 records at a time, or for what provides the best performance. This should free up your application while a set of items is added to a list.
Here is some additional information: async and await while adding elements to List<T>
Related
I have a json file that contains many objects and options.
Each of these kinds:
{"item": "name", "itemId": 78, "data": "Some data", ..., "option": number or string}
There are about 10,000 objects in the file.
And when part of item value("ame", "nam", "na", etc) entered , it should display all the objects and their options that match this part.
RegExp is the only thing that comes to my mind, but at 200mb+ file it starts searching for a long time(2 seconds+)
That's how I'm getting the object right now:
let reg = new RegExp(enteredName, 'gi'), //enteredName for example "nam"
data = await fetch("myFile.json"),
jsonData = await data.json();
let results = jsonData.filter(jsonObj => {
let item = jsonObj.item,
itemId = String(jsonObj.itemId);
return reg.test(item) || reg.test(itemId);
});
But that option is too slow for me.
What method is faster to perform such search using js?
Looking up items by item number should be easy enough by creating a hash table, which others have already suggested. The big problem here is searching for items by name. You could burn a ton of RAM by creating a tree, but I'm going to go out on a limb and guess that you're not necessarily looking for raw lookup speed. Instead, I'm assuming that you just want something that'll update a list on-the-fly as you type, without actually interrupting your typing, is that correct?
To that end, what you need is a search function that won't lock-up the main thread, allowing the DOM to be updated between returned results. Interval timers are one way to tackle this, as they can be set up to iterate through large, time-consuming volumes of data while allowing for other functions (such as DOM updates) to be executed between each iteration.
I've created a Fiddle that does just that:
// Create a big array containing items with names generated randomly for testing purposes
let jsonData = [];
for (i = 0; i < 10000; i++) {
var itemName = '';
jsonData.push({ item: Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15) });
}
// Now on to the actual search part
let returnLimit = 1000; // Maximum number of results to return
let intervalItr = null; // A handle used for iterating through the array with an interval timer
function nameInput (e) {
document.getElementById('output').innerHTML = '';
if (intervalItr) clearInterval(intervalItr); // If we were iterating through a previous search, stop it.
if (e.value.length > 0) search(e.value);
}
let reg, idx
function search (enteredName) {
reg = new RegExp(enteredName, 'i');
idx = 0;
// Kick off the search by creating an interval that'll call searchNext() with a 0ms delay.
// This will prevent the search function from locking the main thread while it's working,
// allowing the DOM to be updated as you type
intervalItr = setInterval(searchNext, 0);
}
function searchNext() {
if (idx >= jsonData.length || idx > returnLimit) {
clearInterval(intervalItr);
return;
}
let item = jsonData[idx].item;
if (reg.test(item)) document.getElementById('output').innerHTML += '<br>' + item;
idx++;
}
https://jsfiddle.net/FlimFlamboyant/we4r36tp/26/
Note that this could also be handled with a WebWorker, but I'm not sure it's strictly necessary.
Additionally, this could be further optimized by utilizing a secondary array that is filled as the search takes place. When you enter an additional character and a new search is started, the new search could begin with this secondary array, switching to the original if it runs out of data.
I have a doubt about how can be affected to speed the use of object data arrays, that is, use it directly or preasign them to simple vars.
I have an array of elements, for example 1000 elements.
Every array item is an object with 10 properties (for example).
And finally I use some of this properties to do 10 calculations.
So I have APPROACH1
var nn = myarray.lenght;
var a1,a2,a3,a4 ... a10;
var cal1,cal2,.. cal10
for (var x=0;x<nn;x++)
{ // assignment
a1=my_array[x].data1;
..
a10 =my_array[x].data10;
// calculations
cal1 = a1*a10 +a2*Math.abs(a3);
...
cal10 = (a8-a7)*4 +Math.sqrt(a9);
}
And APPROACH2
var nn = myarray.lenght;
for (var x=0;x<nn;x++)
{
// calculations
cal1 = my_array[x].data1*my_array[x].data10 +my_array[x].data2*Math.abs(my_array[x].data3);
...
cal10 = (my_array[x].data8-my_array[x].data7)*4 +Math.sqrt(my_array[x].data9);
}
Assign a1 ... a10 values from my_array and then make calculations is faster than make the calculations using my_array[x].properties; or the right is the opposite ?????
I dont know how works the 'js compiler' ....
The kind of short answer is: it depends on your javascript engine, there is no right and wrong here, only "this has worked in the past" and "this don't seem to speed thing up no more".
<tl;dr> If i would not run a jsperf test, i would go with "Cached example" 1 example down: </tl;dr>
A general rule of thumb is(read: was) that if you are going to use an element in an array more then once, it could be faster to cache it in a local variable, and if you were gonna use a property on an object more then once it should also be cached.
Example:
You have this code:
// Data generation (not discussed here)
function GetLotsOfItems() {
var ret = [];
for (var i = 0; i < 1000; i++) {
ret[i] = { calc1: i * 4, calc2: i * 10, calc3: i / 5 };
}
return ret;
}
// Your calculation loop
var myArray = GetLotsOfItems();
for (var i = 0; i < myArray.length; i++) {
var someResult = myArray[i].calc1 + myArray[i].calc2 + myArray[i].calc3;
}
Depending on your browser (read:this REALLY depends on your browser/its javascript engine) you could make this faster in a number of different ways.
You could for example cache the element being used in the calculation loop
Cached example:
// Your cached calculation loop
var myArray = GetLotsOfItems();
var element;
var arrayLen = myArray.length;
for (var i = 0; i < arrayLen ; i++) {
element = myArray[i];
var someResult = element.calc1 + element.calc2 + element.calc3;
}
You could also take this a step further and run it like this:
var myArray = GetLotsOfItems();
var element;
for (var i = myArray.length; i--;) { // Start at last element, travel backwards to the start
element = myArray[i];
var someResult = element.calc1 + element.calc2 + element.calc3;
}
What you do here is you start at the last element, then you use the condition block to see if i > 0, then AFTER that you lower it by one (allowing the loop to run with i==0 (while --i would run from 1000 -> 1), however in modern code this is usually slower because you will read an array backwards, and reading an array in the correct order usually allow for either run-time or compile-time optimization (which is automatic, mind you, so you don't need to do anything for this work), but depending on your javascript engine this might not be applicable, and the backwards going loop could be faster..
However this will, by my experience, run slower in chrome then the second "kinda-optimized" version (i have not tested this in jsperf, but in an CSP solver i wrote 2 years ago i ended caching array elements, but not properties, and i ran my loops from 0 to length.
You should (in most cases) write your code in a way that makes it easy to read and maintain, caching array elements is in my opinion as easy to read (if not easier) then non-cached elements, and they might be faster (they are, at least, not slower), and they are quicker to write if you use an IDE with autocomplete for javascript :P
Respected ppl ....
This is my node.js code ...
https://gist.github.com/SkyKOG/99d47dbe5a2cec97426b
Im trying to parse the data of our exam results ...example ...
http://www.vtualerts.com/results/get_res.php?usn=1MV09IS002&sem=7
Im getting the results ...and i am traversing back for previous seems too ...
All works but traversing back happens in random ... prolly something wrong with the loops ...
json.results = [];
var output = '';
var k = response.query.results.body.div.div[0].table[1].tr.length;
for (var j = 1; j < k; j++) {
for (var i = 0; i <= 5; i++) {
var result_obj = {};
result_obj.subjects = [];
for (key in response.query.results.body.div.div[0].table[1].tr[j].td[i]) {
if (typeof response.query.results.body.div.div[0].table[1].tr[j].td[i].em === "undefined") {
continue;
}
var subject_obj = {};
output += "Subject : " + response.query.results.body.div.div[0].table[1].tr[j].td[i].em + " " + "\n";
var subtext = response.query.results.body.div.div[0].table[1].tr[j].td[i].em + " " + "\n";
subject_obj.subjectname = subtext.replace(/[(].*[)]/, "").trim();
result_obj.subjects.push(subject_obj);
console.log(subject_obj);
break;
}
console.log(result_obj.subjects);
I presume there is something like async concepts which need to implemented correctly to make the reordering of sems in correct order ...
And to get the JSON in this format ...
https://gist.github.com/SkyKOG/3845d6a94cea3b744296
I dont think im pushing the created objects at the right scope ...
Kindly help in this regard .... thank you ...
(I'll answer the ordering part. Suggest making the JSON issue a separate question to fit in with the Q&A format.)
When you make the HTTP request in your code (see the line below) you're bringing a varying delay into the order that responses are executed
new YQL.exec(queryname, function (response) {
You need to track the order of the requests yourself, or use a library to do it for you.
Code it yourself
In order to get around that you need something that keeps track of the original order of the requests. Because of the way closures work you can't just increment a simple counter because it'll be changed in the global scope as your loop progresses. The idiomatic way to solve this is by passing the counter into an immediately executed function (as a value type)
e.g.
var responseData = [];
for ( var i = 0; i < 100; i++ ){
(function(){
...
// http call goes in here somewhere
responseData[i] = data_from_this_response
...
})(i)
}
Use a library
Check out the async.parallel() call in Caolan's excellent library. You pass it an array of functions and it'll return to your callback with an array of the results.
https://github.com/caolan/async/#parallel
You'll need to create a loop that populates the array with curried versions of your function containing the appropriated variables.
Hello I'm working on a problem that requires me to change an set array of numbers into an array that returns the original numbers as a function. So we get a return of a2 instead of a[2].
I dont want the answer I just need a hint. I know i can loop through the array and use .pop() to get the last value of the array, but then I dont know how to convert it to a function from there. any hints?
var numToFun = [1, 2, 3];
var numToFunLength = numToFun.length;
for (var i = 0; i < numToFunLength; i++) {
(function(num){
numToFun.unshift(function() {
return num;
});
}(numToFun.pop()))
}
DEMO
basically it pops out a number from the last, builds a function with that number returned, and put back into the first of the array. after one full cycle, all of them are functions.
here's the catch: how this works, it's up to you to research
why the loop does not look like the straightforward pop-unshift:
for (var i = 0; i < numToFunLength; i++) {
numToFun.unshift(function() { //put into first a function
return numToFun.pop() //that returns a number
});
}
and why i did this: (HINT: performance)
var numToFunLength = numToFun.length;
There's three important steps here:
Extract the number value from the array. Within a loop with an iterator of i, it might look like this:
var num = numArray[i];
This is important, because i will not retain its value that it had when you created the new function - it'll end up with the last value it had, once the for loop is finished. The function itself might look like this:
function() { return num; }
There's no reference to i any more, which is important - to understand better, read about closures. The final step would be to add the new function to the array of functions that you want.
...and you're done!
EDIT: See other's answers for good explanations of how to do this right, I will fix mine also though
As others have pointed out, one of the tricky things in javascript that many struggle with (myself included, obviously) is that scoping variables in javascript is dissimilar to many other languages; scopes are almost purely defined by functions, not the {} blocks of, for example, a for loop, as java/C would be.
So, below you can see (and in other answers here) a scoping function can aid with such a problem.
var numArray = [12, 33, 55];
var funcArray = [];
var numArrLength = numArray.length; // Don't do this in for loop to avoid the check multiple times
for(var j=0; j < numArrLength; j++) {
var scopeMe = function() {
var numToReturn = numArray[j];
console.log('now loading... ' + numToReturn);
var newFunc = function() {
return numToReturn;
};
return newFunc;
}();
funcArray.push(scopeMe);
};
console.log('now me');
console.log(funcArray);
console.log(funcArray[0]());
console.log(funcArray[1]());
console.log(funcArray[2]());
console.log(funcArray[1]()); // To ensure it's repeatable
EDIT my old bad answer below
What you'll want to do is something like
var funcArray = [];
for(...) {
var newFunc = function() {
return numArray.pop();
}
funcArray.push(newFunc);
}
The key here is that functions in javascript can be named variables, and passed around as such :)
how can we make a function from arrays like
$.loader({
js: [
['1.js','3.js','2.js'],
['4.js'],
['5.js']
]
});
into something that does
$.when(
$.getScript("1.js"),
$.getScript('3.js'),
$.getScript('2.js'))
.then(
function(){
$.getScript('4.js').then(function(){
$.getScript('5.js');
})
});
heres what currently im working on
loader: function(arg){
var js = arg;
var phase = 0;
for (var i = 0; i < o.length; i++) {
console.log(o[i]);
$.each(o[i], function(key,src) {
//append javascript
});
};
}
the problem is i dont know where to start.
heres what i have in mind
i divide them into phases.
set phase 0, it will run the fist array but how can i do it ? $.getScript(src) for each src dosent stack. or chain. maybe put it one var sting ? like string + = $.getScript(src). so it will be like $.getScript("1.js").getScript("3.js").getScript("2.js") until there is arrays of string which is 3 in this example. then do something like append $.when( infront of the string then at the back of the string we put .then(function(){ until it finish then somehow closed the string.
ok i dont know ( and here goes the post at stackoverflow )
function lScripts(startAt){
for (var i = startAt; i < js.length; i++) {
if (js[i].constructor == Array){
for (var j = 0; j < js[i].length; j++) {
if(j==js[i].length){
$.getScript('js[i][j]',function(){
lScripts(startAt+1);
});
}else{
$.getScript('js[i][j]');
}
}
}else{
$.getScript('js[i]',function(){
lScripts(startAt+1);
});
}
}
}
lScripts(0);
Just to be clear, are you talking about writing a solution yourself that loads up a bunch of scripts? If so, dont do it yourself. Stand on someone else's shoulders.
RequireJS already does what you're asking.
http://requirejs.org/
It even has documentation for use with jQuery.
http://requirejs.org/docs/jquery.html
Someone else already did the hard work and wrote and tested a good product. Save yourself time and stress.
I hope this is what you mean: http://jsfiddle.net/pimvdb/hdN3Q/1/.
(function($) {
function load(files, index) {
var arr = [];
// fill arr with getScript requests (fake here)
for(var i = 0; i < files[index].length; i++) {
arr.push(console.log(files[index][i]));
}
// call $.when with arr as array of arguments (using apply)
$.when.apply(null, arr).then(function() {
if(index < files.length - 1) {
// if we have to move to the next chunk,
// load that next one
setTimeout(function() {
// timeout is to show they get processed in chunks
load(files, index + 1);
}, 1000);
}
});
}
$.loader = function(obj) {
load(obj.js, 0);
}
})(jQuery);
$.loader({
js: [
['1.js','3.js','2.js'],
['4.js'],
['5.js']
]
});
.apply essentially does the same thing as calling a function. However, because the amount of arguments to $.when differs depending on the input, you can't just call $.when(...) because you don't have a fixed number of arguments. The way to call a function with a variable amount of arguments is using .apply. It works like this:
someFunction(1, 2, 3);
is equal to:
someFunction.apply(null, [1, 2, 3]);
(The null refers to the execution context which is out of scope here.) The great thing about this is that you can build an array of any size. So you can call the function with any variable amount of arguments this way.
In the load function, arr gets filled with getScript functions, and it works the same way:
var arr = [getScript('file1'), getScript('file2')];
$.when.apply(null, arr);
is equal to:
$.when(getScript('file1'), getScript('file2'));
In load, we fill arr with the files of a chunk, e.g. in your case the first chunk is 1.js, 3.js and 2.js. This is done using a for loop but the array you will end up with you can just pass to .apply.
When all files are loaded, .then is called. The function we pass there should load the next chunk, but only if there is a next chunk. If there are 3 chunks, it should go to the next one when index is 0 or 1. When chunk 2 has finished, there is no next chunk so it should not continue to the next one as there isn't a next one.
So we need this if clause:
if(index < files.length - 1) {
Say files.length is 3. Then this if conditional will only pass when index is 0 or 1, which is what we want.