I am trying to move a large amount of data from a mysql database (an amount larger than the max query size) and will need to run it through a loop in chunks to do so. The issue I am having is that even in the small sample size of 50 records, I am consistently missing about half of them as it seems the connection is being closed before the loops have finished. The code snippet I am having trouble with is below:
for(var j = 0; j < maxNum; j+=increment){
// populates the array that can be used to find the mods that have been purchased
console.log("START = " + j + " END = " + (j+increment));
con.query("SELECT * FROM sometable WHERE id BETWEEN " + j
+ " AND " + (j + increment), function(err, rows, fields){
if(!err){
console.log("ROWS = " + rows.length);
for(var a = 0; a < rows.length; a++){
arrayOfObjs.push(rows[a]);
console.log(rows[a].ID);
// If the inner loop is at the end and the while loop is about to end this cycle
if( (a + 1 == rows.length) && ((userCounter + increment - 1) > maxNum)) {
// Loop through all the user objects and if it has records, add it to that item
for(var b = 0; b < arrayOfUsers.length; b++){
arrayOfUsers[b].objs = returnObjById(b, arrayOfMods);
// If the loop is ending
if(b+1 == arrayOfUsers.length){
done1 = true;
// temp force close
shutDown()
}
}
}
}
}
else{
console.log(err);
}
})
}
The maxNum is supposed to represent the total number of users in the table as a whole, and the increment would be the size of the chunks. The returnObjs function does not call anything and is a simple switch that has no effect on the functionality. The shutDown function is a mysql connection end with a process.exit for a callback. That function is what is ending the script before it is finished but as I am new to Node, I do not know how I could adjust this in order to ensure that this does not continue to happen. Any suggestions would be appreciated as this is being done partly as a way to learn Node since it wouldn't be difficult to do in a language I'm familiar with.
I see two possible problems here.
First, conn.query() is probably an async function (by looking at the callback function I see there). That means node won't wait for the callback to be called before continuing execution, resulting in your first loop iterations being executed all before even the first conn.query() callback is called.
I suggest you have a look at this excellent library : async. It helps a lot with async operations, like in your case when you want to wait for the callback before starting another iteration. For that matter, the async.series() function may be helpful.
Secondly, you call to shutdown() seems to automatically happen when the last iteration of the third for loop has finished, but you are still in the first iteration of your first and second for loops. This means j will never be incremented, nor will a. The shutdown() function should be called at the end of the last iteration of your first for loop.
EDIT: I overlooked your if before the third for loop. I'm not sure what the purpose of the (userCounter + increment - 1) > maxNum check is, but if it prevents the third loop from being called when a or j are not at their max value, then you can discard my second point. The async problem is still there though, and is most likely the cause of your problem.
Related
What is an off-by-one error? If I have one, how do I fix it?
An off-by-one error is for example when you intend to perform a loop n times and write something like:
for (int i = 1; i < n; ++i) { ... }
or:
for (int i = 0; i <= n; ++i) { ... }
In the first case the loop will be executed (n - 1) times and in the second case (n + 1) times, giving the name off-by-one. Other variations are possible but in general the loop is executed one too many or one too few times due to an error in the initial value of the loop variable or in the end condition of the loop.
The loop can be written correctly as:
for (int i = 0; i < n; ++i) { ... }
A for loop is just a special case of a while loop. The same kind of error can be made in while loops.
An off-by-one error is when you expect something to be of value N, but in reality it ends up being N-1 or N+1. For example, you were expecting the program to perform an operation 10 times, but it ends up performing 9 or 11 times (one too few or one too many times). In programming this is most commonly seen happening when dealing with "for" loops.
This error happens due to a misjudgement where you do not realize that the number you are using to keep track of your counting may not be the same as the number of things you are counting. In other words, the number you are using to count may not be the same as the total of things you are counting. There is nothing that obligates both things to be the same. Try to count out loud from 0 to 10 and you end up saying 11 numbers in total, but the final number that you say is 10.
One way to prevent the problem is to realize that our brain has a tendency (maybe a cognitive bias) to make that error. Keeping that in mind may help you identify and prevent future situations. But I guess that the best thing you can do to prevent this error is to write unit tests. The tests will help you make sure that your code is running as it should.
Say you have the following code featuring an array and a for loop:
char exampleArray[] = { 'H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd' };
for(int i = 0; i <= 11; i++)
{
print(exampleArray[i])
}
See the issue here? Because I counted my array to have eleven characters in it, I have set my loop to iterate eleven times. However, arrays start at zero in most languages, meaning that when my code goes to print
exampleArray[11]
I will get an index out of bounds error because the array in the example has no value at index eleven.
In this case, I can fix this easily by simply telling my loop to iterate one fewer times.
The easiest way to debug this issue is to print out your upper and lower bounds and see which value generates an index out of bounds error, then set your value to be one greater or one fewer than it is throughout your entire iteration.
Of course, this assumes the error is generated by a loop going one over or one less than the bounds of an array, there are other situations where an index out of bounds error can occur, however, this is the most common case. An index out of bounds will always refer to trying to access data where data does not exist due to the boundaries past not being within the boundaries of data.
A common off-by-one confusion arises because some languages enumerate vectors from zero (C, for example) and other languages from one (R, for example). Thus, a vector x of size n has members running from x[0] to x[n-1] in C but from x[1] to x[n] in R.
You are also confronted with the off-by-one challenge when coding the common idiom for cyclic incrementation:
In C:
i = (i+1)%n
In R:
i <- (i-1)%%n + 1
Off by one error (sometimes called OBOE) crop up when you're trying to target a specific index of a string or array (to slice or access a segment), or when looping over the indices of them.
If we consider Javascript as an example language, indexing starts at zero, not one, which means the last index is always one less than the length of the item. If you try to access an index equal to the length, the program may throw an
"index out of range" reference error
or
print undefined.
When you use string or array methods that take index ranges as arguments, it helps to read the documentation of that language and understand if they are inclusive (the item at the given index is part of what's returned) or not. Here are some examples of off by one errors:
let alphabet = "abcdefghijklmnopqrstuvwxyz";
let len = alphabet.length;
for (let i = 0; i <= len; i++) {
// loops one too many times at the end
console.log(alphabet[i]);
}
for (let j = 1; j < len; j++) {
// loops one too few times and misses the first character at index 0
console.log(alphabet[j]);
}
for (let k = 0; k < len; k++) {
// Goldilocks approves - this is just right
console.log(alphabet[k]);
}
A simple rule of thumb:
int i = 0; // if i is initiated as 0, always use a < or > in the condition
while (i < 10)
System.out.printf("%d ", i++);
int i = 1; // if i is initiated as 1, always use a <= or >= in the condition
while (i <= 10)
System.out.printf("%d ". i++);
So I am having an issue with my solution, and I may be entirely off on what needs to be done. I keep timing out which makes me believe I have the .splice() in an incorrect location.
The problem:
You are given an array of integers. On each move you are allowed to increase exactly one of its element by one. Find the minimal number of moves required to obtain a strictly increasing sequence from the input.
Example
For inputArray = [1, 1, 1], the output should be
arrayChange(inputArray) = 3.
My Pseudo code
First check and see if the current index is greater than the next index. If not, continue check through the entire loop. If so, add one to the next index and test again until true. If you increment the next index by one, add one to a variable "moves". Return moves
function arrayChange(inputArray) {
for( var i = 0; i < inputArray.length; i++){
var addition = (inputArray[i+1]+1)
if(inputArray[i] >= inputArray[i+1]){
inputArray.splice(i,0, addition);
}
}
return inputArray;
}
My Error:
Execution time limit exceeded on test 1: Program exceeded the execution time limit. Make sure that it completes execution in a few seconds for any possible input.
Why your code fails:
for( var i = 0; i < inputArray.length; i++){//iterate until reaching end of array
var addition =(inputArray[i+1]+1);
if(inputArray[i] >= inputArray[i+1]){ //if its not increasing
inputArray.splice(i,0, addition);
}
}
Lets examine a sample input:
arrayChange([1,1]);
So inputArray[i] is 1, and inputArray[i+1] is 1.
Therefore it will go into the if, as 1>=1. So it will add 1+1 into the array,but not at the end but at i, so at position 0. It will look like this:
[2,1,1]
The loop will go on:
[2,2,2....,1,1]
It will never end.
To make it work, you must stop at array.length-1 :
for( var i = 0; i < inputArray.length-1; i++){
and you must insert at the right index,and also remove one:
inputArray.splice(i+1,1,addition);
Non array changing approach:
function numchanges(array){
var before=array[0],counter=0;
for(var i=1;i<array.length;i++){
if(before>=array[i]){
//we need to change array[i] to before+1, so:
counter+=(++before)-array[i];
}else{
before=array[i];
}
}
return counter;
}
console.log(numchanges([1,1,1]));//should return 3
How it works:
A proper strictly monotonically increasing function would have values like this:
[1,2,3,4,5,10]
So it might at least go one up, or it jumps up. So lets take one random array, and its valid counterpart:
[1,1,3,4,5,-1]
[1,2,3,4,5,6]
So the changes needed:
[0,1,0,0,0,7] => 8
So the upper code keeps the valid number ( before ) and the needed change while iterating from left to right. current starts with the first array item:
before=array[0];//1
But we dont need to change the first array element, so we start at i=1.If this number is valid, so
before<array[i]
we just go on:
before=array[i];
i++;//due the for loop
If not, its not valid and we need a correction.
result+=before+1-array[i];
So if the last was 5 (=before) and now weve got -10 (=array[i]), we need to correct it to 6 (=before+1);
result+=6--10//==16
So we need another 16 corrections...
For one of my elements on my page, I want the text to change every ten seconds, and for the class to be changed. Text changing is easy, as is class changing, but I'm having trouble with my for loop, and I feel like I'm missing something.
I want to have the for loop choose a random faction in an array, and then apply that to the element. For my testing, I've been using console.log rather than DOM manipulation.
First, I set up my array:
var factions = ["Enforcers", "Reapers", "Ular Boys", "Roaches"];
Then, I want a variable that is a number chosen at random in reference to this array:
var x = factions[Math.floor(Math.random()*factions.length)];
From that, I want the ability to run the Math.floor and Math.random functions elsewhere.
function reDefine() {
x = factions[Math.floor(Math.random()*factions.length)];
console.log(x);
}
Finally, I want the for loop to run 200 times (I've chosen 200 times because it's far and beyond the time the user will be staying on the site), so I told it to count to 200 (i = 0; i < 200). After that, I wanted each time it iterated, to wait 10s, so I have a Timeout function with a delay of 10000 (milliseconds). Then, the code to reDefine and then, in the case of testing, console.log the new definition of the x variable.
function reChange() {
for (var i = 0; i < 200; i++) {
setTimeout(function() {
reDefine();
console.log("Chosen faction is now: " + x);
}, 10000);
}
}
Instead of counting to 1 (the first iteration), waiting 10000, and then redefining x, it redefines x two hundred times, then logs them all.
Is there something I'm specifically doing wrong here, perhaps with the Timeout function?
Is there something I'm specifically doing wrong here, perhaps with the Timeout function?
Yes! You're scheduling a bunch of deferred callbacks, but not actually waiting until one has finished to schedule the next.
You can fix that with something as simple as:
function reChange(currentIndex) {
setTimeout(function() {
reDefine();
console.log("Chosen faction is now: " + factions[currentIndex]);
// If we haven't gotten to the end of the list, queue up another one
var nextIndex = ++currentIndex;
if (nextIndex < factions.length) {
// Enqueue the next faction
reChange(nextIndex);
}
}, 10000);
}
Make sure to note that the function without the timeout has closure over the value of currentIndex for each call of reChange. That is, the next invocation does not replace currentIndex in any previous timeout, since primitives (including numbers) are passed by value. Closure in JS can be a tricky thing.
The core problem is that your execution right now looks like:
for each item
wait
log
rather than:
for the current item
wait
log
repeat
Because JS is single-threaded (for most intents and purposes), setTimeout adds a callback to be executed later. It doesn't block until the timeout has expired, like a traditional sleep would do.
Currently I am trying to make a little game maninly for myself for having fun while making it and using it, but i got stuck at a part.
I would like to make a memory game where the program generates a certain array filled with the names of the arrows that should change. So e.g.: left,left,up,down,right with this I would like to change the left arrow 1st only to green than back to normal then since the 2nd time it would happen twice since the 1st 2 elements of the array are "left".
So I have been trying to use setTimeout in JS and tried to look it up how to use it and managed to slowly advance but i kidna got lost in all of the explanations.
My current code changes all the arrows that are listed in the array green then changes them back to normal 1 by 1 (actually I can't see the 1st arrow changin at all but i assume it does just the setTimeout doesnt apply to it since the rest is changing).
for (var j = 0; j < narray.length; j++) {
var image = document.getElementById(narray[j]);
image.src = "arrow-"+narray[j]+"-good.png";
function(j){
sesetTimeout(function(){
var image2 = document.getElementById(narray[j]);
image2.src = "arrow-"+narray[j]+".png";
}, j*1000);
})(j);
I know it is really messy but I have totally got lost in the tutorials I have tried to learn from. So if anyone could help me I would be really greatful.
Thanks in advance!
Here's what I think will work.
k=0;
for (var j = 0; j < narray.length; j++) {
var image = document.getElementById(narray[j]);
image.src = "arrow-"+narray[j]+"-good.png";
setTimeout(function(){
var image2 = document.getElementById(narray[k]);
image2.src = "arrow-"+narray[k]+".png";
k++;
}, j*1000+500);
}
So besides the typos, there's an issue with using "j" from the for loop in the timeout functions because they run later. So we introduce "k" to keep track of how many of them we've done. So k will be incremented each time, and here it's a global variable, so as each timeout function actually runs, k will be a different number, and will increment k to be ready for the next one that will be called.
What if you don't use a for loop and you call the function recursive, something like this:
function doSomething(j,l){
if(j>=l) return 'done';
setTimeout(function(){
console.log('step: ' + j + ' from: ' + l);
// add your code here :)
}, j*1000);
doSomething(++j,l);
}
doSomething(0,narray.length); // start with 0 and the length of the array
i want to perform a loop that writes strings in a div without overloading. I have got a list of names in an global array called names. The array is filled with about 2500 names. Now i want to write the names in a div using ajax. But when i want to loop the names, the script is kinda overloading and stops with an alert of the client. This is my loop:
for(var i = 0; i < names.length; i++){
document.getElementById('div').innerHTML += names[i] + "<br/>";
}
I also tried to interlace the loop in smaller steps like 100 (because i thought like that the loop wouldnt overload). I also tried it with window.setTimeout("function(i)",0); with the param i, that steps 100 each time the function gets called, until i am at the length of the array names.
I know there is an easy way. But i can't find that way..... would be pleased by getting help.
Thanks
You can code it like:
document.getElementById('div').innerHTML = names.join('<br/>');
This will set the inner html in single instruction, solving the load issue.
try it this way:
var content = '';
for(var i = 0, n_length = names.length; i < n_length; i++) {
content += names[i] + "<br/>";
}
document.getElementById('div').innerHTML += content; // if the previous content should remain there
Please note that its faster, to store the abort value of a loop. Otherwise it would read the length-property of your array for each iteration of your loop (2500 x).
Also note that DOM-Manipulation is very verys slow, so its smart to reduce it to the lowest level possible. Better build your content separately and write it to the DOM once for all.