Couchbase, reduction too large error - javascript

On my work I using couchbase and I have some problems. From some devices data come to couchbase, and after I calling aggregate view. This view must aggregate values by 2 keys: timestamp and deviceId.
Everything was fine, before I have tried to aggregate more then 10k values. In this case I have reduction error
Map function:
function(doc, meta)
{
if (doc.type == "PeopleCountingIn"&& doc.undefined!=true)
{
emit(doc.id+"#"+doc.time, [doc.in, doc.out, doc.id, doc.time, meta.id]);
}
}
Reduce function:
function(key, values, rereduce)
{
var result =
{
"id":0,
"time":0,
"in" : 0,
"out" : 0,
"docs":[]
};
if (rereduce)
{
result.id=values[0].id;
result.time = values[0].time;
for (i = 0; i<values.length; i++)
{
result.in = result.in + values[i].in;
result.out = result.out + values[i].out;
for (j = 0; j < values[i].docs.length; j++)
{
result.docs.push(values[i].docs[j]);
}
}
}
else
{
result.id = values[0][2];
result.time = values[0][3];
for(i = 0; i < values.length; i++)
{
result.docs.push(values[i][4]);
result.in = result.in + values[i][0];
result.out = result.out + values[i][1];
}
}
return result;
}
Document sample:
{
"id": "12292228#0",
"time": 1401431340,
"in": 0,
"out": 0,
"type": "PeopleCountingIn"
}
UPDATE
Output document:
{"rows":[
{"key":"12201774#0#1401144240","value":{"id":"12201774#0","time":1401144240,"in":0,"out":0,"docs":["12231774#0#1401546080#1792560127"]}},
{"key":"12201774#0#1401202080","value":{"id":"12201774#0","time":1401202080,"in":0,"out":0,"docs":["12201774#0#1401202080#1792560840"]}}
]
}
}
Error occurs in the case where "docs" array length more then 100. And I think in that cases working rereduce function. Is there some way to fix this error exept making count of this array less?

There are a number of limits on the output of map & reduce functions, to prevent indexes taking too long and/or growing too large.
These are in the process of being added to the official documentation, but in the meantime quoting from the issue (MB-11668) tracking the documentation update:
1) indexer_max_doc_size - documents larger then this value are skipped by the
indexer. A message is logged (with document ID, its size, bucket name, view name, etc)
when such a document is encountered. A value of 0 means no limit (like what it used to
be before). Current default value is 1048576 bytes (1Mb).
2) max_kv_size_per_doc - maximum total size (bytes) of KV pairs that can be emitted for
a single document for a single view. When such limit is passed, message is logged (with
document ID, its size, bucket name, view name, etc). A value of 0 means no limit (like what
it used to be before). Current default value is 1048576 bytes (1Mb)
Edit: Additionally, there is a limit of 64kB for the size of a single reduction (output of the reduce(). I suggest you re-work your reduce function to return data within this limit. See MB-7952 for a technical discussion on why this is the case.

Related

Weird comparison performance on Javascript object key lookup

Presentation :
I am working on a piece of code that can compare two javascript Object by looping into the first one (called A) to perform a key lookup in the second one called B (I put value as key and occurence as value).
But when I am measuring the performance of the subkey key lookup of Object A (10 times per amount of data, with data as changing parameters for each 10 times the program runs (100 per row, 200...) I get high timing for the smallest amount of data (so potentially less key in the dict B)
Objects layout :
Object A looks like below:
{
SQL_TABLE_1:
{
column_1:
[
'2jmj7l5rSfeb/vlWAYkK/YBwIDk=',
'3MaRDFGBKvsLLhrLUdplz3wUiOI=',
'PEvUFHDR4HbOYXcj7danOvyRJqs='
'XHvERAKZ4AqU+iWlx2scZXdem80=',
'nSG0lvwlkIe5YxZGTo5binr3pAw=',
'Swuc/7YCU9Ptfrff+KHaJJ1+b7U=',
'N28qqdfezfZbPmK7CaGmj7m7IOQ=',
'ffc7skeffegT1ZytAqjco3EpwUE=',
'2XldayBefzBxsUuu6tMFYHVZEEY=',
'5rC2y6IzadQ1aEy7CvNyr30JJ2k='
]
},
SQL_TABLE_2:
{
column_1:[......]
}
}
Object B field have various size but this size never change in our tests
And Object B looks like:
[
field_1:
{
'2jmj7l5rSfeb/vlWAYkK/YBwIDk=': 1,
'3MaRDFGBKvsLLhrLUdplz3wUiOI=': 1,
'PEvUFHDR4HbOYXcj7danOvyRJqs=': 1,
'XHvERAKZ4AqU+iWlx2scZXdem80=': 4,
'nSG0lvwlkIe5YxZGTo5binr3pAw=': 1,
'Swuc/7YCU9Ptfrff+KHaJJ1+b7U=': 1,
'N28qqdfezfZbPmK7CaGmj7m7IOQ=': 27,
'ffc7skeffegT1ZytAqjco3EpwUE=': 1,
'2XldayBefzBxsUuu6tMFYHVZEEY=': 18,
'5rC2y6IzadQ1aEy7CvNyr30JJ2k=': 1 },
field_2:{......}
]
Timing measurement in the code is structured like this:
sql_field_1:
{
mongo_field_1: 0.003269665241241455, mongo_field_2: 0.0015446391105651855, mongo_field_3: 0.0009834918975830079, mongo_field_4: 0.0004488091468811035,
},
sql_field_2:
{
....
}
Goal
The goal is to perform for each sub-subkey of Object A a key lookup on the Object B subkeys.
Code
Here's the code that cause this behavior:
Object A is called sql_dict_array
Object B is called hash_mongo_dict
for(var field_name in hash_mongo_dict)
{
performance_by_mongo_field[field_name] = {};
result_by_mongo_field[field_name] = {};
// LOOP ON OBJECT A KEYS
for(var table_name in sql_dict_array)
{
// Start of time measurement
var start_time = performance.now();
// there is only one column in our test data
for(var column_name in sql_dict_array[table_name])
{
found_count = 0;
for(var value of sql_dict_array[table_name][column_name])
{
// **SUBKEY LOOPKUP HERE WITH VALUE**
var results = hash_mongo_dict[field_name][value];
// IF NOT UNDEFINED THEN IT HAS BEEN FOUND
// THIS CHECK IS NOT THE BOTTLENECK
if(results != undefined)
{
found_count+=results;
}
}
if(found_count > limit_parameter)
{
console.log("error: too many match between hashes")
process.exit(0)
}
// PERFORMANCE CALCULATION
performance_by_mongo_field[field_name][table_name] = (performance.now() - start_time)/1000;
result_by_mongo_field[field_name][table_name+'.'+column_name] = (found_count/verif_hash_count*100);
}
}
return some results...
}
Testing:
With this code, I expect to have almost constant time whatever the size of the Object B (amount of subkey) but in my code I have higher time when I have only 10 subkeys in the nested object A, and it become stable when reaching 100 keys or more (tested with 6000 keys)
Here's 10 runs for the key lookup code of one key of Object A containing 10 subkeys with 300.000+ data from Object B:
0.2824700818061829 0.2532380700111389 0.2455208191871643 0.2610406551361084 0.2840422649383545 0.2344329071044922 0.2375670108795166 0.23545906591415405 0.23111085414886476 0.2363566837310791
Here's the same comparison but with 4000+ subkeys:
0.0027927708625793456 0.0018292622566223144 0.015235211849212647 0.0036304402351379395 0.002919149875640869 0.004972007751464844 0.014655702114105225 0.003572652339935303 0.0032280778884887697 0.003232938766479492
I will appreciate every advice you can provide me,

Javascript average within multidimension array

I have a 2D array, sorted by names of students and their various 100m sprint sessions. A snippet of the data array may look like this...
Names Time(s)
ANDREWMICHEALSMITH 13.95
JOHNJAMESPETERSON 13.70
JOHNJAMESPETERSON 13.80
JOHNJAMESPETERSON 12.50
PETERMICHEALHEWITT 12.80
PETERMICHEALHEWITT 12.60
VICENTJAMESSMITH 13.10
VICENTJAMESSMITH 13.50
VICENTJAMESSMITH 13.80
VICENTJAMESSMITH 13.00
I'm desperately needing to convert the table to show student name, their average time and the number of timed sessions, the desirable result as per below...
Names Avg Time(s) Sessions
ANDREWMICHEALSMITH 13.95 1
JOHNJAMESPETERSON 13.33 3
PETERMICHEALHEWITT 12.70 2
VICENTJAMESSMITH 13.35 4
I have drafted a JavaScript code so far unfortunately fails at the very first if statement.
Cannot read property "0" from undefined.
The code also doesn't capture student with nth number of sessions, and no provision for number of sessions yet. Quite frankly my Java knowledge is so limited for the task at hand. I'm in desperate need of some help to get to the end result.
I'm truely grateful for any guidance.
StudentList.sort();
for (var i=0; i<StudentList.length; i++){
if(StudentList[i][0] == StudentList[i+1][0]){
if(StudentList[i+1][0] == StudentList[i+2][0]){
if(StudentList[i+2][0] == StudentList[i+3][0]){
if(StudentList[i+3][0] == StudentList[i+4][0]){
StudentList[i][1] = (StudentList[i][1]+StudentList[i+1][1]+StudentList[i+2][1]+StudentList[i+3][1]+StudentList[i+4][1])/5;
StudentList.splice(i+1,3);
}else{
StudentList[i][1] = (StudentList[i][1]+StudentList[i+1][1]+StudentList[i+2][1]+StudentList[i+3][1])/4;
}
}else{
StudentList[i][1] = (StudentList[i][1]+StudentList[i+1][1]+StudentList[i+2][1])/3;
}
}else{
StudentList[i][1] = (StudentList[i][1]+StudentList[i+1][1])/2;
}
}
}
You could use an object as hash table for names and us ean array as result set for counting and storing the data.
var studentList = [['ANDREWMICHEALSMITH', 13.95], ['VICENTJAMESSMITH', 13.00], ['PETERMICHEALHEWITT', 12.80], ['PETERMICHEALHEWITT', 12.60], ['VICENTJAMESSMITH', 13.10], ['JOHNJAMESPETERSON', 13.70], ['JOHNJAMESPETERSON', 12.50], ['VICENTJAMESSMITH', 13.50], ['JOHNJAMESPETERSON', 13.80], ['VICENTJAMESSMITH', 13.80]],
hash = Object.create(null),
result = [],
array;
studentList.forEach(function (a) {
if (!hash[a[0]]) {
hash[a[0]] = { name: a[0], sum: 0, count: 0, avg: 0 };
result.push(hash[a[0]]);
}
hash[a[0]].sum += a[1];
hash[a[0]].count++;
hash[a[0]].avg = (hash[a[0]].sum / hash[a[0]].count).toFixed(2);
});
array = result.map(function (a) {
return [a.name, a.avg, a.count];
});
console.log(array);
console.log(result);
.as-console-wrapper { max-height: 100% !important; top: 0; }

How to keep for loop active until it finishes, then put it back to 0

I have a bot that checks for when people send me a message. I want it so when people send me: "!account" it checks through my json files, if the guy sending the message's ID, matches someones ID in the Json files. If it does, I want it to send him his "credits" and "steamid" so far I've got it kind of working:
ok = 1;
if(ok == 1) {
console.log("[SERVER] "+steamID.getSteamID64()+" is asking for his account information.");
for(r=0;r<config.tradesettings.amountofaccounts+1;r+=1) {
if(config.accounts[r].steamID == steamID.getSteamID64()) {
ok = 0;
console.log("[SERVER] "+ steamID.getSteamID64() +" is asking for his account info, respondig with it.\n")
client.chatMessage(steamID.getSteamID64(), "\nDisplaying your account on this bot\nBotID: "+config.accounts[r].botID+"\nSteamID: "+config.accounts[r].steamID+"\nBalance: "+config.accounts[r].balance);
console.log(r);
return;
} else {
//do nothing
}
}
}
Now that is working, except it isn't checking through all json files. The for loop only get's activated once, so it only checks the first or second json.
How would I make the for loop active until it finishes searching, then set r = 0
Here is the the json file is called config.json, and here it is.
"accounts":[
{
"botID":"0",
"steamID":"2919",
"balance": 54
},
{
"botID":"1",
"steamID":"",
"balance": 0
},
{
"botID":"2",
"steamID":"",
"balance": 0
},
{
"botID":"3",
"steamID":"76561198026027024",
"balance": 0
}
]
To ensure that the loop runs through all the items in the json array, your for loop should compare the value of r with the length of the array config.accounts, or ensure that the value config.tradesettings.amountofaccounts is set to length of the array.
The following code to replace the for loop should make iteration through all the array items.
for(r = 0; r < config.accounts.length - 1; r++) {
if(config.accounts[r].steamID == steamID.getSteamID64()) {
ok = 0;
console.log("[SERVER] "+ steamID.getSteamID64() +" is asking for his account info, respondig with it.\n")
client.chatMessage(steamID.getSteamID64(), "\nDisplaying your account on this bot\nBotID: "+config.accounts[r].botID+"\nSteamID: "+config.accounts[r].steamID+"\nBalance: "+config.accounts[r].balance);
console.log(r);
break;
}
}
return;
This should iterate through all the items in the json array.

Generating a price per item in an html table using a javascript

I have a javascript that is generating a table for me. The elements in the table are gathered in an array of arrays called sep. Sep contains 1152 sub arrays that are of the form:
Sep[0] //["316SS", "K", "-100 to 225°C", "Brass", "1/8", "4'", "4'", "8", "Ungrounded"]
So basically there are 1152 rows, each of which defines a products with 9 parameters. I want to make a for-loop that will create a price for each of the configurations. This is what I have so far:
//PART 1-------------WORKS FINE-----------------------------------
var eopartprice2 = []; //matrix that I want to contain my prices
for (var i = 0; i < sep.length; i++) {
strnum1 = sep[i][5]; //parameter 5 is a length of material
len1 = Number(strnum1.substr(0, strnum1.length - 1));
strnum2 = sep[i][6]; //parameter 6 is another length of material
len2 = Number(strnum2.substr(0, strnum2.length - 1));
strnum3 = sep[i][7]; //parameter 7 is the number of units required
condnum = Number(strnum3.substr(0, strnum3.length));
feetOfMat = len1*len2*condnum; //The product of these is the total feet of req material
//PART 2------------PFCost always = 0.87--------------------------
//Next i need to identify the cost of the material (to multiply by the total feet)
var costOfMat = [0.87, 0.87, 1.77, 0.55] //different costs of the 4 materials
if (sep[i][0] = "304SS") {
var PFCost = costOfMat[0]; //304SS costs 0.87/foot
} else if (sep[i][0] = "316SS") {
var PFCost = costOfMat[1]; //316SS costs 0.87/foot
} else if (sep[i][0] = "Inconel") {
var PFCost = costOfMat[2]; //Inconel costs 1.77/foot
} else if (sep[i][0] = "High Temp. Glass") {
var PFCost = costOfMat[3]; //High Temp. Glass costs 0.55/foot
}
baseMatCost[i] = PFCost*feetOfMat; //I'd like to generate a matrix that
//contains all of the base prices (1 for each row)
//PART 3---------------fitcost always = 36------------------------
//Trying to identify the cost of brass vs. stainless fittings
if (sep[i][3] = "Brass") {
fitcost = 36;
} else if (sep[i][3] = "Stainless Steel") {
fitcost = 37;
}
}
My Problem so far is that I want the prices to be defined based off of whether or not the if statements are satisfied but in both cases (fitcost and PFCost) the values are simply the ones defined in the first if statement.
Lastly I'd like to generate my final price in the eopartprice2 matrix based off adding up the materials generated above + some cost of labor multiplied by some margin.
Also I'm concerned with the speed of how quickly this runs as it will be a live table in my website, and every time I add more to this I feel like it's taking longer and longer to generate. Here's a link to my w3 that I'm working in.
Please, any help would be greatly appreciated :)
In your if statement conditions, you're using a single equals sign. This is an assignment operator, not a comparison operator!
So, an if statement such as if (sep[i][0] = "304SS") is actually assigning the value "304SS"; it is not comparing the value "304SS" to sep[i][0].
To correctly compare the values, you'll want to change the single equals sign to a double equals:
if (sep[i][0] == "304SS").
Note: == will convert types if necessary before comparing. For example: ".87" == 0.87 returns true.

2 dimensional JavaScript Array generated by for loop is being overwritten by last loop result

Im trying to make pseudo-random sequence generator that works just line Linear Feedback Shift Register.
Im doing it in JavaScript because its the only language that I know and im using HTML to create GUI.
User should type in initial value and get schematic diagram and pseudo-random sequence itself.
Here is my JavaScript code:
var UserInput = document.getElementById('ulaz');
var Output = document.getElementById('izlaz');
//variable `data` is an array of objects which I used to store pictures of circuits
// and [taps][3] necessary for shift registers to give max possible length output
// before going into loop which is 2^n-1, where n (`bit` in my code) is number of
//register blocks and number of digits in input value.
function pss (){
var data = [
{
slika:"pic/2bit.png",
tap:[0,1]
},
{
slika:"pic/3bit.png",
tap:[0,2]
},
{
slika:"pic/4bit.png",
tap:[0,3]
},
{
slika:"pic/5bit.png",
tap:[1,4]
},
{
slika:"pic/6bit.png",
tap:[0,5]
},
{
slika:"pic/7bit.png",
tap:[0,6]
},
{
slika:"pic/8bit.png",
tap:[1,2,3,7]
},
{
slika:"pic/9bit.png",
tap:[3,8]
},
{
slika:"pic/10bit.png",
tap:[2,9]
},
{
slika:"pic/11bit.png",
tap:[1,10]
},
{
slika:"pic/12bit.png",
tap:[0,3,5,11]
},
{
slika:"pic/13bit.png",
tap:[0,2,3,12]
},
{
slika:"pic/14bit.png",
tap:[0,2,4,13]
},
{
slika:"pic/15bit.png",
tap:[0,14]
},
{
slika:"pic/16bit.png",
tap:[1,2,4,15]
},
{
slika:"pic/17bit.png",
tap:[2,16]
},
{
slika:"pic/18bit.png",
tap:[6,17]
},
{
slika:"pic/19bit.png",
tap:[0,1,4,18]
},
{
slika:"pic/20bit.png",
tap:[2,19]
},
{
slika:"pic/21bit.png",
tap:[1,20]
},
{
slika:"pic/22bit.png",
tap:[0,21]
},
{
slika:"pic/23bit.png",
tap:[4,22]
},
{
slika:"pic/24bit.png",
tap:[0,2,3,23]
},
{
slika:"pic/25bit.png",
tap:[2,24]
},
{
slika:"pic/26bit.png",
tap:[0,1,5,25]
},
{
slika:"pic/27bit.png",
tap:[0,1,4,26]
},
{
slika:"pic/28bit.png",
tap:[2,27]
},
{
slika:"pic/29bit.png",
tap:[0,28]
},
{
slika:"pic/30bit.png",
tap:[0,3,5,29]
},
{
slika:"pic/31bit.png",
tap:[2,30]
},
{
slika:"pic/32bit.png",
tap:[1,5,6,31]
}
];
var first = UserInput.value.split("");
for (k=0;k<first.length;k++) first[k] = +first[k];
//first is just UserInput separated in one char strings than parsed to integers
var bit = first.length - 2;
// I subtracted 2 here so I can access objects from data
var matrix = [first];
var t = 0;
var between;
var z;
for (i=1; i<Math.pow(2, bit+2)-1; i++){ //here is that 2^n-1. +2 is because i had -2 before. For loop is starting from 1 and ending with <2^n-1 because i already have first array of matrix
for (j=0; j<data[bit].tap.length; j++){
z = data[bit].tap[j];
t = t ^ matrix[i-1][z];
} // this for makes "t" which is all taps XOR-ed. If user input was 101, tap would be [0,2] and t would be 1xor1=0
between = matrix[i-1];
console.log(between);
between.unshift(t);
between.pop();
matrix[i] = between;
t=0; // here Im "shifting registers" or just placing t in front of last generated row and removing its last digit, thus generating new row
}
console.log(matrix);
}
and here is HTML so you can run it.
variable data is an array of objects which I used to store pictures of circuits and taps necessary for shift registers to give max possible length output before going into loop which is 2^n-1, where n (bit in my code) is number of register blocks and number of digits in input value.
So problem is: console.log(between); which logs last generated row is all correct except, ofc, there is no last row because it shows last generated, but than console.log(matrix) which should log complete matrix , shows all rows overwritten by last one.
So for user input 101, matrix should be
101
010
001
100
110
111
011
but is just
011
011
011 ...
I cant figure out what is wrong with it if part before console.log(between); is all fine...
P.S. Code is not finished it wont display solution in HTML, and there still needs to be part of function that makes an array from last column of matrix, which is pseudo-random sequence.
I realized that var between refers to the same array as var matrix[i-1], rather than a new, independent array.
between = matrix[i-1];
So, if you want to store only values of matrix[i-1], not to create reference, you can do this like this:
between = JSON.parse(JSON.stringify(matrix[i-1]));
In js when you copy array in some variable, you create reference of that array by default. There are many ways to avoid this, and you can find many examples here.
I do not know why but I've come to a solution (will investigate more when I get free time).
for (i=1; i<Math.pow(2, bit+2)-1; i++){ //here is that 2^n-1. +2 is because i had -2 before. For loop is starting from 1 and ending with <2^n-1 because i already have first array of matrix
for (j=0; j<data[bit].tap.length; j++){
z = data[bit].tap[j];
t = t ^ matrix[i-1][z];
} // this for makes "t" which is all taps XOR-ed. If user input was 101, tap would be [0,2] and t would be 1xor1=0
between = matrix[i-1];
console.log(between);
between.unshift(t);
between.pop();
// MODIFICATION
var between_string = between;
matrix[i] = between_string.join(); // Turn it to a string
matrix[i] = matrix[i].split(','); // Turn it back to array to keep it working on the for loop above.
// END MODIFICATION
t=0; // here Im "shifting registers" or just placing t in front of last generated row and removing its last digit, thus generating new row
}
Now, when you print it out in the console it'll show you a bidimentional array, although it's weird, sometimes (on my console) it shows int numbers and sometimes mixed with string numbers (respecting the original value of between).
Edit: I tried only using "101" for the input.
Second edit: Okay, I feel ashamed, the reason why it returns [1, "0", "0"] (example) is because of split(',') for "1,0,0"(only tow numbers were preceded by comas). Haha. Sorry.

Categories

Resources