MongoDB MapReduce, Date and % operator - javascript

I have a problem with a python-script witch i use to aggregate large collections into smaller pieces and have them grouped by a timestamp.
map = Code("function(number) {"
"emit({"
"ts : new Date(new Date((this.ts - (this.ts % (60 * number))) * 1000).setSeconds(0))"
"}, 1);"
"}")
reduce = Code("function(key, vals) {"
"var sum = 0;"
"for (var i in vals) {"
"sum += vals[i]"
"}"
"return sum;"
"}")
As you can see, it's a pretty simple MapReduce and the timestamp (ts) should be grouped by anny given number of minutes. I've tested the Javascript here http://jsfiddle.net/QgMzK/1/ and it seems to work fine. But when i run it in Python all timestamps become ISODate("1970-01-01T00:00:00Z").
Any ideas?

Your map function takes one parameter: number, which, when called by map-reduce, will be set to null and after coertion (and some % by zero) will make the date your map is returning ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ"). This after type conversion becomes datetime.datetime(1970, 1, 1, 0, 0).
Remove the parameter and it should work.
edit
To confirm that, try running this code:
from pymongo import Connection
from bson.code import Code
db = Connection().mr_test
for i in xrange(10):
db.things.insert({"x" : i})
map = Code("function(number) {"
"emit({"
"ts : number"
"}, 1);"
"}")
reduce = Code("function(key, vals) {"
"var sum = 0;"
"for (var i in vals) {"
"sum += vals[i]"
"}"
"return sum;"
"}")
result = db.things.map_reduce(map, reduce, "test_results")
for doc in result.find():
print doc
The result on my machine is :
{u'_id': {u'ts': None}, u'value': 10.0}
Note that ts is None in the result, cause number was not set when the mapping function executed.
edit 2
AFAIK the only way to pass a parameter to map is by using scope optional parameter in map_reduce, but then you'll have to remove it from map signature anyway.
So by changing your map to:
map = Code("function() {"
"emit({"
"ts : new Date(new Date((this.ts - (this.ts % (60 * number))) * 1000).setSeconds(0))"
"}, 1);"
"}")
and by calling:
db.whatever.map_reduce(map, reduce, "collection_name", scope = {"number" : the_value_your_function_needs})
you can get the result you want.

Related

Javascript variable being overwriiten after query execution

I am using mysql database with Nodejs. I am running a query like this:
rows = await pool.query(`Select partID,stageID,locationID,QuantityAfter, QuantityChange, rowDeleted from InventoryTransactions
where partID = ${item.partID} and stageID = ${item.stageID} and locationID = ${item.locationID} and orderID = ${orderID}
order by transactionID desc`)
if(rows[0].rowDeleted != 1){ //This means that the quantity in the inventory table and inv trans table is not the same
console.log(`=====================================`)
console.log(Number(rows[0].QuantityChange))
console.log(Number(rows[0].QuantityChange))
console.log(Number(rows[0].QuantityChange))
await pool.query(`Insert into inventorytransactions(OrderID,PartID,LocationID,StageID,QuantityBefore,QuantityChange,QuantityAfter,ITTTimeStamp,rowDeleted,id)
Values(${orderID},${rows[0].partID},${rows[0].locationID},${rows[0].stageID},${parseInt(rows[0].QuantityAfter)},${-parseInt(rows[0].QuantityChange)},${parseInt(rows[0].QuantityAfter)-parseInt(rows[0].QuantityChange)},'${(new Date ((new Date((new Date(new Date())).toISOString() )).getTime() - ((new Date()).getTimezoneOffset()*60000))).toISOString().slice(0, 19).replace('T', ' ')}',true,${id})`)
console.log(Number(rows[0].QuantityChange))
console.log(Number(rows[0].QuantityChange))
console.log(Number(rows[0].QuantityChange))
console.log(`Update inventory set Quantity = quantity - ${Number(rows[0].QuantityChange)} where partID =${rows[0].partID} and stageID = ${rows[0].stageID} and locationID = ${rows[0].locationID}`)
await pool.query(`Update inventory set Quantity = quantity - ${Number(rows[0].QuantityChange)} where partID =${rows[0].partID} and stageID = ${rows[0].stageID} and locationID = ${rows[0].locationID} `)
console.log(`=====================================`)
}
The reason I have so many console.log is because I have encountered a very weird bug. All of the values of rows[0].QuantityChange) should be the same.
But my console shows:
====================================
4
4
4
425
NaN
NaN
NaN
Update inventory set Quantity = quantity - NaN where partID =12462 and stageID = 1 and locationID = 1
{ Error: ER_BAD_FIELD_ERROR: Unknown column 'NaN' in 'field list'
I know 425 is the value of ${id} variable which can be found at the end of the insert sql statement. Which makes no sense to me. What makes even less sense is how are the the bottom values are undefined or NAN
The 425 likely comes from some other code that is running during your await pool.query() statement (not from anything in the code you show us). await blocks the execution of the local function, but it does not block the event queue so other events can run while an await ... is waiting for completion.
Then, you don't show where rows is defined, but it seems possible that it's getting overwritten by some other code that is running during the await. If you show us a lot more context around this code including where rows is defined, we could advise and explain in more detail.
I'm not sure, but, in this part
'${(new Date ((new Date((new Date(new Date())).toISOString() )).getTime() - ((new Date()).getTimezoneOffset()*60000))).toISOString().slice(0, 19).replace('T', ' ')}'
Exist an incorrect use of single quotes, which can be altering the result of the consult. You should scape inner single quotes, or use Double quotes for wrapping the complete instruction.
"${(new Date ((new Date((new Date(new Date())).toISOString() )).getTime() - ((new Date()).getTimezoneOffset()*60000))).toISOString().slice(0, 19).replace('T', ' ')}"

How do I reference a value in an App Maker query (like I do in Apps Script)

I am writing a script to take a stock number, loop through existing stock numbers until a match is NOT found, then assign that unique stock number to the record. My problem is that the usual data[i][2] doesn't seem to reference a 'query' the same way that Apps Script would reference an array.
Fair warning, I'm trying to expand my Apps Script skills in to broader Javascript so I there's a good chance I'm doing it all wrong - I'm all ears if you tell me I'm doing this all incorrectly!
Using the log: data[i][2] gives me 'undefined' whereas data[2] gives me all fields of the third item in my query. Based on this I feel like I just need to learn how to reference it properly.
//Querying my datasource as 'var data'
var query = app.models.UsedVehicles.newQuery();
query.filters.ParentDealType._contains = prefix;
var data = query.run();
//Returns four records which is correct.
var testStockNo = prefix+month+countstring+year;
console.log("Test Stock Number " + j + ": " + testStockNo);
for (i = 0; i < data.length; i++){
console.log("data[i][2]: " + data[i][2]); //results: undefined
console.log("data[2]: " + data[2]); //results: all fields of 3rd query result.
if(data[i][2] === testStockNo){
k++;
break;
}else{
console.log("No Match");
}
}
Even if testStockNo equals the value in field:TStockNo, the log displays:
Test Stock Number 1: C1200118
data[i][2]: undefined
data[2]: Record : { TIndex: 8, TVin8: HS654987, TStockNo: null,
TParentStkNo: GSD6578, TYear: 2010, TMake: NISSAN, TModel: PICKUP,
TMileage: 24356, ParentDealType: C}
No Match
Issue/Solution:
query.run() returns array of records and NOT a array of arrays(2D). You should access the Record value using it's key instead of a index.
Snippets:
console.log("data[i][TStockNo]: " + data[i]['TStockNo']);
console.log("data[i].TStockNo: " + data[i].TStockNo);
console.log("data[2]: " + data[2]);
References:
Query#Run

Converting a string (150000) to time ( hh:mm:ss) in javascript

I am new to JS. Can anybody tell me whether is it possible to convert a string ( eg: "140000") to time (eg: 14:00:00) in Javascript.
You can do this:
Split your text into an array using the split method.
Map over your array and replace every item that has the index can be divided by to with the item + : using the map method.
Join your array to be string again using the join method.
Remove the last : from your result.
let convertToTime = str => str
.split("")
.map((item, index) => (index % 2 != 0) ? item + ':' : item)
.join("")
.slice(0, -1);
console.log(convertToTime("140000")); // 14:00:00
console.log(convertToTime("173003")); // 17:30:03
console.log(convertToTime("225510")); // 22:55:10
console.log(convertToTime("010201")); // 01:02:01
That's a bit tricky, because well... there is no Time object in Javascript.
There is only a Date object, although the name is kind of misleading, because the Date object contains both "Date" and "Time". There is a reason behind this seemingly confusing convention. I'm not going to go into too long of a detail, but to put it shortly, referencing "Time" is unreliable (to human brain).
For that, I'll just recommend reading this answer which explains it quite brilliantly.
Either way, this simple function will do the trick.
toDate(what){}
Upon sucessful string of time conversion, return a newly created Date object. Otherwise return -1. (or you could throw an Exception instead, up to you really)
var str = '143025';
function toDate(what){ // turns string into Date() object
var result;
if (what.length === 6) { // must be in HH:MM:SS length
var timearr = what.match(/.{1,2}/g); // splits time to HH:MM:SS array
if (timearr[0] <= 24 && timearr[1] <= 60 && timearr[2] <= 60){
return new Date('01 January 0001 ' + timearr[0] + ':'+
timearr[1] + ':' + timearr[2] + ' GMT'); // valid time format
}
}
return -1;
}
var mytime = toDate(str);
console.log(mytime.toUTCString());
Note, Date by itself is only expression of serialized numbers, if you want to return the time as result, you need to use the .toString or .toUTCString method from the Date.prototype object.

how to perform aggregation and expression from given string (formula)?

I want to do aggregation operation by identifying "sum(a)" as in the below string and need to perform some logical operation to get sum, avg, count or percentage(i have separate method to do that so don't worry about that).
expression like below,
= '"sum(a)" * 10 + "count(b)"'
Here a and b is key word. Based on the key word i will perform the operation. But i need to identify operation name and key word in the above string.
Then, I need to execute the expression without using eval function for following expression. like below,
"10 * 10 + 22"
Note: I am trying to use Function constructor, but i am facing performance issue.(Use string to call function without eval())
You can use Array.proptotype.reduce:
const values = [2,3,4];
const sum = values.reduce((sum, current) => sum + current, 0);
console.log(sum * 10 + values.length);
function sum(values){
return values.reduce(function(s, n){
return s + n;
}, 0)
}
function count(values){
return values.length
}
alert( sum([2,3,4]) * 10 + count([2,3,4]) )

Postgresql9.3 query optimization

I am using postgresql 9.3 in my node.js application. In my database i have some 7lakhs records now. Also in my database i have json datatype column.
My query is as following:
EXPLAIN ANALYSE select id_0, name_0, id_1, name_1, id_2, name_2, id_3, name_3, id_4, name_4, latitude, longitude, to_char(collecteddate, 'dd/mm/yyyy') as collecteddate, key, value->>'xxxxx' as value from table where
CAST(value->'xxxxx'->> 'aaaaa' as INTEGER)BETWEEN 1 and 43722 and value->'PCA_2011'->> 'aaaaa' NOT LIKE ' ' and
CAST(value->'xxxxx'->> 'bbbbb' as INTEGER)BETWEEN 1 and 100 and value->'xxxx'->> 'bbbbb' NOT LIKE ' '
and leveltype = 'nnnn' and id_1= 'ww' and id_0 = 'uuu' and collecteddate = '2011-03-31';
This query will retrieve almost 1lakh records and takes 3 secs to be executed. I have created index for the json column and also the columns in where conditions. But i think its very long time to execute. Is there any way to reduce the execution time. I am new to this database optimization concepts, is there any optimization techniques to reduce my execution time to some milli seconds. Thanks in advance..
EDIT:
My index definition:
CREATE INDEX index_pop on table (id_0, id_1, collecteddate, leveltype, key, (value->'xxxxx'->>'aaaa'));
My Explain analyses result:
"Bitmap Heap Scan on table (cost=1708.27..59956.46 rows=1 width=132) (actual time=880.576..5137.266 rows=93615 loops=1)"
" Recheck Cond: (((id_0)::text = '356'::text) AND ((id_1)::text = '9'::text) AND (collecteddate = '2011-03-31'::date) AND ((leveltype)::text = 'pppp'::text))"
" Filter: ((((value -> 'xxxx'::text) ->> 'aaaa'::text) !~~ ' '::text) AND (((value -> 'xxxxx'::text) ->> 'bbbb'::text) !~~ ' '::text) AND ((((value -> 'xxxxx'::text) ->> 'aaaaa'::text))::integer >= 1) AND ((((value -> 'PCA (...)"
" Rows Removed by Filter: 4199"
" -> Bitmap Index Scan on index_name (cost=0.00..1708.27 rows=37856 width=0) (actual time=828.856..828.856 rows=97814 loops=1)"
" Index Cond: (((id_0)::text = '356'::text) AND ((id_1)::text = '9'::text) AND (collecteddate = '2011-03-31'::date) AND ((leveltype)::text = 'ppppp'::text))"
"Total runtime: 5211.271 ms"
ALso 1 more thing: Bitmap Index Scan on index_name is different index other than in my where condition index, also y only 1 index is earched??

Categories

Resources