Is there a quicker way to do this? From what I understand it cannot be done in a projection using the aggregation pipeline. Do I have to pre-calculate this? I basically want to emit part of the date (i.e. an hour) in a map function (for a map-reduce). I appreciate you taking the time to help me :-)
db.events.find().snapshot().forEach(
function (e) {
e.StartTime = new Date(e.start_time);
db.events.save(e);
}
)
The Bulk Operations API is the fastest "safe" way to do this:
var bulk = db.events.initializeOrderedBulkOp();
var count = 0;
db.events.find({ },{ "start_time": 1 }).snapshot().forEach(function(e) {
bulk.find({ "_id": e._id }).updateOne({
"$set": { "StartTime": new Date(e.start_time) }
});
count++;
if ( count % 1000 == 0 ) {
bulk.execute();
bulk = db.events.initializeOrderedBulkOp();
}
});
if ( count % 1000 != 0 )
bulk.execute();
That will only send and return from the server once per every 1000 documents read. So the decreased traffic there saves a lot of time, as does working with only the required fields.
If this is absolutely a "one off" operation that does not need to continue to happen in production, and if you are able to do so then you can always use db.eval(). But please read the documentation and warnings there as it is not a very good idea:
db.eval(function() {
db.events.find({ },{ "start_time": 1 }).snapshot().forEach(function(e) {
db.events.update(
{ "_id": e._id },
{ "$set": { "StartTime": new Date(e.start_time) } }
});
]);
But if you are looking for any other way to "convert" a field, there is presently no way for an update operation to refer to an existing value of a field and use it to update another, or even itself. There are exceptions such as $inc or $bit, but these have specific purposes.
Related
I'm working with an API to parse stock market data in NodeJS.
The program works fine up to around 30K transactions per second, which is fine for the slower parts of the day. During market open though, it can exceed 100K transactions per second and the heap explodes to 30GB+, lagging and eventually crashing, despite being on an extremely fast machine.
I'm relatively new to NodeJS, but the bottleneck appears to be the below section of code, I obtained from a sample client that is in an extension of the EventEmitter class.
onMessage( data ){
data = JSON.parse( data )
data.map(( msg ) => {
if( msg.ev === 'status' ){
console.log('Status Update:', msg.message)
}
this.emit(msg.ev, msg)
})
Is there a more efficient way to code this that could give a 2x+ speed improvement? The individual bits of JSON are quite small such as the following:
{
"ev": "T",
"sym": "MSFT",
"x": 4,
"i": "12345",
"z": 3,
"p": 114.125,
"s": 100,
"c": [
0,
12
],
"t": 1536036818784
}
In my comments, I suggested you take the data.map() loop out and send all that at once and then let the client iterate through the individual pieces:
onMessage( data ){
data = JSON.parse( data )
this.emit("multiMsg", data);
}
Then, the client would have this:
socket.on("multiMsg", (data) => {
for (const item of data) {
// process each item here
console.log(item);
}
});
If you're still getting lots of onMessage() calls in rapid fire, you can accumulate them for a short period of time (the time to choose depends upon the acceptable time delay for your app). That also allows you to drastically reduce the number of separate messages you send to the client during high traffic times which can be many, many times more efficient for the server. So, if you would have received 100 calls to onMessage() within 50ms, then this modification will send one accumulated message to the client, instead of 100 separate messages to the client.
Here's an idea how that could work:
const msgQueueTime = 500; // pick an appropriate time delay here
let msgQueue = [];
let msgQueueTimer = null;
onMessage( data ){
data = JSON.parse( data )
if (!msgQueueTimer) {
msgQueueTimer = setTimeout(() => {
// send data we have accumulated
this.emit("multiMsg", msgQueue);
msgQueue = [];
msgQueueTimer = null;
}, msgQueueTime);
}
// add data onto our queue array
msgQueue.push(...data);
}
I am trying to perform 2 operations in one findOneAndUpdate():
Update date in one field lastUpdatedTimestamp, set it to current date (this one works fine in my statement),
Update date in other field expiryTimestamp, by adding 1 day to $currentDate (I couldn't find a way to achieve it so I'm trying to $add 1 day to the the value read from the above field lastUpdatedTimestamp) - (I can't make this one work).
findOneAndUpdate(
{"_id":123},
{ $currentDate: {"lastUpdatedTimestamp":true}, $set: {"expiryTimestamp": {$add: ["$lastUpdatedTimestamp", 24*60*60000]}}}
)
Here's the error I'm receiving:
{ "ok" : 0.0, "errmsg" : "The dollar ($) prefixed field '$add' in 'expiryTimestamp.$add' is not valid for storage.", "code" : 52 }
Is it even possible? I'd appreciate your help.
You can use the setDate() method to set the "expiryTimestamp" value.
db.collection.updateOne(
{ "_id": 123 },
{ "$set": {
"lastUpdatedTimestamp": new Date(),
"expiryTimestamp": new Date().setDate(new Date().getDate() + 1)
}}
)
You don't need to use findOneAndUpdate unless you want to return the new or old document.
The marked answer is wrong in the sense that using new Date() will not use database timestamp which is important if your server and database hosted on different region and also count network time for sending data. To correctly do this, use $currentDate like this:
db.collection.findOneAndUpdate({_id: 123 }, {
$set: { /** update doc here **/ },
$currentDate: { lastUpdatedTimestamp: true}
});
Similarly using updateOne
db.collection.updateOne({_id: 123 }, {
$set: { /** update doc here **/ },
$currentDate: { lastUpdatedTimestamp: true}
});
I have collection named inventory where I have multiple documents that has values for each doc
{ "apples": 2 ,"oranges": 3, "carrots": 5 }
{ "apples": 4, "oranges": 6, "carrots": 9 }
How do I update push all fruits in to a single array on multiple documents like so:
{ "fruits": { "apples":2 ,"oranges":3 }, "carrots": 5 }
First thing to note here is that the example you give is not an array but just a sub-document for "fruits" that has different keys. An "array" in MongoDB would look like this:
{ "fruits": [{ "apples":2 } , { "orange":3 }], "carrot": 5 }
Also, aside from the term "fruits" being subjective, as with no other identifier you would have to specify a "list" of things that qualify as fruits, the other thing to consider is that there is no actual way in MongoDB at present to refer to the existing value of a field when processing an update.
What that means is you need to .find() each document to retrieve the data in order to be able to work with the sort of "re-structure" that you want. This essentially means looping the results an performing an .update() operation for each document.
The Bulk API for MongoDB 2.6 and greater can be of some help here, where at least the "write" operations to the database can be sent in batches, rather than one at a time:
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
var fruits = ["apples","oranges"];
var unset = {};
fruits.forEach(function(fruit) {
unset[fruit] = 1;
});
db.collection.find({}).forEach(function(doc) {
var fields = [];
fruits.forEach(function(fruit) {
if ( doc.hasOwnProperty(fruit) ) {
var subDoc = {};
subDoc[fruit] = doc[fruit];
fields.push(subDoc);
}
});
bulk.find({ "_id": doc._id }).updateOne({
"$unset": unset, "$push": { "fruits": { "$each": fields } }
});
count++;
if ( count % 1000 == 0 ) {
bulk.execute();
var bulk = db.collection.initializeOrderedBulkOp();
}
});
if ( count % 1000 != 0 )
bulk.execute();
That also uses the $each modifier for $push in order to add multiple array entries at once. The $unset operator can be safely called for fields that don't exist in the document so there is no need to check for their presence in the document as is otherwise required when constructing the array of elements to $push.
Of course if you actually want a document like what you gave an example of that is not actually an array, then you construct differently with the $set operator:
var fields = {};
fruits.forEach(function(fruit) {
if ( doc.hasOwnProperty(fruit) )
fields[fruit] = doc[fruit];
});
bulk.find({ "_id": doc._id }).updateOne({
"$unset": unset, "$set": { "fruits": fields }
});
Whatever the case is you need to loop the existing collection. There is no operation that allows you to "take" an existing value in a document and "use it" in order to set a new value from a server side perspective.
I've created a Collection containing 1 million documents, and I'm trying to select 50000 of these records based on the ObjectID, and update one of the values (i'm working in Mongo shell, running on Ubuntu).
Is it possible to define a 50000 document range? It doesn't matter which documents are included in the 50000 range, I simply want to ringfence a definite number of records and run an update operation using the primary id so that I can measure the performance time.
The code I've tried to run doesn't work:
use Assignment
var _start = new Date()
db.FlightsDate.update({$set:{Airtime: 8888}}).limit(50000).hint({_id:1});
var _end = new Date();
print("Time to Bulk Update AirTime key for 50000 documents… " + ((_end _start)/1000));
...i'm gathering that MongoDB needs me to include a query in the command to specify which docs are to be updated (I now understand from reading other posts that .limit won't constrain the number of records than an .update writes to).
Please can anyone advise a method that'll enable me to define the number of records to be updated?
Grateful for advice.
R,
Jon
If you are simply looking for a "range" that covers 50,000 of the documents in the collection then your best approach is to query and find the "starting" and "ending" documents of your range first. Then apply that "range" specification to your update.
var start_id = db.FlightsDate.find({}).limit(1).toArray()[0]._id;
var end_id = db.FlightsDate.find({}).skip(49999).limit(1).toArray()[0]._id;
var _start = new Date();
db.FlightsDate.update(
{ "_id": { "$gte": start_id, "$lte": end_id } },
{ "$set"; { "Airtime": 8888 } },
{ "multi": true }
);
var _end = new Date();
( _end - _start )/1000;
If you then wanted the next 50,000 in an additional range then :
var start_id = db.FlightsDate.find(
{ "_id": { "$gt": end_id } }
).limit(1).toArray()[0]._id;
var end_id = db.FlightsDate.find(
{ "_id": { "$gt": end_id } }
).skip(49999).limit(1).toArray()[0]._id;
And do it all again.
The point is you need to know where to "start" and when to "end" within a range to limit your update to just 50,000 documents without any other criteria to do so.
Also note the usage of "multi" in the update method there. By default, .update() does not "update" any more than one document, essentially being the first match. So what you mean to do is update "all documents in the range" and that is why you need to apply "multi" here.
I'm building a Sencha Touch app that utilizes the awesome calendar plugin https://github.com/SwarmOnline/Ext.ux.TouchCalendar , however, some custom implementation is needed in order to utilize the events functionality.
I've already tied the events template to a store, which fetches data from the server. It works as planned, but the problem is the plugin looks for ALL records in the store and counts each one as an event (because in the event model, it looks for "date" as the start and end point). So every day looks like has an event, even though those without "items" are blank, see: http://cl.ly/image/3j461O2L2Y1k. I only want to display events with "items"
My data from the server that comes back in the following format (many days do not have "items"):
[
{
"day":28,
"iscurrentmonth":false,
"issunday":false,
"date":"2013-05-28",
"items":[
{
"id":134513,
"title":"Subject",
"typeid":3,
"typename":"Essay",
"author":"Bryan Fisher",
"classname":"English 9A",
"classid":344499,
"courseid":60555
},
{
"id":134485,
"title":"Subject",
"typeid":3,
"typename":"Essay",
"author":"Bryan Fisher",
"classname":"English 10",
"classid":344500,
"courseid":60555
}
]
}
]
So, I have to change the data array's structure into the following format:
[
{
"date":"2013-05-28",
"id":134513,
"title":"Subject",
"typeid":3,
"typename":"Essay",
"author":"Bryan Fisher",
"classname":"English 9A",
"classid":344499
},
{
"date":"2013-05-28",
"id":134485,
"title":"Subject",
"typeid":3,
"typename":"Essay",
"author":"Bryan Fisher",
"classname":"English 10",
"classid":344500
}
]
How can I change the original object to match the new format? (take the "date" and insert it into the "items" node) ?
I am completely open to something like underscore.js
Thanks in advance
Maybe I over thought this whole thing...
A bit of a hack...
in TouchCalendarEvents.js I added the following method to check for an empty event div
hideOthers: function(){
var bar = $('.event-bar');
for (var i = 0; i < bar.length; i++){
var allBars = bar[i];
if (allBars.innerHTML == ''){
console.log('number ' + i + 'is Empty!' );
allBars.remove();
}
}
},
and call it in refreshEvents
refreshEvents: function(){
// scroll the parent calendar to the top so we're calculating positions from the base line.
if(this.calendar.getScrollable()){
this.calendar.getScrollable().getScroller().scrollTo(0,0);
}
this.removeEvents();
this.getViewModeProcessor().generateEventBars(); // in turn calls this.renderEventBars(this.eventBarStore);
this.createEventWrapper();
this.hideOthers();
if (this.getAllowEventDragAndDrop()) {
this.createDroppableRegion();
}
},
Works well enough for now!