I am running a script on a large dataset to expand existing information. e.g:
...
{
id : 1234567890
},
{
id : 1234567891
},
...
becomes
...
{
id : 1234567890,
Name : "Joe"
},
{
id : 1234567891,
Name : "Bob"
},
...
I am doing this via the following code:
for(var cur in members)
{
curMember = members[cur];
// fetch account based on curMember.id to 'curAccount'
if(curAccount != null)
{
curMember.DisplayName = curAccount.DisplayName;
}
}
For the most part, this works as expected. However, once in a while (in the order of tens of thousands of entries), the result looks like this:
...
{
id : 1234567891,
Name : "Bob",
Name : "Bob"
},
...
I now have data which is in an invalid format and cannot be read by the DB, since duplicate property names doesn't make sense. It is occurring for random entries when the script is re-run, not the same ones every time. I need either a way to PREVENT this from happening, or to DETECT that it has happened so I can simply reprocess the entry. Anyone know what's going on here?
EDIT: After further investigation, the problem appears to occur only when the objects being modified come from a MongoDB query. It seems that if code explicitly sets a value to the same element name more than once, the field will be duplicated. All elements of the same name appear to be set to the most recently specified value. If it is only assigned once as in my original problem, it is only duplicated very rarely. I am using MongoDB 2.4.1.
Got it all figured out. MongoDB has a bug up to shell version 2.4.1 which allows duplicate element names to be set for query result objects. Version 2.4.3, released just this Monday, has a fix. See https://jira.mongodb.org/browse/SERVER-9066.
I don't really get your problem. If you apply identical property names to an object in ECMAscript, that property will just get overwritten. The construct in your snippet, can never be exist in that form on a live-object (excluding JSON strings).
If you just want to detect the attempt to create a property which is already there, you either need to have that object reference cached beforehand (so you can loop its keys) - or -
you need to apply ES5 strict mode.
"use strict";
at the top of your file or function. That will assure that your interpreter will throw an exception on the attempt to create two identical property keys. You can of course, use a try - catch statement to intercept that failure then.
Seems like you cannot intercept errors which get thrown because of strict mode violation.
Related
Should I store objects in an Array or inside an Object with top importance given Write Speed?
I'm trying to decide whether data should be stored as an array of objects, or using nested objects inside a mongodb document.
In this particular case, I'm keeping track of a set of continually updating files that I add and update and the file name acts as a key and the number of lines processed within the file.
the document looks something like this
{
t_id:1220,
some-other-info: {}, // there's other info here not updated frequently
files: {
log1-txt: {filename:"log1.txt",numlines:233,filesize:19928},
log2-txt: {filename:"log2.txt",numlines:2,filesize:843}
}
}
or this
{
t_id:1220,
some-other-info: {},
files:[
{filename:"log1.txt",numlines:233,filesize:19928},
{filename:"log2.txt",numlines:2,filesize:843}
]
}
I am making an assumption that handling a document, especially when it comes to updates, it is easier to deal with objects, because the location of the object can be determined by the name; unlike an array, where I have to look through each object's value until I find the match.
Because the object key will have periods, I will need to convert (or drop) the periods to create a valid key (fi.le.log to filelog or fi-le-log).
I'm not worried about the files' possible duplicate names emerging (such as fi.le.log and fi-le.log) so I would prefer to use Objects, because the number of files is relatively small, but the updates are frequent.
Or would it be better to handle this data in a separate collection for best write performance...
{
"_id": ObjectId('56d9f1202d777d9806000003'),"t_id": "1220","filename": "log1.txt","filesize": 1843,"numlines": 554
},
{
"_id": ObjectId('56d9f1392d777d9806000004'),"t_id": "1220","filename": "log2.txt","filesize": 5231,"numlines": 3027
}
From what I understand you are talking about write speed, without any read consideration. So we have to think about how you will insert/update your document.
We have to compare (assuming you know the _id you are replacing, replace {key} by the key name, in your example log1-txt or log2-txt):
db.Col.update({ _id: '' }, { $set: { 'files.{key}': object }})
vs
db.Col.update({ _id: '', 'files.filename': '{key}'}, { $set: { 'files.$': object }})
The second one means that MongoDB have to browse the array, find the matching index and update it. The first one means MongoDB just update the specified field.
The worst:
The second command will not work if the matching filename is not present in the array! So you have to execute it, check if nMatched is 0, and create it if it is so. That's really bad write speed (see here MongoDB: upsert sub-document).
If you will never/almost never use read queries / aggregation framework on this collection: go for the first one, that will be faster. If you want to aggregate, unwind, do some analytics on the files you parsed to have statistics about file size and line numbers, you may consider using the second one, you will avoid some headache.
Pure write speed will be better with the first solution.
I have one object called bookings, and inside it I have several properties, and i want extend with Object.assign, like this:
let data = Object.assign(booking, {
hiw: event.hiw[booking.locale],
tip: event.tip[booking.locale],
start: moment(event.start).format('L')
});
But when I print the data, the result will be the same object from the source (booking), so hiw, tip and start will be ignored, but... if I try to do:
let data = Object.assign({}, {
hiw: event.hiw[booking.locale],
tip: event.tip[booking.locale],
start: moment(event.start).format('L')
});
This will work perfect.
My question is: what am I doing wrong here? Why can't I extend booking and I can extend the empty object?
That's definitely not a problem with async code, when i try to extend booking, he already exist with all properties inside.
I also was trying to use underscore extend method, and the behavior is exactly the same.
Mongoose documents (model instances) are special: they will only print properties that are present in the schema.
If properties aren't defined there, they won't show up when you console.log() them (and also not if you convert the document to a plain JS object with obj.toObject()).
This means that using Object.assign() will only work if you assign properties that are also present in the schema. Any properties that aren't declared will not be shown (nor saved to the database).
If your intention is to use the document for output, you should convert it to a proper JS object first before assigning to it:
let data = Object.assign(booking.toObject(), {
hiw : event.hiw[booking.locale],
tip : event.tip[booking.locale],
start : moment(event.start).format('L')
});
I have an IndexedDB of changes. I add an item like this, and then log the result to check the key has been created successfully:
_this._idb.add('steps', step).done(function (items) {
var item = items[0];
_logger.log("ADDED STEP", { id: item.__id__, step: item }, "CT");
});
The output from this is as expected:
...as you can see, the id has been added to the object when it is stored.
However, when I query the db to getback a list of objects, using this code:
this._idb.steps.query('timestamp').bound(start, end).execute().done(function (results) {
_logger.log("Results", results, "CT");
}
I don't get the id as part of the object that is returned:
... and the lack of id makes updating and deleting impossible.
How can I get the id of the item when I query indexed db using db.js - or am I approaching this in the wrong way, and is there something else I should be doing?
(Note: I'm using TypeScript to compile the JS, but I don't think that's especially relevant to this question)
This is expected behaviour, you're only going to get the __id__ property if you don't define a keyPath in your db schema.
Because there's no keyPath defined the value is not associated with it in indexeddb, it's only added to the resulting object after it has been added, because at that point in time we know the auto-incremented value that IndexedDB has assigned to it.
Since the value isn't really part of the object I don't have any way to assign it to the object when it comes out during a query, maybe I could use the position in the array but that's more likely to be wrong than right.
If you want the ID to be persisted against the object then you need to define a keyPath as part of the object store schema and the property will be added to the resulting object and available and it will be on the object returned from a query.
Disclaimer - I wrote db.js
Looking at the source, __id__ is only defined when your keyPath is null in the add() method. From what I'm seeing, you'll never see this in a query() response.
In IDB null keyPaths are allowed only when using auto-incrementing ("out-of-line") keys. So if you're getting the object back, it should have an auto-incrementing key on it or some other keyPath.
The __ prefix in JavaScript usually means the developer intended it to be a "private" property. I'm guessing this is for internal use and you shouldn't be counting on this in your application code.
Consider using explicit, so-called "in-line" keys on your object store.
The goal of db.js is easy and simple to use. Your is advanced use case.
As shown in this example
javascript-use-variable-as-object-name
I am using eval to use a DOM attribute to select an element from an array. Though there is no direct way for the user to change the input, I want to be as secure as possible and make sure that the variable is indeed an integer before I evaluated it.
Which of the following would be the best, most secure, way?
$(".listitem").click(function(){
var id = $(this).attr("record-id");
if(!isNaN(new Number(id))){
Storage.search.nearby.currec = rowsHolder[eval(id)];
}else{
// send email to admin, shut down
}
});
or
$(".listitem").click(function(){
var id = $(this).attr("record-id");
if(parseInt(id)){
Storage.search.nearby.currec = rowsHolder[eval(id)];
}else{
// send email to admin, shut down
}
});
More, but not required info:
Basically I am pulling down a large JSON string from online, containing an array of records. Upon building a table from the info using a for statement ( for(i in array) ), I push each row into an array called rowsHolder and give the tr an attribute of record-id="i". Then when the user clicks the row, I call the method you see above. I am using PhoneGap with JQuery Mobile.
As always, thanks for the input
-D
There is absolutely no reason to use eval here.
If your id is kind of a number, use parseFloat(id) to get it. Unnecessary as it would be converted back to a string when used as a property name, though.
If your id is an integer, use parseInt(id, 10) to get it. Unnecessary as it would be converted back to a string when used as a property name, though.
If your id is a string, just let it be a string. The property name you use it for would be one anyway.
I'm writing an application for Google Chrome (targeted audience is an internal team) that allows a user to manipulate elements from within an iframe. The user is able to use her mouse to select DOM elements and to perform various actions to them, such as changing colors, fonts, etc.
I'm using a nodeIterator method to select only elements that have IDs or class names. Then for each of those elements, I add some element-specific properties to an object, and push that object to an array. Then, I open an IndexedDB database and add each object in the array to the database.
My problem is this: Everything works fine so long as I don't include a reference to the element in the object.
// Works fine
array.push({
width : currentNode.offsetWidth,
height : currentNode.offsetHeight,
top : currentNode.style.top;
left : currentNode.style.left;
});
// Doesn't work
array.push({
elem : currentNode,
width : currentNode.offsetWidth,
height : currentNode.offsetHeight,
top : currentNode.style.top;
left : currentNode.style.left;
});
Google chrome fails silently (nothing in the console at all) after trying to add the first element to the IndexedDB store.
My question is this: Has anyone else experienced this behavior and is this a browser-specific bug?
I'll distill my code to JSfiddle tomorrow. Thanks in advance.
IndexedDB store structured clone of your object. Basically your data will converted into JSON object, these exclude Element or Node data type.
However fail silently is not an expected behaviour. Accordingly to the structured clone algorithm, it should throw DataCloneError.
Is it necessary to save the DOM element? Can you just save the ID of the DOM element and retrieve the element back by its ID?
The indexeddb is only capable of storing data that doesn't have circular references. There is maybe one thing you can try. Sometime ago I wrote a blog post on how you can serialize and deserialize functions to JSON. Maybe this can help you, but I would advace you not to store complete elements unless there is no other option. This will add a lot of unnecessary data into your database, and it's possible you'll lose information when serializing to JSON.
You should get an exception in chrome (I just tried on Chrome 23) from the put() itself, which means if you have an onerror handler, it won't get called because the exception gets called first:
i.e. if you have
req = db.transaction("foo", "readwrite").objectStore("foo").put({...data with dom nodes })
req.onsuccess = ...
req.onerror = ...
The exception will be thrown by the first line.