With documents like:
{
_id: 123,
events: {
someEvent:{
created: ISODate("2015-06-27T16:51:03.000Z"),
metadata: {
some: "thing"
}
},
anotherEvent:{
created: ISODate("2015-06-27T16:51:01.000Z"),
metadata: {
some: "thing"
}
}
}
}
This is a simplified example of the data. The events object can have between 200 and 3,000 fields. There are thousands of documents like this.
I'm looking to use mapreduce on the collection so I only return one of the events (the one with the latest 'created' date) for each document in the collection.
Is this possible?
Yes it's possible. MapReduce is a bit of a "blunt edged sword" for this, but your element structure is "not great" and a possible 3000 elements needs something like this:
db.collection.mapReduce(
function() {
var doc = this;
var plucked = Object.keys(doc.events)
.map(function(key) {
var myObj = doc.events[key];
myObj.key = key;
return myObj;
})
.sort(function(a,b) {
return ( a.created > b.created )
? -1 : ( a.created < b.created )
? 1 : 0;
})[0];
emit(this._id, plucked);
},
function() {},
{ "out": { "inline": 1 } }
);
So basically that cylces though "events" and reshapes the data somewhat. Then you .sort() on the "created" key of the results in "descencing order" and just pull the first array element.
The "reducer" does nothing here. But this is just a way for the server to do the filtering.
Related
Scenario:
I am making a generic function that returns a boolean depending on logical AND statements, however, the function being generic accept multiple type of objects and arrays, and the statements can vary depending on the objects.
at the moment I have something like this
private async myFunction(
myArray: myArrObj[],
myObj : myObj,
): Promise<boolean> {
return (
myArr.some(
(a) =>
a.status1=== "*" ||
a.status1 === myObj.status1.status1Id
) &&
myArr.some(
(a) =>
a.status2=== "*" ||
a.status2 === myObj.status2.status2Id
) &&
myArr.some(
(a) =>
a.status3=== "*" ||
a.status3 === myObj.status3.status3Id
) &&
myArr.some(
(a) =>
a.status4=== "*" ||
a.status4 === myObj.status4.status4Id
)
)
}
the issue is not being able to know what kind of array is passed and how many checks are needed, how can I make a return? My idea was storing each array.some method in an array and join them with " && ", this approach would require to execute something from a string, which I'm not sure is the most secure thing to do, since eval is not secure at all.
to get the myObj statuses I could just use a for loop and store the the the property in a string.
I can't come up with a good solution, so feel free to propose something new if my idea is not good enough
As noted by others in the comments, it would help if you had a reproducible example with sample data. That being said, from your comment:
but the statuses and id's have different names, some id's are .nameId, and some are just .id , but the statuses themselves have the same name, so instead of status1 and obStatus1 it really should be status1 and status1
Breaking this down:
but the statuses and id's have different names, some id's are .nameId, and some are just .id
You could try to see if nameId exists and fall back to id.
but the statuses themselves have the same name, so instead of status1 and obStatus1 it really should be status1 and status1
When myArr entries share keys with myObj, then you could simply loop through myObj's keys.
async function myFunction(myArr, myObj) {
// Fallback value for if .nameId and .id both don't exist.
// Falling back to `undefined` would cause a bug / false positives.
const notFound = Symbol();
// Loop through every key:value pair in the input object.
return Object.entries(myObj).every(([myObjKey, myObjValue]) => {
// Handle both `.nameId` and `.id`
const id = myObjValue[`${myObjKey}Id`] ?? myObjValue.id ?? notFound;
// If `myArrObj`'s children only ever contain exactly
// a single key { status2: { someRandomKey: 123 } }, then you
// could use myObjValue[Object.keys(myObjValue)[0]];
// For this key--for example "status1"--is there *any* array entry
// in `myArrObj` that has the same key and value or "*"?
return myArr.some((a) => {
return a[myObjKey] === '*' || a[myObjKey] === id;
});
});
}
With the following sample data:
const sampleArr = [
{ status3: "*" },
{ status2: 234 },
{ status1: 123, thisIsAnUnusedKey: true },
{ status4: 456 },
{ name: "Foobar" },
{ thisIsAnUnusedArrayEntry: true },
];
const sampleObj = {
status1: {
status1Id: 123,
},
status2: {
status2Id: 234,
},
status3: {
status3Id: 345,
},
status4: {
// Different key
id: 456,
},
name: {
// Different dataType
nameId: "Foobar"
}
};
myFunction(sampleArr, sampleObj).then(console.log); // Logs `true`
I want to perform a query on this collection to determine which documents have any keys in things that match a certain value. Is this possible?
I have a collection of documents like:
{
"things": {
"thing1": "red",
"thing2": "blue",
"thing3": "green"
}
}
EDIT: for conciseness
If you don't know what the keys will be and you need it to be interactive, then you'll need to use the (notoriously performance challenged) $where operator like so (in the shell):
db.test.find({$where: function() {
for (var field in this.settings) {
if (this.settings[field] == "red") return true;
}
return false;
}})
If you have a large collection, this may be too slow for your purposes, but it's your only option if your set of keys is unknown.
MongoDB 3.6 Update
You can now do this without $where by using the $objectToArray aggregation operator:
db.test.aggregate([
// Project things as a key/value array, along with the original doc
{$project: {
array: {$objectToArray: '$things'},
doc: '$$ROOT'
}},
// Match the docs with a field value of 'red'
{$match: {'array.v': 'red'}},
// Re-project the original doc
{$replaceRoot: {newRoot: '$doc'}}
])
I'd suggest a schema change so that you can actually do reasonable queries in MongoDB.
From:
{
"userId": "12347",
"settings": {
"SettingA": "blue",
"SettingB": "blue",
"SettingC": "green"
}
}
to:
{
"userId": "12347",
"settings": [
{ name: "SettingA", value: "blue" },
{ name: "SettingB", value: "blue" },
{ name: "SettingC", value: "green" }
]
}
Then, you could index on "settings.value", and do a query like:
db.settings.ensureIndex({ "settings.value" : 1})
db.settings.find({ "settings.value" : "blue" })
The change really is simple ..., as it moves the setting name and setting value to fully indexable fields, and stores the list of settings as an array.
If you can't change the schema, you could try #JohnnyHK's solution, but be warned that it's basically worst case in terms of performance and it won't work effectively with indexes.
Sadly, none of the previous answers address the fact that mongo can contain nested values in arrays or nested objects.
THIS IS THE CORRECT QUERY:
{$where: function() {
var deepIterate = function (obj, value) {
for (var field in obj) {
if (obj[field] == value){
return true;
}
var found = false;
if ( typeof obj[field] === 'object') {
found = deepIterate(obj[field], value)
if (found) { return true; }
}
}
return false;
};
return deepIterate(this, "573c79aef4ef4b9a9523028f")
}}
Since calling typeof on array or nested object will return 'object' this means that the query will iterate on all nested elements and will iterate through all of them until the key with value will be found.
You can check previous answers with a nested value and the results will be far from desired.
Stringifying the whole object is a hit on performance since it has to iterate through all memory sectors one by one trying to match them. And creates a copy of the object as a string in ram memory (both inefficient since query uses more ram and slow since function context already has a loaded object).
The query itself can work with objectId, string, int and any basic javascript type you wish.
I have an application where an object is used to display a tree view of files on a user's system. It's structured like so:
[{
text: 'C:/',
type: 'dir',
nodes: [
{
text: 'foo',
type: 'dir',
nodes: [] // And so on
},
{
text: 'bar',
type: 'file'
}
}]
In keeping with conventions, I'd like directories to be displayed first and files to be displayed second. Unfortunately, my data is retrieved in alphabetical order regardless of item type.
To remedy this I wrote a nice recursive function
var sort = function (subtree)
{
subtree = _.sortBy(subtree, function (item)
{
if (item.nodes)
{
sort(item.nodes)
}
return item.type
});
}
var tree = someTreeData;
sort(tree);
I'm using lodash to sort each of the nodes arrays alphabetically by file type. Unfortunately the subtree does not appear to reference the tree object as when I log its output it remains unsorted. How can I remedy this?
You can use JavaScript’s built-in Array.prototype.sort function, which does sort in-place. It accepts two arguments and performs a comparison. Note that sorting item.notes inside the sortBy key extractor is kind of inappropriate.
function isDirectory(node) {
return !!node.nodes;
}
function sortTree(subtree) {
subtree.sort(function (a, b) {
return a.type < b.type ? -1 :
a.type > b.type ? 1 : 0;
});
subtree
.filter(isDirectory)
.forEach(function (node) {
sortTree(node.nodes);
});
}
I have a an array of objects, that contains and array of objects, that contains an array of objects.
What I want to do is to filter on a property in the last array of objects.
Here is an example dataset
var sampleData=[
{
"name":"Cooking",
"shelve":[
{
"name":"Shelve 1",
"drawers":[
{
"Name":"Left"
},
{
"Name":"Middle"
}
]
},
{
"name":"Shelve 2",
"drawers":[
{
"Name":"Middle side"
},
{
"Name":"Left"
},
{
"Name":"Up"
}
]
}
]
},
{
"name":"DBs",
"shelves":[
{
"name":"Shelve 3",
"drawers":[
{
"Name":"asdfasdf"
},
{
"Name":"New Test 12"
}
]
}
]
}
];
Basically what I would like to do is to be able to get back the objects where drawer.name contains a substring. I would like for it to return the full object graph so if I searched for Middle I would expect to get back
var sampleData=[
{
"name":"Cooking",
"shelve":[
{
"name":"Shelve 1",
"drawers":[
{
"Name":"Middle"
}
]
},
{
"name":"Shelve 2",
"drawers":[
{
"Name":"Middle side"
}
]
}
]
}
];
I was hoping that using underscore would help and I initially just tried to nest filters but that did not work.
I tried the following and while found would evaluate to true when it should it would not filter as I hoped.
var secs = _.filter(sampleData, function(section) {
_.filter(section.shelve, function(shelve) {
_.filter(shelve.drawers, function(drawer) {
var found = drawer.Name.indexOf('Middle') !== -1;
if(found) {
var xa = 'found it!!';
}
return drawer.Name.indexOf('Middle') !== -1;
});
});
});
Here is a quick jsfiddler example. http://jsfiddle.net/cnalk/GtfNj/3/
Unfortunately, a basic array filter like _.filter doesn't provide a way to do two-step filtering (first, by values of an array; then by how many values passed), so you need to write your own filter function for each layer, such as:
function filter_shelf( shelf ) {
var filtered = _.omit( shelf, 'drawers' );
filtered.drawers = _.filter( shelf.drawers, filter_drawer );
return filtered.drawers.length && filtered;
}
Complete fiddle: http://jsfiddle.net/GtfNj/5/
Each step considers the property it needs to filter, filters it by the step "below" it, then returns a filtered copy of itself or false if the property ended up empty from the filter. The step "above" it then filters based on that result. And so on...
Ps. your sample data uses shelve or shelves inconsistently.
I'm quite new to Backbone so I am getting into some problems I can't quite figure out.
I have a Backbone collection with a bit over 100 items. I want to filter these with an array of ids, that is working fine, but I want the order of the items also based on this array's order of items. That is not working. The other sorting methods seems to be asciibetical based, that's not what I need either. Is it possible to get items using this filter, and then also put them into the collection in the order I've defined?
I have an array of id's that I filter with, this array looks like this:
var dDefaultItems = ['1','2','146','3','4','9','26','8','96','10','11','54','145','273','38'];
The code for the collection and filtering looks like this:
var ChannelCollection = Backbone.Collection.extend({
fetch : function() {
var params = _.extend({}, arguments, {
data : {
"groupnumber" : "1000"
}
});
this.constructor.__super__.fetch.apply(this, [params]);
},
model : Channel,
url : function () {
return utility.apiUrl('/myurl/tothething');
},
filterData: function(params) {
this.originalModels = this.models.slice();
_.each(params, function(val, key){
if (typeof val !== 'object') val = [ val ];
this.models = _.filter(this.models, function(model){
return _.indexOf(val, model.get(key)) !== -1;
}, this);
}, this);
return this.reset(this.models).toJSON();
},
parse : function(json) {
return json.channelInfoList;
}
});
Then I render this in a view with this code (there's other bits of code for defining model and other attributes that I don't think is relevant, I may be wrong, but I'm thinking someone will know what I need from looking at this.)
var ChannelListView = Backbone.View.extend({
initialize: function() {
var _this = this;
currentChannelList = new ChannelCollection();
currentChannelList.once("sync", function() {
_this.render();
});
currentChannelList.fetch();
},
render : function() {
var _this = this;
$(_this.el).empty();
dust.render("list-channels", { channelList : currentChannelList.filterData({id: dDefaultItems})} , function(err, html) {
var $el = $(html).appendTo($(_this.el));
});
}
});
Backbone collections are automatically sorted by the order of insertion, unless you implement Collection#comparator. The problem is that your filtering algorithm is not producing an ordered output.
If you need to maintain an ordered collection only when filtering by id, I would suggest implementing a separate method, because seach by id is far faster compared to search by arbitrary attributes:
filterById: function(idArray) {
return this.reset(_.map(idArray, function(id) { return this.get(id); }, this));
}
Usage:
collection.filterById(['1', '2', '146', '3', '4', '9', '26', '8', '96', '10', '11', '54',' 145', '273', '38']);