So I have two Sequelize models with this relationship:
models.Note.belongsToMany(models.Topic, {
through: 'noteTopicRelation'
});
models.Topic.belongsToMany(models.Note, {
through: 'noteTopicRelation'
});
I can make a successful query to the Note model like so, getting all the Notes that belong to the Topic with the id of 2:
models.Note.findAll({
include: [{
model: models.Topic,
through: 'noteTopicRelation',
}]
where: {
'$topics.id$': 2
}
})
However, what if I only want a Note that has multiple specific Topics associated with it (i.e. a Note that is associated with Topics of ids 1, 4, 6)?
I have tried adding this operator on my where:
where: {
'$topics.id$': {$overlap: [1, 4, 6]}
}
But getting an error:
operator does not exist: uuid && text[]
Am I using Op.overlap incorrectly? Is there another way to achieve this result? Thank you!
EDIT: and just to clarify (sorry if this wasn't clear in my original post), I want to get the notes that are strictly associated with all of those Topics. Performing a '$topics.id$': [1, 4, 6] will get me notes that are associated with any of those Topics.
I think you want $in rather than $overlap; the latter maps to the PostgreSQL && operator which is meant for range types, not lists.
So I suggest trying:
where: {
'$topics.id$': [1, 4, 6]
}
The above will get notes which have ANY of the topic IDs (posted before the question was edited to say that only notes with ALL of the provided topic.ids should be returned).
As per the link to the discussion on the Sequelize github issues page in the comments below; one way to get notes with ALL of the topic IDs, you'll need to do something like the following:
var topicIds = [1, 4, 6];
models.NoteTopicRelation
.findAll({
include: [
{model: models.Topic, where: {id: topicIds}}
],
group: ['note_id'],
having: ['COUNT(*) >= ?', topicIds.length]
})
.then((noteTopicItems) => models.Note.find({
where: {id: noteTopicItems.map((item) => item.note_id)}
}))
.then((notes) => {
// do something with notes
});
Note that this method only reliably works if the link table (ie noteTopicRelation in your case) has only unique pairs of note_id & topic_id - ie. there is a unique key of some sort on these fields. Otherwise a topic can be assigned to a note more than once, which will throw up the COUNT(*). I believe the default "through" tables that Sequelize creates have a unique key on both fields; there are legitimate cases where this might not be desired however so I thought it worth mentioning.
Also note that I've made some assumptions about the column/property names of your noteTopicRelation model in the above query so you'll probably need to tweak them.
Another thing to note - the join from NoteTopicRelation to Topic isn't really necessary in the example case; you could achieve the same thing more efficiently using where: {topic_id: topicIds} (which would avoid the join to Topic) if you are only wanting to filter by topic.id. I've left the join there in case you're actually wanting to query on e.g. topic name or include other attributes from the Topic model/table in your where clause (e.g. an enabled attribute or similar).
Related
I'm trying to find the best way to group by category and iterate products in O (n) to get some insights from the categories.
I have the sample data:
[
{
"code": 25754,
"description": "ADAPTADOR BLUETOOH USB RECEPTOR DE AUDIO P2",
"price": 5.0,
"stock": 10,
"category": {
"id": 1,
"name": "Adapters"
}
},
{
"code": 20212,
"description": "ADAPTADOR CONECTOR HDMI FEMEA L / FEMEA",
"price": 2.8,
"stock": 20,
"category": {
"id": 2,
"name": "Eletronics"
}
},
]
I need to invert the relationship, having a list of categories with corresponding products, and for that i wrote this solution
function group_by_categories(products) {
const categories = {}
for (const product of products) {
const { category, ...cleanedProduct } = product
categories[category.id] = categories[category.id] || category
categories[category.id].products = categories[category.id].products || []
categories[category.id].products.push(cleanedProduct)
}
return Object.values(categories)
}
// returns
[
{ id: 1, name: 'Adapters', products: [ [Object] ] },
{ id: 2, name: 'Eletronics', products: [ [Object] ] }
]
But I am struggling in two things.
Is it the best way to reverse the relationship? How can I replicate this in another language like C, where I have no objects to use as unique keys?
Once you have this type of data, the only way to iterate categories and products (see how many items a category has, for example) is in O (n²)?
I appreciate all the help, even if you can only answer one question. Also, sorry for my bad English, I'm trying to be as clear as possible here.
So you have 3 issues. 1) Use the code/id as keys instead, 2) Use sets instead of arrays, and 3) use an appropriate data structure to avoid duplicating work that you've already done.
You really just want to map the connections not all the data. I believe code is probably unique to the product, so that is your key. Your category id is also likely unique, so you only need to consider that. The mapping structures should only concern themselves with the minimal amount of unique data. This may increase performance a fair bit as well as the amount of data that get's copied is probably 1/10 to 1/100 of yours in terms of # of characters (of course translating that exactly to time saved is difficult). Of course, that's a minor point compared to the O(N^2) performance, but just that would likely speed things up by a bit by that alone.
You should be using sets as well as the hash (object). Idea here is O(1) lookup, O(1) size check, O(1) inclusion test (IE: Does the category X have code Y in it?), and O(1) mapping back to the original data (code Y is a product with this information).
The other key thing is you really just want to map the connections not all the data. I believe code is probably unique to the product, so that is your key. Your category mapping structure should only concern itself with the minimal amount of unique data (this will increase performance by a lot as well).
Keep in mind, if you have access to an actual database that is probably the more ideal solution, 95% or so of the time once you start wanting to do more complex queries. I would say you almost certainly should be using one, there's probably a database that will suit your needs.
That being said, what you need is this if you don't need to go too far with your queries. It looks like you need to answer these three questions:
Given the code, what is the record (sample data)? This is just a simple code:product object. IE: {25754: {code: ..., price: ..., stock: ..., ...}, {20212: {code: ..., price: ..., stock: ..., ...}}
Given a category, what are the codes in that category? In this case you have a category_id:set(codes) lookup. Very important that you add codes to a set and not a list/array. Sets have O(1) to add/delete/inclusion, while lists/arrays have O(1) add, O(N) delete and O(N) inclusion check.
Given a category, how many are in that category? This is just a data[category].size check (length instead of size in some languages).
Main thing is to use dictionaries and sets for performance.
Time to build the lookups is likely O(P) where P is the total number products. Performance for the queries should be O(1) for each one you need to do.
To avoid O(N^2) performance the lookup should only be calculated once. Should you need to add/remove products from categories, you should adjust the lookup itself and not rebuild it every time. This may mean storing it in a database, building when you first run the app and keeping it in memory, or if the # of products isn't too much, building it at each request (by only using the minimal amount of data to build it, it may be more practical). Generally speaking, even with 1000 products, it should take probably milliseconds to build the category lookup and iterate over them.
Basically your category code should look like his after you've written out the methods.
...
category_lookup = build_category_lookup() # O(P)
product_lookup = build_product_lookup() # O(P)
...
products_for_category = product_lookup[category_id] # O(1)
products_count = products_for_category.length # O(1) | C: sizeof(products_for_category)
...
(Mostly code in ruby these days, so snake_case)
This is a document from my profiles collection:
{
_id: ObjectId("5f2ba3a43feccd0004b8698c")
userID: "238906554584137728",
serverID: "533691583845892100",
username: "W.M.K",
money: 15775,
__v: 4,
items: [...],
serverName: "W-15i: Overworld",
gladium: 7959.33,
stocks: [{
_id: {...},
stockID: "605b26d309a48348e05d88c9",
name: "GOOGL",
amount: 1,
desc: "Alphabet Inc's (Google) Stock."
}]
}
I'm using const profile = await Profile.findOne({userID: userID, 'stocks.stockName': stock.name})
EDIT: For one stock, it's as easy as profile.stocks[0].amount -=1. But when I have multiple stocks, how do I get the amount and manipulate it like amount -= 1?
there are few ways you can go through an array of objects in mongoose... you can map through the array, you can also use dot notation in an update. Either way one thing you'll need to do is mark the document modified because mongo doesn't detect changes in arrays of objects. To do this, make the changes and then use the markModified() function on the document. It takes a parameter of the array field name that was modified. in this case.... profile.markModified('stocks')
This is done after changes and before the save.
You can use dot notation with mongoose and you can also iterate through the array via map or forEach
The benefit of using map is that you can double it up with .filter() to filter the array for certain stocks
As already stated by Jeremy, it is basically a matter of array manipulations.
Here is one example:
for(let i = 0; i < profile.stocks.length; i++) {
profile.stocks[i].amount -= 1;
}
If you just want to update the profile on your database, you could also look into some very advanced mongodb querys (I think it's called "aggregation"), but that is probably overkill.
I'm trying to find all documents that do not contain at least one document with a specific field value. For example here is a sample collection:
{ _id : 1,
docs : [
{ foo : 1,
bar : 2},
{ foo : 3,
bar : 3}
]
},
{ _id : 2,
docs : [
{ foo : 2,
bar : 2},
{ foo : 3,
bar : 3}
]
}
I want to find every record where there is not a document in the docs block that does not contain at least one record with foo = 1. In the example above, only the second document should be returned.
I have tried the following, but it only tells me if there are any that don't match (which returns document 1.
db.collection.find({"docs": { $not: {$elemMatch: {foo: 1 } } } })
UPDATE: The query above actually does work. As many times happens, my data was wrong, not my code.
I have also looked at the $nin operator but the examples only show when the array contains a list of primitive values, not an additional document. When I've tried to do this with something like the following, it looks for the EXACT document rather than just the foo field I want.
db.collection.find({"docs": { $nin: {'foo':1 } } })
Is there anyway to accomplish this with the basic operators?
Using $nin will work, but you have the syntax wrong. It should be:
db.collection.find({'docs.foo': {$nin: [1]}})
Use the $ne operator:
db.collection.find({'docs.foo': {$ne: 1}})
Update: I'd advise against using $nin in this case.
{'docs.foo': {$ne: 1}} takes all elements of docs, and for each of them it checks whether the foo field equals 1 or not. If it finds a match, it discards the document from the result list.
{'docs.foo': {$nin: [1]}} takes all elements of docs, and for each element it checks whether its foo field matches any of the members of the array [1]. This is a Cartesian product, you compare an array to another array, each element to each element. Although MongoDB might be smart and optimize this query, I assume you only use $nin because "it has do to something with arrays". But if you understand what you do here, you'll realize $nin is superfluous, and has possibly subpar performance.
Example
I have two collections, one for Posts and one for Labels that look like this:
Post {
"_id": "WZTEGgknysdfXcQBi",
"title": "ASD",
"labels": {},
"author": "TMviRL8otm3ZsddSt",
"createdAt": "2016-01-14T08:42:42.343Z",
"date": "2016-01-14T08:42:42.343Z"
}
Label {
"_id": "9NCNPGH8F5MWNzjkA",
"color": "#333",
"name": "Grey Label",
"author": "TMviRL8otm3ZsddSt"
}
What I want to achieve is to have Posts with multiple Labels.
Problem is I cannot insert label data into the post.
I have a template to add new post and in there I am repeating over the labels. Then in the helpers I check which label is checked and store it into an array, but I cannot insert that array in the Posts collection.
'submit .add-entry': function(event) {
var title = event.target.title.value;
var description = event.target.description.value;
var checkedLabels = $('.label-checkbox:checked');
//initiate empty array
var labelsArray = [];
//go over the checked labels
for(i = 0; i < checkedLabels.length; i++){
var label = checkedLabels[i].value;
// store ids into array
labelsArray.push(label)
};
Posts.insert({
title: title,
description: description,
labels: labelsArray
});
Q1: Should I insert all the tags data or only the ID fetch more details from the Tags collection based on that ID?
Q2: How can I insert that array of labels into a Post? The code above does not work because it needs an Object
Q3 What is the best way to achieve such relationship between collections?
(Q1) You can insert all tag IDs to that labels list and keep that updated (Q3) I think that's a good practice; in your development process try to publish using composite collections with this package: https://github.com/englue/meteor-publish-composite
(Q2) Inserting is easy:
Post.insert({title:'Abc', labels:['one', 'two']});
You would need to use $addToSet to update. Here's an example of a possible method:
let setModifier = {
$addToSet: {
labels: myArrayOfLabelIDs
}
};
return Post.update({
_id: id
}, setModifier);
Note: Never use inserts on the client (like you did). My example can be added as a method on the server (use check() inside that) and can be called from the client upon submit like:
Meteor.call('myMethod', id, labelsArray)
Q1: Should I insert all the tags data or only the ID fetch more
details from the Tags collection based on that ID?
It depends: does the label change? If it mutable, then storing the ID only is best so that fetching the Label by ID will always return the proper data. Otherwise you need to update every Post with the new label info.
Q2: How can I insert that array of labels into a Post? The code above
does not work because it needs an Object
If the code does not work, it is likely that you are using some sort of schema that is not matched: to insert an array in place of an object, you need to modify the schema to accept an Array. You did not post details of a schema.
Q3 What is the best way to achieve such relationship between
collections?
You should look at 'denormalization': this is the term used for this type of things in a NoSQL database like MongoDB.
There is no straight answer, it depends on the relationship (1-1, 1-many, many-1) and the ratio of each 'many': in some cases it is best to nest data into objects if they are immutable or if dealing with updates is not too much of a pain.
this series of blog posts should help you get a better understanding:
http://blog.mongodb.org/post/87200945828/6-rules-of-thumb-for-mongodb-schema-design-part-1
I am trying to update a value in the nested array but can't get it to work.
My object is like this
{
"_id": {
"$oid": "1"
},
"array1": [
{
"_id": "12",
"array2": [
{
"_id": "123",
"answeredBy": [], // need to push "success"
},
{
"_id": "124",
"answeredBy": [],
}
],
}
]
}
I need to push a value to "answeredBy" array.
In the below example, I tried pushing "success" string to the "answeredBy" array of the "123 _id" object but it does not work.
callback = function(err,value){
if(err){
res.send(err);
}else{
res.send(value);
}
};
conditions = {
"_id": 1,
"array1._id": 12,
"array2._id": 123
};
updates = {
$push: {
"array2.$.answeredBy": "success"
}
};
options = {
upsert: true
};
Model.update(conditions, updates, options, callback);
I found this link, but its answer only says I should use object like structure instead of array's. This cannot be applied in my situation. I really need my object to be nested in arrays
It would be great if you can help me out here. I've been spending hours to figure this out.
Thank you in advance!
General Scope and Explanation
There are a few things wrong with what you are doing here. Firstly your query conditions. You are referring to several _id values where you should not need to, and at least one of which is not on the top level.
In order to get into a "nested" value and also presuming that _id value is unique and would not appear in any other document, you query form should be like this:
Model.update(
{ "array1.array2._id": "123" },
{ "$push": { "array1.0.array2.$.answeredBy": "success" } },
function(err,numAffected) {
// something with the result in here
}
);
Now that would actually work, but really it is only a fluke that it does as there are very good reasons why it should not work for you.
The important reading is in the official documentation for the positional $ operator under the subject of "Nested Arrays". What this says is:
The positional $ operator cannot be used for queries which traverse more than one array, such as queries that traverse arrays nested within other arrays, because the replacement for the $ placeholder is a single value
Specifically what that means is the element that will be matched and returned in the positional placeholder is the value of the index from the first matching array. This means in your case the matching index on the "top" level array.
So if you look at the query notation as shown, we have "hardcoded" the first ( or 0 index ) position in the top level array, and it just so happens that the matching element within "array2" is also the zero index entry.
To demonstrate this you can change the matching _id value to "124" and the result will $push an new entry onto the element with _id "123" as they are both in the zero index entry of "array1" and that is the value returned to the placeholder.
So that is the general problem with nesting arrays. You could remove one of the levels and you would still be able to $push to the correct element in your "top" array, but there would still be multiple levels.
Try to avoid nesting arrays as you will run into update problems as is shown.
The general case is to "flatten" the things you "think" are "levels" and actually make theses "attributes" on the final detail items. For example, the "flattened" form of the structure in the question should be something like:
{
"answers": [
{ "by": "success", "type2": "123", "type1": "12" }
]
}
Or even when accepting the inner array is $push only, and never updated:
{
"array": [
{ "type1": "12", "type2": "123", "answeredBy": ["success"] },
{ "type1": "12", "type2": "124", "answeredBy": [] }
]
}
Which both lend themselves to atomic updates within the scope of the positional $ operator
MongoDB 3.6 and Above
From MongoDB 3.6 there are new features available to work with nested arrays. This uses the positional filtered $[<identifier>] syntax in order to match the specific elements and apply different conditions through arrayFilters in the update statement:
Model.update(
{
"_id": 1,
"array1": {
"$elemMatch": {
"_id": "12","array2._id": "123"
}
}
},
{
"$push": { "array1.$[outer].array2.$[inner].answeredBy": "success" }
},
{
"arrayFilters": [{ "outer._id": "12" },{ "inner._id": "123" }]
}
)
The "arrayFilters" as passed to the options for .update() or even
.updateOne(), .updateMany(), .findOneAndUpdate() or .bulkWrite() method specifies the conditions to match on the identifier given in the update statement. Any elements that match the condition given will be updated.
Because the structure is "nested", we actually use "multiple filters" as is specified with an "array" of filter definitions as shown. The marked "identifier" is used in matching against the positional filtered $[<identifier>] syntax actually used in the update block of the statement. In this case inner and outer are the identifiers used for each condition as specified with the nested chain.
This new expansion makes the update of nested array content possible, but it does not really help with the practicality of "querying" such data, so the same caveats apply as explained earlier.
You typically really "mean" to express as "attributes", even if your brain initially thinks "nesting", it's just usually a reaction to how you believe the "previous relational parts" come together. In reality you really need more denormalization.
Also see How to Update Multiple Array Elements in mongodb, since these new update operators actually match and update "multiple array elements" rather than just the first, which has been the previous action of positional updates.
NOTE Somewhat ironically, since this is specified in the "options" argument for .update() and like methods, the syntax is generally compatible with all recent release driver versions.
However this is not true of the mongo shell, since the way the method is implemented there ( "ironically for backward compatibility" ) the arrayFilters argument is not recognized and removed by an internal method that parses the options in order to deliver "backward compatibility" with prior MongoDB server versions and a "legacy" .update() API call syntax.
So if you want to use the command in the mongo shell or other "shell based" products ( notably Robo 3T ) you need a latest version from either the development branch or production release as of 3.6 or greater.
See also positional all $[] which also updates "multiple array elements" but without applying to specified conditions and applies to all elements in the array where that is the desired action.
I know this is a very old question, but I just struggled with this problem myself, and found, what I believe to be, a better answer.
A way to solve this problem is to use Sub-Documents. This is done by nesting schemas within your schemas
MainSchema = new mongoose.Schema({
array1: [Array1Schema]
})
Array1Schema = new mongoose.Schema({
array2: [Array2Schema]
})
Array2Schema = new mongoose.Schema({
answeredBy": [...]
})
This way the object will look like the one you show, but now each array are filled with sub-documents. This makes it possible to dot your way into the sub-document you want. Instead of using a .update you then use a .find or .findOne to get the document you want to update.
Main.findOne((
{
_id: 1
}
)
.exec(
function(err, result){
result.array1.id(12).array2.id(123).answeredBy.push('success')
result.save(function(err){
console.log(result)
});
}
)
Haven't used the .push() function this way myself, so the syntax might not be right, but I have used both .set() and .remove(), and both works perfectly fine.