Develop Reference

Trouble With Mongoose ETL Process and Deeply Nested Array

I'm working on building a database from a set of CSV files and am running into issues pushing elements to a deeply nested array. I've seen examples for 2D arrays and the use of positional operators but I can't quite figure out how to use them for my situation.
What I am trying to do read the CSV file which has columns for the answer_id and the url string for the photo. I want to push the url string to the photos array for the corresponding answer_id. When I try to use the code below, I get a long error message which starts with:
MongoBulkWriteError: Cannot create field 'answers' in element
{
results: [
{
question_id: "1",
_id: ObjectId('6332e0b015c1d1f4eccebf4e'),
answers: [
{
answer_id: "5",
//....
}
],
}
]
}
I formatted the above message to make things easier to read. It may be worth noting the first row of my CSV file has '5' in the answer_id column which makes me think things are failing at the first try to update my document.
Here is an example of my code:
const ExampleModel = new Schema({
example_id: String,
results: [
{
question_id: String,
answers: [
{
answer_id: String,
photos: [
{ url: String },
]
}
]
}
]
});
// Example Operation
// row.answer_id comes from CSV file
updateOne: {
'filter': {
'results.answers.answer_id': row.answer_id,
},
'update': {
'$push': {
'results.answers.$.photos' : { 'url': 'test'}
}
}
}
I guess my question is can I update an array this deeply nested using Mongoose?

MongoDB: return just the document from an array that matches a certain query (JS)

I have a MongoDB collection in the form of:
users: {
{
user_id: 1000,
activities: [
{id: 1, activity: 'swimming'},
{id: 2, activity: 'running'},
{id: 3, activity: 'biking'},...
]
},...
}
and I want to get the activity document that matches a specific ID. For example, if I query using {id: 1}, I want an output of {id: 1, activity: 'swimming'}. Currenlty, I'm trying to use findOne({activities: {$elemMatch: {id: 1}}}), but it returns the entire user document (the user id, and all the activities).
The code I'm using is:
id = req.body.queryID;
db.collection('users').findOne({activities: {$elemMatch: {id}}})
.then((document) => {
console.log(document);
// more code after this
});
I've also tried to query using aggregate() and findOne({}, {activities: {$elemMatch: {id}}}), but I haven't been able to figure it out.
I am on MongoDB version 4.4.8 according to db.version(). Any help is appreciated!

Your attempts look close, I think you just need to put them together. findOne takes two optional arguments, a query and a projection. The query tells MongoDB what document to look for and return. The projection document tells MongoDB what field(s) to include in the document returned by the query.
What you have here:
db.collection('users').findOne({ activities: { $elemMatch: { id }}})
Passes one argument, the query. It will look for a document where there is one element in the activities array with an id equal to the value of the variable id. You could also write this as:
db.collection('users').findOne({ "activities.id": id })
You'd also like to only return documents in the activities array with a matching ID. This is when you'd do something like your other attempt, where $elemMatch is in the second argument, the projection:
db.collection('users').findOne({}, {activities: {$elemMatch: {id}}})
But because you passed an empty query, MongoDB will return any document, not necessarily one that contains an activity with a matching ID.
So if you pass both a query and the projection, I think you should get what you're looking for. It might look like this:
db.collection('users').findOne({ "activities.id": id }, { activities: { $elemMatch: { id }}})
This will include only the _id field and the matching activities document if one exists. If you want to include other fields of the document, they need to be explicitly included. See this documentation about projection.

You can use the aggregate method with the unwind, match and project pipelines
db.users.aggregate([
{$unwind: "$activities"},
{$match: {"activities.id": id}},
{$project: {id: "$activities.id", activity: "$activities.activity", _id: 0}}
])
NOTE: This code is written using the mongo shell syntax.
The query does the following
starts with the unwind pipeline which first pulls out all the items in the activities array.
The match pipeline finds the array element that has the id we're looking for
The project pipeline returns the output as desired
Hope this helps

Please try as follow:
db.users.aggregate([
{
'$unwind': '$activities'
}, {
'$match': {
'id': id
}
}, {
'$project': {
'activity': '$activities.activity',
'id': '$activities.id',
'_id':0
}
}
]);

Sequelize exclude orderBy from subQuery

Trying to use sequelize findAndCountAll method, to get items.
I've to use distinct, and offset with limit due to my task.
The problem is, that in associated model i've column with array type, and i need to order parent model by that array length.
My query looks like :
const { rows, count } = await this.repo.findAndCountAll({
where: { someField: someValue },
col: 'someCol',
distinct: true,
include: [
{
model: someNestedModel,
as: 'someNestedModelAssociation',
include: [{ model: someInnerNestedModel, as: 'someInnerNestedAssociation' }]
}
],
// eslint-disable-next-line #typescript-eslint/ban-ts-comment
//#ts-ignore
order: this.getOrderByOptions(sortOrder, orderBy),
limit,
offset
});
getOrderByOptions(sortOrder, orderBy) {
switch (orderBy) {
case Sort_By_Array_Length:
return Sequelize.literal('json_array_length(someModelName.someColumnWithArrayName) ASC');
default:
return [[orderBy, sortOrder]];
}
}
The problem is, that my order by query is used both in subQuery and mainQuery.
And using it into subQuery leads to error, cz there is no such field.
If i use subQuery:false flag, it works, but then i got messed with returning results, due to problems with subQuery:false and offset&limits.
So the question is, is there a way, to exclude orderBy field from subQuery?
P.S. Models have many to many association with through table.

Subsequent MongoDB aggregate() calls operate on the result of the previous call!? (using NodeJS driver)

Consider the following code:
const db = mongodb.getDb();
// Results count
let count = (await db.collection('sec').aggregate(aggregateQuery)
.group({ _id: null, count: { $sum: 1 } })
.project({ _id: 0, count: 1 })
.toArray());
console.dir(count)
// test
let test = await db.collection('sec').aggregate(aggregateQuery).toArray();
console.dir(test)
The aggregate(aggregateQuery) filters the collection ad performs a lookup with another collection.
The first query counts all documents resulting from the aggregate, and it outputs [ { count: 2086 } ] as expected.
In my mind, test should contain a bunch of documents filtered from the collection (according to the aggregate operations), but it is just [ { count: 2086 } ], as in the first query!
If in the second query instead of aggregate(aggregateQuery) I put find(), it outputs all the collection's documents, as expected, so I guess there is something I am missing in the aggregate() function...

The object passed as a parameter inside aggregate (aggregateQuery in this case) is modified by the following functions (group and project), so when it is used again in the second aggregate call, it is different and yields different results.

MongoDB complex faceted search not yielding expected result

I am trying to build in one aggregation query a match text search followed by another match query to avoid sending multiple queries to the database, but I am not receiving any result.
Here is what my query looks like:
query = await Product.aggregate([
{ $match: { $text: { $search: userQuery } } },
{ $addFields: { score: { $meta: "textScore" } } },
{
$facet: {
Forfait: [
{
$match: { Forfait: "midnight" }
}
]
}
}
]).exec();
Here is what I am trying to accomplish:
1. Perform a text search on user input on the collection
2. and also perform a match on the collection on the forfait field
The query returns nothing, it seems that it is performing a sort of logical AND query and short-circuits but I don't know how I can perform the two queries in one aggregation.
I have tried following a similar issue here MongoDB Facet Error pipeline requires text score metadata, but there is no text score available which reflects my query.
Any directions or help will be greatly appreciated.
Below is what the document model looks just a simple doc
{
id: String
title: String
description: String
forfait: String
}