Develop Reference

Pass query criteria to mongoDB aggregation

our current setup is: SPA frontend, Azure functions with mongoose middleware, MongoDB
(Maybe first read the question***)
Since we have a lot of documents in our DB and our customer wants to query them we are facing the following problem:
The user is assigned to his organization. He wants to search for Doc1s he has not responded to.
Doc1
{
_id
organization -> partitionKey
content
}
By creating doc2 with reference to doc1 he can respond.
Doc2
{
_id
organization -> partitionKey
Doc1ref
content
}
We have a 1:n relationship.
At the moment we filter just by query criteria of doc1 with limit and skip options.
But the new requirement is to filter the same way by referring doc2s.
I was thinking of:
Doing it in my code => Problem: after we have read with limit=100 and I filter it by my code, the result is not 100 anymore.
Extending doc1 by doc2 arrays => Must be the last option
Dynamic aggregation, Prepared in the code and executed at runtime => Don't want to user dynamic aggregations and the benefits of mongoose are almost lost.
Create a MongoDB view with lookup aggregation (populating doc1 by doc1.respondedOrganizations) => Problem is see here is the performance. When searching a lot of documents and then joining them by a non partitionKey.
*** So, I come to my question:
Is it possible to pass a virtual (not existing) query criteria...
doc1.find({ alreadyResponded : my.organization } )
...and use it as input variable in an aggregation
{
$lookup: {
from: Doc2s,
localField: _id,
foreignField: Doc1ref,
as: < output array field >
pipeline: [{
$match: {
$organization: {
$eq: $$alreadyResponded
}]
}
}
It would reduce query performance extremly.
Thanks

Firebase Realtime Database - Best practice for additional data nodes (lookups and initial load)

I have a Firebase Realtime Database with the below structure.
I wish to fetch all "notes" a user has access to and, initially, only show titles for the notes.
notes: {
"noteId-1345" : {
"access" : {
"author": "1234567890"
"members": {
"1234567890": 0 <--- Author
"0987654321": 1 <--- Member
}
},
"data" : {
"title": "Hello",
"content": "Konichiwa!",
"comment": "123"
}
}
}
To be able to fetch all notes a user has access to I have expanded my data model by keeping an additional user_notes node in the root:
Whenever I associate a user (update of members) with a note, I write that relation both in /notes/$noteid and in /user_notes/$uid.
user_notes: {
"$uid": {
"noteId-1345": true
}
}
When fetching initial data I only need the "notes" the user has access to - including titles.
(I only fetch the entire "note" if a user wants to view a complete "note")
I begin by fetching the ids for notes the user has access to and then I have to do another lookup to fetch the titles.
let titles = []
database.ref(`user_notes/${uid}`)
.on('value', (snaps) => {
snaps.forEach((snap) => {
const noteId = snap.key
database.ref(`notes/${noteId}/data/title`).on('value', (noteSnap) => {
const title = noteSnap.val()
titles.push(title)
}
})
})
Is this the most efficient approach? - It seems inefficient to do double lookups.
Should I store title, and other data needed for initial load, in the user_notes node as well to avoid double lookups?
What is considered to be best practice in cases like this when using a NoSQL database?
Kind regards /K

What you're doing is indeed the common approach. It is not nearly as slow as you may initially think, since Firebase pipelines the requests over a single connection.
A few things to consider:
I'd typically move the members for each note under a top-level node note_members. Separating the types of data typically makes it much easier to keep your security rules reasonable, and is documented under keep your data structure flat.
If you'd like to get rid of the lookup, you can consider storing the title of each note under each user_notes node where you have the ID. You'd essentially replace the true with the name:
user_notes: {
"$uid": {
"noteId-1345": "Hello"
}
}
This simplifies the lookup code (its main advantage) and makes it a bit faster.
This sort of data duplication is quite common when using Firebase and other NoSQL databases, you trade write complexity and extra data storage, for reach simplicity and scalability.

Mongo check if a document already exists

In the MEAN app I'm currently building, the client-side makes a $http POST request to my API with a JSON array of soundcloud track data specific to that user. What I now want to achieve is for those tracks to be saved to my app database under a 'tracks' table. That way I'm then able to load tracks for that user from the database and also have the ability to create unique client URLs (/tracks/:track)
Some example data:
{
artist: "Nicole Moudaber"
artwork: "https://i1.sndcdn.com/artworks-000087731284-gevxfm-large.jpg?e76cf77"
source: "soundcloud"
stream: "https://api.soundcloud.com/tracks/162626499/stream.mp3?client_id=7d7e31b7e9ae5dc73586fcd143574550"
title: "In The MOOD - Episode 14"
}
This data is then passed to the API like so:
app.post('/tracks/add/new', function (req, res) {
var newTrack;
for (var i = 0; i < req.body.length; i++) {
newTrack = new tracksTable({
for_user: req.user._id,
title: req.body[i].title,
artist: req.body[i].artist,
artwork: req.body[i].artwork,
source: req.body[i].source,
stream: req.body[i].stream
});
tracksTable.find({'for_user': req.user._id, stream: req.body[i].stream}, function (err, trackTableData) {
if (err)
console.log('MongoDB Error: ' + err);
// stuck here - read below
});
}
});
The point at which I'm stuck, as marked above is this: I need to check if that track already exists in the database for that user, if it doesn't then save it. Then, once the loop has finished and all tracks have either been saved or ignored, a 200 response needs to be sent back to my client.
I've tried several methods so far and nothing seems to work, I've really hit a wall and so help/advice on this would be greatly appreciated.

Create a compound index and make it unique.
Using the index mentioned above will ensure that there are no documents which have the same for_user and stream.
trackSchema.ensureIndex( {for_user:1, stream:1}, {unique, true} )
Now use the mongoDB batch operation to insert multiple documents.
//docs is the array of tracks you are going to insert.
trackTable.collection.insert(docs, options, function(err,savedDocs){
//savedDocs is the array of docs saved.
//By checking savedDocs you can see how many tracks were actually inserted
})
Make sure to validate your objects as by using .collection we are bypassing mongoose.

Make a unique _id based on user and track. In mongo you can pass in the _id that you want to use.
Example {_id : "NicoleMoudaber InTheMOODEpisode14",
artist: "Nicole Moudaber"
artwork: "https://i1.sndcdn.com/artworks-000087731284-gevxfm-large.jpg?e76cf77"
source: "soundcloud"
stream: "https://api.soundcloud.com/tracks/162626499/stream.mp3? client_id=7d7e31b7e9ae5dc73586fcd143574550"
title: "In The MOOD - Episode 14"}
_id must be unique and won't let you insert another document with the same _id. You could also use this to find the record later db.collection.find({_id : NicoleMoudaber InTheMOODEpisode14})
or you could find all tracks for db.collection.find({_id : /^NicoleMoudaber/}) and it will still use the index.
There is another method to this that I can explain if you dont' like this one.
Both options will work in a sharded environment as well as a single replica set. "Unique" indexes do not work in a sharded environment.

Soundcloud API provides a track id, just use it.
then before inserting datas you make a
tracks.find({id_soundcloud : 25645456}).exec(function(err,track){
if(track.length){ console.log("do nothing")}else {//insert}
});

Trying to simplify access to private posts with Express

I'm learning Node.js with MongoDB and Express and it is going quite well.
I have my user registration working fine and every user can create posts.
Now I'm trying something more complicated, I'd like user to create private posts and only user who created it and other allowed users can see the post.
I did something and it seems to work but I think it can be done in some better way.
What I have now to get the post at this address www.mywebsite.com/post-title is this:
Post
.findOne({ permalink: req.params.permalink })
.exec(function(err, post) {
if (!post) {
req.flash('errors', { msg: 'Post not found' });
return res.redirect('/');
} else {
if (post._creator == req.user.id) {
res.render('post/home', {
title: post.name,
post: post
});
} else {
req.flash('errors', { msg: 'You are not allowed to see this post' });
res.redirect('/');
}
}
});
It works fine but if I wish to add few more options to this post and create another link like: www.mywebsite.com/post-title/tags to get that page I have to repeat the whole code posted above...
I wish to find a way to match easily the owner of the post or allowed people and a way to get the post through the permalink without useing fineOne for every get...
Is this possible?
Does it make sense?
Thanks

I might have helped you to "simplify" your question and just explain what you wanted to do. But those who take the time to read all of it would eventually see that you basically want to
" List all posts including private and allowed posts for the current user.. "
Which is basically the simplified version of the question.
So all you basically need are some fields on your "Post" document that allow the access control:
{
"title": "this is the title",
"permalink": "some/sort/of/slug",
"body": "post body here",
"creator": "Bill",
"_private": true,
"_allowed": ["Ted","Fred"]
}
So basically you are not going to care about the "_allowed" list where "private" is false, but you do want to care where this is true. So you want this logic in the query rather than evaluating it per document retrieved:
Post.find(
{
"$or": [
{ "_private": false },
{
"_private": true,
"$or": [
{ "creator": req.user.id },
{ "_allowed": req.user.id }
]
}
}
},
function(err,docs) {
So essentially your logic is based of a nested $or operation which either allows the public posts to display or otherwise where the post is private then only the "creator" $or the "_allowed" users will receive this in any query.
The logic applies to whether you are retrieving a list of posts for paging results or whether recalling an individual post for a single in depth display.

Chaining models relations in node.js (node-orm) like in Rails

I'm building a relatively big NodeJS application, and I'm currently trying to figure out how to fetch the data I need from the DB. Here is a part of my models :
One user has one role, which has access to many modules (where there's a table role_modules to link roles and modules).
In Rails, I would do something like user.role.modules to retrieve the list of the modules he has access to. In NodeJS it's a bit more complicated. I'm using node-orm2 along with PostgreSQL. Here is what I have so far:
req.models.user.find({email: req.body.user}, function(err, user) {
user[0].getRole(function(err, role) {
role.getModules(function(err, modules) {
var list_modules = Array();
modules.forEach(function(item) {
console.log(item);
list_modules.push(item.name);
})
But I can't do this, because item only contains role_id and module_id. If I want to have the name, I would have to do item.getModule(function() {...}), but the results would be asynchronous ... so I don't really see how I could end up with an array containing the names of the modules a user has access to ... have any idea?
Also, isn't that much slower than actually running a single SQL query with many JOIN? Because as I see it, the ORM makes multiple queries to get the data I want here...
Thank you!

I wrote an ORM called bookshelf.js that aims to simplify associations and eager loading relations between models in SQL. This is what your query would probably look like to load the role & modules on a user given your description:
var Module = Bookshelf.Model.extend({
tableName: 'modules'
});
var Role = Bookshelf.Model.extend({
tableName: 'roles',
modules: function() {
return this.belongsToMany(Module);
}
});
var User = Bookshelf.Model.extend({
tableName: 'users'
role: function() {
return this.hasOne(Role);
}
});
User.forge({email: req.body.user})
.fetch({
require: true,
withRelated: ['role.modules']
})
.then(function(user) {
// user, now has the role and associated modules eager loaded
console.log(user.related('role'));
console.log(user.related('role').related('modules'))
}, function(err) {
// gets here if no user was found.
});
Might be worth taking a look at.