This question already has an answer here:
Speed up fetching posts for my social network app by using query instead of observing a single event repeatedly
(1 answer)
Closed 6 years ago.
Considering this data structure:
{
"users":{
"user-1":{
"groups":{
"group-key-1":true,
"group-key-3":true
}
}
},
"groups":{
"group-key-1":{
"name":"My group 1"
},
"group-key-2":{
"name":"My group 2"
},
"group-key-3":{
"name":"My group 3"
},
"group-key-4":{
"name":"My group 4"
},
}
}
I could get the groups of a particular user with:
firebase.database().ref('users/user-1/groups').on(...)
Which would give me this object:
{
"group-key-1":true,
"group-key-3":true
}
So now I want to retrieve the info for those groups.
My initial instinct would be to loop those keys and do this:
var data = snapshot.val()
var keys = Object.keys(data)
for (var i = 0; i < keys.length; i++) {
firebase.database().ref('groups/' + keys[i]).on(...)
}
Is this the appropriate way to call multiple keys on the same endpoint in Firebase?
Does Firebase provide a better way to solve this problem?
Is this the appropriate way to call multiple keys on the same endpoint
in Firebase?
Yes, generally this is a good solution to retrieve every group this way.
Does Firebase provide a better way to solve this problem?
I don't think Firebase provides any other function/query that could help in this case.
Anyway, this could be improved saving the ref in a variable and looping on the keys of the object directly. Also, if you just need to retrieve those data once, you should use once() instead of on()
var groupRef = firebase.database().ref('groups/')
var data = snapshot.val()
for (var groupID in data) {
groupRef.child(data[groupID]).once(...)
}
Another way could be using Firebase's functions for the data snapshots, like forEach
snapshot.forEach(function(childSnapshot) {
groupRef.child(childSnapshot.key).once(...)
})
Yes, you are going in the right way. As written in this question firebase will pipeline your requests and you won't have to be concerned about performance and roundtrip time. As soon as your iteration starts you will be receiving firebase data. So keep in mind to handle it properly in your ui.
Another option, that will depends on how big your /groups data will grow, is to keep a snapshot (could be a $firebaseObject if you are using angularfire) of the entire /groups branch that will refresh whenever data changes. Then you just have to track this snapshot. But again, if you are planning to play with a big amount of groups your current solution is a better choice.
Also, I recommend you to take care of setting an on event for each group you retrieve. Keep in mind that the callback will be triggered whenever the data changes (depends on the setted event). Depending on your use case you should consider using .once("value" since it will trigger the data only once, making it more reliable, more performatic and will avoid handling callbacks when you are not expecting them.
Related
For the context I'm using firebase/nodeJS on my application.
I'm deleting an user of a collection called workers. To do so I update the field deleted : true on the worker document.
I'm not deleting the document, because I want to keep their infos if needed.
But when I delete a worker, I must remove him from all the properties is working on (they have a field called workers, which is an array of all the workers associated.)
Hence, I must browse all the properties of the collection associated, check if the worker is on it and remove him.
Is there a way to do so ?
I saw this page : https://firebase.google.com/docs/firestore/manage-data/add-data#node.js_11
But don't know how to use theses ressources to do so.
So I finally did it alone.
First, I had to fetch all the properties where the work is working on, to do this I used array-contains.
let docsProperties = await db.collection('user').doc(id).collection("properties")
.where("workers", "array-contains", workerId).get();
Then I wanted to browse all theses properties and remove the worker. Thought about doing a ForEach first. But the problem is I wanted to use an await .update() with arrayRemove and async methods aren't usable inside ForEach.
So I used promises.
let promises = [];
docsProperties.forEach(doc => {
promises.push(doc.ref.update({
workers: admin.firestore.FieldValue.arrayRemove(workerId)
})); });
return Promise.all(promises);
arrayRemove is a method from the firebase documentation to remove all the occurences of an element on a following array field.
So I browsed all properties, added all promises and executed them!
worked perfectly, if anyone got questions don't hesitate.
I have db structure like this:
datas
-data1
--name
--city
--date
--logs
---log1
---log2
---log3
-data2
--name
...
Now, I released putting 'logs' inside 'data' parent was a huge mistake because its user generated child and growing up fast (so much data under it) and causes delay on downloading 'data1' parent naturally.
Normally I am pulling 'data1' with this:
database().ref('datas/' + this.state.dataID).on('value', function(snapshot) {
... })
I hope i could explain my problem, I just basically ignore 'logs' child (I need name,city,date)
As there project started and users already using this, I need a proper way.
Is there a way to do this on firebase side ?
I don't think you'll have an easy way out of this one...
Queries are deep by default: they always return the entire subtree.
https://firebase.google.com/docs/firestore/rtdb-vs-firestore#querying
I can see only two options:
Migrate the logs to a different location (if it's really a huge amount of data, you could use something like BiqQuery https://cloud.google.com/bigquery or if it's events, you could store them in Google Analytics, it really depends on the volume and type of logs)
Attach multiple listeners instead of a single one (depending on the amount of entries that might be a viable interim solution):
let response={
name:null,
city:null,
date:null
}
const refs = ['name', 'city', 'date'].map(key=>database().ref(`datas/${this.state.dataID}/${key}')
refs.forEach(ref=>ref.on('value',snapshot=>{
})
My Meteor client receives data from the server and stores it in minimongo. This data is guaranteed not to change during their session, so I don't need Meteor's reactivity. The static data just happens to arrive by that route; let's just take that as a given.
The data looks like this:
{_id: 'abc...', val: {...}}
On the client, is it more efficient for me to look up values using:
val_I_need = Collection.findOne({id})
or to create a JavaScript object:
data = {}
Collection.find().fetch().map((x) => {data[x._id] = x.val})
and use it for look ups:
val_I_need = data[id]
Is there a tipping point, either in terms of the size of the data or the number of look ups, where the more efficient method changes, or outweighs the initial cost of building the object?
FindOne may be more efficient on larger datasets because it looks up using cursors where _id is an indexed key while your find().fetch() approach requires to get all docs and then iterate manually by mapping.
Note, that findOne could also be replaced by .find({_id:desiredId}).fetch()[0](assuming it returns the desired doc).
More on this in the mongo documentation on query performance.
However, if it concerns only one object that is afterwards not reactively tracked, I would rather load it via a "findOne"-returning method from the server:
export const getOne = new ValidatedMethod({
name: "getOne",
validate(query) {
// validate query schema
// ...
},
run(query) {
// CHECK PERMISSIONS
// ...
return MyCollection.findOne(query);
});
This avoids using publications / subscriptions and thus minimongo for this collection on the current client template. Think about that pub/sub has already some reactivity initialized to observe the collection and thus eats up some computation somewhere.
My gut feeling is that you'll never hit a point where the performance gain of putting it in an object makes a noticeable difference.
It's more likely that your bottleneck will be in the pub/sub mechanism, as it can take a while to send all documents to the client.
You'll see a much more noticeable difference for a large dataset by retrieving the data using a Meteor method.
At which point you've got it in a plain old javascript object anyway and so end up with the small performance gain of native object lookups as well.
I'am using MongoDB with a nodejs REST service which exposes my data stored inside. I have a question about how to interrogate my data which uses $ref.
Here is a sample of an Object which contains a reference to another object (detail) in anther collection :
{
"_id" : ObjectId("5962c7b53b6a02100a000085"),
"Title" : "test",
"detail" : {
"$ref" : "ObjDetail",
"$id" : ObjectId("5270c7b11f6a02100a000001")
},
"foo" : bar
}
Actually, using Node.js and mongodb module, I do the following :
db.collection("Obj").findOne({"_id" : new ObjectID("5962c7b53b6a02100a000085"},
function(err, item) {
db.collection(item.$ref).findOne({"_id" : item.$id}, function(err,subItem){
...
});
});
In fact I make 2 queries, and get 2 objects. It's a kind of "lazy loading" (not exactly but almost)
My question is simple : is it possible to retrieve the whole object graph in one query ?
Thank you
No, you can't.
To resolve DBRefs, your application must perform additional queries to return the referenced documents. Many drivers have helper methods that form the query for the DBRef automatically. The drivers do not automatically resolve DBRefs into documents.
From the MongoDB docs http://docs.mongodb.org/manual/reference/database-references/.
Is it possible to fetch parent object along with it's $ref using single MongoDB query?
No, it's not possible.
Mongo have no inner support for refs, so it up to your application to populate them (see Brett's answer).
But is it possible to fetch parent object with all its ref's with a single node.js command?
Yes, it's possible. You can do it with Mongoose. It has build-in ref's population support. You'll need to change your data model a little bit to make it work, but it's pretty much what you're looking for. Of course, to do so Mongoose will make the same two MongoDB queries that you did.
Answer of Vladimir is not still valid as the db.dereference method was deleted from MongoDB Nodejs API:
https://www.mongodb.com/blog/post/introducing-nodejs-mongodb-20-driver
The db instance object has been simplified. We've removed the following methods:
db.dereference due to db references being deprecated in the server
No, very few drivers for MongoDb include special support for a DBRef. There are two reasons:
MongoDb doesn't have any special commands to make retrieval of referenced documents possible. So, drivers that do add support are artificially populating the resulting objects.
The more, "bare metal" the API, the less it makes sense. In fact, as. MongoDb collections are schema-less, if the NodeJs driver brought back the primary document with all references realized, if the code then saved the document without breaking the references, it would result in an embedded subdocument. Of course, that would be a mess.
Unless your field values vary, I wouldn't bother with a DBRef type and would instead just store the ObjectId directly. As you can see, a DBRef really offers no benefit except to require lots of duplicate disk space for each reference, as a richer object must stored along with its type information. Either way, you should consider the potentially unnecessary overhead of storing a string containing the referenced collection's documents.
Many developers and MongoDb, Inc. have added an object document mapping layer on top of the existing base drivers. One popular option for MongoDb and Nodejs is Mongoose. As the MongoDb server has no real awareness of referenced documents, the responsibility of the references moves to the client. As it's more common to consistently reference a particular collection from a given document, Mongoose makes it possible to define the reference as a Schema. Mongoose is not schema-less.
If you accept having and using a Schema is useful, then Mongoose is definitely worth looking at. It can efficiently fetch a batch of related documents (from a single collection) from a set of documents. It always is using the native driver, but it generally does operations extremely efficiently and takes some of the drudgery out of more complex application architectures.
I would strongly suggest you have a look at the populate method (here) to see what it's capable of doing.
Demo /* Demo would be a Mongoose Model that you've defined */
.findById(theObjectId)
.populate('detail')
.exec(function (err, doc) {
if (err) return handleError(err);
// do something with the single doc that was returned
})
If instead of findById, which always returns a single document, find were used, with populate, all returned documents' details property will be populated automatically. It's smart too that it would request the same referenced documents multiple times.
If you don't use Mongoose, I'd suggest you consider a caching layer to avoid doing client side reference joins when possible and use the $in query operator to batch as much as possible.
I reach the desired result with next example:
collection.find({}, function (err, cursor) {
cursor.toArray(function (err, docs) {
var count = docs.length - 1;
for (i in docs) {
(function (docs, i) {
db.dereference(docs[i].ref, function(err, doc) {
docs[i].ref = doc;
if (i == count) {
(function (docs) {
console.log(docs);
})(docs);
}
});
})(docs, i)
}
});
});
Not sure that it solution is best of the best, but It is simplest solution that i found.
I've just started to look into using Meteor for an upcoming project, and have a question about data persistence. It sounds like you have two options: First, you can declare a "name" when instantiating a new Collection which will create a database collection which will be saved upon alteration.
Chatrooms = new Meteor.Collection("chatrooms");
The other option is to create an anonymous collection, which won't be saved.
Chatrooms = new Meteor.Collection();
But what do I do if I want to populate a Collection from the database, but not save it upon alteration on the client-side? For instance, I might want to create a collection of user Movies that will be displayed in a grid -- each having their own absolute positioning based upon the sorting and filtering applied to the collection. Upon changes to the collection, the associated views (or templates) will be re-rendered to reflect those changes. But I don't necessarily want these absolute positions to be stored in a database...
Any ideas?
I'm not very clear about your question. But perhaps, you can bind the absolute position into the collection data? They are just normal javascript objects. And the collection data will only be changed through insert/update/remove function call.
I ended up doing something like this:
movies: function() {
var movies = Movies.find().fetch();
_.each(movies, function(movie, index){
movie.left = index * 2;
movie.top = index * 2;
});
return movies;
},
Basically, 'fetch()' allows me to deal with pure JSON objects, making it easier to iterate through them and alter them without performing 'update' commands.