Firebase querying - issue with indexing unique ids - javascript

My database structure looks like this (simplified):
{
"articles": {
"-uniqueId1": {
"title": "Article 1",
"category": "news"
},
"-uniqueId2": {
"title": "Article 2",
"category": "other"
},
"-uniqueId3": {
"title": "Article 3",
"category": "news"
},
"-uniqueId4": {
"title": "Article 4",
"category": "news"
}
},
"articlesByCategory": {
"news": {
"-uniqueId1": true,
"-uniqueId3": true,
"-uniqueId4": true
},
"other": {
"-uniqueId2": true
}
}
}
The query needs to fetch articles where a specific article's category isn't within. Does that make sense? Say, if uniqueId2 is of category "other", the query would only fetch articles within "news" and all other existing categories. But since the list may contain, say millions of articles, I have to be as specific as possible and not fetch articles that doesn't match this exact criteria. Therefor, a query like below would be ideal:
const ref = firebase.database().ref("articlesByCategory");
ref.orderByChild("-uniqueId2").equalTo(null).once("value", function(snapshot) {
console.log(snapshot.key);
});
Here I am searching for categories where a specific property is absent (where a property equals to null). This is explained here: Firebase: Query to exclude data based on a condition
However, doing a query like this requires me to index every article's unique id on "/articlesByCategory" in the security rules. But it would be unrealistic and non-optimal to dynamically add new article ids inside the ".indexOn" array as that would end up with millions of unique (auto-generated) ids:
{
"articles": {
".read": true,
".write": true
},
"articlesByCategory": {
".read": true,
".write": true,
".indexOn": ["-uniqueId1", "-uniqueId2", "-uniqueId3", "-uniqueId4"...] // List would be endless!
}
}
So how is this achievable? Why am I seeing these sorts of structures (inverted indexing) as ideal solutions everywhere on StackOverflow, but no one seems to tackle this issue?

Firebase only can query order statements on nodes with auto-generated ids, and when you've defined your own keys, firebase is unable to perform any sorting/ordering
The ideal way to generate keys is
firebase.getInstance().getReference("articles").push(myArticle);
this will result in a slightly different database structure like
{
"12dakj137_9": { // this is a auto-generated id
"article1": {
"title": "Article 1",
"category": "news"
},
"12asjh_123": {
"title": "Article 2",
"category": "other"
},...
}
Now firebase is able to order/sort and the way you do this is by
firebase.getInstance().child("articles").orderByChild("category").equalTo("other");

Related

How to find one document in MongoDB by unique slug from Next.js?

My MongoDB data looks like this:
[
{
"Post": "this is a post",
"_id": ObjectId("630f3c32c1a580642a9ff4a0"),
"slug": "this-is-a-title",
"title": "This is a title"
},
{
"Post": "this is a post",
"_id": ObjectId("630f3c32c1a580642a9ff4a1"),
"slug": "this-is-a-title-b",
"title": "This is a title B"
}
]
All slug are unique, how can I find one document by that unique slug?
I am not using Mongoose.
you can use the find method...
the query works like this, in the find method you need to specify the property that you want to compare (in this case "slug") and the you can use the $eq (equal to) operator with your value.
db.YOURCOLLECTION.find({
slug: {
$eq: 'this-is-a-title'
},
})
here the playground with a working example based on your data:
https://mongoplayground.net/p/X_1MAuhKBS4
official docs
https://www.mongodb.com/docs/manual/reference/operator/query/eq/

Is there any Dojo treegrid limits to avoid "Sorry, an error occurred" message appear?

I cannot find any info if there is any limitations for dojo treegrid JSON store. Here is my simple store. It works perfect but fails if it has thousands of items. So is there a limit for number of items or childItems? Or is there a limit for JSON object size?
{
"identifier": "id",
"label": "name",
"items": [
{
"id": "id1",
"type": "year",
"year": "2018",
"childItems": [
{
"id": "id0",
"projname": "Project 1"
},
{
.....
}
]
},
{
.....
}
]
}
Dojo treegrid mostly shows this error message when it finds multiple items with the same id, so you want to make sure all "id" attributes in your "items" list have unique value. In my test setup I was able to load more than 20000 rows so most probably you have malformed data. Supply following option to grid to log any fetch related errors:
onFetchError: function(err, req){
console.log(err);
}
Hope it helps.

Firebase Fanout Structure List Query

For the Firebase fanout data structure example of users and groups, what would be the most efficient way to retrieve user/member ("users") detail data for a given group from the "groups" list? Let's say the goal was to display the "name" property for each member of group "techpioneers"?
{
"users": {
"alovelace": {
"name": "Ada Lovelace",
"groups": {
"techpioneers": true,
"womentechmakers": true
}
},
},
"groups": {
"techpioneers": {
"name": "Historical Tech Pioneers",
"members": {
"alovelace": true,
"ghopper": true
}
},
...
}
}
Would using a combination of orderByChild() and equalTo() be the best approach to find users who have "techpioneers" key and get at each of their data such as the "name" property? How would you access a property such as "name" once you've determined which users are part of the target group?
let usersRef = firebase.database().ref('users')
usersRef.orderByChild("groups").equalTo('techpioneers').on('child_added', (snapshot) => {
// how would you access user "alovelace" name property?
console.log(snapshot.val().name);
});
Thank you for any help you can provide.

MongoDb: How to get a field (sub document) from a document?

Consider this example collection:
{
"_id:"0,
"firstname":"Tom",
"children" : {
"childA":{
"toys":{
'toy 1':'batman',
'toy 2':'car',
'toy 3':'train',
}
"movies": {
'movie 1': "Ironman"
'movie 2': "Deathwish"
}
},
"childB":{
"toys":{
'toy 1':'doll',
'toy 2':'bike',
'toy 3':'xbox',
}
"movies": {
'movie 1': "Frozen"
'movie 2': "Barbie"
}
}
}
}
Now I would like to retrieve ONLY the movies from a particular document.
I have tried something like this:
movies = users.find_one({'_id': 0}, {'_id': 0, 'children.ChildA.movies': 1})
However, I get the whole field structure from 'children' down to 'movies' and it's content. How do I just do a query and retrieve only the content of 'movies'?
To be specific I want to end up with this:
{
'movie 1': "Frozen"
'movie 2': "Barbie"
}
The problem here is your current data structure is not really great for querying. This is mostly because you are using "keys" to actually represent "data points", and while it might initially seem to be a logical idea it is actually a very bad practice.
So rather than do something like assign "childA" and "childB" as keys of an object or "sub-document", you are better off assigning these are "values" to a generic key name in a structure like this:
{
"_id:"0,
"firstname":"Tom",
"children" : [
{
"name": "childA",
"toys": [
"batman",
"car",
"train"
],
"movies": [
"Ironman"
"Deathwish"
]
},
{
"name": "childB",
"toys": [
"doll",
"bike",
"xbox",
],
"movies": [
"Frozen",
"Barbie"
]
}
]
}
Not the best as there are nested arrays, which can be a potential problem but there are workarounds to this as well ( but later ), but the main point here is this is a lot better than defining the data in "keys". And the main problem with "keys" that are not consistently named is that MongoDB does not generally allow any way to "wildcard" these names, so you are stuck with naming and "absolute path" in order to access elements as in:
children -> childA -> toys
children -> childB -> toys
And that in a nutshell is bad, and compared to this:
"children.toys"
From the sample prepared above, then I would say that is a whole lot better approach to organizing your data.
Even so, just getting back something such as a "unique list of movies" is out of scope for standard .find() type queries in MongoDB. This actually requires something more of "document manipulation" and is well supported in the aggregation framework for MongoDB. This has extensive capabilities for manipulation that is not present in the query methods, and as a per document response with the above structure then you can do this:
db.collection.aggregate([
# De-normalize the array content first
{ "$unwind": "$children" },
# De-normalize the content from the inner array as well
{ "$unwind": "$children.movies" },
# Group back, well optionally, but just the "movies" per document
{ "$group": {
"_id": "$_id",
"movies": { "$addToSet": "$children.movies" }
}}
])
So now the "list" response in the document only contains the "unique" movies, which corresponds more to what you are asking. Alternately you could just $push instead and make a "non-unique" list. But stupidly that is actually the same as this:
db.collection.find({},{ "_id": False, "children.movies": True })
As a "collection wide" concept, then you could simplify this a lot by simply using the .distinct() method. Which basically forms a list of "distinct" keys based on the input you provide. This playes with arrays really well:
db.collection.distinct("children.toys")
And that is essentially a collection wide analysis of all the "distinct" occurrences for each"toys" value in the collection, and returned as a simple "array".
But as for you existing structure, it deserves a solution to explain, but you really must understand that the explanation is horrible. The problem here is that the "native" and optimized methods available to general queries and aggregation methods are not available at all and the only option available is JavaScript based processing. Which even though a little better through "v8" engine integration, is still really a complete slouch when compared side by side with native code methods.
So from the "original" form that you have, ( JavaScript form, functions have to be so easy to translate") :
db.collection.mapReduce(
// Mapper
function() {
var id this._id;
children = this.children;
Object.keys(children).forEach(function(child) {
Object.keys(child).forEach(function(childKey) {
Object.keys(childKey).forEach(function(toy) {
emit(
id, { "toys": [children[childkey]["toys"][toy]] }
);
});
});
});
},
// Reducer
function(key,values) {
var output = { "toys": [] };
values.forEach(function(value) {
value.toys.forEach(function(toy) {
if ( ouput.toys.indexOf( toy ) == -1 )
output.toys.push( toy );
});
});
},
{
"out": { "inline": 1 }
}
)
So JavaScript evaluation is the "horrible" approach as this is much slower in execution, and you see the "traversing" code that needs to be implemented. Bad news for performance, so don't do it. Change the structure instead.
As a final part, you could model this differently to avoid the "nested array" concept. And understand that the only real problem with a "nested array" is that "updating" a nested element is really impossible without reading in the whole document and modifying it.
So $push and $pull methods work fine. But using a "positional" $ operator just does not work as the "outer" array index is always the "first" matched element. So if this really was a problem for you then you could do something like this, for example:
{
"_id:"0,
"firstname":"Tom",
"childtoys" : [
{
"name": "childA",
"toy": "batman"
}.
{
"name": "childA",
"toy": "car"
},
{
"name": "childA",
"toy": "train"
},
{
"name": "childB",
"toy": "doll"
},
{
"name": "childB",
"toy": "bike"
},
{
"name": "childB",
"toy": "xbox"
}
],
"childMovies": [
{
"name": "childA"
"movie": "Ironman"
},
{
"name": "childA",
"movie": "Deathwish"
},
{
"name": "childB",
"movie": "Frozen"
},
{
"name": "childB",
"movie": "Barbie"
}
]
}
That would be one way to avoid the problem with nested updates if you did indeed need to "update" items on a regular basis rather than just $push and $pull items to the "toys" and "movies" arrays.
But the overall message here is to design your data around the access patterns you actually use. MongoDB does generally not like things with a "strict path" in the terms of being able to query or otherwise flexibly issue updates.
Projections in MongoDB make use of '1' and '0' , not 'True'/'False'.
Moreover ensure that the fields are specified in the right cases(uppercase/lowercase)
The query should be as below:
db.users.findOne({'_id': 0}, {'_id': 0, 'children.childA.movies': 1})
Which will result in :
{
"children" : {
"childA" : {
"movies" : {
"movie 1" : "Ironman",
"movie 2" : "Deathwish"
}
}
}
}

JSON parsing in JS

I am getting a JSON in response from server:
{
"width": "765",
"height": "990",
"srcPath": "http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_MERGED_/1273.pdf",
"coverPage": "",
"documents": [
{
"index": "1",
"text": "Archiving Microsoft® Office SharePoint® Server 2007 Data with the Hitachi Content Archive Platform and Hitachi Data Discovery for Microsoft SharePoint",
"type": "doc",
"id": "HDS_054227~201106290029",
"children": [
{
"text": "Page 1",
"leaf": "true",
"pageLocation": "http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_IMAGES_/HDS_054227~201106290029/image_1.png"
},
{
"text": "Page 2",
"leaf": "true",
"pageLocation": "http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_IMAGES_/HDS_054227~201106290029/image_2.png"
}
]
},
{
"index": "11",
"text": "Brocade FCoE Enabling Server I/O Consolidation",
"type": "doc",
"id": "HDS_053732~201105261741",
"children": [
{
"text": "Page 1",
"leaf": "true",
"pageLocation": "http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_IMAGES_/HDS_053732~201105261741/image_1.png"
},
{
"text": "Page 2",
"leaf": "true",
"pageLocation": "http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_IMAGES_/HDS_053732~201105261741/image_2.png"
}
]
}
]
}
And I want to get pagelocation of the children.
Can anyone tell me how to do this?
Hi
i also want to get indexes from this and then want to get pagelocations of that particular children. Can you tell me how would i do that?
And also when i when i am getting indexes array it is returning me ,, only and not the index nos.
I am using following code for that :
indexes=response.documents.map(function(e){ return e.children.index; })
Thanks & Regards
If you're interested in simply retrieving all the page locations, you can do it using filter:
var locations = [];
json.documents.forEach(function(e,i) {
e.children.forEach(function(e2,i2) {
locations.push(e2.pageLocation);
)}
});
// returns flat array like [item1,item2,item3,item4]
You can get an array of arrays using map:
var locations = [];
var locations = json.documents.map(function(e) {
return e.children.map(function(e2) {
return e2.pageLocation;
});
});
// returns 2-dimensional array like [[item1,item2],[item1,item2]]
Your json response is an appropriate javascript object So you can access all elements of the object like you do as in back end.
here, you have an array of object of the type documents and each document object has array of objects of the type children. so
syntax would be
myjson.documents[0].children[0].pagelocation
( = http://192.168.5.13:8888/ebook/user_content/_ADMIN_/_IMAGES_/HDS_054227~201106290029/image_1.png)
will give you the very first page location..
and so on

Categories

Resources