From the docs:
You can also chain multiple where() methods to create more specific queries (logical AND).
How can I perform an OR query?
Example:
Give me all documents where the field status is open OR upcoming
Give me all documents where the field status == open OR createdAt <= <somedatetime>
OR isn't supported as it's hard for the server to scale it (requires keeping state to dedup). The work around is to issue 2 queries, one for each condition, and dedup on the client.
Edit (Nov 2019):
Cloud Firestore now supports IN queries which are a limited type of OR query.
For the example above you could do:
// Get all documents in 'foo' where status is open or upcmoming
db.collection('foo').where('status','in',['open','upcoming']).get()
However it's still not possible to do a general OR condition involving multiple fields.
With the recent addition of IN queries, Firestore supports "up to 10 equality clauses on the same field with a logical OR"
A possible solution to (1) would be:
documents.where('status', 'in', ['open', 'upcoming']);
See Firebase Guides: Query Operators | in and array-contains-any
suggest to give value for status as well.
ex.
{ name: "a", statusValue = 10, status = 'open' }
{ name: "b", statusValue = 20, status = 'upcoming'}
{ name: "c", statusValue = 30, status = 'close'}
you can query by ref.where('statusValue', '<=', 20) then both 'a' and 'b' will found.
this can save your query cost and performance.
btw, it is not fix all case.
I would have no "status" field, but status related fields, updating them to true or false based on request, like
{ name: "a", status_open: true, status_upcoming: false, status_closed: false}
However, check Firebase Cloud Functions. You could have a function listening status changes, updating status related properties like
{ name: "a", status: "open", status_open: true, status_upcoming: false, status_closed: false}
one or the other, your query could be just
...where('status_open','==',true)...
Hope it helps.
This doesn't solve all cases, but for "enum" fields, you can emulate an "OR" query by making a separate boolean field for each enum-value, then adding a where("enum_<value>", "==", false) for every value that isn't part of the "OR" clause you want.
For example, consider your first desired query:
Give me all documents where the field status is open OR upcoming
You can accomplish this by splitting the status: string field into multiple boolean fields, one for each enum-value:
status_open: bool
status_upcoming: bool
status_suspended: bool
status_closed: bool
To perform your "where status is open or upcoming" query, you then do this:
where("status_suspended", "==", false).where("status_closed", "==", false)
How does this work? Well, because it's an enum, you know one of the values must have true assigned. So if you can determine that all of the other values don't match for a given entry, then by deduction it must match one of the values you originally were looking for.
See also
in/not-in/array-contains-in: https://firebase.google.com/docs/firestore/query-data/queries#in_and_array-contains-any
!=: https://firebase.googleblog.com/2020/09/cloud-firestore-not-equal-queries.html
I don't like everyone saying it's not possible.
it is if you create another "hacky" field in the model to build a composite...
for instance, create an array for each document that has all logical or elements
then query for .where("field", arrayContains: [...]
you can bind two Observables using the rxjs merge operator.
Here you have an example.
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/observable/merge';
...
getCombinatedStatus(): Observable<any> {
return Observable.merge(this.db.collection('foo', ref => ref.where('status','==','open')).valueChanges(),
this.db.collection('foo', ref => ref.where('status','==','upcoming')).valueChanges());
}
Then you can subscribe to the new Observable updates using the above method:
getCombinatedStatus.subscribe(results => console.log(results);
I hope this can help you, greetings from Chile!!
We have the same problem just now, luckily the only possible values for ours are A,B,C,D (4) so we have to query for things like A||B, A||C, A||B||C, D, etc
As of like a few months ago firebase supports a new query array-contains so what we do is make an array and we pre-process the OR values to the array
if (a) {
array addObject:#"a"
}
if (b) {
array addObject:#"b"
}
if (a||b) {
array addObject:#"a||b"
}
etc
And we do this for all 4! values or however many combos there are.
THEN we can simply check the query [document arrayContains:#"a||c"] or whatever type of condition we need.
So if something only qualified for conditional A of our 4 conditionals (A,B,C,D) then its array would contain the following literal strings: #["A", "A||B", "A||C", "A||D", "A||B||C", "A||B||D", "A||C||D", "A||B||C||D"]
Then for any of those OR combinations we can just search array-contains on whatever we may want (e.g. "A||C")
Note: This is only a reasonable approach if you have a few number of possible values to compare OR with.
More info on Array-contains here, since it's newish to firebase docs
If you have a limited number of fields, definitely create new fields with true and false like in the example above. However, if you don't know what the fields are until runtime, you have to just combine queries.
Here is a tags OR example...
// the ids of students in class
const students = [studentID1, studentID2,...];
// get all docs where student.studentID1 = true
const results = this.afs.collection('classes',
ref => ref.where(`students.${students[0]}`, '==', true)
).valueChanges({ idField: 'id' }).pipe(
switchMap((r: any) => {
// get all docs where student.studentID2...studentIDX = true
const docs = students.slice(1).map(
(student: any) => this.afs.collection('classes',
ref => ref.where(`students.${student}`, '==', true)
).valueChanges({ idField: 'id' })
);
return combineLatest(docs).pipe(
// combine results by reducing array
map((a: any[]) => {
const g: [] = a.reduce(
(acc: any[], cur: any) => acc.concat(cur)
).concat(r);
// filter out duplicates by 'id' field
return g.filter(
(b: any, n: number, a: any[]) => a.findIndex(
(v: any) => v.id === b.id) === n
);
}),
);
})
);
Unfortunately there is no other way to combine more than 10 items (use array-contains-any if < 10 items).
There is also no other way to avoid duplicate reads, as you don't know the ID fields that will be matched by the search. Luckily, Firebase has good caching.
For those of you that like promises...
const p = await results.pipe(take(1)).toPromise();
For more info on this, see this article I wrote.
J
OR isn't supported
But if you need that you can do It in your code
Ex : if i want query products where (Size Equal Xl OR XXL : AND Gender is Male)
productsCollectionRef
//1* first get query where can firestore handle it
.whereEqualTo("gender", "Male")
.addSnapshotListener((queryDocumentSnapshots, e) -> {
if (queryDocumentSnapshots == null)
return;
List<Product> productList = new ArrayList<>();
for (DocumentSnapshot snapshot : queryDocumentSnapshots.getDocuments()) {
Product product = snapshot.toObject(Product.class);
//2* then check your query OR Condition because firestore just support AND Condition
if (product.getSize().equals("XL") || product.getSize().equals("XXL"))
productList.add(product);
}
liveData.setValue(productList);
});
For Flutter dart language use this:
db.collection("projects").where("status", whereIn: ["public", "unlisted", "secret"]);
actually I found #Dan McGrath answer working here is a rewriting of his answer:
private void query() {
FirebaseFirestore db = FirebaseFirestore.getInstance();
db.collection("STATUS")
.whereIn("status", Arrays.asList("open", "upcoming")) // you can add up to 10 different values like : Arrays.asList("open", "upcoming", "Pending", "In Progress", ...)
.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#Nullable QuerySnapshot queryDocumentSnapshots, #Nullable FirebaseFirestoreException e) {
for (DocumentSnapshot documentSnapshot : queryDocumentSnapshots) {
// I assume you have a model class called MyStatus
MyStatus status= documentSnapshot.toObject(MyStatus.class);
if (status!= null) {
//do somthing...!
}
}
}
});
}
And old slash escaping bug left us with some messed up data, like so:
{
suggestions: [
"ok",
"not ok /////////// ... 10s of KBs of this ... //////",
]
}
I would like to just pull those bad values out of the array. My first idea was to $pull based on a regex that matches 4 "/" characters, but it appears that regexes to not work on large strings:
db.notes.count({suggestions: /\/\/\/\//}) // returns 0
db.notes.count({suggestions: {$regex: "////"}}) // returns 0
My next idea was to use a $where query to find documents that have suggestion strings that are longer than 1000. That query works:
db.notes.count({
suggestions: {$exists: true},
$where: function() {
return !!this.suggestions.filter(function (item) {
return (item || "").length > 1000;
}).length
}
})
// returns a plausible number
But a $where query can't be used as the condition in a $pull update.
db.notes.update({
suggestions: {$exists: true},
}, {
$pull: {
suggestions: {
$where: function() {
return !!this.suggestions.filter(function (item) {
return (item || "").length > 1000;
}).length
}
}
}
})
throws
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 81,
"errmsg" : "no context for parsing $where"
}
})
I'm running out of ideas. Will I have to iterate over the entire collection, and $set: {suggestions: suggestions.filter(...)} for each document individually? Is there no better way to clean bad values out of an array of large strings in MongoDB?
(I'm only adding the "javascript" tag to get SO to format the code correctly)
The simple solution pointed out in the question comments should have worked. It does work with a test case that is a recreation of the original problem. Regexes can match on large strings, there is no special restriction there.
db.notes.updateOne({suggestions: /\/\//}, { "$pull": {suggestions: /\/\//}})
Since this didn't work for me, I ended up going with what the question discussed: updating all documents individually by filtering the array elements based on string length:
db.notes.find({
suggestions: {$exists: true}
}).forEach(function(doc) {
doc.suggestions = doc.suggestions.filter(function(item) {
return (item || "").length <= 1000;
}); db.notes.save(doc);
});
It ran slow, but that wasn't really a problem in this case.
Quick question from someone new to Mongo.
I have a collection of documents that (simplified) look like this:
{"_id":<objectID>, "name":"fakeName", "seeds":[1231,2341,0842,1341,3451, ...]}
What I really need is a $pop that pops 2 or 3 items off my list of seeds, but $pop currently only works for one item, so I'm trying to look for another way to accomplish the same thing.
The first thing I looked at was doing $push/$each/$slice with an empty "each", like:
update: { $push: { order: { $each: [ ], $slice: ?}}}
The problem here is that I don't know exactly how long I want my new slice to be (I want it to be "current size - number of seeds I popped"). If the $slice modifier worked like the $slice projection, this would be easy, I could just do $slice: [ #of seeds, ], but it doesn't so that doesn't work.
The next thing I looked at was getting the side of the array and using that as an input to $slice, like:
update: { $push: { seeds: { $each: [ ], $slice: {$subtract: [{$size:"$seeds"}, <number of seeds to pop>]}}}}
But Mongo tells me "value for $slice must be numeric value and not an Object", so apparently the result of $subtract is an Object not a number.
Then I tried to see if I could "remove" items from the array based on an empty query with a $limit, but apparently limit gets applied later in the pipeline, so I couldn't manage to make that work.
Any other suggestions, or am I out of luck and need to go back to the drawing board?
Thanks so much for help/input.
MongoDB does not presently have any method of referencing the existing values of fields in a singular update statement. The only exceptions are operators such as $inc an $mul which can act on the present value and alter it according to a set rule.
This is in part due to the compatibility of the "phrasing" of operations to act over multiple documents, whether that is the case or not. But what you are asking for is some "variable" operation that allows the "length" of an array to be tested and used as in "input parameter" to another method. This is not supported.
So the best you can do is read the document content and test the length of the array in code, then perform the $slice update as you first surmized, or alternately you could use the aggregation framework to work out the "possible lengths" of arrays ( assuming a lot of duplication ) and then work on "multi" updates for those documents that match the conditions, of course assuming that you want to do this over more than a single document.
First form:
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
db.collection.find().forEach(function(doc) {
if ( doc.order.length > 2 ) {
bulk.find({ "_id": doc._id })
.updateOne({
"$push": {
"order": { "$each": [], "$slice": doc.order.length - 2 }
}
});
count ++;
}
if ( (count % 1000) == 0 && ( count > 1 ) ) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
});
if ( count % 1000 != 0 )
bulk = db.collection.initializeOrderedBulkOp();
Second form:
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
db.collection.aggregate([
{ "$group": { "_id": { "$size": "$order" } }},
{ "$match": { "$_id": { "$gt": 2 } }}
]).forEach(function(doc) {
bulk.find({ "order": { "$size": doc._id } })
.update(
"$push": {
"order": { "$each": [], "$slice": doc._id - 2 }
}
});
count ++;
if ( count % 1000 == 0 ) {
bulk.execute();
bulk = db.collection.initializeOrderedBulkOp();
}
});
if ( count % 1000 != 0 )
bulk = db.collection.initializeOrderedBulkOp();
Noting that in both cases there is some logic to consider the length of the arrays in order not to "empty" them, or produce an undesired $slice operation.
Another possibly alternative is to use the projection form of $slice in the query to get the last n elements and then $pull the matching elements from the array. Of course the identifier used for such an operation would have to be "unique", but it is a valid case where uniqueness is assured.
So whatever your case, you cannot do this in a singular update statement without having some prior knowledge of the current state of the document to be modified. The different listings though give you ways to approach "emulating" this, albeit not in a single statement.
I'm trying to use map-reduce to understand when this can be helpful.
So I have a collection named "actions" with 100k docs like this:
{
"profile_id":1111,
"action_id":2222
}
Now I'm trying to do map-reduce examples. I'm trying to get a list of "all users and total actions each one has". Is this possible? My code:
db.fbooklikes.mapReduce(
function(){
emit(this.profile_id, this.action_id);
},
function(keyProfile, valueAction){
return Array.sum(valueAction);
},
{
out:"example"
}
)
.. This is not working. The result is:
"counts" : {
"input" : 100000,
"emit" : 100000,
"reduce" : 1146,
"output" : 13
},
"ok" : 1,
"_o" : {
"result" : "map_reduce_example",
"timeMillis" : 2539,
"counts" : {
"input" : 100000,
"emit" : 100000,
"reduce" : 1146,
"output" : 13
},
"ok" : 1
},
What I'm trying to do is something possible with map-reduce?
Well yes you can use it, but the more refined response is that there are likely better tools for doing what you want.
MapReduce is handy for some tasks, but usually best suited when something else does not apply. The inclusion of mapReduce in MongoDB pre-dates the introduction of the aggregation framework, which is generally what you should be using when you can:
db.fbooklikes.aggregate([
{ "$group": {
"_id": "$profile_id",
"count": { "$sum": 1 }
}}
])
Which will simply return the counts for the all documents in the collection grouped by each value of "profile_id".
MapReduce requires JavaScript evaluation and therefore runs much slower than the native code functions implemented by the aggregation framework. Sometimes you have to use it, but in simple cases it is best not to, and there are some quirks that you need to understand:
db.fbooklikes.mapReduce(
function(){
emit(this.profile_id, 1);
},
function(key,values){
return Array.sum(values);
},
{
out: { "inline": 1 }
}
)
The biggest thing people miss with mapReduce is the fact that the reducer is almost never called just once per emitted key. In fact it will process output in "chunks", thus "reducing" down part of that output and placing it back to be "reduced" again against other output until there is only a single value for that key.
For this reason it is important to emit the same type of data from the reduce function as is sent from the "map" function. It's a sticky point that can lead to weird results when you don't understand that part of the function. It is in fact the underlying way that mapReduce can deal with large values of results for a single key value and reduce them.
But generally speaking, you should be using the aggregation framework where possible, and where a problem requires some special calculations that would not be possible there, or otherwise has some complex document traversal where you need to inspect with JavaScript, then that is where you use mapReduce.
You don't want to sum the action ids, you want to count them. So you want something like the following
var map = function () {
emit(this.profile_id, { action_ids : [this.action_id], count : 1 });
}
var reduce = function(profile_id, values) {
var value = { action_ids: [], count: 0 };
for (var i = 0; i < values.length; i++) {
value.count += values[i].count;
value.action_ids.push.apply(value.action_ids, values[i].action_ids);
}
return value;
}
db.fbooklikes.mapReduce(map, reduce, { out:"example" });
This will give you an array of action ids and a count for each profile id. The count could be obtained by accessing the length of the action_ids array, but I thought I would keep it separate to make the example clearer.
I have to deal with inconsistent documents in MongoDB collection where some field might be numeric or might have NaN value. I need to update it with $inc. But looks like if it have NaN value $inc have no effect. What options available for atomic document update?
Well this seems to lead to two logical conclusions. The first being that if there are NaN values present in a field then how to identify them? Consider the following sample, let's call the collection "nantest"
{ "_id" : ObjectId("54055993b145d1c015a1ad41"), "n" : NaN }
{ "_id" : ObjectId("540559e8b145d1c015a1ad42"), "n" : Infinity }
{ "_id" : ObjectId("54055b59b145d1c015a1ad43"), "n" : 1 }
{ "_id" : ObjectId("54055ea1b145d1c015a1ad44"), "n" : -Infinity }
So both NaN and Infinity or -Infinity are representative of "non-numbers" that have somehow emerged in your data. The best way to find these documents where that field is set that way is to use the $where operator for a JavaScript evaluated query condition. Not efficient but it what you have got:
db.nantest.find({
"$where": "return isNaN(this.n) || Math.abs(this.n) == Infinity"
})
So that gives a way of finding the data that is the problem. From here you could jump through hoops and decide that where this was encountered you would just reset it to 0 before incrementing, essentially issuing two update statements where the first one would not match a document to update if the value was correct:
db.nantest.update(
{ "$where": "return isNaN(this.n) || Math.abs(this.n) == Infinity" },
{ "$set": { "n": 0 } }
);
db.nantest.update(
{ },
{ "$inc": { "n": 1 } }
);
But really when you look at that, why would you want to patch your code to cater for this when you can just patch the data. So the logical thing to finally conclude is just update all the Nan and possibly Infinity values to a standard reset number in one statement:
db.nantest.update(
{ "$where": "return isNaN(this.n) || Math.abs(this.n) == Infinity" },
{ "$set": { "n": 0 } },
{ "multi": true }
);
Run one statement and then you don't have to change your code and simply process increments as you should normally expect.
If your trouble is knowing which fields have the Nan values present in order to invoke updates to fix them, then consider something along the lines of this mapReduce process to inspect the fields:
db.nantest.mapReduce(
function () {
var doc = this;
delete doc._id;
Object.keys( doc ).forEach(function(key) {
if ( isNaN( doc[key] ) || Math.abs(doc[key]) == Infinity )
emit( key, 1 );
});
},
function (key,values) {
return Array.sum( values );
},
{ "out": { "inline": 1 } }
)
For which you might need to add some complexity to for more nested documents, but this tells you which fields can possibly contain the errant values so you can construct update statements to fix them.
It would seem that rather than bending your code to suit this you "should be" doing:
Find the source that is causing the numbers to appear and fix that.
Identify the field or fields that contain these values
Process one off update statement to fix the data all at once.
Minimal messing with your code and it both fixes the "source" of the problem and the "result" of that data corruption that was introduced.