I'm using react and fetching docs from my firestore collection. and when i fetch the docs i assigned a conditional query and the query direction changes depending on the value provided.
const getData = async () => {
const constraints = [];
if (price)
constraints.push(orderBy("price", price == "1" ? "desc" : "asc"));
if (date)
constraints.push(orderBy("postedDate", date == "1" ? "desc" : "asc"));
if (type)
constraints.push(orderBy("type", type == "1" ? "desc" : "asc"));
// there are more conditional queries here. 8 more to be exact
const livings = collection(db, "livingPosts");
let q = query(livings, ...constraints);
const qSnapshot = await getDocs(q);
const dataQ = qSnapshot.docs.map((doc) => ({
...doc.data(),
id: doc.id,
}));
// console.log(dataQ);
setDatas(dataQ);
};
useEffect(() => {
getData();
}, []);
as you can see in the above code i've implemented conditional query. and there are more queries that i have not included. so my question is that how can i create indexes for all of these queries?
i created indexes with the link firebase provided me to create it. but it made me create 4 composite indexes just for 2 queries. (for price and date), the order is
price - asc , date - asc
price - desc , date - asc
price - asc , date - desc
price - desc , date - asc
these were the indexes. so do i have to create every possible indexes like these? if so there are number of combinations i have to do and the max number of indexes are 200. please show me the right way
So do i have to create every possible indexes like these
Yes you need to create all the indexes corresponding to the queries your app may potentially execute.
For that you can use the URL present in the error messages (which means that you need to execute all the possible queries, i.e. play all the scenarii) or you can use the Firebase (or Google Cloud) consoles to build them manually.
For the default limit of maximum 200 composite indexes you can actually contact the Firebase support to ask for an increase of this maximum number.
In Postgres, the overlap ("&&") operator of arrays are used to check if two arrays have any element in common.
For example,
Query
Result
ARRAY[1,4,3] && ARRAY[2,1]
true
ARRAY[1,4,3] && ARRAY[2,5]
false
Is there anything similar in GraphQL?
Also, is there anything I can use in Hasura for this?
A more specific real use case I can think of is as follows. (I just made them up just for clarification and did not run or validate the syntax.)
// Assume a you want to find all restaurants with any of the selected cuisines.
const restaurantA = {..., cuisineType: ['Fusion', 'Chinese', 'Japanese', 'Korean']}
const restaurantB = {..., cuisineType: ['Vegetarian', 'Thai']}
const restaurantC = {..., cuisineType: ['European']}
// Assume the data above is stored in Postgres (as an array in column field, not using a bridge table.
const SQL_1 = "SELECT *
FROM restaurants
WHERE cuisine_type && ['Korean', 'Japanese']" // Expect restaurantA and restaurantC
const SQL_2 = "SELECT *
FROM restaurants
WHERE cuisine_type && ['Vegetarian', 'European']" // Expect restaurantB and restaurantC
### GOAL: I want to use GraphQL, Apollo, Hasura to have the same effect as SQL_1 and SQL_2
From the docs:
You can also chain multiple where() methods to create more specific queries (logical AND).
How can I perform an OR query?
Example:
Give me all documents where the field status is open OR upcoming
Give me all documents where the field status == open OR createdAt <= <somedatetime>
OR isn't supported as it's hard for the server to scale it (requires keeping state to dedup). The work around is to issue 2 queries, one for each condition, and dedup on the client.
Edit (Nov 2019):
Cloud Firestore now supports IN queries which are a limited type of OR query.
For the example above you could do:
// Get all documents in 'foo' where status is open or upcmoming
db.collection('foo').where('status','in',['open','upcoming']).get()
However it's still not possible to do a general OR condition involving multiple fields.
With the recent addition of IN queries, Firestore supports "up to 10 equality clauses on the same field with a logical OR"
A possible solution to (1) would be:
documents.where('status', 'in', ['open', 'upcoming']);
See Firebase Guides: Query Operators | in and array-contains-any
suggest to give value for status as well.
ex.
{ name: "a", statusValue = 10, status = 'open' }
{ name: "b", statusValue = 20, status = 'upcoming'}
{ name: "c", statusValue = 30, status = 'close'}
you can query by ref.where('statusValue', '<=', 20) then both 'a' and 'b' will found.
this can save your query cost and performance.
btw, it is not fix all case.
I would have no "status" field, but status related fields, updating them to true or false based on request, like
{ name: "a", status_open: true, status_upcoming: false, status_closed: false}
However, check Firebase Cloud Functions. You could have a function listening status changes, updating status related properties like
{ name: "a", status: "open", status_open: true, status_upcoming: false, status_closed: false}
one or the other, your query could be just
...where('status_open','==',true)...
Hope it helps.
This doesn't solve all cases, but for "enum" fields, you can emulate an "OR" query by making a separate boolean field for each enum-value, then adding a where("enum_<value>", "==", false) for every value that isn't part of the "OR" clause you want.
For example, consider your first desired query:
Give me all documents where the field status is open OR upcoming
You can accomplish this by splitting the status: string field into multiple boolean fields, one for each enum-value:
status_open: bool
status_upcoming: bool
status_suspended: bool
status_closed: bool
To perform your "where status is open or upcoming" query, you then do this:
where("status_suspended", "==", false).where("status_closed", "==", false)
How does this work? Well, because it's an enum, you know one of the values must have true assigned. So if you can determine that all of the other values don't match for a given entry, then by deduction it must match one of the values you originally were looking for.
See also
in/not-in/array-contains-in: https://firebase.google.com/docs/firestore/query-data/queries#in_and_array-contains-any
!=: https://firebase.googleblog.com/2020/09/cloud-firestore-not-equal-queries.html
I don't like everyone saying it's not possible.
it is if you create another "hacky" field in the model to build a composite...
for instance, create an array for each document that has all logical or elements
then query for .where("field", arrayContains: [...]
you can bind two Observables using the rxjs merge operator.
Here you have an example.
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/observable/merge';
...
getCombinatedStatus(): Observable<any> {
return Observable.merge(this.db.collection('foo', ref => ref.where('status','==','open')).valueChanges(),
this.db.collection('foo', ref => ref.where('status','==','upcoming')).valueChanges());
}
Then you can subscribe to the new Observable updates using the above method:
getCombinatedStatus.subscribe(results => console.log(results);
I hope this can help you, greetings from Chile!!
We have the same problem just now, luckily the only possible values for ours are A,B,C,D (4) so we have to query for things like A||B, A||C, A||B||C, D, etc
As of like a few months ago firebase supports a new query array-contains so what we do is make an array and we pre-process the OR values to the array
if (a) {
array addObject:#"a"
}
if (b) {
array addObject:#"b"
}
if (a||b) {
array addObject:#"a||b"
}
etc
And we do this for all 4! values or however many combos there are.
THEN we can simply check the query [document arrayContains:#"a||c"] or whatever type of condition we need.
So if something only qualified for conditional A of our 4 conditionals (A,B,C,D) then its array would contain the following literal strings: #["A", "A||B", "A||C", "A||D", "A||B||C", "A||B||D", "A||C||D", "A||B||C||D"]
Then for any of those OR combinations we can just search array-contains on whatever we may want (e.g. "A||C")
Note: This is only a reasonable approach if you have a few number of possible values to compare OR with.
More info on Array-contains here, since it's newish to firebase docs
If you have a limited number of fields, definitely create new fields with true and false like in the example above. However, if you don't know what the fields are until runtime, you have to just combine queries.
Here is a tags OR example...
// the ids of students in class
const students = [studentID1, studentID2,...];
// get all docs where student.studentID1 = true
const results = this.afs.collection('classes',
ref => ref.where(`students.${students[0]}`, '==', true)
).valueChanges({ idField: 'id' }).pipe(
switchMap((r: any) => {
// get all docs where student.studentID2...studentIDX = true
const docs = students.slice(1).map(
(student: any) => this.afs.collection('classes',
ref => ref.where(`students.${student}`, '==', true)
).valueChanges({ idField: 'id' })
);
return combineLatest(docs).pipe(
// combine results by reducing array
map((a: any[]) => {
const g: [] = a.reduce(
(acc: any[], cur: any) => acc.concat(cur)
).concat(r);
// filter out duplicates by 'id' field
return g.filter(
(b: any, n: number, a: any[]) => a.findIndex(
(v: any) => v.id === b.id) === n
);
}),
);
})
);
Unfortunately there is no other way to combine more than 10 items (use array-contains-any if < 10 items).
There is also no other way to avoid duplicate reads, as you don't know the ID fields that will be matched by the search. Luckily, Firebase has good caching.
For those of you that like promises...
const p = await results.pipe(take(1)).toPromise();
For more info on this, see this article I wrote.
J
OR isn't supported
But if you need that you can do It in your code
Ex : if i want query products where (Size Equal Xl OR XXL : AND Gender is Male)
productsCollectionRef
//1* first get query where can firestore handle it
.whereEqualTo("gender", "Male")
.addSnapshotListener((queryDocumentSnapshots, e) -> {
if (queryDocumentSnapshots == null)
return;
List<Product> productList = new ArrayList<>();
for (DocumentSnapshot snapshot : queryDocumentSnapshots.getDocuments()) {
Product product = snapshot.toObject(Product.class);
//2* then check your query OR Condition because firestore just support AND Condition
if (product.getSize().equals("XL") || product.getSize().equals("XXL"))
productList.add(product);
}
liveData.setValue(productList);
});
For Flutter dart language use this:
db.collection("projects").where("status", whereIn: ["public", "unlisted", "secret"]);
actually I found #Dan McGrath answer working here is a rewriting of his answer:
private void query() {
FirebaseFirestore db = FirebaseFirestore.getInstance();
db.collection("STATUS")
.whereIn("status", Arrays.asList("open", "upcoming")) // you can add up to 10 different values like : Arrays.asList("open", "upcoming", "Pending", "In Progress", ...)
.addSnapshotListener(new EventListener<QuerySnapshot>() {
#Override
public void onEvent(#Nullable QuerySnapshot queryDocumentSnapshots, #Nullable FirebaseFirestoreException e) {
for (DocumentSnapshot documentSnapshot : queryDocumentSnapshots) {
// I assume you have a model class called MyStatus
MyStatus status= documentSnapshot.toObject(MyStatus.class);
if (status!= null) {
//do somthing...!
}
}
}
});
}
I'm setting up a test to ensure that a faceted Solr query 'contents' are correctly displayed within a page element, using javascript.
The Solr query result, which I've named "ryanlinkstransmissionpage", is;
{ Transmission: [ 'Manual', 12104, 'Automatic', 9858 ] }
What I would like to do is extract the 'Manual' and 'Automatic' only, so I can then test that these values are displayed on a page.
However, it is more the functionality involved in this that I cannot get my head around, as I will be using this method on other Solr query results.
To possibly complicate things, this Solr query result "ryanlinkstransmissionpage" is from a dynamic 'live' Solr, so the values may change each time it's run (so there may be more or less values within this array when it's tested on the following day for example).
I've tried a few javascript commands, but to no avail.
JSON.parse(ryanlinkstransmissionpage)
JSON.stringify(ryanlinkstransmissionpage)
Object.values(ryanlinkstransmissionpage)
Any help would be greatly appreciated. Thanks.
If possible, i highyl recommend changing the transmission field to be an object, rather than an array. That will give you far greater ability to read the data within.
Ignoring that, are you looking to extract the string values and the number values that follow them? ie. "Manual" and "12104"? Or are you simply trying to assert that the string values are present on the page?
Either way, here are two possible approaches.
const ryanlinkstransmissionpage = { Transmission: [ 'Manual', 12104, 'Automatic', 9858 ] };
// Pull out the string values
const strngVals = ryanlinkstransmissionpage.Transmission.filter(val => typeof val === 'string');
// Pull out the string values and the numbers that follow
const strngNumVals = ryanlinkstransmissionpage.Transmission.reduce((keyVals, val, idx, srcArr) => {
if (typeof val === 'string') keyVals[val] = srcArr[idx + 1];
return keyVals;
}, {});
The reduce approach is not stable or robust to changes in data provided from this Solr query result you refer to, nor is it tested. #shrug
Javascript has a built in method called Array.prototype.find(() =>). If you just want to check if this value exists to ensure its on the page, you can simply do:
const ryanlinkstransmissionpage = { Transmission: [ 'Manual', 12104, 'Automatic', 9858 ] };
const manual = ryanlinkstransmissionpage.Transmission.find((ele) => ele === 'Manual'); // returns 'Manual'
const automatic = ryanlinkstransmissionpage.Transmission.find((ele) => ele === 'Automatic'); // returns 'Automatic'
console.log(automatic);
console.log(manual);
// or
const findInArray = (arr, toFind) => {
const result = arr.find((ele) => ele === toFind);
return !!result;
}
console.log(findInArray(ryanlinkstransmissionpage.Transmission, 'Automatic')); // true
console.log(findInArray(ryanlinkstransmissionpage.Transmission, 'HelloWorld')); // false
console.log(findInArray(ryanlinkstransmissionpage.Transmission, 'Manual')); // true
Im creating my batch and inserting it to collection using command i specified below
batch = []
time = 1.day.ago
(1..2000).each{ |i| a = {:name => 'invbatch2k'+i.to_s, :user_id => BSON::ObjectId.from_string('533956cd4d616323cf000000'), :out_id => 'out', :created_at => time, :updated_at => time, :random => '0.5' }; batch.push a; }
Invitation.collection.insert batch
As stated above, every single invitation record has user_id fields value set to '533956cd4d616323cf000000'
after inserting my batch with created_at: 1.day.ago i get:
2.1.1 :102 > Invitation.lte(created_at: 1.week.ago).count
=> 48
2.1.1 :103 > Invitation.lte(created_at: Date.today).count
=> 2048
also:
2.1.1 :104 > Invitation.lte(created_at: 1.week.ago).where(user_id: '533956cd4d616323cf000000').count
=> 14
2.1.1 :105 > Invitation.where(user_id: '533956cd4d616323cf000000').count
=> 2014
Also, I've got a map reduce which counts invitations sent by each unique User (both total and sent to unique out_id)
class Invitation
[...]
def self.get_user_invites_count
map = %q{
function() {
var user_id = this.user_id;
emit(user_id, {user_id : this.user_id, out_id: this.out_id, count: 1, countUnique: 1})
}
}
reduce = %q{
function(key, values) {
var result = {
user_id: key,
count: 0,
countUnique : 0
};
var values_arr = [];
values.forEach(function(value) {
values_arr.push(value.out_id);
result.count += 1
});
var unique = values_arr.filter(function(item, i, ar){ return ar.indexOf(item) === i; });
result.countUnique = unique.length;
return result;
}
}
map_reduce(map,reduce).out(inline: true).to_a.map{|d| d['value']} rescue []
end
end
The issue is:
Invitation.lte(created_at: Date.today.end_of_day).get_user_invites_count
returns
[{"user_id"=>BSON::ObjectId('533956cd4d616323cf000000'), "count"=>49.0, "countUnique"=>2.0} ...]
instead of "count" => 2014, "countUnique" => 6.0 while:
Invitation.lte(created_at: 1.week.ago).get_user_invites_count returns:
[{"user_id"=>BSON::ObjectId('533956cd4d616323cf000000'), "count"=>14.0, "countUnique"=>6.0} ...]
Data provided by query, is accurate before inserting the batch.
I cant wrap my head around whats going on here. Am i missing something?
The part that you seemed to have missed in the documentation seem to be the problem here:
MongoDB can invoke the reduce function more than once for the same key. In this case, the previous output from the reduce function for that key will become one of the input values to the next reduce function invocation for that key.
And also later:
the type of the return object must be identical to the type of the value emitted by the map function to ensure that the following operations is true:
So what you see is your reduce function is returning a signature different to the input it receives from the mapper. This is important since the reducer may not get all of the values for a given key in a single pass. Instead it gets some of them, "reduces" the result and that reduced output may be combined with other values for the key ( possibly also reduced ) in a further pass through the reduce function.
As a result of your fields not matching, subsequent reduce passes do not see those values and do not count towards your totals. So you need to align the signatures of the values:
def self.get_user_invites_count
map = %q{
function() {
var user_id = this.user_id;
emit(user_id, {out_id: this.out_id, count: 1, countUnique: 0})
}
}
reduce = %q{
function(key, values) {
var result = {
out_id: null,
count: 0,
countUnique : 0
};
var values_arr = [];
values.forEach(function(value) {
if (value.out_id != null)
values_arr.push(value.out_id);
result.count += value.count;
result.countUnique += value.countUnique;
});
var unique = values_arr.filter(function(item, i, ar){ return ar.indexOf(item) === i; });
result.countUnique += unique.length;
return result;
}
}
map_reduce(map,reduce).out(inline: true).to_a.map{|d| d['value']} rescue []
end
You also do not need user_id in the values emitted or kept as it is already the "key" value for the mapReduce. The remaining alterations consider that both "count" and "countUnique" can contain an exiting value that needs to be considered, where you were simply resetting the value to 0 on each pass.
Then of course if the "input" has already been through a "reduce" pass, then you do not need the "out_id" values to be filtered for "uniqueness" as you already have the count and that is now included. So any null values are not added to the array of things to count, which is also "added" to the total rather than replacing it.
So the reducer does get called several times. For 20 key values the input will likely not be split, which is why your sample with less input works. For pretty much anything more than that, then the "groups" of the same key values will be split up, which is how mapReduce optimizes for large data processing. As the "reduced" output will be sent back to the reducer again, you need to be mindful that you are considering the values you already sent to output in the previous pass.