I cannot figure out how to proceed with an Upsert & "multiple" onConflict constraints. I want to push a data batch in a Supabase table.
My data array would be structured as follows:
items = [
{ date: "2023-01-26", url: "https://wwww.hello.com"},
{ date: "2023-01-26", url: "https://wwww.goodbye.com"},
...]
I would like to use the Upsert method to push this new batch in my Supabase table, unless if it already exists. To check if it already exists, I would like to use the date, and the url as onConflict criteria, if I understood well.
When I'm running this method
const { error } = await supabase
.from('items')
.upsert(items, { onConflict: ['date','url'] })
.select();
I'm having the following error:
{
code: '42P10',
details: null,
hint: null,
message: 'there is no unique or exclusion constraint matching the ON CONFLICT specification'
}
What am I missing? Where am I wrong?
You can pass more than one column in the upsert into by adding a column in a string (instead of using an array):
const { data, error } = await supabase
.from('items')
.upsert(items, { onConflict: 'date, url'} )
Postgres performs unique index inference as mentioned in https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT
It is necessary to have unique or indexes for this to work, as you can read in the documentation above:
INSERT into tables that lack unique indexes will not be blocked by
concurrent activity. Tables with unique indexes might block if
concurrent sessions perform actions that lock or modify rows matching
the unique index values being inserted; the details are covered in
Section 64.5. ON CONFLICT can be used to specify an alternative action
to raising a unique constraint or exclusion constraint violation
error.
I have to execute an INSERT INTO query using mysql or mysql2 in node.js
The values passed into the query utilizes a spatial geometry function ST_GeomFromGeoJSON() on geoMap column which is of type GEOMETRY.
Here is the simplified code:
const insertQuery = `INSERT INTO maps (name, geoMap) VALUES ?`;
const insertData = [];
geoMapList.forEach((geoMap) => {
insertData.push([
geoMap.name,
`ST_GeomFromGeoJSON(${JSON.stringify(geoMap.map)}, 2)`
]);
});
this.connection.query(insertQuery, [insertData]);
The above code does not work and throws the error
Cannot get geometry object from data you send to the GEOMETRY field
I believe this is because the ST_GeomFromGeoJSON() is not parsed as a function but as a string by MySQL.
How can this be resolved?
You can't put MySQL function calls in the parameter array, that data is all treated as literals.
Call the function in the SQL, with the JSON string as its parameter.
I don't think you can use node-mysql's bulk-insert for this, though, so you'll need to insert each row separately.
const insertQuery = `INSERT INTO maps(name, geoMap) VALUES (?, ST_GeomFromGeoJSON(?))`
geoMapList.forEach((geomap) =>
this.connection.query(insertQuery, [geomap.name, JSON.stringify(geomap.map)])
);
To avoid calling query() thousands of times, you can put multiple values lists in the query.
const values = Array(geoMapList.length).fill('(?, ST_GeomFromGeoJSON(?))).join(',');
const insertQuery = `INSERT INTO maps(name, geoMap) VALUES ${values}`;
const params = geoMapList.flatMap(({name, map}) => [name, JSON.stringify(map)]);
this.connection.query(insertquery, params);
Since this will create a very long INSERT query, make sure you increase the MySQL max_allowed_packet setting. You can also split geoMapList into smaller batches, rather than doing everything in a single query.
I thought I read that you can query subcollections with the new Firebase Firestore, but I don't see any examples. For example I have my Firestore setup in the following way:
Dances [collection]
danceName
Songs [collection]
songName
How would I be able to query "Find all dances where songName == 'X'"
Update 2019-05-07
Today we released collection group queries, and these allow you to query across subcollections.
So, for example in the web SDK:
db.collectionGroup('Songs')
.where('songName', '==', 'X')
.get()
This would match documents in any collection where the last part of the collection path is 'Songs'.
Your original question was about finding dances where songName == 'X', and this still isn't possible directly, however, for each Song that matched you can load its parent.
Original answer
This is a feature which does not yet exist. It's called a "collection group query" and would allow you query all songs regardless of which dance contained them. This is something we intend to support but don't have a concrete timeline on when it's coming.
The alternative structure at this point is to make songs a top-level collection and make which dance the song is a part of a property of the song.
UPDATE
Now Firestore supports array-contains
Having these documents
{danceName: 'Danca name 1', songName: ['Title1','Title2']}
{danceName: 'Danca name 2', songName: ['Title3']}
do it this way
collection("Dances")
.where("songName", "array-contains", "Title1")
.get()...
#Nelson.b.austin Since firestore does not have that yet, I suggest you to have a flat structure, meaning:
Dances = {
danceName: 'Dance name 1',
songName_Title1: true,
songName_Title2: true,
songName_Title3: false
}
Having it in that way, you can get it done:
var songTitle = 'Title1';
var dances = db.collection("Dances");
var query = dances.where("songName_"+songTitle, "==", true);
I hope this helps.
UPDATE 2019
Firestore have released Collection Group Queries. See Gil's answer above or the official Collection Group Query Documentation
Previous Answer
As stated by Gil Gilbert, it seems as if collection group queries is currently in the works. In the mean time it is probably better to use root level collections and just link between these collection using the document UID's.
For those who don't already know, Jeff Delaney has some incredible guides and resources for anyone working with Firebase (and Angular) on AngularFirebase.
Firestore NoSQL Relational Data Modeling - Here he breaks down the basics of NoSQL and Firestore DB structuring
Advanced Data Modeling With Firestore by Example - These are more advanced techniques to keep in the back of your mind. A great read for those wanting to take their Firestore skills to the next level
What if you store songs as an object instead of as a collection? Each dance as, with songs as a field: type Object (not a collection)
{
danceName: "My Dance",
songs: {
"aNameOfASong": true,
"aNameOfAnotherSong": true,
}
}
then you could query for all dances with aNameOfASong:
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
NEW UPDATE July 8, 2019:
db.collectionGroup('Songs')
.where('songName', isEqualTo:'X')
.get()
I have found a solution.
Please check this.
var museums = Firestore.instance.collectionGroup('Songs').where('songName', isEqualTo: "X");
museums.getDocuments().then((querySnapshot) {
setState(() {
songCounts= querySnapshot.documents.length.toString();
});
});
And then you can see Data, Rules, Indexes, Usage tabs in your cloud firestore from console.firebase.google.com.
Finally, you should set indexes in the indexes tab.
Fill in collection ID and some field value here.
Then Select the collection group option.
Enjoy it. Thanks
You can always search like this:-
this.key$ = new BehaviorSubject(null);
return this.key$.switchMap(key =>
this.angFirestore
.collection("dances").doc("danceName").collections("songs", ref =>
ref
.where("songName", "==", X)
)
.snapshotChanges()
.map(actions => {
if (actions.toString()) {
return actions.map(a => {
const data = a.payload.doc.data() as Dance;
const id = a.payload.doc.id;
return { id, ...data };
});
} else {
return false;
}
})
);
Query limitations
Cloud Firestore does not support the following types of queries:
Queries with range filters on different fields.
Single queries across multiple collections or subcollections. Each query runs against a single collection of documents. For more
information about how your data structure affects your queries, see
Choose a Data Structure.
Logical OR queries. In this case, you should create a separate query for each OR condition and merge the query results in your app.
Queries with a != clause. In this case, you should split the query into a greater-than query and a less-than query. For example, although
the query clause where("age", "!=", "30") is not supported, you can
get the same result set by combining two queries, one with the clause
where("age", "<", "30") and one with the clause where("age", ">", 30).
I'm working with Observables here and the AngularFire wrapper but here's how I managed to do that.
It's kind of crazy, I'm still learning about observables and I possibly overdid it. But it was a nice exercise.
Some explanation (not an RxJS expert):
songId$ is an observable that will emit ids
dance$ is an observable that reads that id and then gets only the first value.
it then queries the collectionGroup of all songs to find all instances of it.
Based on the instances it traverses to the parent Dances and get their ids.
Now that we have all the Dance ids we need to query them to get their data. But I wanted it to perform well so instead of querying one by one I batch them in buckets of 10 (the maximum angular will take for an in query.
We end up with N buckets and need to do N queries on firestore to get their values.
once we do the queries on firestore we still need to actually parse the data from that.
and finally we can merge all the query results to get a single array with all the Dances in it.
type Song = {id: string, name: string};
type Dance = {id: string, name: string, songs: Song[]};
const songId$: Observable<Song> = new Observable();
const dance$ = songId$.pipe(
take(1), // Only take 1 song name
switchMap( v =>
// Query across collectionGroup to get all instances.
this.db.collectionGroup('songs', ref =>
ref.where('id', '==', v.id)).get()
),
switchMap( v => {
// map the Song to the parent Dance, return the Dance ids
const obs: string[] = [];
v.docs.forEach(docRef => {
// We invoke parent twice to go from doc->collection->doc
obs.push(docRef.ref.parent.parent.id);
});
// Because we return an array here this one emit becomes N
return obs;
}),
// Firebase IN support up to 10 values so we partition the data to query the Dances
bufferCount(10),
mergeMap( v => { // query every partition in parallel
return this.db.collection('dances', ref => {
return ref.where( firebase.firestore.FieldPath.documentId(), 'in', v);
}).get();
}),
switchMap( v => {
// Almost there now just need to extract the data from the QuerySnapshots
const obs: Dance[] = [];
v.docs.forEach(docRef => {
obs.push({
...docRef.data(),
id: docRef.id
} as Dance);
});
return of(obs);
}),
// And finally we reduce the docs fetched into a single array.
reduce((acc, value) => acc.concat(value), []),
);
const parentDances = await dance$.toPromise();
I copy pasted my code and changed the variable names to yours, not sure if there are any errors, but it worked fine for me. Let me know if you find any errors or can suggest a better way to test it with maybe some mock firestore.
var songs = []
db.collection('Dances')
.where('songs.aNameOfASong', '==', true)
.get()
.then(function(querySnapshot) {
var songLength = querySnapshot.size
var i=0;
querySnapshot.forEach(function(doc) {
songs.push(doc.data())
i ++;
if(songLength===i){
console.log(songs
}
console.log(doc.id, " => ", doc.data());
});
})
.catch(function(error) {
console.log("Error getting documents: ", error);
});
It could be better to use a flat data structure.
The docs specify the pros and cons of different data structures on this page.
Specifically about the limitations of structures with sub-collections:
You can't easily delete subcollections, or perform compound queries across subcollections.
Contrasted with the purported advantages of a flat data structure:
Root-level collections offer the most flexibility and scalability, along with powerful querying within each collection.
I'm querying documents by timestamp and it is returning an empty array. However it works when I use "==" ex:.where('date', '==',timestamp), and returns empty array when I use '>=' or '<='.
I have tried to convert timestamp into date object and string also but no success.
Note: The date field in firestore is of type Timestamp.
I'm querying documents with date greater than '2018-08-03' in the collection.
Below is a picture of the collection of transactions (left) and the document(right) which should be part of the array of documents returned, because the date is greater than '2018-08-03'
Below is my code.
const firstDay = new Date('2018-08-03');
const timestamp1 = admin.firestore.Timestamp.fromDate(firstDay);
const trans = [];
const docRef = db.collection('Users').doc(uid).collection('transactions').where('item_id', '==', item_id)
.where('date', '>=', timestamp1);
await docRef.get()
.then((snapshot) => {
snapshot.forEach((doc) => {
trans.push({ transaction_id: doc.id, transaction: doc.data() });
});
})
.catch(err => new Error('cannot get the documents', err));
Expected result: should be an array with transactions with date greater than specified above.
Actual result: empty array.
Since its working for equality ==, I assumed that >= and <= would work. Anything I'm missing here?
Logs after trying ">" (equal to timestamp1)
Firestore (2.3.0) 2019-10-03T14:01:25.013Z AObRG [ClientPool.acquire]: Re-using existing client with 100 remaining operations
Firestore (2.3.0) 2019-10-03T14:01:25.015Z AObRG [Firestore.readStream]: Sending request: {"parent":"projects/valuemo-000/databases/(default)/documents/Users/Xmr3vKT19OSST02DSTDMt0jSq692","structuredQuery":{"from":[{"collectionId":"transactions"}],"where":{"compositeFilter":{"op":"AND","filters":[{"fieldFilter":{"field":{"fieldPath":"item_id"},"op":"EQUAL","value":{"stringValue":"zGRkpP3QkzI89zKyDjZ7FPDrXd5G3Bco5ENlR"}}},{"fieldFilter":{"field":{"fieldPath":"date"},"op":"EQUAL","value":{"timestampValue":{"seconds":1533254400}}}}]}}}}
Firestore (2.3.0) 2019-10-03T14:01:25.232Z AObRG [Firestore._initializeStream]: Received stream error: { Error: The query requires a COLLECTION_ASC index
for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB
at Http2CallStream.call.on (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call.js:68:41)
at Http2CallStream.emit (events.js:194:15)
at process.nextTick (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call-stream.js:71:22)
at process.internalTickCallback (internal/process/next_tick.js:70:11)
code: 9,
details:
'The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB',
metadata: Metadata { options: undefined, internalRepr: Map {} } }
Firestore (2.3.0) 2019-10-03T14:01:25.256Z AObRG [Firestore._initializeStream]: Received initial error: { Error: The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB
at Http2CallStream.call.on (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call.js:68:41)
at Http2CallStream.emit (events.js:194:15)
at process.nextTick (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call-stream.js:71:22)
at process.internalTickCallback (internal/process/next_tick.js:70:11)
code: 9,
details:
'The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB',
metadata: Metadata { options: undefined, internalRepr: Map {} } }
Firestore (2.3.0) 2019-10-03T14:01:25.260Z AObRG [Firestore._retry]: Request failed with unrecoverable error: { Error: The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB
at Http2CallStream.call.on (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call.js:68:41)
at Http2CallStream.emit (events.js:194:15)
at process.nextTick (D:\Projects\Valuemo-firebase\node_modules\#grpc\grpc-js\build\src\call-stream.js:71:22)
at process.internalTickCallback (internal/process/next_tick.js:70:11)
code: 9,
details:
'The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet. See its status here: https://console.firebase.google.com/project/valuemo-000/database/firestore/indexes/single_field?create_exemption=ClJwcm9qZWN0cy92YWx1ZW1vLTAwMC9kYXRhYmFzZXMvKGRlZmF1bHQpL2NvbGxlY3Rpb25Hcm91cHMvdHJhbnNhY3Rpb25zL2ZpZWxkcy9kYXRlEAEaCAoEZGF0ZRAB',
metadata: Metadata { options: undefined, internalRepr: Map {} } }
So I was testing it out, with what I believe to be a very similar set up to what you presented, and I was actually able to reproduce the same behaviour.
After testing different possibilities, I found that the issue for me had nothing to do with the datatype of the fields, but with the configuration of the composite index.
The composite index configuration required to make this query work was the following:
According to the Queries supported by composite indexes section of the documentation, the indexes for these kind of compound queries should have the equality filtered field first, this is not explicitly said, but it is how the docs present it in the examples.
Let me know if this managed to resolve the issue for you.
Take your date and convert into javascript date object. Then do getTime() on converted date object it will return a number. Then get firestore timestamp and do getSeconds().Compare seconds value of firestore timestamp with the variable storing output of getTime().
The query requires a COLLECTION_ASC index for collection transactions and field date. That index is not ready yet.
Why not create the index as the error is suggesting? This might fundamentally solve your issue.
I want to create a query with which I can check if a document has related documents, e.g. I have a document
{
name:"test1",
status:"CREATED"
date:"2018-12-12"
user:"foo"
}
and i want to retreive all documents where the user has no document with status:"OPEN"
I had the idea to project a field with the number of documents which are open for the user & day and filter for 0, but I don't know how I could do that.
So when i have the following documents:
it should only return foo2 and foo4 because foo, foo3 and foo5 already have documents with status:"OPEN"
Please find the below code to get all the documents for users for whom no 'OPEN' status exists:
db.collection.find({
user: {
$nin: db.collection.find({status: 'OPEN'}).map((row) => row.user)
}
});
$nin: to specify and exclusion list.
.map((row) => row.user): to convert array of filtered documents to array of just the string containing user ids.
Here is the logic:
1.First query all the doc with status OPEN
2.Now you have a sets of data with OPEN only doc
4.Now create a query where you pass all the userId of this set in query and fetch those values from collection which have userId not equal to the passed ones
Example:
db.collectionName.find({status:'OPEN'});// set1
db.collectionName.find({userId:{$not : 'userid1'}},{userId:{$not : 'userid2'}},....);
This is not good code but this is all you can do now or add some other fields in db to do it more effectively