Related
I am currently trying to make a script where I pass my own unique key in firebase as primary key. It needs to work like this
StudentData
-Student Unique id
- referadID : "active",
- referadID2 : "inactive",
- referadID3 : "active",
-Student Unique id2
-referadIDnew : "active",
Here every student refers another student and once a refered student join that refered student who joined has his own unique id which stores value of active or inactive. The problem here is that if I use push it will generate a unique id which I don't want because it will make it difficult to work while using sorting. I don't want to do it. How can I create my own unique key in studentUniqueid2. I tried using transaction instead of push but the issue is that transaction instead of adding another key value pair would simply update the whole key leaving only the latest student unique id in the table which I don't want. Here is my code
let createData=adminDatabase.ref('/ReferralSystem/joinedtransactionlog');
createData.transaction(function(currentData){
if(currentData !== InviterId){
return {
[InviterId]: {
[member.id] : "active"
}
}
}else{
console.log("curretn data exists");
}
});
I am using javascript and working on firebase realtime database system with platform of Admin.
If I understand correctly, you have two UIDs: one for the inviter and the other for the invitee/member. In that case, you can write the structure in your question with:
createData.child(inviterUID).child(memberUID).set("active");
Doing this with a simple set operation will scale much better than the transaction you tried, since you're creating contention on the entire StudentData node while this merely performs an idempotent write.
There is no need to prevent writing the data if it already exists, as the operation is idempotent: subsequent writes don't change the result of the first write.
I have a Firestore collection named channels, and I'd like to get the list of channels based on an array of IDs and order it by the createdAt field, this is my function :
const getChannels = () => {
const q = query(
collection(db, "channels"),
where(documentId(), "in", [
"F0mnR5rNdhwSLPZ57pTP",
"G8p6TWSopLN4dNHJLH8d",
"wMWMlJwa3m3lYINNjCLT",
]),
orderBy("createdAt")
);
const unsubscribe = onSnapshot(q, (snapshot) => {
snapshot.docs.map((doc) => {
console.log(doc.data());
});
});
return unsubscribe;
};
But I'm getting this error
FirebaseError: inequality filter property and first sort order must be the same: __name__ and createdAt.
It only works if I orderBy documentId().
I'm aware there is a limitation in the docs about this, but I'm wondering if there is a workaround for this type of situation.
Also the answer for this question isn't working anymore I guess.
The title of your question indicates that you are trying to use where and orderBy for different fields. But note that you are using documentId() in the where condition to filter, which is not a field in the Firestore document.
So if you filter is based on documentId(), you can use only documentId() in orderBy() clause, that also in ascending order because currently Firestore does not support sorting in descending order of documentId() which is mentioned in this answer.
Let’s take a look at the following examples -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId()).get();
The above will work and sort the documents based on documentId() after filtering based on documentId().
But it is not relevant to apply an orderBy() clause based on the documentId(), because without applying the orderBy() clause also yields the same result as, by default, Firestore query gives documents in ascending order of documentId(). That means the following also yields the same result -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).get();
Now Firestore doesn’t support to sort in descending order of documentId() which means the following will not work -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId(),"desc").get();
This will ask to create an index -
The query requires an index. You can create it here:
But if you go there to create an index it will say -
__name__ only indexes are not supported.
Now let's come to your query. What you are trying to do is to filter based on documentId() and then orderBy() based on createdAt field which is not possible and it will give the following error-
inequality filter property and first sort order must be the same.
You may think to use two orderBy() clauses, something like this -
const data=await db.collection("users").where(admin.firestore.FieldPath.documentId(),"in",["104","102","101"]).orderBy(admin.firestore.FieldPath.documentId()).orderBy(“createdAt”
).get();
Which will not work and give the following error
order by clause cannot contain more fields after the key
I am not sure of your use case but it’s not a great idea to filter based on documentId(). If it is required to filter based on documentId(), I would suggest creating a field in the Firestore document which will contain the documentIds and filter based on that.
Now considering the title of the question, yes it is possible to use where() and orderBy() clauses for different fields in Firestore. There are some limitations and you need to stick to that -
If you include a filter with a range comparison (<, <=, >, >=), your first ordering must be on the same field.
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“createdAt”).get();
The above query doesn't work.
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“number”).get();
The above query works and you can still use further orderBy() on different fields, something like following -
const data=await db.collection("users").where(“number”,">=", “101”).orderBy(“number”).orderBy(“createdAt”).get();
You cannot order your query by any field included in an equality (=) or in clause.
const data=await db.collection("users").where(“number”,"in",["104","102","101"]).orderBy(“number”).get();
const data=await db.collection("users").where(“number”,"==", “101”).orderBy(“number”).get();
The above two don’t work.
Firestore's speed and efficiency comes almost ENTIRELY from it's use of indexes. Inequalities (INCLUDING in and not-in) are accomplished by sorting by the index, and using the value as a "cut-off" - thus REQUIRING (whether you want it or not) the orderby() to be on the same field as the inequality.
The "answer not working anymore" was never really working in the first place, as the above shows. If you aren't trying to paginate, do the obvious and "filter" by the document ID's and sort on the client.
BUT...
...more importantly, it is ALMOST NEVER useful nor performant to use documentId's to select from the database, unless you both copy it to a field, AND are looking for a SPECIFIC id. In almost all cases, it would be FAR better to use a query on another field (however you got the list of documentId's in the first place), then orderBy. Yes, the inequality/orderBy is a limitation, but it's there for a reason.
Going forward, an important design decision is to understand what questions you want your data to answer, and design your entire database schema to support those queries - this is the fundamental nature of NoSQL.
Problem:The other link that you have shared before perfectly works and the only solutions available is to create an index. However the reason you are not able to do a where and order with the above example is because you cannot create an index with the document id and createdAt.
Solution: To do so add the document id as one of the field say docID in the document then create an index with the fields docID and createdAt. This should be working for you.
Note: I have not physically tested this. Will update once I have checked it
I've searched around but didn't find if it's possible.
I've this MySQL query:
INSERT INTO table (id,a,b,c,d,e,f,g) VALUES (1,2,3,4,5,6,7,8)
Field id has a "unique index", so there can't be two of them. Now if the same id is already present in the database, I'd like to update it. But do I really have to specify all these field again, like:
INSERT INTO table (id,a,b,c,d,e,f,g) VALUES (1,2,3,4,5,6,7,8)
ON DUPLICATE KEY UPDATE a=2,b=3,c=4,d=5,e=6,f=7,g=8
Or:
INSERT INTO table (id,a,b,c,d,e,f,g) VALUES (1,2,3,4,5,6,7,8)
ON DUPLICATE KEY UPDATE a=VALUES(a),b=VALUES(b),c=VALUES(c),d=VALUES(d),e=VALUES(e),f=VALUES(f),g=VALUES(g)
I've specified everything already in the insert...
A extra note, I'd like to use the work around to get the ID to!
id=LAST_INSERT_ID(id)
I hope somebody can tell me what the most efficient way is.
The UPDATE statement is given so that older fields can be updated to new value. If your older values are the same as your new ones, why would you need to update it in any case?
For eg. if your columns a to g are already set as 2 to 8; there would be no need to re-update it.
Alternatively, you can use:
INSERT INTO table (id,a,b,c,d,e,f,g)
VALUES (1,2,3,4,5,6,7,8)
ON DUPLICATE KEY
UPDATE a=a, b=b, c=c, d=d, e=e, f=f, g=g;
To get the id from LAST_INSERT_ID; you need to specify the backend app you're using for the same.
For LuaSQL, a conn:getlastautoid() fetches the value.
There is a MySQL specific extension to SQL that may be what you want - REPLACE INTO
However it does not work quite the same as 'ON DUPLICATE UPDATE'
It deletes the old row that clashes with the new row and then inserts the new row. So long as you don't have a primary key on the table that would be fine, but if you do, then if any other table references that primary key
You can't reference the values in the old rows so you can't do an equivalent of
INSERT INTO mytable (id, a, b, c) values ( 1, 2, 3, 4)
ON DUPLICATE KEY UPDATE
id=1, a=2, b=3, c=c + 1;
I'd like to use the work around to get the ID to!
That should work — last_insert_id() should have the correct value so long as your primary key is auto-incrementing.
However as I said, if you actually use that primary key in other tables, REPLACE INTO probably won't be acceptable to you, as it deletes the old row that clashed via the unique key.
Someone else suggested before you can reduce some typing by doing:
INSERT INTO `tableName` (`a`,`b`,`c`) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE `a`=VALUES(`a`), `b`=VALUES(`b`), `c`=VALUES(`c`);
There is no other way, I have to specify everything twice. First for the insert, second in the update case.
Here is a solution to your problem:
I've tried to solve problem like yours & I want to suggest to test from simple aspect.
Follow these steps: Learn from simple solution.
Step 1: Create a table schema using this SQL Query:
CREATE TABLE IF NOT EXISTS `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(30) NOT NULL,
`password` varchar(32) NOT NULL,
`status` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `no_duplicate` (`username`,`password`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1;
Step 2: Create an index of two columns to prevent duplicate data using following SQL Query:
ALTER TABLE `user` ADD INDEX no_duplicate (`username`, `password`);
or, Create an index of two column from GUI as follows:
Step 3: Update if exist, insert if not using following queries:
INSERT INTO `user`(`username`, `password`) VALUES ('ersks','Nepal') ON DUPLICATE KEY UPDATE `username`='master',`password`='Nepal';
INSERT INTO `user`(`username`, `password`) VALUES ('master','Nepal') ON DUPLICATE KEY UPDATE `username`='ersks',`password`='Nepal';
Just in case you are able to utilize a scripting language to prepare your SQL queries, you could reuse field=value pairs by using SET instead of (a,b,c) VALUES(a,b,c).
An example with PHP:
$pairs = "a=$a,b=$b,c=$c";
$query = "INSERT INTO $table SET $pairs ON DUPLICATE KEY UPDATE $pairs";
Example table:
CREATE TABLE IF NOT EXISTS `tester` (
`a` int(11) NOT NULL,
`b` varchar(50) NOT NULL,
`c` text NOT NULL,
UNIQUE KEY `a` (`a`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
I know it's late, but i hope someone will be helped of this answer
INSERT INTO t1 (a,b,c) VALUES (1,2,3),(4,5,6)
ON DUPLICATE KEY UPDATE c=VALUES(a)+VALUES(b);
You can read the tutorial below here :
https://mariadb.com/kb/en/library/insert-on-duplicate-key-update/
http://www.mysqltutorial.org/mysql-insert-or-update-on-duplicate-key-update/
You may want to consider using REPLACE INTO syntax, but be warned, upon duplicate PRIMARY / UNIQUE key, it DELETES the row and INSERTS a new one.
You won't need to re-specify all the fields. However, you should consider the possible performance reduction (depends on your table design).
Caveats:
If you have AUTO_INCREMENT primary key, it will be given a new one
Indexes will probably need to be updated
With MySQL v8.0.19 and above you can do this:
mysql doc
INSERT INTO mytable(fielda, fieldb, fieldc)
VALUES("2022-01-01", 97, "hello")
AS NEW(newfielda, newfieldb, newfieldc)
ON DUPLICATE KEY UPDATE
fielda=newfielda,
fieldb=newfieldb,
fieldc=newfieldc;
SIDENOTE: Also if you want a conditional in the on duplicate key update part there is a twist in MySQL. If you update fielda as the first argument and include it inside the IF clause for fieldb it will already be updated to the new value! Move it to the end or alike. Let's say fielda is a date like in the example and you want to update only if the date is newer than the previous:
INSERT INTO mytable(fielda, fieldb)
VALUES("2022-01-01", 97)
AS NEW(newfielda, newfieldb, newfieldc)
ON DUPLICATE KEY UPDATE
fielda=IF(fielda<STR_TO_DATE(newfielda,'%Y-%m-%d %H:%i:%s'),newfielda,fielda),
fieldb=IF(fielda<STR_TO_DATE(newfielda,'%Y-%m-%d %H:%i:%s'),newfieldb,fieldb);
in this case fieldb would never be updated because of the <! you need to move the update of fielda below it or check with <= or =...!
INSERT INTO mytable(fielda, fieldb)
VALUES("2022-01-01", 97)
AS NEW(newfielda, newfieldb, newfieldc)
ON DUPLICATE KEY UPDATE
fielda=IF(fielda<STR_TO_DATE(newfielda,'%Y-%m-%d %H:%i:%s'),newfielda,fielda),
fieldb=IF(fielda=STR_TO_DATE(newfielda,'%Y-%m-%d %H:%i:%s'),newfieldb,fieldb);
This works as expected with using = since fielda is already updated to its new value before reaching the if clause of fieldb... Personally i like <= the most in such a case if you ever rearrange the statement...
you can use insert ignore for such case, it will ignore if it gets duplicate records
INSERT IGNORE
... ; -- without ON DUPLICATE KEY
I currently have a few issues with my Firestore querying technique. As per this stackoverflow post I made recently, Querying with two array with firestore security rules
The answer proposed to add the the "ids" into a object, with the key as the id, and the value simply being "true". I have completed this, and now my structure looks like so:
This leaves me with this query:
db.collection('Depots')
.where(`products.${productId}`, '==', true)
.where(`users.${userId}`, '==', true)
.where('created', '>', 1585998560500)
.orderBy('created', 'asc')
.get();
This query leaves me with throwing an error, asking to create an index:
The query requires an index. You can create it here: ...
However, this tries to index the specific object key, i.e. QXooVYGBIFWKo6C so products.QXooVYGBIFWKo6C. Which is certianly not what I want, as this query changes, and can have an infinite number of possibilities, which means I would have to create another index for each key entry in order to query it.
Is there any way to solve this issue? I am assuming it needs to index this query due to the different operators used in the query, so I was wondering if there were any workarounds to this issue.
Thank you very much in advance.
What you have here is a map field, for which indexes should usually be created automatically.
That indeed means that you'll have as many indexes as you have products, which means:
You are limited in how many products you can have, as there is a maximum of 40,000 index entries per document.
You pay more per document, as you pay for the storage of each index.
If these are not what you want, you'll have to switch back to your original model, with the query limitations you had there. There doesn't seem to be a solution that fits both of your requirements.
After our discussion in chat, this is the starting point I would suggest. Who knows what the end architecture would look like, but I think this or very close to this. You say that a user can exist in multiple depots at the same time and multiple depots can contain the same products, also at the same time. You also said that a depot can never have more than 40 users at a given time, so an array of 40 users would certainly not encroach on Firestore's document limit of 1,048,576 bytes.
[collection]
<documentId>
- field: value
[depots]
<UUID>
- depotId: string "depot456"
- productCount: num 5,000
<UUID>
- depotId: string "depot789"
- productCount: num 4,500
[products]
<UUID>
- productId: string "lotion123"
- depotId: string "depot456"
- users: [string] ["user10", "user27", "user33"]
<UUID>
- productId: string "lotion123"
- depotId: string "depot789"
- users: [string] ["user10", "user17", "user50"]
[users]
<userId>
- depots: [string] ["depot456", "depot999"]
<userId>
- depots: [string] ["depot333", "depot999"]
In NoSQL, storage is cheap and computation isn't so denormalize your data as much as you need to make your queries possible and efficient (fast and cheap).
To find all depots in a single query where user10 and lotion123 are both true, query the products collection where productId equals x and users array-contains y and collect the depotId values from those results. If you want to preserve the array-contains operation for something else, you'd have to denormalize your data further (replace the array for a single user). Or you could split this query into two separate queries.
With this model, when a user leaves a depot, get all products where users array-contains that user and remove that userId from the array. And when a user joins a depot, get all products where depotId equals x and append that userId to the array.
Watch this video, and others by Rick, to get a solid handle on NoSQL: https://www.youtube.com/watch?v=HaEPXoXVf2k
#danwillm If you are not sure about the number of users and products then your DB structure seems unfit for this situation because there are size and length limitations of the firestore document.
You should rather create a separate collection for products and users i.e normalize your data and have a reference for the user in the product collection.
User :
{
userId: documentId,
name: John,
...otherInfo
}
Product :
{
productId: documentId,
createdBy: userId,
createdOn:date,
productName:"exa",
...otherInfo
}
This way you there will be the size of the document would be limited, i.e try avoiding using maps/arrays in firestore if you are not sure about there size.
Also, in this case, the number of queries would be increased but you don't need many indexes in this case.
Below is a snipet of code that I am having trouble with. The purpose is to check duplicate entries in the database and return "h" with a boolean if true or false. For testing purposes I am returning a true boolean for "h" but by the time the alert(duplicate_count); line gets executed the duplicate_count is still 0. Even though the alert for a +1 gets executed.
To me it seems like the function updateUserFields is taking longer to execute so it's taking longer to finish before getting to the alert.
Any ideas or suggestions? Thanks!
var duplicate_count = 0
for (var i = 0; i < skill_id.length; i++) {
function updateUserFields(h) {
if(h) {
duplicate_count++;
alert("count +1");
} else {
alert("none found");
}
}
var g = new cfc_mentoring_find_mentor();
g.setCallbackHandler(updateUserFields);
g.is_relationship_duplicate(resource_id, mentee_id, section_id[i], skill_id[i], active_ind,table);
};
alert(duplicate_count);
There is no reason whatsoever to use client-side JavaScript/jQuery to remove duplicates from your database. Security concerns aside (and there are a lot of those), there is a much easier way to make sure the entries in your database are unique: use SQL.
SQL is capable of expressing the requirement that there be no duplicates in a table column, and the database engine will enforce that for you, never letting you insert a duplicate entry in the first place. The syntax varies very slightly by database engine, but whenever you create the table you can specify that a column must be unique.
Let's use SQLite as our example database engine. The relevant part of your problem is right now probably expressed with tables something like this:
CREATE TABLE Person(
id INTEGER PRIMARY KEY ASC,
-- Other fields here
);
CREATE TABLE MentorRelationship(
id INTEGER PRIMARY KEY ASC,
mentorID INTEGER,
menteeID INTEGER,
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
However, you can make enforce uniqueness i.e. require that any (mentorID, menteeID) pair is unique, by changing the pair (mentorID, menteeID) to be the primary key. This works because you are only allowed one copy of each primary key. Then, the MentorRelationship table becomes
CREATE TABLE MentorRelationship(
mentorID INTEGER,
menteeID INTEGER,
PRIMARY KEY (mentorID, menteeID),
FOREIGN KEY (mentorID) REFERENCES Person(id),
FOREIGN KEY (menteeID) REFERENCES Person(id)
);
EDIT: As per the comment, alerting the user to duplicates but not actually removing them
This is still much better with SQL than with JavaScript. When you do this in JavaScript, you read one database row at a time, send it over the network, wait for it to come to your page, process it, throw it away, and then request the next one. With SQL, all the hard work is done by the database engine, and you don't lose time by transferring unnecessary data over the network. Using the first set of table definitions above, you could write
SELECT mentorID, menteeID
FROM MentorRelationship
GROUP BY mentorID, menteeID
HAVING COUNT(*) > 1;
which will return all the (mentorID, menteeID) pairs that occur more than once.
Once you have a query like this working on the server (and are also pulling out all the information you want to show to the user, which is presumably more than just a pair of IDs), you need to send this over the network to the user's web browser. Essentially, on the server side you map a URL to return this information in some convenient form (JSON, XML, etc.), and on the client side you read this information by contacting that URL with an AJAX call (see jQuery's website for some code examples), and then display that information to the user. No need to write in JavaScript what a database engine will execute orders of magnitude faster.
EDIT 2: As per the second comment, checking whether an item is already in the database
Almost everything I said in the first edit applies, except for two changes: the schema and the query. The schema should become the second of the two schemas I posted, since you don't want the database engine to allow duplicates. Also, the query should be simply
SELECT COUNT(*) > 0
FROM MentorRelationship
WHERE mentorID = #mentorID AND menteeID = #menteeID;
where #mentorID and #menteeID are the items that the user selected, and are inserted into the query by a query builder library and not by string concatenation. Then, the server will get a true value if the item is already in the database, and a false value otherwise. The server can send that back to the client via AJAX as before, and the client (that's your JavaScript page) can alert the user if the item is already in the database.