I'm using an extremely simple Cloud Function to (try to) keep my realtime database in sync with the Firestore.
exports.copyDocument = functions.database.ref('/invoices/{companyId}/{documentId}')
.onWrite((change, context) => {
if (!change.after.exists()) {
return null;
}
return admin.firestore().collection('companies').doc(context.params.companyId).collection('invoices')
.doc(context.params.documentId)
.set(change.after.val());
});
Unfortunately, I am seeing issues where sometimes the Cloud Firestore document does not have the latest copy of the realtime DB data. It's infrequent, but nonetheless impacting my end users.
Why is this happening?
Two ideas I had were -
The Firestore is possibly unavailable to write to in the Cloud Function, and I don't have retries enabled in my Cloud Function, so it could just be erroring out. Enabling retries will solve my problem. However, I have absolutely 0 error logs of this happening.
User makes change A, then user makes change B 2-3 seconds later. In this case, it's possible, I guess, that the Cloud Function trigger for change A hasn't executed, but the trigger for change B quickly executes, then the change A trigger executes and copies 'stale' data. Possible remedies would be to fetch the latest version again from the realtime database in my Cloud Function (not ideal, it's nice using change.after.val()), or perhaps keep some incrementing integer on my document, and instead copy it to the Firestore using a transaction that compares versions instead of a simple set().
The only error message I do see in my Cloud Function error logs is:
The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability
But those docs indicate that this error is always retried, even without explicitly enabling function retries.
I am leaning towards issue 1 - the Firestore just 'blips' sometimes, retries aren't enabled, and I'm not logging the failed promise properly. If this is the case - how can I fix this logging?
Related
We are running an application on Firestore and got a simple trigger that when order's details are created or updated some of it's information should be rewritten in the parent order collection.
The function for this got following code
export const updateOrderDetails = functions
.region(FUNCTION_REGION)
.firestore.document("orders/{orderId}/details/pickupAndDropoff")
.onWrite(async (change, context) => {
return await admin
.firestore()
.collection("orders")
.doc(context.params.orderId)
.set({ pickupAndDropoff: change.after.data() }, { merge: true });
});
It was work fine before, but now at random about every third of its executions is delayed. Sometimes by few minutes. In Cloud Function logs we see normal execution times <200ms, so it seems the trigger runs after a huge pause.
What's worse from time to time our change.after.data() is undefined, but we never delete anything - it's just updates and creates.
It was working fine, we did not changed nothing since last week, but now it started to have this unexpected delays. We've also checked the firebase status, but there are no malfunctions in firebase functions service. What can be the cause of this?
The problem can be due to the monotonically increasing orderId as the parameter passed here:
...
.collection("orders")
.doc(context.params.orderId)
...
If you can check once if the orderId passed here is monotonically increasing with each request? It can lead to hotspots which impacts latency.
To explain, I think the write rate must be changing at different day's and time's - as the user traffic using the application or load testing requests changes - which is creating the unexpected kind of behaviour. At low write rate, the requests are working as expected most of the time. At high write rate, the requests are facing hotspot situation in the firestore as mentioned in the firestore documentation resulting in delays (latency issue).
Here is the relevant link to firestore best practices documentation.
Thanks to Frank van Puffelen suggestion we've sent this question directly to Firebase support and after their internal investigation we've got the reply from an engineering team that it was in fact an infrastructure malfunction.
The reply I got from them was:
I escalated the issue to recover more information. So far it appears that there was an issue with pub/sub delivering and creating the event. The Firestore team is also communicating with the pub/sub team to investigate the issue and prevent future incidents.
It seems that the best way to deal with such problems is to quickly write directly to Firebase support team, because as they mentioned in the automatic reply I got after sending a support ticket:
For Firebase outages not listed on the status dashboard, we'll respond within 4 hours.
which seems to be the best option.
I want to reset a specific value in my Firebase Realtime Database every day at 12:00 AM. I'm using Firebase Cloud Functions to do this. This is the code that I have:
exports.dailyReset = functions.pubsub.schedule('0 0 * * *').onRun((context) => {
exports.resetValue = functions.database.ref('/users/{userId}')
.onWrite((change, context) => {
var ref = change.data.ref;
ref.update({
"daily": 0
});
return ref.update({"daily": 0});
});
return null;
});
Upon testing the script, it doesn't work. It's not resetting the value in my Firebase Realtime Database. Can someone tell me how do I fix this?
It's not possible to use the Functions SDK to write to the database. functions can only be used to establish triggers that run when the triggering event occurs.
In other words, functions.database.ref('/users/{userId}').onWrite() isn't going to write anything at all.
If you want to write to Realtime Database from a nodejs program, you should use the Firebase Admin SDK to write data.
The Cloud Functions triggers in your index.js file have to be known and fixed when run firebase deploy. That means you can't dynamically create triggers from inside a Cloud Functions, as you're trying to do.
The common approaches for dynamic scheduling are:
Have a single Cloud Function that runs periodically and then executes the tasks for the past time period.
Dynamically schedule Cloud Functions with Cloud Tasks, as Doug describes in his blog post How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL).
But in your case, why do you even need the onWrite trigger? Can't you just import the Admin SDK, user that to read all users, and then delete them?
I have looked up for this issue but could not find any sufficient information about it.
Google Cloud Spanner client libraries handles sessions automatically and its limit is 10.000 sessions for each node, no problem till here.
I have a micro serviced application which also has Google Cloud Functions. I am doing some specific database jobs on Cloud Functions and I'm also calling those functions continuously. After a little while, Cloud Spanner is starting to throw an error;
Too many active sessions in database, limit is 10000. Increase the node count to allow more sessions.
I know about the limits, but there is not any operation that will cause my app to exceed those limits.
After I noticed this, I have two questions which I could not find any answer;
1- Does Cloud Functions creates new session for every call? (I am using HTTP Trigger)
Here is what I did so far;
1- Here is example cloud functions declaration of mine;
exports.myFunction = function myFunction(req, res) {}
I was declaring my database instance out of this scope before I realize this issue;
const db = Spanner({projectId: '[my-project]'}).instance('[my-cs-instance]').database('[my-database]');
exports.myFunction = function myFunction(req, res) {}
After this issue, I have put it in the scope like this, and closed the database session after I'm done;
exports.myFunction = function myFunction(req, res) {
const db = Spanner({projectId: '[my-project]'}).instance('[my-cs-instance]').database('[my-database]');
// codes
db.close();
}
That didn't change anything, it still exceeds the session limit after a while.
Do you have any experience what causes this? Is this related to Cloud Functions or Cloud Spanner itself?
2- If every transaction object use one connection at a time, what happens in this scenario.
I have a REST endpoint other than these Cloud Functions. It creates a database instance when its starting to listen HTTP endpoints and I am not creating any other instance in its lifecycle anymore. At that endpoint, I am making CRUDs and I am using transactions and they all use the same instance which I created at the start of process. My experience is;
Sometimes transactions or other CRUD operations works with a bit delay which does not happen all the time.
My question is;
Is that because when transaction starts to work, does it lock the connection and all other operations should wait until it ends? If so, should I create independent database instances for transactions on that endpoint?
Thanks in advance
This now has been fixed per the issue opened at #89 and the fix at #91, and logged as #71987137 at Google Issue Trackers.
If any issue persists, please report at Google issue tracker they will re-open to examine.
I have written a cloud function that is working well.
Sometimes this function is being executed more than one time for the same user (I made sure that only one request is being requested from the client android app).
After some debugging I noticed that this problem happens if the connection is bad. I may be or may not be correct.
How to overcome such an issue?
As in the comments I also don't believe the client SDKs would duplicate a Cloud Function call on a bad connection. The expected behaviour would be for them to throw a network-related exception on the client side and not call it again. A problem would arise if the Cloud Function runs successfully and the client is only unable to get the result back.
I can think of the following solutions, with no more details about the Cloud Function itself:
Try to make the function idempotent - Meaning that even if it runs twice the end result is the same, assuming the same input/parameters.
Cache the results and manually throttle the function - This is more complicated, and is only needed if the network problem persists and you can't work around eliminating the side effects of the function. You would have to create another Class to cache the results of the function (maybe keyed by the parameters) and returning the same result with no computation and side effects.
I have breezeJs running in an angular app on mobile device (cordova), which talks to .Net WebApi.
Everything works great, except once in a while the device will get PrimaryKey violations (from my SQL Server).
I think I narrowed it down to only happening when data connection is shakey on the device.
The only way I can figure these primary key violations are happening is somehow the server is Saving Changes, but the mobile connection drops out before the response can come back from server that everything saved OK.
What is supposed to happen when BreezeJS doesn't hear back from server after calling SaveChanges?
Anyone familiar with BreezeJS know of a way to handle this scenario?
I've had to handle the same scenario in my project. The approach I took was two part:
Add automatic retries to failed ajax requests. I'm using breeze with jQuery, so I googled "jQuery retry ajax". There's many different implementations, mine is somewhat custom, all center around hijacking the onerror callback as well as the deferred's fail handler to inject retry logic. I'm sure Angular will have similar means of retrying dropped requests.
In the saveChanges fail handler, add logic like this:
...
function isConcurrencyException(reason: any) {
return reason && reason.message && /Store update, insert, or delete statement affected an unexpected number of rows/.test(reason.message);
}
function isConnectionFailure(reason: any): boolean {
return reason && reason.hasOwnProperty('status') && reason.status === 0
}
entityManager.saveChanges()
.then(... yay ...)
.fail(function(reason) {
if (isConnectionFailure(reason)) {
// retry attempts failed to reach server.
// notify user and save to local storage....
return;
}
if (isConcurrencyException(reason)) {
// EF is not letting me save the entities again because my previous save (or another user's save) moved the concurrency stamps on the record. There's also the possibility that a record I'm try to save was deleted by another user.
// recover... in my case I kept it simple and simply attempt to reload the entity. If nothing is returned I know the entity was deleted. Otherwise I now have the latest version. In either case a message is shown to the user.
return;
}
if (reason.entityErrors) {
// We have an "entityErrors" property... this means the saved failed due to server-side validation errors.
// do whatever you do to handle validation errors...
return;
}
// an unexpected exception. let it bubble up.
throw reason;
})
.done(); // terminate the promise chain (may not be an equivalent in Angular, not sure).
One of the ways you can test spotty connections is to use Fiddler's AutoResponder tab. Set up a *.drop rule with a regex that matches your breeze route and check the "Enable Automatic Responses" box when you want to simulate dropped requests.
This is a somewhat messy problem to solve- no one size fits all answer, hope this helps.
NOTE
Ward makes a good point in the comments below. This approach is not suitable in situations where the entity's primary key is generated on the server (which would be the case if your db uses identity columns for PKs) because the retry logic could cause duplicate inserts.