It is known that firebase bills with number of doc reads. there is a daily limit to firestore doc read (50k read per day). but when i fetch in my react app it prompts the quota exceed error.
FirebaseError: Request failed with error: Quota exceeded.
what i actually did was i created an async function to get documents, and called it. but this way it listens to every change in firestore, and read all over again. i haven't even opened the console b/c it listens to changes as well.
let me explain what i'm building. I'm building an amazon type web. and in the home page it display the products. so to prevent more read i need to limit the number of fetched docs. initially, the first items fetched should be 21 products. and when the user reach the end of the list they click the load more button and it loads an additional 21 products.
so my questions are,
How can i Limit the number of docs fetched? even if it's limited does it cache the previously loaded docs or it reloads them again? and if i use snapshot listener does that reload every docs or updates the latest one? is there a right way to fetch firestore documents?
this is how i used it
const getData = async () => {
const constraints = [];
if (price)
constraints.push(orderBy("price", price == "1" ? "desc" : "asc"));
if (date)
constraints.push(orderBy("postedDate", date == "1" ? "desc" : "asc"));
const livings = collection(db, "livingPosts");
let q = query(livings, ...constraints);
const qSnapshot = await getDocs(q);
const dataQ = qSnapshot.docs.map((doc) => ({
...doc.data(),
id: doc.id,
}));
// console.log(dataQ);
setDatas(dataQ);
};
getData();
You should use the pagination options suggested in firebase here
You can limit docs fetched by for example 21 using the .limit option see example here
In case of snapshot listeners, if you read 100 docs then it is considered 100 reads and if there is any update to any of those 100 docs...it is pushed to client, so it is considered an additional read AFAIK.
best option to use query, pagination and limit to achieve what you want, as per what you have mentioned. it does not need snapshot listeners.
Example of how to do this available in a old post which you can adapt to current firebase version
Related
I am using the getCountFromServer() for my React App to display the number of docs in a collection.
However, when the user deletes a doc on the front end or creates one, the count is not updated in real-time.
Is it possible to fix this or must the user always reload the page to see the latest data?
The documentation says, emphasis mine:
count() aggregation queries are currently only supported via direct server response. Queries are served only by the Cloud Firestore backend, skipping the local cache and any buffered updates. This behavior is identical to operations performed inside Cloud Firestore transactions. You cannot currently use count() queries with real-time listeners and offline queries.
So, no, you can't get realtime count updates. You will have to make the request again to get an update. Whether or not you require a page refresh is up to you.
As #DougStevenson mentioned in his comment, there is no way you can get real-time updates when you're using count(). It's true you can call count() every time it is needed, but there is however a workaround.
To solve this, you should consider storing a counter in a document at a known path, and then increment its value every time a new document is added or decrement the value every time a document is deleted. To get real-time updates, simply attach a persistent listener on that document and you'll be up to date with the number of documents in the collection.
Besides that, if your collection grows in size (> 10,000,000 documents), then the above solution might be the only solution you have. For that, please check #FrankvanPuffelen's answer from the following post:
How fast is counting documents in Cloud Firestore?
If you are using getCountFromServer() to load the initial count and then want to decrement it for all active users when a document is deleted, you can use a listener on the same collection and update the state on frontend like this:
import { collection, query, where, onSnapshot } from "firebase/firestore";
// initial state loaded by getCountFromServer()
const [count, setCount] = useState(10);
const q = query(collection(db, "collectionName"));
const unsubscribe = onSnapshot(q, (snapshot) => {
snapshot.docChanges().forEach((change) => {
if (change.type === "removed") {
console.log("Removed document: ", change.doc.data());
// TODO: Decrement the count in state
// e.g. setCount(count - 1)
}
});
});
This way you won't have to calculate the count again or maintain the count anywhere.
Here is my snipping code of React Native which I am using for my mobile application. I can't wait 30sec for just 50k data. Imagine the time it will take to load in 500k of data or more
useEffect(() => {
const fetchData = () => {
setLoading(true);
firestore().collection('FData')
.orderBy('counter', 'desc') //order using firestore counter
.limit(50) //limiting value
.onSnapshot(snapshot => {
var items = [];
snapshot.forEach((doc)=> {
items.push({ id: doc.id, ...doc.data() });
});
setLoading(false);
setList(items);
})
};
fetchData();
}, []);
You can try to sort them in the app.
But I think your main problem - you trying to get 50k items by ONE request. It's to much. 10 item it's ok. More than 100 - it's not ok.
I can't wait 30sec for just 50k data.
Reading 50k documents, it's way too much data to read in a single go. Bear in mind that, that the query performance in Firestore depends on the number of documents you request and not on the number of documents you search.
So it doesn't really matter if you search for 10 documents in a collection of 100 documents or in a collection that contains 100 MIL documents, the response time will always be the same. So to be able to read data fast, I recommend you load it in smaller chunks. This practice is called pagination and it can be really simple implemented using query cursors if take a look at the official documentation.
Besides that, since everything in Firestore it's about the number of reads you perform, reading 50k documents at once, can be a bit expensive.
I have an angular app where i am querying my firebase database as below:
constructor() {
this.getData();
}
getData() {
this.projectSubscription$ = this.dataService.getAllProjects()
.pipe(
map((projects: any) =>
projects.map(sc=> ({ key: sc.key, ...sc.payload.val() }))
),
switchMap(appUsers => this.dataService.getAllAppUsers()
.pipe(
map((admins: any) =>
appUsers.map(proj =>{
const match: any = admins.find(admin => admin.key === proj.admin);
return {...proj, imgArr: this.mapObjectToArray(proj.images), adminUser: match.payload.val()}
})
)
)
)
).subscribe(res => {
this.loadingState = false;
this.projects = res.reverse();
});
}
mapObjectToArray = (obj: any) => {
const mappedDatas = [];
for (const key in obj) {
if (Object.prototype.hasOwnProperty.call(obj, key)) {
mappedDatas.push({ ...obj[key], id: key });
}
}
return mappedDatas;
};
And here is what I am querying inside dataService:
getAllProjects() {
return this.afDatabase.list('/projects/', ref=>ref.orderByChild('createdAt')).snapshotChanges();
}
getAllAppUsers() {
return this.afDatabase.list('/appUsers/', ref=>ref.orderByChild('name')).snapshotChanges();
}
The problem I am facing with this is I have 400 rows of data which I am trying to load and it is taking around 30seconds to load which is insanely high. Any idea how can I query this in a faster time?
We have no way to know whether the 30s is reasonable, as that depends on the amount of data loaded, the connection latency and bandwidth of the client, and more factors we can't know/control.
But one thing to keep in mind is that you're performing 400 queries to get the users of each individual app, which is likely not great for performance.
Things you could consider:
Pre-load all the users once, and then use that list for each project.
Duplicate the name of each user into each project, so that you don't need to join any data at all.
If you come from a background in relational databases the latter may be counterintuitive, but it is actually very common in NoSQL data modeling and is one of the reasons NoSQL databases scale so well.
I propose 3 solutions.
1. Pagination
Instead of returning all those documents on app load, limit them to just 10 and keep record of the last one. Then display the 10 (or any arbitrary base number)
Then make the UI in such a way that the user has to click next or when the user scrolls, you fetch the next set based on the previous last document's field's info.
I'm supposing you need to display all the fetched data in some table or list so having the UI paginate the data should make sense.
2. Loader
Show some loader UI on website load. Then when all the documents have fetched, you hide the loader and show the data as you want. You can use some custom stuff for loader, or choose from any of the abundant libraries out there, or use mat-progress-spinner from Angular Material
3. onCall Cloud Function
What if you try getting them through an onCall cloud function? It night be faster because it's just one request that the app will make and Firebase's Cloud Functions are very fast within Google's data centers.
Given that the user's network might be slow to iterate the documents but the cloud function will return all at once and that might give you what you want.
I guess you could go for this option only if you really really need to display all that data at once on website load.
... Note on cost
Fetching 400 or more documents every time a given website loads might be expensive. It'll be expensive if the website is visited very frequently by very many users. Firebase cost will increase as you are charged per document read too.
Check to see if you could optimise the data structure to avoid fetching this much.
This doesn't apply to you if this some admin dashboard or if fetching all users like this is done rarely making cost to not be high in that case.
I recently developped a firebase schedule function to retrieve data on every monday.
I tested it during the previous days and it was working correctly. However, this morning, I discovered that my query wasn't able anymore to retrieve data as it used to do. I now have an empty array at QuerySnapshot.docs. You can find my code below:
const functions = require("firebase-functions");
const admin = require("firebase-admin");
admin.initializeApp();
const db = admin.firestore();
exports.scheduledFunction = functions.pubsub.schedule("0 0 * * 1").onRun(async () => {
console.log("start");
const querySnapshotnext = await db.collection("Next_challenges").orderBy("createdAt").get();
console.log("Let see querySnapshot :", querySnapshotnext); //it works, I can see QuerySnapshot object
console.log("Let see docs :", querySnapshotnext.docs); //have an empty array [] even if it should not, it wasn't empty few days ago
console.log("let see the data of the first doc : ", querySnapshotnext.docs[0].data()); //is of course undefined
return null;
});
You can find my database below with the doc that is normally selected:
The rules of my databases are the following :
I don't really understand why my code isn't working anymore, I think it's certainly related to some parameters stuffs and don't really know how I could debug this by myself so don't hesitate to give me some tips so I can be more autonomous with firebase. Thank you :)
Firestore's documentation has a note that says,
An orderBy() clause also filters for existence of the given field. The result set will not include documents that do not contain the given field.
You have .orderBy("createdAt") in your query but there isn't any createdAt field in the documents (at least the one in screenshot). Did you change your document structure recently that is causing this issue?
Also the Admin SDK bypasses any security rules so I'd recommend setting them to false if you don't need to access data directly from client.
I'm having slow performance issues with Firestore while retrieving basic data stored in a document compared to the realtime database with 1/10 ratio.
Using Firestore, it takes an average of 3000 ms on the first call
this.db.collection(‘testCol’)
.doc(‘testDoc’)
.valueChanges().forEach((data) => {
console.log(data);//3000 ms later
});
Using the realtime database, it takes an average of 300 ms on the first call
this.db.database.ref(‘/test’).once(‘value’).then(data => {
console.log(data); //300ms later
});
This is a screenshot of the network console :
I'm running the Javascript SDK v4.50 with AngularFire2 v5.0 rc.2.
Did anyone experience this issue ?
UPDATE: 12th Feb 2018 - iOS Firestore SDK v0.10.0
Similar to some other commenters, I've also noticed a slower response on the first get request (with subsequent requests taking ~100ms). For me it's not as bad as 30s, but maybe around 2-3s when I have good connectivity, which is enough to provide a bad user experience when my app starts up.
Firebase have advised that they're aware of this "cold start" issue and they're working on a long term fix for it - no ETA unfortunately. I think it's a separate issue that when I have poor connectivity, it can take ages (over 30s) before get requests decide to read from cache.
Whilst Firebase fix all these issues, I've started using the new disableNetwork() and enableNetwork() methods (available in Firestore v0.10.0) to manually control the online/offline state of Firebase. Though I've had to be very careful where I use it in my code, as there's a Firestore bug that can cause a crash under certain scenarios.
UPDATE: 15th Nov 2017 - iOS Firestore SDK v0.9.2
It seems the slow performance issue has now been fixed. I've re-run the tests described below and the time it takes for Firestore to return the 100 documents now seems to be consistently around 100ms.
Not sure if this was a fix in the latest SDK v0.9.2 or if it was a backend fix (or both), but I suggest everyone updates their Firebase pods. My app is noticeably more responsive - similar to the way it was on the Realtime DB.
I've also discovered Firestore to be much slower than Realtime DB, especially when reading from lots of documents.
Updated tests (with latest iOS Firestore SDK v0.9.0):
I set up a test project in iOS Swift using both RTDB and Firestore and ran 100 sequential read operations on each. For the RTDB, I tested the observeSingleEvent and observe methods on each of the 100 top level nodes. For Firestore, I used the getDocument and addSnapshotListener methods at each of the 100 documents in the TestCol collection. I ran the tests with disk persistence on and off. Please refer to the attached image, which shows the data structure for each database.
I ran the test 10 times for each database on the same device and a stable wifi network. Existing observers and listeners were destroyed before each new run.
Realtime DB observeSingleEvent method:
func rtdbObserveSingle() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observeSingleEvent(of: .value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Realtime DB observe method:
func rtdbObserve() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observe(.value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore getDocument method:
func fsGetDocument() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).getDocument() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore addSnapshotListener method:
func fsAddSnapshotListener() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).addSnapshotListener() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Each method essentially prints the unix timestamp in milliseconds when the method starts executing and then prints another unix timestamp when each read operation returns. I took the difference between the initial timestamp and the last timestamp to return.
RESULTS - Disk persistence disabled:
RESULTS - Disk persistence enabled:
Data Structure:
When the Firestore getDocument / addSnapshotListener methods get stuck, it seems to get stuck for durations that are roughly multiples of 30 seconds. Perhaps this could help the Firebase team isolate where in the SDK it's getting stuck?
Update Date March 02, 2018
It looks like this is a known issue and the engineers at Firestore are working on a fix. After a few email exchanges and code sharing with a Firestore engineer on this issue, this was his response as of today.
"You are actually correct. Upon further checking, this slowness on getDocuments() API is a known behavior in Cloud Firestore beta. Our engineers are aware of this performance issue tagged as "cold starts", but don't worry as we are doing our best to improve Firestore query performance.
We are already working on a long-term fix but I can't share any timelines or specifics at the moment. While Firestore is still on beta, expect that there will be more improvements to come."
So hopefully this will get knocked out soon.
Using Swift / iOS
After dealing with this for about 3 days it seems the issue is definitely the get() ie .getDocuments and .getDocument. Things I thought were causing the extreme yet intermittent delays but don't appear to be the case:
Not so great network connectivity
Repeated calls via looping over .getDocument()
Chaining get() calls
Firestore Cold starting
Fetching multiple documents (Fetching 1 small doc caused 20sec delays)
Caching (I disabled offline persistence but this did nothing.)
I was able to rule all of these out as I noticed this issue didn't happen with every Firestore database call I was making. Only retrievals using get(). For kicks I replaced .getDocument with .addSnapshotListener to retrieve my data and voila. Instant retrieval each time including the first call. No cold starts. So far no issues with the .addSnapshotListener, only getDocument(s).
For now, I'm simply dropping the .getDocument() where time is of the essence and replacing it with .addSnapshotListener then using
for document in querySnapshot!.documents{
// do some magical unicorn stuff here with my document.data()
}
... in order to keep moving until this gets worked out by Firestore.
Almost 3 years later, firestore being well out of beta and I can confirm that this horrible problem still persists ;-(
On our mobile app we use the javascript / node.js firebase client. After a lot of testing to find out why our app's startup time is around 10sec we identified what to attribute 70% of that time to... Well, to firebase's and firestore's performance and cold start issues:
firebase.auth().onAuthStateChanged() fires approx. after 1.5 - 2sec, already quite bad.
If it returns a user, we use its ID to get the user document from firestore. This is the first call to firestore and the corresponding get() takes 4 - 5sec. Subsequent get() of the same or other documents take approx. 500ms.
So in total the user initialization takes 6 - 7 sec, completely unacceptable. And we can't do anything about it. We can't test disabling persistence, since in the javascript client there's no such option, persistence is always enabled by default, so not calling enablePersistence() won't change anything.
I had this issue until this morning. My Firestore query via iOS/Swift would take around 20 seconds to complete a simple, fully indexed query - with non-proportional query times for 1 item returned - all the way up to 3,000.
My solution was to disable offline data persistence. In my case, it didn't suit the needs of our Firestore database - which has large portions of its data updated every day.
iOS & Android users have this option enabled by default, whilst web users have it disabled by default. It makes Firestore seem insanely slow if you're querying a huge collection of documents. Basically it caches a copy of whichever data you're querying (and whichever collection you're querying - I believe it caches all documents within) which can lead to high Memory usage.
In my case, it caused a huge wait for every query until the device had cached the data required - hence the non-proportional query times for the increasing numbers of items to return from the exact same collection. This is because it took the same amount of time to cache the collection in each query.
Offline Data - from the Cloud Firestore Docs
I performed some benchmarking to display this effect (with offline persistence enabled) from the same queried collection, but with different amounts of items returned using the .limit parameter:
Now at 100 items returned (with offline persistence disabled), my query takes less than 1 second to complete.
My Firestore query code is below:
let db = Firestore.firestore()
self.date = Date()
let ref = db.collection("collection").whereField("Int", isEqualTo: SomeInt).order(by: "AnotherInt", descending: true).limit(to: 100)
ref.getDocuments() { (querySnapshot, err) in
if let err = err {
print("Error getting documents: \(err)")
} else {
for document in querySnapshot!.documents {
let data = document.data()
//Do things
}
print("QUERY DONE")
let currentTime = Date()
let components = Calendar.current.dateComponents([.second], from: self.date, to: currentTime)
let seconds = components.second!
print("Elapsed time for Firestore query -> \(seconds)s")
// Benchmark result
}
}
well, from what I'm currently doing and research by using nexus 5X in emulator and real android phone Huawei P8,
Firestore and Cloud Storage are both give me a headache of slow response
when I do first document.get() and first storage.getDownloadUrl()
It give me more than 60 seconds response on each request. The slow response only happen in real android phone. Not in emulator. Another strange thing.
After the first encounter, the rest request is smooth.
Here is the simple code where I meet the slow response.
var dbuserref = dbFireStore.collection('user').where('email','==',email);
const querySnapshot = await dbuserref.get();
var url = await defaultStorage.ref(document.data().image_path).getDownloadURL();
I also found link that is researching the same.
https://reformatcode.com/code/android/firestore-document-get-performance