Develop Reference

Angular: Increase Query Loading Time in Firebase Database

I have an angular app where i am querying my firebase database as below:
constructor() {
this.getData();
}
getData() {
this.projectSubscription$ = this.dataService.getAllProjects()
.pipe(
map((projects: any) =>
projects.map(sc=> ({ key: sc.key, ...sc.payload.val() }))
),
switchMap(appUsers => this.dataService.getAllAppUsers()
.pipe(
map((admins: any) =>
appUsers.map(proj =>{
const match: any = admins.find(admin => admin.key === proj.admin);
return {...proj, imgArr: this.mapObjectToArray(proj.images), adminUser: match.payload.val()}
})
)
)
)
).subscribe(res => {
this.loadingState = false;
this.projects = res.reverse();
});
}
mapObjectToArray = (obj: any) => {
const mappedDatas = [];
for (const key in obj) {
if (Object.prototype.hasOwnProperty.call(obj, key)) {
mappedDatas.push({ ...obj[key], id: key });
}
}
return mappedDatas;
};
And here is what I am querying inside dataService:
getAllProjects() {
return this.afDatabase.list('/projects/', ref=>ref.orderByChild('createdAt')).snapshotChanges();
}
getAllAppUsers() {
return this.afDatabase.list('/appUsers/', ref=>ref.orderByChild('name')).snapshotChanges();
}
The problem I am facing with this is I have 400 rows of data which I am trying to load and it is taking around 30seconds to load which is insanely high. Any idea how can I query this in a faster time?

We have no way to know whether the 30s is reasonable, as that depends on the amount of data loaded, the connection latency and bandwidth of the client, and more factors we can't know/control.
But one thing to keep in mind is that you're performing 400 queries to get the users of each individual app, which is likely not great for performance.
Things you could consider:
Pre-load all the users once, and then use that list for each project.
Duplicate the name of each user into each project, so that you don't need to join any data at all.
If you come from a background in relational databases the latter may be counterintuitive, but it is actually very common in NoSQL data modeling and is one of the reasons NoSQL databases scale so well.

I propose 3 solutions.
1. Pagination
Instead of returning all those documents on app load, limit them to just 10 and keep record of the last one. Then display the 10 (or any arbitrary base number)
Then make the UI in such a way that the user has to click next or when the user scrolls, you fetch the next set based on the previous last document's field's info.
I'm supposing you need to display all the fetched data in some table or list so having the UI paginate the data should make sense.
2. Loader
Show some loader UI on website load. Then when all the documents have fetched, you hide the loader and show the data as you want. You can use some custom stuff for loader, or choose from any of the abundant libraries out there, or use mat-progress-spinner from Angular Material
3. onCall Cloud Function
What if you try getting them through an onCall cloud function? It night be faster because it's just one request that the app will make and Firebase's Cloud Functions are very fast within Google's data centers.
Given that the user's network might be slow to iterate the documents but the cloud function will return all at once and that might give you what you want.
I guess you could go for this option only if you really really need to display all that data at once on website load.
... Note on cost
Fetching 400 or more documents every time a given website loads might be expensive. It'll be expensive if the website is visited very frequently by very many users. Firebase cost will increase as you are charged per document read too.
Check to see if you could optimise the data structure to avoid fetching this much.
This doesn't apply to you if this some admin dashboard or if fetching all users like this is done rarely making cost to not be high in that case.

Using ExpressJS, how do I extract different data from a SQLite table using the URL?

I want to access resources from a SQLite database table. I have one table for accounts, one for movies and one for reviews. The reviews-table is constructed like this:
CREATE TABLE IF NOT EXISTS reviews (
id INTEGER PRIMARY KEY,
authorId INTEGER,
movieId INTEGER,
review TEXT,
rating INTEGER,
FOREIGN KEY('authorId') REFERENCES 'accounts'('id'),
FOREIGN KEY('movieId') REFERENCES 'movies'('id')
)
What I want to do is that I want to be able to get all reviews, made by one author. But I also want to be able to get all reviews, made for the same movie. Below is my code for getting all reviews made by the same user/author. The code looks the same when getting the reviews based on the movie, with a few changes.
Both of them does what I want them to do. But of course only the one written first in the file are running.
const authorId = request.params.authorId;
const query = "SELECT * FROM reviews WHERE authorId = ?";
const values = [authorId];
db.all(query, values, function (error, review) {
if (error) {
response.status(500).end();
} else if (!review) {
response.status(404).end();
} else {
response.status(200).json(review);
}
});
});
The url will look the same no matter which of them I want running; http://localhost:3000/reviews/3. How can I differentiate the url so that they know which one should run?
I have tried to experiment with query strings, but I'm not sure how that works, and after hours of searching for something that worked on my code, I gave up.
I have also been thinking about using something like
app.use(function (request, response, next) {
if (request.method == "GET" && request.uri == "/reviews/:authorId") {
response.send("Hello World");
} else {
next;
}
});
This didn't work, and it didn't work if I tried to remove ":authorId" from the url either. The page just keeps loading.
So how do I solve this?

The most dynamic would be a single route /reviews and use the query string with the params like ?author=123 or ?movie=123, they can be combined like ?author=123&movie=123. As you want to return JSON the code will be used via API, so the pretty path is usually not as important as when it is a web-url. To make the implementation effective, most people use a function where you can drop the query object in and get the where-clause, or use an ORM.
In express, when you have routers like '/reviews/:authorId' and then '/reviews/:movieId', then the second one will never be called, because the first one will always match. That is something to be careful about when organizing your express routes.

Dialogflow to firestore inline editor javascript: sorting and filtering on different parameters of collection

I have a database in Google cloud firestore with a collection of documents that each includes the following parameters:
country, headline, URL, date.
I need to return two separate things depending on the input in Dialogflow. The first thing is the headlines of the 3 latest news and the second thing is the headlines of the 3 latest news in France.
I am to do it in Dialogflows inline editor, which is javascript, but it doesn't seem entirely the same as javascript e.g. in Node.
I have already solved the first problem with the following code:
function TopNews_Global(agent) {
agent.add('Here are the latest Global news');
let documents = db.orderBy('Date', 'desc').limit(3);
return documents.get()
.then(snapshot => {
snapshot.forEach(doc => {
agent.add('-' + doc.data().Headline + '.');
agent.add('Link:(' + doc.data().URL + ')');
});
return Promise.resolve('Read complete');
}).catch(() => {
agent.add('Error reading entry from the Firestore database.');
});
}
Where db is my collection in firebase.
The following code is my solution to the second thing I want to return, and this is where I am stuck. It is not possible to filter and order over two different fields as I do. But there MUST be a way around this - it's a pretty simple thing I wanna do.
https://firebase.google.com/docs/firestore/manage-data/export-import
function TopNews_France(agent) {
agent.add('Here are the latest French news');
let documents = db.orderBy('Date', 'desc').where('Country', '==', 'France').limit(3);
return documents.get()
.then(snapshot => {
snapshot.forEach(doc => {
agent.add('-' + doc.data().Headline + '.');
agent.add('Link:(' + doc.data().URL + ')');
});
return Promise.resolve('Read complete');
}).catch(() => {
agent.add('Error reading entry from the Firestore database.');
});
}

Assuming that as you state, db is a collection object, your query is a valid one for Firestore.
db.orderBy('Date', 'desc').where('Country', '==', 'France').limit(3);
Firestore does allow queries across multiple fields, assuming that the only orderBy or range queries are on the same field. Thus, your query is valid (one orderBy and one non-range where on different fields).
You didn't mention what error you are getting, but I suspect it is this (nominally visible in a Javascript console):
The query requires an index. You can create it here: (url)
That is because this is a query which requires a composite index -- it sorts on one field and filters on another.
Thus, to fix this, you just need to create the index. In an ideal case, you can just click on the URL in that error message and be brought to the exact point in the Firebase console to create the exact index you need. If for some reason you need to create it manually, you can also do that through the console. After it completes building, your query should work.
In this case, you will need to create an index for whatever collection db is pointing at, with the fields Date (in descending order) and Country (in either descending or ascending order -- the equality won't care), and with collection scope.

How to perform sql "LIKE" operation on firebase?

I am using firebase for data storage. The data structure is like this:
products:{
product1:{
name:"chocolate",
}
product2:{
name:"chochocho",
}
}
I want to perform an auto complete operation for this data, and normally i write the query like this:
"select name from PRODUCTS where productname LIKE '%" + keyword + "%'";
So, for my situation, for example, if user types "cho", i need to bring both "chocolate" and "chochocho" as result. I thought about bringing all data under "products" block, and then do the query at the client, but this may need a lot of memory for a big database. So, how can i perform sql LIKE operation?
Thanks

Update: With the release of Cloud Functions for Firebase, there's another elegant way to do this as well by linking Firebase to Algolia via Functions. The tradeoff here is that the Functions/Algolia is pretty much zero maintenance, but probably at increased cost over roll-your-own in Node.
There are no content searches in Firebase at present. Many of the more common search scenarios, such as searching by attribute will be baked into Firebase as the API continues to expand.
In the meantime, it's certainly possible to grow your own. However, searching is a vast topic (think creating a real-time data store vast), greatly underestimated, and a critical feature of your application--not one you want to ad hoc or even depend on someone like Firebase to provide on your behalf. So it's typically simpler to employ a scalable third party tool to handle indexing, searching, tag/pattern matching, fuzzy logic, weighted rankings, et al.
The Firebase blog features a blog post on indexing with ElasticSearch which outlines a straightforward approach to integrating a quick, but extremely powerful, search engine into your Firebase backend.
Essentially, it's done in two steps. Monitor the data and index it:
var Firebase = require('firebase');
var ElasticClient = require('elasticsearchclient')
// initialize our ElasticSearch API
var client = new ElasticClient({ host: 'localhost', port: 9200 });
// listen for changes to Firebase data
var fb = new Firebase('<INSTANCE>.firebaseio.com/widgets');
fb.on('child_added', createOrUpdateIndex);
fb.on('child_changed', createOrUpdateIndex);
fb.on('child_removed', removeIndex);
function createOrUpdateIndex(snap) {
client.index(this.index, this.type, snap.val(), snap.name())
.on('data', function(data) { console.log('indexed ', snap.name()); })
.on('error', function(err) { /* handle errors */ });
}
function removeIndex(snap) {
client.deleteDocument(this.index, this.type, snap.name(), function(error, data) {
if( error ) console.error('failed to delete', snap.name(), error);
else console.log('deleted', snap.name());
});
}
Query the index when you want to do a search:
<script src="elastic.min.js"></script>
<script src="elastic-jquery-client.min.js"></script>
<script>
ejs.client = ejs.jQueryClient('http://localhost:9200');
client.search({
index: 'firebase',
type: 'widget',
body: ejs.Request().query(ejs.MatchQuery('title', 'foo'))
}, function (error, response) {
// handle response
});
</script>
There's an example, and a third party lib to simplify integration, here.

I believe you can do :
admin
.database()
.ref('/vals')
.orderByChild('name')
.startAt('cho')
.endAt("cho\uf8ff")
.once('value')
.then(c => res.send(c.val()));
this will find vals whose name are starting with cho.
source

The elastic search solution basically binds to add set del and offers a get by wich you can accomplish text searches.
It then saves the contents in mongodb.
While I love and reccomand elastic search for the maturity of the project, the same can be done without another server, using only the firebase database.
That's what I mean:
(https://github.com/metaschema/oxyzen)
for the indexing part basically the function:
JSON stringifies a document.
removes all the property names and JSON to leave only the data
(regex).
removes all xml tags (therefore also html) and attributes (remember
old guidance, "data should not be in xml attributes") to leave only
the pure text if xml or html was present.
removes all special chars and substitute with space (regex)
substitutes all instances of multiple spaces with one space (regex)
splits to spaces and cycles:
for each word adds refs to the document in some index structure in
your db tha basically contains childs named with words with childs
named with an escaped version of "ref/inthedatabase/dockey"
then inserts the document as a normal firebase application would do
in the oxyzen implementation, subsequent updates of the document ACTUALLY reads the index and updates it, removing the words that don't match anymore, and adding the new ones.
subsequent searches of words can directly find documents in the words child. multiple words searches are implemented using hits

SQL"LIKE" operation on firebase is possible
let node = await db.ref('yourPath').orderByChild('yourKey').startAt('!').endAt('SUBSTRING\uf8ff').once('value');

This query work for me, it look like the below statement in MySQL
select * from StoreAds where University Like %ps%;
query = database.getReference().child("StoreAds").orderByChild("University").startAt("ps").endAt("\uf8ff");

mongodb getting user info for every document

I'm trying to display a forum/category. I need to get the latest posts. The problem is that I also need data on the user for each post as well as the latest reply.
db.post.find({
"inForum": forumID,
},
{
'sort': [['date', -1]]
},
function(err, cursor) {
cursor.count(function(err, count) {
cursor.skip(skip).limit(20).toArray(function(err, posts) {
var complete = _.after(nodes.length, function () {
res.send(posts)
});
// for every post get its author info and the latest post info
posts.forEach(function (post) {
var users = _.pluck(posts, 'user');
user.load(users, function (profiles) {
_.each(posts,
function(post, k) {
if (profiles[post.user]) post.fieldAvatar = profiles[post.user].fieldAvatar;
});
if (post.latestReply) {
post.load(post.latestReply.id, function (latestReply) {
if (latestReply) post.latestReply = latestReply
complete()
})
}
else {
complete()
}
})
})
});
})
})
This is what I'm doing and it seems really slow / really inelegant to me. Am I doing this correctly and is there any advice for speeding this up?
Thanks.

The best thing you should do here is to embed some information for the author of the posts (username & email or avatar) into those posts so that you don't make multiple queries to the database, one should suffice (sure you have some duplicate data, but the performance is optimal).
If you don't want to / can't do that you can also modify your second query to find all authors in [array_of_ids_of_the_posts]. That would reduce your [number_of_posts] queries into only one.

You could use some caching. For example you could save the users in an dictionary during the loop so you only have to fetch it on the first occurrence from mongodb.
Maybe you could create some kind of thread model where you save basic information about the containing posts, so you only have to go through the threads.
You could save the result of the function and delete it when a new post is added .. so won't go through all posts on every call.
You should not use a document storage like a sql database. Maybe it is better to generate the forum page directly when a post is created/edited and save the whole data in a document, so you only have to make one read call to mongo to show it.