I am developing a web application using NodeJS & SailsJS frameworks. Now I am going to develop searching functionality. There are around 5000 records from which I want to search on one attribute.
I know I can search it using mogodb query. What if I get all the records in javascript at frontend and search from it? What is good way to search? At backend using db query or at fronend using javascript searching?
If you search in the frontend then you have to load the entire dataset into the frontend and keep it synchronised for every query. This is not a good idea.
Use database queries - that is what they are designed for, and you only need to transfer the results.
It's all about your app and users expectations on it. You definitely shouldn't use client-side search if you have:
Short-living data which couldn't be cached (like list of users who are online).
Huge dataset which a) couldn't be cached or b) wouldn't be cached (most visitors woudn't use search). But the size limit depends on the app.
Complex computation intensive search (like full-text search).
In other cases it can work. And searching even millions of data records could run under 100 ms, what is faster than common network delay required to receive a response from server.
Advantages of client search:
fast: no network latency.
powerful queries: query can use all JS capabilities with engine optimization advantages.
Disadvantages:
load full dataset (critical on huge amounts of data).
require synchronization strategy: full reload, partial updates, CRDT, etc.
Do it in backend only using db query, which is good practice.It will reduce execution time.
Should not do this kind of check in client side as you have to send the whole database to client and loop through the records several times to fetch the desired records.
Related
I have a .csv that I want to use as a database and run SQL queries on it from the browser. (Ideally I want to upload the .csv, first. But It could also be stored). Thought this could be done with Django and a Postgres database. Are there simpler ways of accomplishing this?
Is WebSQL an option? Is there something else, I haven't thought of?
Ideally I would want to avoid SQL injections. I tried searching on stack overflow and found this (Display SQL query results in php), but it's not what I'm looking for.
Basically the desired functionality is: when one comes to webpage, they can run SQL queries on the data in the .csv. They type queries in an HTML form and submit the form and then the results would be shown on the same page with actual query.
Use an in-browser library to load the data from the csv file, for example Papa Parse, then equally using an in-browser library, but this time for SQLite, create an empty in-memory database, populate it with the loaded data from the csv file, and then query the database with the same library.
It appears that you are asking if you trigger/run SQL queries against some SQL database directly from a UI. While this is theoretically possible, in practice it is a very bad idea. The reason it is a bad idea is that to do so you would have to open one or more database ports to the outside. This in turn would expose the database to DOS (denial of service) and other types of malicious attacks.
The proper way to proceed would be to place your database behind the backend of your web application. Then, expose one or more endpoints in your backend which in turn talk privately to the database. Finally, allow your UI to hit the backend endpoints to run whatever SQL logic you want.
I'm making a list of tasks to learn how to use PouchDB / CouchDB, the application is quite simple, would have authentication and the user would create their tasks.
My question is regarding how to store each user's information in the database. Should I create a database for each user with their tasks? Or is there a way to put all of the tasks of all users into a database called "Tasks" and somehow filter the synchronization so that PouchDB does not synchronize the whole database (including other users' tasks) that is in the server?
(I have read the pouchdb documentation a few times and I have not been able to define this, if it is documented, please inform me where.)
You can use both approaches to fulfill your use case:
Database per user
A database per user, is the db-per-user pattern in CouchDB. CouchDB can handle the database creation/deletion each time a user is created/deleted in CouchDB. In this case each PouchDB client will replicate the complete user database.
You can enable it in the server config
This is a proper approach if the users data is isolated and you don't need to share information between users. In this case you can have some scalability issues if you need you sync many user databases with another one in CouchDB. See this post.
Single database for every user
You need to use the filtered-replication feature in CouchDB/PouchDB. This post explains how to use it.
With this approach you can replicate a subset of the CouchDB database in PouchDB
As you have a single database is easier to share info between users
But, this approach has some performance problems. The filtering process is very inefficient. As it has to process the whole dataset, including the deleted documents to determine the set of documents to be included in the replication. This filtering is done in a couchdb external process in the server which add more cost to the process.
If you need to use the filtering approach it is better to use a Mango Selector for this purpose as it is evaluated in the CouchDB main process and it could be indexed. See options.selector in the PouchDB replication filtering options.
Conclusion
Which is better? depends on your use case... In any case you should consider the scalability issues in both cases:
In the case of filtered replication, you will face some issues as the number of documents grow if you have to filter the complete dataset. This is reported to be 10x faster when using mango selectors.
In the case of db-per-user, you will have some issues if you need to consolidate the different user databases in a single one when the number of users grow.
Both pattern are valid. The only difference is that in order to use the filtered replication, you need to provide access to the main database.
Since it's in javascript, it's easy to get credentials and then access the main database. This would give users the ability to see everyone's data.
A more secure approach would be to use a database-per-user pattern. Each database will be protected by the user's credentials.
I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
Please, I need an expert advice on how to efficiently test a finished Node.js Application.
What I want to do is
1. I want to run the test to stimulate for example 100 users are all inserting data into a mongoDB
2. 100 users retriving records from the db
3. 100 users are maybe deleting from the database, I want to check how the system will perform in situations like that.
I read about loadtest in npm and it seems to me as a good candidate, but I don't know if I can use this to pass data(post/get) request into the database to actually, see how the system will respond to situation like 100 users all posting real data into the database.
I suppose ,loadtest helps with checking the response time of eg 20sec with maybe 40 concurrent request, which is my basic undertanding of the module, I don't know if it has other functionlities like what I am expecting to do.
Any advice or clue to go about this will be appreciated,because I wish to avoid re-engineering the wheel if possible
Thank you
You can use loadtest module.
But first you should define routes for crud processes and call
loadtest -c 10 --rps 100 http://example.com/api/collection
--rps meaning as request per second
You can get it from: https://github.com/alexfernandez/loadtest
You can use Apache JMeter to conduct load test against:
Web Application frontend - record and replay supported
Web Services
MongoDB directly
any combination of all above.
There's no reason you absolutely need to use a node.js utility to conduct load tests on node.js apps. In fact, I would suggest you use a language that supports multiple threads. With node, I'd worry about blocking the event loop, waiting on I/O and getting inaccurate results.
I recently tried vegeta and am very happy with it. You can use it as is without having to write any Golang code (although it's open source and you can modify it as you please). It supports URLs with headers and POSTs with payloads. It is written in Golang, which does support multithreading, and it even reports its latency. You can get reports in html, json and plain text right out of the box
Finally, vegeta scales well. It seems you'd like to issue POSTs, GETs and DELETEs. You can spin up an instance to do your GET loads, another one for POSTs and one for DELETES. Or you can POST a bunch of data, then DELETE the data on the same VM. When all of the VMs are done running the tests, you can look at the results separately or aggregate them.
I'm trying to create a simple in-browser web app to display the contents on a given rethink table with some nice formatting. I'm having trouble finding a way to actually connect to rethink without having to use node.js. All I want to do is get the data out and then run it through some styling/layout stuff. Node + dependencies are overkill for a tiny browser-only app.
Unfortunately, you're going to need a server. It might be node.js or it might be another language, but you'll need a server.
RethinkDB is not Firebase. It can't be queried from your browser. If you absolutely need browser side querying and can't have a server, you should use Firbase.
If you want to use RethinkDB, you can just have a very thin server that just redirects your queries to RethinkDB. This can be done over HTTP or over WebSockets.
Why
Ultimately, the reason why you don't want to query your database from the browser is security. RethinkDB has no users or read only accounts. That means that if your database is accessible from your browsers, anyone can come and delete all your databases (including your system tables) with a simple query.
For example:
r.db('rethinkdb').tableList().forEach(function (tableName) {
return r.db('rethinkdb').tableDrop(tableName);
});
And now, all your database is gone :).
Keep in mind that this is something the RethinkDB team is aware of and working on.
https://github.com/rethinkdb/rethinkdb/issues/218