Please, I need an expert advice on how to efficiently test a finished Node.js Application.
What I want to do is
1. I want to run the test to stimulate for example 100 users are all inserting data into a mongoDB
2. 100 users retriving records from the db
3. 100 users are maybe deleting from the database, I want to check how the system will perform in situations like that.
I read about loadtest in npm and it seems to me as a good candidate, but I don't know if I can use this to pass data(post/get) request into the database to actually, see how the system will respond to situation like 100 users all posting real data into the database.
I suppose ,loadtest helps with checking the response time of eg 20sec with maybe 40 concurrent request, which is my basic undertanding of the module, I don't know if it has other functionlities like what I am expecting to do.
Any advice or clue to go about this will be appreciated,because I wish to avoid re-engineering the wheel if possible
Thank you
You can use loadtest module.
But first you should define routes for crud processes and call
loadtest -c 10 --rps 100 http://example.com/api/collection
--rps meaning as request per second
You can get it from: https://github.com/alexfernandez/loadtest
You can use Apache JMeter to conduct load test against:
Web Application frontend - record and replay supported
Web Services
MongoDB directly
any combination of all above.
There's no reason you absolutely need to use a node.js utility to conduct load tests on node.js apps. In fact, I would suggest you use a language that supports multiple threads. With node, I'd worry about blocking the event loop, waiting on I/O and getting inaccurate results.
I recently tried vegeta and am very happy with it. You can use it as is without having to write any Golang code (although it's open source and you can modify it as you please). It supports URLs with headers and POSTs with payloads. It is written in Golang, which does support multithreading, and it even reports its latency. You can get reports in html, json and plain text right out of the box
Finally, vegeta scales well. It seems you'd like to issue POSTs, GETs and DELETEs. You can spin up an instance to do your GET loads, another one for POSTs and one for DELETES. Or you can POST a bunch of data, then DELETE the data on the same VM. When all of the VMs are done running the tests, you can look at the results separately or aggregate them.
Related
I'm making a list of tasks to learn how to use PouchDB / CouchDB, the application is quite simple, would have authentication and the user would create their tasks.
My question is regarding how to store each user's information in the database. Should I create a database for each user with their tasks? Or is there a way to put all of the tasks of all users into a database called "Tasks" and somehow filter the synchronization so that PouchDB does not synchronize the whole database (including other users' tasks) that is in the server?
(I have read the pouchdb documentation a few times and I have not been able to define this, if it is documented, please inform me where.)
You can use both approaches to fulfill your use case:
Database per user
A database per user, is the db-per-user pattern in CouchDB. CouchDB can handle the database creation/deletion each time a user is created/deleted in CouchDB. In this case each PouchDB client will replicate the complete user database.
You can enable it in the server config
This is a proper approach if the users data is isolated and you don't need to share information between users. In this case you can have some scalability issues if you need you sync many user databases with another one in CouchDB. See this post.
Single database for every user
You need to use the filtered-replication feature in CouchDB/PouchDB. This post explains how to use it.
With this approach you can replicate a subset of the CouchDB database in PouchDB
As you have a single database is easier to share info between users
But, this approach has some performance problems. The filtering process is very inefficient. As it has to process the whole dataset, including the deleted documents to determine the set of documents to be included in the replication. This filtering is done in a couchdb external process in the server which add more cost to the process.
If you need to use the filtering approach it is better to use a Mango Selector for this purpose as it is evaluated in the CouchDB main process and it could be indexed. See options.selector in the PouchDB replication filtering options.
Conclusion
Which is better? depends on your use case... In any case you should consider the scalability issues in both cases:
In the case of filtered replication, you will face some issues as the number of documents grow if you have to filter the complete dataset. This is reported to be 10x faster when using mango selectors.
In the case of db-per-user, you will have some issues if you need to consolidate the different user databases in a single one when the number of users grow.
Both pattern are valid. The only difference is that in order to use the filtered replication, you need to provide access to the main database.
Since it's in javascript, it's easy to get credentials and then access the main database. This would give users the ability to see everyone's data.
A more secure approach would be to use a database-per-user pattern. Each database will be protected by the user's credentials.
I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
I am developing a web application using NodeJS & SailsJS frameworks. Now I am going to develop searching functionality. There are around 5000 records from which I want to search on one attribute.
I know I can search it using mogodb query. What if I get all the records in javascript at frontend and search from it? What is good way to search? At backend using db query or at fronend using javascript searching?
If you search in the frontend then you have to load the entire dataset into the frontend and keep it synchronised for every query. This is not a good idea.
Use database queries - that is what they are designed for, and you only need to transfer the results.
It's all about your app and users expectations on it. You definitely shouldn't use client-side search if you have:
Short-living data which couldn't be cached (like list of users who are online).
Huge dataset which a) couldn't be cached or b) wouldn't be cached (most visitors woudn't use search). But the size limit depends on the app.
Complex computation intensive search (like full-text search).
In other cases it can work. And searching even millions of data records could run under 100 ms, what is faster than common network delay required to receive a response from server.
Advantages of client search:
fast: no network latency.
powerful queries: query can use all JS capabilities with engine optimization advantages.
Disadvantages:
load full dataset (critical on huge amounts of data).
require synchronization strategy: full reload, partial updates, CRDT, etc.
Do it in backend only using db query, which is good practice.It will reduce execution time.
Should not do this kind of check in client side as you have to send the whole database to client and loop through the records several times to fetch the desired records.
I'm trying to create a simple in-browser web app to display the contents on a given rethink table with some nice formatting. I'm having trouble finding a way to actually connect to rethink without having to use node.js. All I want to do is get the data out and then run it through some styling/layout stuff. Node + dependencies are overkill for a tiny browser-only app.
Unfortunately, you're going to need a server. It might be node.js or it might be another language, but you'll need a server.
RethinkDB is not Firebase. It can't be queried from your browser. If you absolutely need browser side querying and can't have a server, you should use Firbase.
If you want to use RethinkDB, you can just have a very thin server that just redirects your queries to RethinkDB. This can be done over HTTP or over WebSockets.
Why
Ultimately, the reason why you don't want to query your database from the browser is security. RethinkDB has no users or read only accounts. That means that if your database is accessible from your browsers, anyone can come and delete all your databases (including your system tables) with a simple query.
For example:
r.db('rethinkdb').tableList().forEach(function (tableName) {
return r.db('rethinkdb').tableDrop(tableName);
});
And now, all your database is gone :).
Keep in mind that this is something the RethinkDB team is aware of and working on.
https://github.com/rethinkdb/rethinkdb/issues/218
I have a CQRS application with eventual consistency between the event store and the read model. In it I have a list of items and under the list a "Create new" button. When a user successfully creates a new item he is directed back to the list but since the read model has not been updated yet (eventual consistency) the item is missing in the list.
I want to fake the entry in the list until the read model has been updated.
How do I best do that and how do I remove it when the new item is present in the actual list? I expect delays of about 60 seconds for the read model to catch up.
I do realize that there are simpler ways to achieve this behavior without CQRS but the rest of the application really benefits from CQRS.
If it matters the application is a c# mvc4 application. I've been thinking of solutions involving HTML5 Web Storage but want to know what the best practice is for solving this kind of problem.
In this situation, you can present the result in the UI with total confidence. There is no difference in presenting this information directly and reading it from the read model.
Your domain objects are up to date with the UI and that's what really matters here. Moreover, if you valid correctly your AR state in every operation and you keep track of the concurrency with the AR's version then you're safe and your model will be protected against invalid operations.
At the end, what are the probability of your UI going out of sync? This can happen if you there are many users modifying the information you're displaying at the same time. This can be avoided by creating task based UI and following the rule 'one command/operation in the AR per request'.
The read model can be unsynced until the denormalizers do their job.
In the other hand, if the command will generate a conversation (long running operation) between a saga and AR's then you cannot do this and must warn the user about it.
It doesn't matter that's a asp.net mvc app. The only solution I see, besides just telling the user to wait a bit, is to have another but this time synchronous event handler that generate the same model (of course the actual model generation should be encapsulated in a service) and sends it to a memory cache.
Being everything in memory makes it very fast and being synchronous means it's automatically executed before the request ends. I'm assuming the command is executed syncronously too.
Then in your query repository you also consider results from cache, removing it if that result is already returned by the db.
Personally, for things that I know I want to be available to the user and where the read model generation is trivial, I would use only synchronous event handlers. The user doesn't mind waiting a few seconds when submitting something and if updating a read model takes a few seconds, you know you have a backend problem.
I see that eventual consistency is applicable to application only if application environment has multiple front-end servers hosting application and all these servers has own copy of read model. All servers uses same copy of event store.
When something is changed to event store, read model that is used to read result to user must be updated in sync with event store. Rest of servers and read models managed by them can be updated using eventual consistency.
This way result to user (list of items) can be read from local read model copy because it is already updated in sync. No need for special complex fake updates/rollbacks.
Only case when user can see incomplete list is that user hits F5 to refresh list after update change and load balancing directs user request to front-end server which read model is not yet updated (60 second delay), but this can be avoided so that load balancing does not change users server in middle of session.
So, if application has only one front-end server, eventual consistency is not very usable or it does not give any benefits without some special fake updates/rollbacks with read model...