I have read a lot of article about how NodeJs works. But I still can not figure out exactly how the internal threads of Nodejs proceed IO operations.
In this answer https://stackoverflow.com/a/20346545/1813428 , he said there are 4 internal threads in the thread pool of NodeJs to process I/O operations . So what if I have 1000 request coming at the same time , every request want to do I/O operations like retrieve an enormous data from the database . NodeJs will deliver these request to those 4 worker threads respectively without blocking the main thread . So the maximum number of I/O operations that NodeJs can handle at the same time is 4 operations. Am I wrong?.
If I am right , where will the remaining requests will handle?. The main single thread is non blocking and keep driving the request to corresponding operators , so where will these requests go while all the workers thread is full of task? .
In the image below , all of the internal worker threads are full of task , assume all of them need to retrieve a lot of data from the database and the main single thread keep driving new requests to these workers, where will these requests go? Does it have a internal task queuse to store these requests?
The single, per-process thread pool provided by libuv creates 4 threads by default. The UV_THREADPOOL_SIZE environment variable can be used to alter the number of threads created when the node process starts, up to a maximum value of 1024 (as of libuv version 1.30.0).
When all of these threads are blocked, further requests to use them are queued. The API method to request a thread is called uv_queue_work.
This thread pool is used for any system calls that will result in blocking IO, which includes local file system operations. It can also be used to reduce the effect of CPU intensive operations, as #Andrey mentions.
Non-blocking IO, as supported by most networking operations, don't need to use the thread pool.
If the source code for the database driver you're using is available and you're able to find reference to uv_queue_work then it is probably using the thread pool.
The libuv thread pool documentation provides more technical details, if required.
In the image below , all of the internal worker threads are full of task , assume all of them need to retrieve a lot of data from the database and the main single thread keep driving new requests to these workers
This is not how node.js use those threads.
As per Node.js documentation, the threads are used like this:
All requests and responses are "handled" in the main thread. Your callbacks (and code after await) simply take turns to execute. The "loop" between the javascript interpreter and the "event loop" is usually just a while loop.
Apart from worker_threads that you yourself start there are only 4 things node.js use threads for: waiting for DNS response, disk I/O, the built-in crypto library and the built-in zip library. Worker_threads are the only places where node.js execute javascript outside the main thread. All other use of threads execute C/C++ code.
If you are want to know more then I've written several answers to related questions:
Node js architecture and performance
how node.js server is better than thread based server
node js - what happens to incoming events during callback excution
Does javascript process using an elastic racetrack algorithm
Is there any other way to implement a "listening" function without an infinite while loop?
no, main use case for thread pool is offloading CPU intensive operations. IO is performed in one thread - you don't need multiple threads if you are waiting external data in parallel, and event loop is exactly a technique to organise execution flow so that you wait for as much as possible in parallel
Example:
You need to send 100 emails with a question (y/n) and another one with number of answered "y". It takes about 30 second to write email and two hours on average for reply + 10 seconds to read response. You start by writing all 100 emails ( 50 minutes of time ), then you wait alert sound which wakes you up every time reply arrives, and as you receive answers you increase number of "y". in ~2 hours and 50 minutes you're done. This is example of async IO and event loop ( no thread pools )
Blocking example: send email, wait for answer, repeat. Takes 4 days (two if you can clone another you )
Async thread pool example: each response is in the language you don't know. You have 4 translator friends. You email text to them, and they email translated text back to you (Or, more accurate: you print text and put it into "needs translation" folder. Whenever translator available, text is pulled from folder)
Related
I'm trying to understand node.js single threaded architecture and the eventloop to make our application more efficient. So consider this scenario where I have to make several database calls for an http api call. I can do it using Promise.all() or using a separate await.
example:
Using async/await
await inserToTable1();
await insertToTable2();
await updateTable3();
Using Promise.all() I can do the same by
await Promise.all[inserToTable1(), insertToTable2(), updateTable3()]
Here for one API hit at a given time, Promise.all() will be quicker to return the response as it fires the DB calls in parallel. But, if I have 1000 API hits per second, will there be any difference? For this scenario, is Promise.all() better for the eventloop?
Update
Assume the following,
By 1000 API hits, I meant the overall traffic to the application. Consider there are 20-25 APIs. Out of these a few might do DB operations, a few might make a few http calls, etc. Also, at no point we will be reaching the DB pool max connections.
Thanks in advance!!
As usual when it comes to system design, the answer is: it depends.
There are a lot of factors that determines the performance of either. In general, awaiting a single Promise.all() waits for all requests in parallel.
Event Loop
The event loop uses exactly 0% CPU time to wait for a request. See my answer to this related question for an explanation of how exactly the event loop works: Performance of NodeJS with large amount of callbacks
So from the event loop point of view there is no real difference between requesting sequentially and requesting in parallel with a Promise.all(). So if this is the core of your question I guess the answer is there is no difference between the two.
However, processing the callbacks does take CPU time. Again, the time to complete executing all the callbacks are the same. So from the point of view of CPU performance again there is no difference between the two.
Making requests in parallel does reduce overall execution time however. Firstly if the service is multithreaded you are essentially using it's multithreadedness by making parallel requests. This is what makes node.js fast even though it's single threaded.
Even if the service you are requesting from isn't multithreaded and actually handle requests sequentially, or if the server you're requesting from is a single core CPU (rare these days but you can still rent single-core virtual machines) then parallel requests reduces networking overhead since your OS can send multiple requests in a single Ethernet frame thus amortizing the overhead of packet headers over several requests. This does have a diminishing return beyond around half a dozen parallel requests however.
One Thousand Requests
You've hypothesized making 1000 requests. Weather or not awaiting 1000 promises in parallel actually causes parallel requests depends on how the API works at the network level.
Connection pools.
Lots of database libraries implement connection pools. That is, the library will open some number of connections to the database, for example 5, and reuse the connections.
In some implementation, making 1000 requests via such a library will cause the low-level networking code of the library to batch them 5 requests at a time. This means that at most you can have 5 parallel requests (assuming a pool of 5 connections). In this case it is perfectly safe to make 1000 parallel requests.
Some implementations however have a growable connection pool. In such implementations making 1000 parallel requests will cause your software to open 1000 sockets to access the remote resource. In such cases how safe it is to make 1000 parallel requests will depend on weather the remote server allows this.
Connection limit.
Most databases such as Mysql and Postgresql allows the admin to configure a connection limit, for example 5, such that the database will reject more than the limited number of connections per IP address. If you use a library that does not automatically manage maximum connections to your database then your database will accept the first 5 requests and reject the remaining until another slot is available (it's possible that a connection is freed before node.js finishes opening the 1000th socket). In this case you cannot successfully make 1000 parallel requests - you need to manage how many parallel requests you make.
Some API services also limit the number of connections you can make in parallel. Google Maps for example limits you to 500 requests per second. Therefore awaiting 1000 parallel requests will cause 50% of your requests to fail and possibly cause your API key or IP address to be banned.
Networking limits.
There is a theoretical limit on the number of sockets your machine or a server can open. However this number is extremely high so it's not worth discussing here.
However, all OSes that is currently in existence limit the maximum number of open sockets. On Linux (eg Ubuntu & Android) and Unix (eg MacOSX and iOS) sockets are implemented as file descriptors. And there is a maximum number of file descriptors allocated per process.
For Linux this number usually defaults to 1024 files. Note that a process opens 3 file descriptors by default: stdin, stdout and stderr. That leaves 1021 file descriptors shared by files and sockets. So your 1000 request in parallel skirts very close to this number and may fail if two clients try to make 1000 parallel requests at the same time.
This number can be increased but it does have a hard limit. The current maximum number of file descriptors you can configure on Linux is 590432. However this extreme configuration only works properly on a single user system with no daemons (or other background programs) running.
What to do?
The first rule when writing networking code is try not to break the network. Be reasonable in the number of requests you make at any one time. You can batch your requests to the limit of what the service expects.
With async/await it's easy. You can do something like this:
let parallel_requests = 10;
while (one_thousand_requests.length > 0) {
let batch = [];
for (let i=0;i<parallel_requests;i++) {
let req = one_thousand_requests.pop();
if (req) {
batch.push(req());
}
}
await Promise.all(batch);
}
Generally the more requests you can make in parallel the better (shorter) overall process time will be. I guess this is what you wanted to hear. But you need to balance parallelism with the factors above. 5 is generally OK. 10 maybe. 100 will depend on the server responding to the requests. 1000 or more and the admin who installed the server will probably have to tune his OS.
await approach will suspend the function execution for every await call and execute them sequentially while Promise.all can execute things parallel (in async) and return success when all of them are successful.
So it's better to use Promise.all if your three (inserToTable1(), insertToTable2(), table3()) methods are independent.
The ability of javascript to execute other stuff while a heavy operations are happening by suspending is achieved through event loops and call stacks.
Event Loops
The decoupling of the caller from the response allows for the JavaScript runtime to do other things while waiting for your asynchronous operation to complete and their callbacks to fire.
JavaScript runtimes contain a message queue which stores a list of messages to be processed and their associated callback functions. These messages are queued in response to external events (such as a mouse being clicked or receiving the response to an HTTP request) given a callback function has been provided.
The Event Loop has one simple job — to monitor the Call Stack and the Callback Queue. If the Call Stack is empty, it will take the first event from the queue and will push it to the Call Stack, which effectively runs it.
I have multiple jobs (functions) that process data in my DB.
They should be run indefinitely and concurrently. I was wondering about the
best way to run them. Should I write a bash file that starts node somejob.js for each job or should I use node workers from a JavaScript file, or some other method altogether?
While I don't know the best way to do such a job, but I have recently worked on a similar problem.
Since this is a broad question, but for illustration, I will give you an example of a mailing service.
I was asked to make a mailing service which can be used by other services,
to queue non similar raw emails with custom templates until it is sent.
A program (or worker) which I named dispatcher runs indefinitely, and checks for queued email
in DB. Fetched atmost n queued email (our email service has some threshold).
and send them concurrently, wait for some seconds and do that again.
To run dispatcher indefinitely I have used async.forever.
To run concurrent jobs, I have used async.map.
It seems you can do this in JS itself instead of using bash file (for some cron stuff). You can find lot of other useful methods here async
I try to understand how node.js works and although I have read this article: When is the thread pool used? I am not sure what happens if all worker threads are busy and another async I/O operation is ready to be executed.
If I got this http://www.future-processing.pl/blog/on-problems-with-threads-in-node-js/ article right, the event loop is blocked until a worker thread is free to take care of an additional I/O operation. This would imply that if five users tried to access a webpage simultaneously (like their profile page which lets say requires a db-query), the 5th user will be blocked until the first db-query is done and this worker thread is free again?
I/O in general does not block the event loop (there are some exceptions like the crypto module currently and things like fs.*Sync() methods) and especially in the case of network I/O, the libuv thread pool is not used at all (only for things like dns (currently) and fs operations).
If your database driver is written in C++ as a node.js addon, there is a chance that it could either block the event loop (it's doing something synchronous) or it could be using the libuv thread pool. However if the database driver is only written in JavaScript, it typically uses some sort of network I/O, which as previously mentioned does not block anything and does not use the libuv thread pool.
So with your example, depending on how the database driver is implemented, all 5 users could be serviced at once. For example, MySQL at the protocol level only supports one outstanding query at a time per connection. So what most MySQL drivers for node.js will do is queue up additional queries until the current query has finished. However, it's entirely possible for a MySQL driver to internally maintain some sort of connection pool so that you have greater concurrency.
However, if each of the 5 requests were instead causing something to be from disk, then it's possible that the 5th request will have to wait until one of the other 4 fs requests have finished, due to the current default size of the libuv thread pool. This doesn't mean the event loop itself is blocked, as it can still service new incoming requests and other things, but the 5th client will just have to wait a bit longer.
Is there any kind of queue for operations which are carried out in worker threads?
Yes. libuv manages this queue. It is possible for all of the worker threads to be occupied, in which case new asynchronous requests that go to the worker pool will not make progress. Note that an "asynchronous FS request" must still be completed somewhere, and will block the worker thread on which it is being handled until it completes.
I'm sure it's a commonly asked question but didn't find a concrete answer.
I kind of understand the basic concept of NodeJS and it's asynchronous/non-blocking nature of processing I/O.
For argument sake, let's take a simple example of a HTTP server written in node that executes the unix command 'find /' and writes the result to the http response (therefore displaying the result of the command on the user's browser).
Let's assume that this takes 3 seconds.
Let's assume that there are two users 'A' and 'B' requesting through their browsers exactly at the same time.
As I understand the user's requests are queued in the event queue (Message A, Message B). The message also has a reference to it's associated callback to be executed once the processing is done.
Since, the event loop is single threaded and processes the events one by one,
In my above example, Will it take 6 seconds for the Callback of "User B" to get triggered? [3 for "User A"s event processing and 3 for it's own event processing]
This sounds like I'm missing something here?
The worst is if 100 users are requesting at the same millisecond? The 100th event owner is going to be the most unfortunate user and has to wait for eternity.
As I understand, there is only one event queue in the runtime, the above problem can applicable to any user in any part of the application. For example, a slow Database Query in web page X would slow down the a different user in web page Y?
Fundamentally, I see a problem in serial processing of events and serial execution of their associated callbacks.
Am I missing something here?
A properly written node.js server will use async I/O and communication for any networking, disk I/O, timers or communication with other processes. When written this way, multiple http requests can be worked on in parallel. Though the node.js code that processes any given request is only run one at a time, anytime one request is waiting for I/O (which is typically much of the time of a request), then other requests can run.
The end result is that all requests appear to progress at the same time (though in reality, the work on them is interwoven). The Javascript event queue is the mechanism for serializing the work among all the various requests. Whenever an async operation finishes it's work or wishes to notify the main JS thread of some event, it puts something in the event queue. When the current thread of JS execution finishes (even if it has its own async operations in progress), the JS engine looks in the event queue and then executes the next item in that queue (usually some form of a callback) and, in that way, the next queued operation proceeds.
In your specific example, when you fire up another process and then asynchronously wait for its result, the current thread of execution finishes and then the next item in the event queue gets to run. If that next item is another http request, then that request starts processing. When this second request, then hits some async point, it's thread of execution finishes and again the next item in the event queue runs. In this way, new http requests get started and asynchronous callbacks from async operations that have finished get to run. Things happen in roughly a FIFO (first-in, first-out) order for how they are put in the event queue. I say "roughly" because there are actually different types of events and not all are serialized equally, but for the purpose of this discussion that implementation detail can be ignored.
So, if three http requests arrive at the exact same time, then one will run until it hits an async point. Then, the next will run until it hits an async point. Then, the third will run until it hits an async point. Then, whichever request finishes its first async operation will get a callback from that async operation and it will run until it is done or hits another async point. And, so on...
Since much of what usually causes a web server to take much time to respond is usually some sort of I/O operation (disk or networking) which can all be programmed asynchronously in node.js, this whole process generally works quite well and its actually a lot more efficient with server resources than using a separate thread per request. The one time that it doesn't work very well is if there's a heavy compute-intensive or some long running, but not asynchronous operation that ties up the main node.js thread for long periods of time. Because the node.js system is a cooperative CPU sharing system, if you have a long running operation that ties up the main node.js thread, it will hog the system (there is no pre-emptive sharing at all with other operations like there could be with a mutli-threaded system). Hogging the system makes all other requests wait until the first one is done. The node.js answer to some CPU hogging computation would be to move that one operation to another process and communicate asynchronously with that other process from the node.js thread - thus preserving the async model for the single node.js thread.
For node.js database operations, the database will generally provide an async interface for node.js programming to use the database in an async fashion and then it is up to the implementation of the database interface to actually implement the interface in an async fashion. This will likely be done by communicating with some other process where the actual database logic is implemented (probably communicating via TCP). That actual database logic may use actual threads or not - that's an implementation detail that is up to the database itself. What is important to node.js is that the computation and database work is out of the node.js thread in some other process, perhaps even on another host so it does not block the node.js thread.
I'd like some opinions on the practical implications of moving processing that would traditionally be done on the server to be handled instead by the client in a node.js web app.
Example case study:
The user uploads a CSV file containing a years worth of their bank statement entries. We want to parse the file, categorise each entry and calculate cumulative values for each category so that we can store the newly categorised statement in a db and display spending analysis to the user.
The entries are categorised by matching strings in the descriptions. There are many categories and many entries and it takes a fair amount of time to process.
In our node.js server, we can happily free up the event loop whilst waiting for network responses and so on, but if there is any data crunching or similar processing, the server will be blocked from responding to requests, and this seems unavoidable.
Traditionally, the CSV file would be passed to the server, the server would process, save in db, and send back the output of the processing.
It seems to make sense in our single threaded node.js server that this processing is handled by the browser, and the output displayed and sent to server to be stored. Of course the client will have to wait while this is done, but their processing will not be preventing the server from responding to requests from other clients.
I'm interested to see if anyone has had experience build apps using this model.
So, the question is.. are there any issues in getting browsers rather than the server to handle, wherever possible, any processing that will block the event loop? Is this a good/sensible/viable approach to node.js application development?
I don't think trusting client processed data is a good idea.
Instead you should look into creating a work queue that a separate process listens on, separating the CPU intensive tasks from your node.js process handling HTTP requests.
My proposed data flow would be:
HTTP upload request
App server (save raw file somewhere the worker process can access)
Notification to 'csv' work queue
Worker processes uploaded csv file.
Although perfectly possible, simply shifting the processing to the client machine does not solve the basic problem.
Now the client's event loop is blocked, preventing the user from interacting with the browser. Browsers tend to detect this problem and stop execution of the page's script altogether. Something your users will certainly hate.
There is no way around either delegating or splitting up the work-load.
Using a second process (for example a 2nd node instance) for doing the number crunching server-side has the added benefit of allowing the operating system to use a 2nd CPU core. Ideally you run as many Node instances as you have CPU cores in the server and balance your work-load between them. Have a look at the diode module for some inspiration on how to implement multi-process communication in node.