I'm trying to understand node.js single threaded architecture and the eventloop to make our application more efficient. So consider this scenario where I have to make several database calls for an http api call. I can do it using Promise.all() or using a separate await.
example:
Using async/await
await inserToTable1();
await insertToTable2();
await updateTable3();
Using Promise.all() I can do the same by
await Promise.all[inserToTable1(), insertToTable2(), updateTable3()]
Here for one API hit at a given time, Promise.all() will be quicker to return the response as it fires the DB calls in parallel. But, if I have 1000 API hits per second, will there be any difference? For this scenario, is Promise.all() better for the eventloop?
Update
Assume the following,
By 1000 API hits, I meant the overall traffic to the application. Consider there are 20-25 APIs. Out of these a few might do DB operations, a few might make a few http calls, etc. Also, at no point we will be reaching the DB pool max connections.
Thanks in advance!!
As usual when it comes to system design, the answer is: it depends.
There are a lot of factors that determines the performance of either. In general, awaiting a single Promise.all() waits for all requests in parallel.
Event Loop
The event loop uses exactly 0% CPU time to wait for a request. See my answer to this related question for an explanation of how exactly the event loop works: Performance of NodeJS with large amount of callbacks
So from the event loop point of view there is no real difference between requesting sequentially and requesting in parallel with a Promise.all(). So if this is the core of your question I guess the answer is there is no difference between the two.
However, processing the callbacks does take CPU time. Again, the time to complete executing all the callbacks are the same. So from the point of view of CPU performance again there is no difference between the two.
Making requests in parallel does reduce overall execution time however. Firstly if the service is multithreaded you are essentially using it's multithreadedness by making parallel requests. This is what makes node.js fast even though it's single threaded.
Even if the service you are requesting from isn't multithreaded and actually handle requests sequentially, or if the server you're requesting from is a single core CPU (rare these days but you can still rent single-core virtual machines) then parallel requests reduces networking overhead since your OS can send multiple requests in a single Ethernet frame thus amortizing the overhead of packet headers over several requests. This does have a diminishing return beyond around half a dozen parallel requests however.
One Thousand Requests
You've hypothesized making 1000 requests. Weather or not awaiting 1000 promises in parallel actually causes parallel requests depends on how the API works at the network level.
Connection pools.
Lots of database libraries implement connection pools. That is, the library will open some number of connections to the database, for example 5, and reuse the connections.
In some implementation, making 1000 requests via such a library will cause the low-level networking code of the library to batch them 5 requests at a time. This means that at most you can have 5 parallel requests (assuming a pool of 5 connections). In this case it is perfectly safe to make 1000 parallel requests.
Some implementations however have a growable connection pool. In such implementations making 1000 parallel requests will cause your software to open 1000 sockets to access the remote resource. In such cases how safe it is to make 1000 parallel requests will depend on weather the remote server allows this.
Connection limit.
Most databases such as Mysql and Postgresql allows the admin to configure a connection limit, for example 5, such that the database will reject more than the limited number of connections per IP address. If you use a library that does not automatically manage maximum connections to your database then your database will accept the first 5 requests and reject the remaining until another slot is available (it's possible that a connection is freed before node.js finishes opening the 1000th socket). In this case you cannot successfully make 1000 parallel requests - you need to manage how many parallel requests you make.
Some API services also limit the number of connections you can make in parallel. Google Maps for example limits you to 500 requests per second. Therefore awaiting 1000 parallel requests will cause 50% of your requests to fail and possibly cause your API key or IP address to be banned.
Networking limits.
There is a theoretical limit on the number of sockets your machine or a server can open. However this number is extremely high so it's not worth discussing here.
However, all OSes that is currently in existence limit the maximum number of open sockets. On Linux (eg Ubuntu & Android) and Unix (eg MacOSX and iOS) sockets are implemented as file descriptors. And there is a maximum number of file descriptors allocated per process.
For Linux this number usually defaults to 1024 files. Note that a process opens 3 file descriptors by default: stdin, stdout and stderr. That leaves 1021 file descriptors shared by files and sockets. So your 1000 request in parallel skirts very close to this number and may fail if two clients try to make 1000 parallel requests at the same time.
This number can be increased but it does have a hard limit. The current maximum number of file descriptors you can configure on Linux is 590432. However this extreme configuration only works properly on a single user system with no daemons (or other background programs) running.
What to do?
The first rule when writing networking code is try not to break the network. Be reasonable in the number of requests you make at any one time. You can batch your requests to the limit of what the service expects.
With async/await it's easy. You can do something like this:
let parallel_requests = 10;
while (one_thousand_requests.length > 0) {
let batch = [];
for (let i=0;i<parallel_requests;i++) {
let req = one_thousand_requests.pop();
if (req) {
batch.push(req());
}
}
await Promise.all(batch);
}
Generally the more requests you can make in parallel the better (shorter) overall process time will be. I guess this is what you wanted to hear. But you need to balance parallelism with the factors above. 5 is generally OK. 10 maybe. 100 will depend on the server responding to the requests. 1000 or more and the admin who installed the server will probably have to tune his OS.
await approach will suspend the function execution for every await call and execute them sequentially while Promise.all can execute things parallel (in async) and return success when all of them are successful.
So it's better to use Promise.all if your three (inserToTable1(), insertToTable2(), table3()) methods are independent.
The ability of javascript to execute other stuff while a heavy operations are happening by suspending is achieved through event loops and call stacks.
Event Loops
The decoupling of the caller from the response allows for the JavaScript runtime to do other things while waiting for your asynchronous operation to complete and their callbacks to fire.
JavaScript runtimes contain a message queue which stores a list of messages to be processed and their associated callback functions. These messages are queued in response to external events (such as a mouse being clicked or receiving the response to an HTTP request) given a callback function has been provided.
The Event Loop has one simple job — to monitor the Call Stack and the Callback Queue. If the Call Stack is empty, it will take the first event from the queue and will push it to the Call Stack, which effectively runs it.
Related
I have been searching if there is a way to handle large scale HTTP requests with Express.js. I know Node.js has 'workers' starting at version 13 to do the multithreading, but in terms of handling multiple HTTP requests to a single endpoint, how would I go about doing that and scaling up?
For example, if 10,000 requests come at the same time, another thread would open up and deal with the other request to speed up the process. Does it do it automatically or do I need to configure something on it?
Thanks!
Express.js is designed with the event loop philosophy, expecting each request-response cycle to take negligible total CPU time (ignoring any waiting times due to I/O). This allows an Express.js server to efficiently respond to many requests on a single thread. To make this work, it is expected that any request that demands more than negligible CPU time will create a Worker thread, or spawn a process, to do the computation away from the event loop thread. This responsibility falls to the programmer.
If your Express.js app is still not able to keep up with the volume of the requests even though this philosophy is being followed, the solution is to scale up above its level using a load balancer, whether on a web server level (e.g. Apache httpd's mod_load_balancer), or on DNS level.
I am trying to create a API service where any client can pass me his HTTP request and time in seconds after which he would like his HTTP request to be executed.
I can think of two approaches here to make it happen:
Create a lambda with nodeJS and use setTimeout to wait
Create a step
function to wait for x number of seconds would call my lambda to
execute HTTP request
What I am trying to understand is what are the pros and cons of each. setTimeout looks easy to implement and with no obvious flaws. Is there any reason I should go for step functions?
In a flash, I can think of following, will update if I get more points in mind
Lambda can timeout. Whereas StepFunctions can wait for e.g. a year also.
Lambda will cost for the time it is waiting but StepFunction will not.
There is a limit on concurrent executions for lambda totalling to 1000 whereas a StepFunction can have maximum 1,000,000 open executions.
Link:
https://docs.aws.amazon.com/step-functions/latest/dg/limits.html#service-limits-state-machine-executions
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html#per-function-concurrency
I don't have any knowledge about aws step functions, so I can't say anything about that.
But using a setTimeout in nodejs with a possible really large delay, can be a real problem if you want to create a stable an maintainable API because fo the following reasons:
if you wan to roll out any updates for your application, you need to either kill waiting requests, or you need to keep the application online until the last request finished before you can shut down the old code.
if your application crashes, all currently waiting requests are lost.
if the system nodejs is running one needs to restart to install a critical security update then all pending requests are lost.
So setTimeout should not be used for this kind of takes. If you want to do that in node, you would need to store/mirror those requests in a persistent storage.
I'm sure it's a commonly asked question but didn't find a concrete answer.
I kind of understand the basic concept of NodeJS and it's asynchronous/non-blocking nature of processing I/O.
For argument sake, let's take a simple example of a HTTP server written in node that executes the unix command 'find /' and writes the result to the http response (therefore displaying the result of the command on the user's browser).
Let's assume that this takes 3 seconds.
Let's assume that there are two users 'A' and 'B' requesting through their browsers exactly at the same time.
As I understand the user's requests are queued in the event queue (Message A, Message B). The message also has a reference to it's associated callback to be executed once the processing is done.
Since, the event loop is single threaded and processes the events one by one,
In my above example, Will it take 6 seconds for the Callback of "User B" to get triggered? [3 for "User A"s event processing and 3 for it's own event processing]
This sounds like I'm missing something here?
The worst is if 100 users are requesting at the same millisecond? The 100th event owner is going to be the most unfortunate user and has to wait for eternity.
As I understand, there is only one event queue in the runtime, the above problem can applicable to any user in any part of the application. For example, a slow Database Query in web page X would slow down the a different user in web page Y?
Fundamentally, I see a problem in serial processing of events and serial execution of their associated callbacks.
Am I missing something here?
A properly written node.js server will use async I/O and communication for any networking, disk I/O, timers or communication with other processes. When written this way, multiple http requests can be worked on in parallel. Though the node.js code that processes any given request is only run one at a time, anytime one request is waiting for I/O (which is typically much of the time of a request), then other requests can run.
The end result is that all requests appear to progress at the same time (though in reality, the work on them is interwoven). The Javascript event queue is the mechanism for serializing the work among all the various requests. Whenever an async operation finishes it's work or wishes to notify the main JS thread of some event, it puts something in the event queue. When the current thread of JS execution finishes (even if it has its own async operations in progress), the JS engine looks in the event queue and then executes the next item in that queue (usually some form of a callback) and, in that way, the next queued operation proceeds.
In your specific example, when you fire up another process and then asynchronously wait for its result, the current thread of execution finishes and then the next item in the event queue gets to run. If that next item is another http request, then that request starts processing. When this second request, then hits some async point, it's thread of execution finishes and again the next item in the event queue runs. In this way, new http requests get started and asynchronous callbacks from async operations that have finished get to run. Things happen in roughly a FIFO (first-in, first-out) order for how they are put in the event queue. I say "roughly" because there are actually different types of events and not all are serialized equally, but for the purpose of this discussion that implementation detail can be ignored.
So, if three http requests arrive at the exact same time, then one will run until it hits an async point. Then, the next will run until it hits an async point. Then, the third will run until it hits an async point. Then, whichever request finishes its first async operation will get a callback from that async operation and it will run until it is done or hits another async point. And, so on...
Since much of what usually causes a web server to take much time to respond is usually some sort of I/O operation (disk or networking) which can all be programmed asynchronously in node.js, this whole process generally works quite well and its actually a lot more efficient with server resources than using a separate thread per request. The one time that it doesn't work very well is if there's a heavy compute-intensive or some long running, but not asynchronous operation that ties up the main node.js thread for long periods of time. Because the node.js system is a cooperative CPU sharing system, if you have a long running operation that ties up the main node.js thread, it will hog the system (there is no pre-emptive sharing at all with other operations like there could be with a mutli-threaded system). Hogging the system makes all other requests wait until the first one is done. The node.js answer to some CPU hogging computation would be to move that one operation to another process and communicate asynchronously with that other process from the node.js thread - thus preserving the async model for the single node.js thread.
For node.js database operations, the database will generally provide an async interface for node.js programming to use the database in an async fashion and then it is up to the implementation of the database interface to actually implement the interface in an async fashion. This will likely be done by communicating with some other process where the actual database logic is implemented (probably communicating via TCP). That actual database logic may use actual threads or not - that's an implementation detail that is up to the database itself. What is important to node.js is that the computation and database work is out of the node.js thread in some other process, perhaps even on another host so it does not block the node.js thread.
I have read a lot of article about how NodeJs works. But I still can not figure out exactly how the internal threads of Nodejs proceed IO operations.
In this answer https://stackoverflow.com/a/20346545/1813428 , he said there are 4 internal threads in the thread pool of NodeJs to process I/O operations . So what if I have 1000 request coming at the same time , every request want to do I/O operations like retrieve an enormous data from the database . NodeJs will deliver these request to those 4 worker threads respectively without blocking the main thread . So the maximum number of I/O operations that NodeJs can handle at the same time is 4 operations. Am I wrong?.
If I am right , where will the remaining requests will handle?. The main single thread is non blocking and keep driving the request to corresponding operators , so where will these requests go while all the workers thread is full of task? .
In the image below , all of the internal worker threads are full of task , assume all of them need to retrieve a lot of data from the database and the main single thread keep driving new requests to these workers, where will these requests go? Does it have a internal task queuse to store these requests?
The single, per-process thread pool provided by libuv creates 4 threads by default. The UV_THREADPOOL_SIZE environment variable can be used to alter the number of threads created when the node process starts, up to a maximum value of 1024 (as of libuv version 1.30.0).
When all of these threads are blocked, further requests to use them are queued. The API method to request a thread is called uv_queue_work.
This thread pool is used for any system calls that will result in blocking IO, which includes local file system operations. It can also be used to reduce the effect of CPU intensive operations, as #Andrey mentions.
Non-blocking IO, as supported by most networking operations, don't need to use the thread pool.
If the source code for the database driver you're using is available and you're able to find reference to uv_queue_work then it is probably using the thread pool.
The libuv thread pool documentation provides more technical details, if required.
In the image below , all of the internal worker threads are full of task , assume all of them need to retrieve a lot of data from the database and the main single thread keep driving new requests to these workers
This is not how node.js use those threads.
As per Node.js documentation, the threads are used like this:
All requests and responses are "handled" in the main thread. Your callbacks (and code after await) simply take turns to execute. The "loop" between the javascript interpreter and the "event loop" is usually just a while loop.
Apart from worker_threads that you yourself start there are only 4 things node.js use threads for: waiting for DNS response, disk I/O, the built-in crypto library and the built-in zip library. Worker_threads are the only places where node.js execute javascript outside the main thread. All other use of threads execute C/C++ code.
If you are want to know more then I've written several answers to related questions:
Node js architecture and performance
how node.js server is better than thread based server
node js - what happens to incoming events during callback excution
Does javascript process using an elastic racetrack algorithm
Is there any other way to implement a "listening" function without an infinite while loop?
no, main use case for thread pool is offloading CPU intensive operations. IO is performed in one thread - you don't need multiple threads if you are waiting external data in parallel, and event loop is exactly a technique to organise execution flow so that you wait for as much as possible in parallel
Example:
You need to send 100 emails with a question (y/n) and another one with number of answered "y". It takes about 30 second to write email and two hours on average for reply + 10 seconds to read response. You start by writing all 100 emails ( 50 minutes of time ), then you wait alert sound which wakes you up every time reply arrives, and as you receive answers you increase number of "y". in ~2 hours and 50 minutes you're done. This is example of async IO and event loop ( no thread pools )
Blocking example: send email, wait for answer, repeat. Takes 4 days (two if you can clone another you )
Async thread pool example: each response is in the language you don't know. You have 4 translator friends. You email text to them, and they email translated text back to you (Or, more accurate: you print text and put it into "needs translation" folder. Whenever translator available, text is pulled from folder)
I am making a node.js server that makes a lot of ajax requests to download json or html code, to different websites (not ddos/dos). For this language, I was wondering how many requests I can make at the same time without problems. Like is it ok to do like
for(i=0;i<1000;i+=1) {
callajax();
}
or do I have to do
function call() {
callajax(call);
}
this sorta calls the next ajax call, when the current one finishes.
If the top one is ok, how many ajax calls can I call at the same time before I have to wait till they return. I don't want problems with non-returning ajax requests.
Can anyone share some insight into this?
Also, lets say I have two servers running on the same wifi network. If both of them runs their node.js server making ajax requests, is that the same thing as doing it from 1 server?
This will depend upon several variables.
How many simultaneous sockets is node.js configured to allow (this is a configuration option within node.js)?
If all the requests are going to the same host, then how many simultaneous connections can that host handle?
If the requests take a little time for the receiving server to process, how fast can it process each request and will it be able to process the last request in less time than your timeout value is set to?
In general, it would be best to limit the number of simultaneous requests you send to the same host to something manageable like 5-10 because in most cases you won't actually get better throughput by sending more simultaneous requests, but you could exhaust some resources or hit timeouts with a lot of simultaneous requests.
The actual max value is dependent upon a lot of configuration choices and specifics related to the type of request and would have to be discovered via testing.