Develop Reference

Why is LIBUV needed in Node JS?

So, maybe this question is too noob and novice to be asked but I still have no clue why LIBUV got a place in Node JS Architecture? So here is my understanding of NodeJs architecture.
Node Js is built over V8
V8 is capable of running code written with EcmaScript standards.
V8 is written in C++.
So if you want to give any new functionality we can embed V8 in our C++ project and attach new code with new Embedded V8 in C++.
Now here is the doubt,
Since V8 supports EcmaScript Javascript that means it has the capability to run callbacks written with the standards of EcmaScript.
So we can add code for File System Access, HTTP server & DB access in C++ since there are libraries (header files) that gives that functionality since Java is written in C++ (correct me if I am wrong) and Java has the capability to do the same.
Now if we can add this functionality in C++ where does the place for Libuv come into the picture of NodeJs architecture.
Thanks in advance and
Happy Coding :)

Check the docs below -
https://nodejs.org/en/docs/meta/topics/dependencies/#libuv
Another important dependency is libuv, a C library that is used to
abstract non-blocking I/O operations to a consistent interface across
all supported platforms. It provides mechanisms to handle file system,
DNS, network, child processes, pipes, signal handling, polling and
streaming. It also includes a thread pool for offloading work for some
things that can't be done asynchronously at the operating system
level.
So to sum it up, V8 provides the functionalities related to running JS files, but to use system resources like Network, Files, etc., libuv is used. Also it provides a threading model for accessing the resources mentioned.

The libuv module has a responsibility that is relevant for some particular functions in the standard library. for SOME standard library function calls, the node C++ side and libuv decide to do expensive calculations outside of the event loop entirely.They make something called a thread pool that thread pool is a series of four threads that can be used for running computationally intensive tasks such as hashing functions.
By default libuv creates four threads in this thread pool. So that means that in addition to that thread used for the event loop there are four other threads that can be used to offload expensive calculations that need to occur inside of our application. Many of the functions include in the node standard library will automatically make use of this thread pool.
Now the presence of this thread pool is very significant. Well clearly Node.js is not truly single threaded
Libuv also gives node access to the operating system’s underlying file system such as networking. So just as the node standard library has some functions that make use of libuv thread pool it also has some functions that make use of code that is built into the underlying operating system through libuv.
Simple Http request
const https=require(“https”)
const start=Date.now()
https.request(“https://www.google.com”,res=>{
res.on(“data”,()=>{} )
res.on(“end”,()=>{console.log(Date.now()-start) }) }).end()
So in this case libuv sees that we are attempting to make an HTTP request. Neither libuv nor node has any code to handle all of this low level operations that are involved with a network request. Instead libuv delegates the request making to the underlying operating system. So it's actually our operating system that does the real HTTP request Libuv is used to issue the request and then it just waits on the operating system to emit a signal that some response has come back to the request. So because Libuv is delegating the work done to the operating system the operating system itself decides whether to make a new threat or not. Or just generally how to handle the entire process of making the request.

If anyone stumbles upon this and since it lacks a good answer to the OP's question, I will try to take on this.
TLDR;
Javascript language is not asynchronous
Javascript language is not multi-threaded
Callbacks themselves are not asynchronous, they are just mean to piggyback your code to an asynchronous operation.
Let's go over your doubts one by one.
1. Since V8 supports EcmaScript Javascript that means it has the capability to run callbacks written with the standards of EcmaScript.
Callbacks don't mean that the operation is asynchronous. A callback has got nothing to do with asynchronous execution. Callback is just a way to piggyback your function so that it executes after 'something asynchronous'.
// example of synchronous callback
function main(cb) {
console.log('main code of the function');
cb(); // callback invocation here
}
main(function () {
console.log('in callback');
});
Now an example of asynchronous callback
function getDataFromNetwork(url, cb) {
ajaxCall(url).then(cb);
}
getDataFromNetwork('http://some-endpoint', function (data) {
console.log(data);
});
This is an asynchronous call with a callback. Here getDataFromNetwork function is asynchronous not the callback. The point is that callbacks are just a mechanism of running a code after something. In an asynchronous operation, this becomes a necessity. How else we are going to do that? right?
No!
Nowadays we have async-await where you can run a code after the asynchronous function completes without using callbacks.
So you get that? Callbacks are not asynchronous. And that's not the point of having libuv.
2. So we can add code for File System Access, Http server & DB access in C++ since there are libraries (header files) that gives that functionality since Java is written in C++ (correct me if I am wrong) and Java has the capability to do the same.
Yes we can add lots of code for File System Access, Http server. But Why? We do already have a lot of libraries to do that. And yes its already written in C thats how NodeJS executes them.
Java already has that?
Right, but thats also a part of JVM rather than the core Java language, just like libuv is part of NodeJS runtime rather than the core Javascript language. In this regard both Java and NodeJS are similar. Its just that Java has its own C++ layer and NodeJS borrows libuv for that. BTW libuv was primarily built for NodeJS
3. Now if we can add these functionality in C++ where does the place for Libuv come in to the picture of NodeJs architecture.
I answered how these functionalities are already in C++, now lets see where libuv fits in this picture of the whole architecture.
Lets take an ajax/network call for example. Who do you think executes this?
NodeJS? No, It just gives instruction to its C++ API (Node API).
then is it Node API? No, It just gives instruction to the libuv
then is it libuv? Yes, it is
Same goes for timers, file access, child processes etc.
Also think when a lot of network calls, file access are fired within a NodeJS program, on what process it runs? who schedules them? who notify about the results and failure.
This is a lot to do. Java has its own thread pool to do that. Java has its own schedular to schedule the threads. and since Java provides threads to end user(programmers) as well. It makes sense to implement all that stuff using Java threads.
But NodeJS is single-threaded. Why it should have threads to execute I/O operations when it can borrow it from another library without making them a part of Javascript? After all, we aren't going to provide threads to the programmer so why bother?
Also Historically, Javascript was only meant to run in browsers. The only asynchronous operation browsers had access to were network requests, no file access, no DB. So we did not have a lot of bedrock already to build upon.

Why is org/arangodb/request synchronous?

Why is the new JavaScript module request synchronous? Is it supposed to be only used in a job queue?
Is there any way to make asynchronous http(s) requests in ArangoDB?

Full disclosure: I'm part of ArangoDB's development team and primarily work on Foxx and everything JavaScript. I'm also the guy who wrote the org/arangodb/request module.
ArangoDB is a different environment than Node.js, despite sharing many similarities (such as using the V8 JavaScript engine). Unlike Node.js (or the browser), ArangoDB uses a thread-based concurrency model and doesn't feature an Event Loop. However the threads are not exposed in JavaScript (and in fact in V8 every thread is fully isolated) so you normally don't even have to think of them.
In the browser and in Node.js functions like setTimeout work by delaying code execution via the Event Loop (until a certain amount of time has passed or until an external event has occurred).
In ArangoDB the code is always executed linearly. For example, incoming HTTP requests are passed to Foxx controllers in JavaScript and the response is sent as soon as the controller returns. Even if you could use setTimeout, the external resources you were working with (or even "internal" ones like the document collections and transactions) would likely be already gone by the time the delayed code could execute.
Because of this, the request function provided by the org/arangodb/request module is also entirely synchronous. Instead of returning a promise or taking a callback it directly returns the incoming response data. It is also decidedly not the same module as request on npm but rather a synchronous implementation based on that module's API to the extent that implementing its API is possible outside Node.js (e.g. not including streams and returning the remote response instead of taking callbacks).
If you come from a Node.js/io.js background, this may feel wrong because non-blocking IO can achieve higher throughput, but keep in mind that the design goals of ArangoDB and Node.js are very different. Node.js is built around streams and network connections. ArangoDB is built as a persistent data storage and has to deal with transactions and locks instead.
It is probably not the best idea to access external APIs directly from your Foxx controllers if you have a high likelihood of serious network latency or if the external API's response is not essential to the client response. This is what the Foxx queues are for. Transactional e-mails are a prime example for this.
While Foxx is very versatile, its primary focus is to allow you to move most of your application (especially logic that benefits from running closer to the data) directly into the database. For small to medium scale projects that, you can probably get away with doing external API calls in-bounds. But if your application is primarily concerned with talking to other services over the network, running that code in a database is probably not the optimal solution.
Luckily ArangoDB plays well with others, so it's easy to move your network-intensive code out of Foxx if you find that it becomes a performance bottleneck at higher loads. Foxx doesn't eliminate the need for application servers, but it can considerably reduce their complexity.
As a correction to Brian's answer: sadly promises won't let you write async code in a synchronous environment either. The Promises/A+ spec defines promises as having to be executed asynchronously. Where they aren't natively supported they still have to be built on top of existing functions like setTimeout or process.nextTick, neither of which ArangoDB implements.

JavaScript Execution Engine Unspecified?

I started to learn JavaScript recently. I've been working in the creation of applications with Node.js and Angular for a few months now.
One of the main aspects that was puzzling me was how it is possible to write asynchronous code in JavaScript in which I do not have to worry about things like thread synchronization, race conditions, etc.
So, I found a couple of interesting articles([1],[2]) that explained how I can be guaranteed that any piece of code that I write will always be executed by a single thread at the time. Bottom line, all my asynchronous code is simply scheduled to be executed at some point within an event loop. This sounds pretty much like the OS scheduler would work in a machine with a single processor, where every process is scheduled to use the processor for a limited amount of time, giving us the fake sense of parallelism. And the callbacks would be like interrupts.
The articles do not provide any particular references, so I thought that the best source on how the JavaScript execution engine work should certainly be the language specification, and so I got me the latest copy of EcmaScript 5.1.
To my great surprise I discovered that this execution behavior is not specified there. How come? This looks like a fundamental design choice done in all JavaScript execution engines in browsers and in node. Interestingly, I have not been able to find a place where this is specified for any specific engine. In fact, I have no clue how people find out this is the way things work to the point that is so categorically affirmed in books and blogs like the ones cited above.
So, I have a set of what I consider interesting questions. I would appreciate any answers providing insights, remarks or simply references pointing me in the right direction to understand the following:
Since the EcmaScript does not specify that the JavaScript execution engine should work with an event loop, how come may implementations of JavaScript seem to work this way, not only in browsers, but also in Node.js?
Does that mean I could implement a new JavaScript engine which is EcmaScript-compatible that in fact provides true multithreading capabilities with features like sychronization locks, conditions, etc?
Does this execution model using an event loop precludes me from taking advantage of multicores if I want to execute an intense CPU-bound task? I mean, I can surely divide the task in chunks (as explained in one of the articles), but this is still executed serially, not in parallel. So, how could a JavaScript engine take advantage of multicores to run my code?
Do you know of any other reputable sources where this behavior for any particular JavaScript engine implementation is formally specified?
How could the code be portable between libraries and engines if we cannot assume a few things about the execution environments?
It looks like too many questions, perhaps making this post too broad to be answered. If it gets closed I will try to ask them in different threads. But they all revolve around the fact that I want to understand better why JavScript and Node were designed with an event loop, and if this is specified somewhere (besides the browsers source code) that I could read and gain a deeper understanding of designs and decisions taken here and more importantly, to know exactly what is the source of information for people writing books and posts about it.

There are certain assumptions/weak references you make which lead you to this conclusion. Some of them are:
ECMAScript ECMA-XXX vs JavaScript vs JavaScriptEngine:
ECMAscript is a language specification, given by ECMA International. JavaScript is the most widely used web language that conforms to ECMAscript. For most part ECMAScript and JavaScript are synonymous (remember there is ActionScript). JavaScriptEngine is the implementation (interpreter) of JavaScript language code. It is a program in flesh and bones worked from ground-up unlike ECMAScript which only describes JavaScript's end goals and behaviour and JavaScript the code that uses the ECMAScript standard. You will find that an engine will do more than just conform to ECMAScript standard. They are at the ends of the specification/implementation spectrum. Example of this is ECMA-262/JavaScript/V8.
Event loop in browser vs Event loop in node.JS (JSEngine vs JSEnvironment):
This looks like a fundamental design choice done in all JavaScript execution engines in browsers and in node.
If you are using node.JS you may have used core libraries fs/net/http. These use event emitters which are hooked with the event loop provided with libuv. This is an extension to the JavaScriptEngine V8, forming node.JS platform. The event loop here involves objects like threads, sockets, files or abstract requests. But the event did not originate here. It was in first used in browsers. A browser implements a DOM which requires events for working with HTML elements. See the DOM specification and one implemented for Mozilla. They use events and require a event loop built on top of the JSEngine for browser use. Chrome adds DOM interface to the V8 engine it embeds.
Yes, you will feel this is common, because of the necessary DOM API in all browsers. Node developers brought forward this novel evented processing to server with the help of libuv which provides non-blocking, asynchronous abstraction for low-level operations required on server. As pointed already, not all server frameworks use event loop. Take example of Rhino which literally uses Java Classes for file,sockets (everything). If you actually use core Java IO, file operations are synchronous.
Now answering your questions in order:
explained in point 2 above
Yes, you can. Take a look at Rhino, there are many others. It may be possible in node but node is geared to be a high performance webserver and that might be against its zen.
Like I said event loop sits on JSEngine. It is a design pattern, that works best with IO. Multi-threaded design works better with high CPU-loads. If you want to use multiple cores in node.JS take a look at cluster module. For browsers you have webworkers
That varies from engine to engine. And how it is embedded. Browsers will have DOM and therefore event loop. Servers can vary. Check their specifications.
For browser it is possible to make it portable between them to a good extent. No promises for server.

Event loop doesn't have anything to do with javascript itself, it's a part of environment, not js engine. Since javascript was designed primarily to manipulate user interface, it was used heavily with event loop. But event loop is a part of UI implementation, not just in javascript, but in any language.
Yes, you can. But it will not be just engine, more like environment/platform. I think (but not quite sure) that you can use threads and related stuff in Rhino.
Yes, it does. In node this is usually solved by spawning more processes and in browser you can use WebWorkers.
I can't imagine a better source then specification. If something isn't there, it's just not a part of javascript (aka EcmaScript)

I have spent a good amount of time today trying to find the answers to my own questions, guided by some of the comments and other answers left for me here. I share my findings here in case others may consider them useful.
Event-Driven Design in JavaScript for Browsers
The decision to design JavaScript this way seems mostly related to the requirements of the DOM Event Architecture. In this specification we can find explicit requirements related to the implementation of events order and the event loop. The HTML5 specification goes even further, and define the terms explicitly and state specific requirements for the event loop implementation.
This must have certainly driven the design of the JavaScript execution engines in browsers. In this article Timing and Synchronization in JavaScript published by Opera we can clearly see that these requirements are the driving force behind the design of the Opera browser. Also in this another article from Mozilla, named Concurrency Model and Event Loop, we can find a clear explanation of the same event-driven design concepts as implemented by Mozilla (although the document seems outdated).
The use of an event loop to deal with this kind of applications is not new.
Handling user input is the most complex aspect of interactive
programming. An application may be sensitive to multiple input
devices, such as mouse and keyboard, and may multiplex these among
multiple input devices (e.g. different windows). Managing this
many-to-many mapping is usually in the province of of User Interface
Management Systems (UIMS) toolkits. Since most UIMS are implemented
in sequential languages they must resort to various techniques to
emulate the necessary concurrency. Typically this toolkits use an
event-loop that monitors the stream of input events and maps the events to call-back functions (or event handlers) provided by the
application programmer.
- Jonh H. Reppy - Concurrent Programming in ML
The use of event loops is present in other famous UI toolkits like Java Swing and Winforms. In Java all UI work must be done within the EventDispatchThread whearas in Winforms all UI work must be done within the thread that created the Window object. So, even when these languages support true multithreading they still require all UI code to be run in a single thread of execution.
Douglas Crockford explains the history of the event loop in JavaScript in this great video called Loopage (worth watching).
Event-Driven Design in JavaScript for Node
Now, the decision of using an event-driven design for Node.js is a bit less evident. Crockford gives a good explanation in the video shared above. But also, in the book, The Past, Present and Future of JavaScript, its author Axel Rauschmayer says:
2009—Node.js, JavaScript on the server. Node.js lets you implement
servers that perform well under load. To do so, it uses event-driven
non-blocking I/O and JavaScript (via V8). Node.js creator Ryan Dahl
mentions the following reasons for choosing JavaScript:
“Because it’s bare and does not come with I/O APIs.” [Node.js can thus introduce its own non-blocking APIs.]
“Web developers use it already.” [JavaScript is a widely known language, especially in a web context.]
“DOM API is event-based. Everyone is already used to running without threads and on an event loop.” [Web developers are not scared of
callbacks.]
So, it looks like Ryan Dahl, creator of Node.js, took into account the current design of JavaScript in browsers to decide which should be the implementation of his non-blocking, event-driven solution for Node.js.
The latest implementation of Node.js seems to use a library called libuv, designed for the implementation of this kind of applications. This library is a core part of the design of node. We can find the definition of event loops in its documentation. Evidently this plays an important role in the current implementation of Node.js.
About Other EcmaScript Compatible Engines
The EcmaScript specification does not provide requirements about how the concurrency needs to be handled in JavaScript. Therefore, this is decided by the implementation of the language. Other models of concurrency could easily be used without making the implementation incompatible with the standard.
The best two examples I found were the new Nashorn JavaScript Engine created for Oracle for the JDK8, and Rhino JavaScript Engine created by Mozilla. They both are EcmaScript compatible, and they both allow the creation of Java classes. Nothing in these engines requires the use of event-driven programming to deal with concurrency. These engines have access to the Java class library and since they run on top of the JVM they probably have access to other concurrency models offered in this platform.
Consider the following example take from JavaScript, The Definitive Guide to illustrate how to use Rhino JavaScript.
print(x); // Global print function prints to the console
version(170); // Tell Rhino we want JS 1.7 language features
load(filename,...); // Load and execute one or more files of JavaScript code
readFile(file); // Read a text file and return its contents as a string
readUrl(url); // Read the textual contents of a URL and return as a string
spawn(f); // Run f() or load and execute file f in a new thread
runCommand(cmd, // Run a system command with zero or more command-line args
[args...]);
quit() // Make Rhino exit
You can see a new thread can be spawned to run a JavaScript file in an independent thread of execution.
About Event-Driven Design, Multicores and True Concurrency
The best explanation I found on this subject comes from the book JavaScript The Definitive Guide. In this book, David Flanagan explains:
One of the fundamental features of client-side JavaScript is that it
is single-threaded: a browser will never run two event handlers at the
same time, and it will never trigger a timer while an event handler is
running, for example. Concurrent updates to application state or to
the document are simply not possible, and client-side programmers do
not need to think about, or even understand, concurrent programming. A
corollary is that client-side JavaScript functions must not run too
long: otherwise they will tie up the event loop and the web browser
will become unresponsive to user input. This is the reason that Ajax
APIs are always asynchronous and the reason that client-side
JavaScript cannot have a simple, synchronous load() or require()
function for loading JavaScript libraries.
The Web Workers specification very carefully relaxes the
single-threaded requirement for client-side JavaScript. The “workers”
it defines are effectively parallel threads of execution. Web workers
live in a self-contained execution environment, however, with no
access to the Window or Document object and can communicate with the
main thread only through asynchronous message passing. This means that
concurrent modifications of the DOM are still not possible, but it
also means that there is now a way to use synchronous APIs and write
long-running functions that do not stall the event loop and hang the
browser. Creating a new worker is not a heavyweight operation like
opening a new browser window, but workers are not flyweight threads
either, and it does not make sense to create new workers to perform
trivial operations. Complex web applications may find it useful to
create tens of workers, but it is unlikely that an application with
hundreds or thousands of workers would be practical.
What About Node.js True Parallelism?
Node.js is a fast-evolving technology, and perhaps that's why it is difficult to find opinions that are up-to-date. But basically, since it follows the same event-driven model as the browsers do, it is impossible to simply program a piece of code and expect it will take advantage of our multiple cores in the server. Since Node.js is implemented using non-blocking technologies, we could assume that every time we do some form of I/O (i.e. read a file, send something through a socket, write to a database, etc.), under the hood, the node engine could be spawning multiple threads and maybe taking advantage of the cores, but our code would still be run serially.
These days, it looks like node.js clustering is the solution for this problem. There are also some libraries like Node Worker that seem to implement the Web Worker concept in node. These libraries basically let us spawn new independent processes within node.js. (Although I have not experimented with this yet).
What About Portability?
It looks like there is no way that, in terms of the concurrency models, we can guarantee that all these libraries will play nice in all environments.
Although in the realm of browsers they all seem to work similarly, and since Node.js runs in an event loop, many things may still work, but there not guarantees that this should work in other engines. I guess this is probably one of the disadvantages of EcmaScript compared to other more extensive specifications like those defining the Java Virtual Machine or the CLR.
Perhaps something gets standardize later. In the future of EcmaScript, more concurrency ideas are being discussed today. See the EcmaSript Wiki: Strawman Proposals Communicating Event-Loop Concurrency and Distribution

Advantages of Web Workers and how they were achieved before?

I have read about Web Workers on http://www.whatwg.org/specs/web-apps/current-work/multipage/workers.html and I think I understand their purpose, but I am wondering if one of the main purposes of web workers, namely "allows long tasks to be executed without yielding to keep the page responsive." could be already achieved without web workers?
Like Registering Callbaks also allow long tasks to be executed, and only interrupt when they are ready, wtihout blocking, isn't that the same?

Callbacks allow you to manage concurrency. That is handling tasks. Not always in an easy way.
Not only do webworkers allow you to do concurrency in an easier way, they also let you have parallelism, that is tasks really running in parallel : they don't necessarily block each other and they don't block the UI.
In order to have a long javascript based running task in your browser before web worker, you had to micro-manage it to cut it in small parts in order to allow the UI to keep responsive. And of course having more than one long running task was more complex.

We know web browsers increased a lot over the past few years and it is primarily because of lot of work done on its engines, ex- V8 (Google), Chakra (Microsoft). The JavaScript so far runs in a single thread. The problem with single-threaded architecture is that it blocks the code and UI becomes unresponsive in case of running the complex script. There are various ways to solve this problem:
Offload work to the server, but to make apps faster fat client is preferred
Use asynchronous calls, but many complex ecosystem of async calls & promises could
lead into callback hell
Leverage multi-threading. Interesting!
Web Workers solve this issue by providing the capability of multi-threading in JavaScript.

Browser-side node.js or non-blocking javascript?

I am fascinated with non-blocking architectures. While I haven't used Node.js, I have a grasp of it conceptually. Also, I have been developing an event-driven web app so I have a fundamental understanding of event programming.
How do you write non-blocking javascript in the browser? I imagine this must differ in some ways from how Node does it. My app, for example, allows users to load huge amounts of data (serialized to JSON). This data is parsed to reconstitute the application state. This is a heavy operation that can cause the browser to lock for a time.
I believe using web workers is one way. (This seemed to be the obvious choice, however, Node accomplishes a non-blocking, event-driven architecture I believe without using Web Workers so I guess there must be another way.) I believe timers can also play a role. I read about TameJS and some other libraries that extend the javascript language. I am interested in javascript libraries that use native javascript without introducing a new language syntax.
Links to resources, libraries and practical examples are most appreciated.
EDIT:
Learned more and I realize that what I am talking about falls under the term "Futures".
jQuery implements this however, it always uses XHR to call a server where the server does the processing before returning the result and what I am after is doing the same thing without calling the server, where the client does the processing but in a non-blocking manner.
http://www.erichynds.com/jquery/using-deferreds-in-jquery/

Three are two methods of doing non-blocking work on the browser
Web Workers. WebWorkers create a new isolated thread for you to do computation in, however browser support tells you that IE<10 hates you.
not doing it, expensive work in a blocking fashion should not be done on the client, send an ajax request to a server to do this, then have the server return the results.
Poor man's threads:
There are a few hacks you can use:
emulate time splicing by using setTimeout. This basically means that after every "lump" of work you give the browser some room to be responsive by calling setTimeout(doMore, 10). This is basically writing your own process scheduler in a really poor non optimized manner, use web workers instead
creating a "new process" by creating a iframe with it's own html document. In this iframe you can do computation without blocking your own html document from being responsive.

What do you mean by non-blocking specifically?
The longest operations, Ajax-calls, are already non-blocking (async)
If you need some long-running function to run "somewhere" and then do something, you can call
setTimeout(function, 0)
and call the callback from the function.
And you can also read on promises and here as well