Develop Reference

Writing custom, true Asynchronous functions in Javascript/Node

How do the NodeJS built in functions achieve their asynchronicity?
Am I able to write my own custom asynchronous functions that execute outside of the main thread? Or do I have to leverage the built in functions?

Just a side note, true asynchronous doesn't really mean anything. But we can assume you mean parallelism?.
Now depending on what your doing, you might find there is little to no benefit in using threads in node. Take for example: nodes file system, as long as you don't use the sync versions, it's going to automatically run multiple requests in parallel, because node is just going to pass these requests to worker threads.
It's the reason when people say Node is single threaded, it's actually incorrect, it's just the JS engine that is. You can even prove this by looking at the number of threads a nodeJs process takes using your process monitor of choice.
So then you might ask, so why do we have worker threads in node?. Well the V8 JS engine that node uses is pretty fast these days, so lets say you wanted to calculate PI to a million digits using JS, you could do this in the main thread without blocking. But it would be a shame not to use those extra CPU cores that modern PC's have and keep the main thread doing other things while PI is been calculated inside another thread.
So what about File IO in node, would this benefit been in a worker thread?.. Well this depends on what you do with the result of the file-io, if you was just reading and then writing blocks of data, then no there would be no benefit, but if say you was reading a file and then doing some heavy calculations on these files with Javascript (eg. some custom image compression etc), then again a worker thread would help.
So in a nutshell, worker threads are great when you need to use Javascript for some heavy calculations, using them for just simple IO may in fact slow things down, due to IPC overheads.
You don't mention in your question what your trying to run in parallel, so it's hard to say if doing so would be of benefit.

Javascript is mono-thread, if you want to create 'thread' you can use https://nodejs.org/api/worker_threads.html.
But you may have heard about async function and promises in javascript, async function return a promise by default and promise are NOT thread. You can create async function like this :
async function toto() {
return 0;
}
toto().then((d) => console.log(d));
console.log('hello');
Here you will display hello then 0
but remember that even the .then() will be executed after it's a promise so that not running in parallel, it will just be executed later.

How to return synchronous result to asynchronous function call

In Node 13, an external 3rd party library calls my code:
const myInput = myCode.run(somVar); // it doesn't use await
As my code then has to perform nested synchronous calls, how could I provide an appropriate return value to the 3rd party library that is not a promise, but the result of my promises? Ideally something like this:
const run = (inputVar) =>{
let result
(async ()=>{
result = await doSyncCalls(inputVar);
})(); // code should not proceed until after await
return result;
} // will return undefined, but ideally should return doSyncCalls result
deasync would be a good solution, except it has an unresolved bug that causes nested promises to not resolve.

Well, there is no way to synchronously return (from the callback) a value that you obtain asynchronously within the callback. Javascript does not currently support that. So, these are your options:
Put the asynchronous code in a child_process. Then run that child process with something like child_process.execFileSync(). That allows you to run the asynchronous code in the child process, but have the parent process block and wait for the result. This is a hack because it blocks the parent while waiting for the result. But, it can be made to work.
Before calling the library function, prefetch whatever value it is you will need when the callback is called. This allows you to use regular asynchronous programming before you call the library and then once you have the desired value in some sort of cache, you can then call the 3rd party library and when it calls you back and wants the value, you will have it synchronously in a cache somewhere. Obviously, this only works if you can figure what value or range of possible values will be required in the callback so you can pre-fetch them.
Modify the code in the library to add support for an asycnhronous callback.
Redesign code to work a different way that doesn't require this library or that can use some other library that doesn't have this problem.
Write or find some native code add-on (like deasync) that lets you block somehow during the asynchronous operation while still letting the event queue do what it needs to do to process the asynchronous completion. This would have to hook deep into the internals of the V8 engine. Or fix, deasync so it works for your case.
Write a blocking add-on in native code that could carry out your asynchronous operation in native code while blocking the V8 engine.
FYI, since everything but #2, #3 and #4 all block the main JS thread which is generally a bad thing to do in any server environment, and you've said that #2 was not practical, my preference would be #3 or #4. Since you don't share the actual code and actual detailed library and problem, we can't help you with any specifics.
Probably the most straightforward solution to implement is #1 (package the code up into a child process that you run synchronously), but it blocks the app while running which is a downside.

setTimeout to achieve asynchronicity in Node

What is the point of doing setTimeout(fx, 0) in node?
This is not asynchronous or even non-blocking, as the async function is really the setTimeout, not your fx, and after setTimeout has run asynchronously, you will end up running fx which will block your code anyway.
Doing the setTimeout with 0 to call a function fx will just wait until the stack is empty to run fx, but then while fx is running you won't be able to accept any requests, right?
So is setTimeout(fx, 0) just a way of telling node 'hey, run this whenever you can'? Is there any way to trully run async functions in Node?

If your question is:
Can node run functions in parallel at the same time?
Then the answer is yes, but you have to use a web worker.
The paradigm of asynchronosity in node is different from traditional definitions. The expectation is that you don't run too many ultra-long running functions in node. This way, effective asynchronosity is achieved.
Node is good for some things, not for others, just like any environment.
For a more detailed answer, refer here
As for setTimeout(...,0) calls; sometimes giving a break during a time consuming task to allow calls in the queue have their share of processing can be required. Dividing tasks in different ways can save you from these; but still, this is not really a hack, it is just the way event queues work. Also, using process.nextTick for this aim is much better since when you use setTimeout, calculation and checks of the time passed will be necessary while process.nextTick is simply what we really want: "Hey task, go back to end of the queue, you have used your share!"

If I understand your question correctly the answer would be the following:
As JavaScript is a single threaded language it is still able to deal with two things separately.
Using setTimeout(fx, 0) allows you to push the function or operation within the setTimeout Function in a "waiting qeue". As soon as the Stack of operations is completed the function gets put onto the execution stack and gets executed.
More detailed information about that can be found in this video

I know that callback function runs asynchronously, but why?

Which part of syntax provides the information that this function should run in other thread and be non-blocking?
Let's consider simple asynchronous I/O in node.js
var fs = require('fs');
var path = process.argv[2];
fs.readFile(path, 'utf8', function(err,data) {
var lines = data.split('\n');
console.log(lines.length-1);
});
What exactly makes the trick that it happens in background? Could anyone explain it precisely or paste a link to some good resource? Everywhere I looked there is plenty of info about what callback is, but nobody explains why it actually works like that.
This is not the specific question about node.js, it's about general concept of callback in each programming language.
EDIT:
Probably the example I provided is not best here. So let's do not consider this node.js code snippet. I'm asking generally - what makes the trick that program keeps executing when encounter callback function. What is in syntax
that makes callback concept a non-blocking one?
Thanks in advance!

There is nothing in the syntax that tells you your callback is executed asynchronously. Callbacks can be asynchronous, such as:
setTimeout(function(){
console.log("this is async");
}, 100);
or it can be synchronous, such as:
an_array.forEach(function(x){
console.log("this is sync");
});
So, how can you know if a function will invoke the callback synchronously or asynchronously? The only reliable way is to read the documentation.
You can also write a test to find out if documentation is not available:
var t = "this is async";
some_function(function(){
t = "this is sync";
});
console.log(t);
How asynchronous code work
Javascript, per se, doesn't have any feature to make functions asynchronous. If you want to write an asynchronous function you have two options:
Use another asynchronous function such as setTimeout or web workers to execute your logic.
Write it in C.
As for how the C coded functions (such as setTimeout) implement asynchronous execution? It all has to do with the event loop (or mostly).
The Event Loop
Inside the web browser there is this piece of code that is used for networking. Originally, the networking code could only download one thing: the HTML page itself. When Mosaic invented the <img> tag the networking code evolved to download multiple resources. Then Netscape implemented progressive rendering of images, they had to make the networking code asynchronous so that they can draw the page before all images are loaded and update each image progressively and individually. This is the origin of the event loop.
In the heart of the browser there is an event loop that evolved from asynchronous networking code. So it's not surprising that it uses an I/O primitive as its core: select() (or something similar such as poll, epoll etc. depending on OS).
The select() function in C allows you to wait for multiple I/O operations in a single thread without needing to spawn additional threads. select() looks something like:
select (max, readlist, writelist, errlist, timeout)
To have it wait for an I/O (from a socket or disk) you'd add the file descriptor to the readlist and it will return when there is data available on any of your I/O channels. Once it returns you can continue processing the data.
The javascript interpreter saves your callback and then calls the select() function. When select() returns the interpreter figures out which callback is associated with which I/O channel and then calls it.
Conveniently, select() also allows you to specify a timeout value. By carefully managing the timeout passed to select() you can cause callbacks to be called at some time in the future. This is how setTimeout and setInterval are implemented. The interpreter keeps a list of all timeouts and calculates what it needs to pass as timeout to select(). Then when select() returns in addition to finding out if there are any callbacks that needs to be called due to an I/O operation the interpreter also checks for any expired timeouts that needs to be called.
So select() alone covers almost all the functionality necessary to implement asynchronous functions. But modern browsers also have web workers. In the case of web workers the browser spawns threads to execute javascript code asynchronously. To communicate back to the main thread the workers must still interact with the event loop (the select() function).
Node.js also spawns threads when dealing with file/disk I/O. When the I/O operation completes it communicates back with the main event loop to cause the appropriate callbacks to execute.
Hopefully this answers your question. I've always wanted to write this answer but was to busy to do so previously. If you want to know more about non-blocking I/O programming in C I suggest you take a read this: http://www.gnu.org/software/libc/manual/html_node/Waiting-for-I_002fO.html
For more information see also:
Is nodejs representing Reactor or Proactor design pattern?
Performance of NodeJS with large amount of callbacks

First of all, if something is not Async, it means it's blocking. So the javascript runner stops on that line until that function is over (that's what a readFileSync would do).
As we all know, fs is a IO library, so that kind of things take time (tell the hardware to read some files is not something done right away), so it makes a lot of sense that anything that does not require only the CPU, it's async, because it takes time, and does not need to freeze the rest of the code for waiting another piece of hardware (while the CPU is idle).
I hope this solves your doubts.

A callback is not necessarily asynchronous. Execution depends entirely on how fs.readFile decides to treat the function parameter.
In JavaScript, you can execute a function asynchronously using for example setTimeout.
Discussion and resources:
How does node.js implement non-blocking I/O?
Concurrency model and Event Loop
Wikipedia:
There are two types of callbacks, differing in how they control data flow at runtime: blocking callbacks (also known as synchronous callbacks or just callbacks) and deferred callbacks (also known as asynchronous callbacks).

Node.js - Pausing script execution in another thread

.:Disclaimer:.
Please note that I am trying to achieve something with node.js which might go against its design principles. However as node.js goes against the norm of "javascript is for client side", I am also trying something different here. Please bear with me.
.:Background:.
I have a requirement where Java Scripts need to be narrative (read from beginning to end) for simplistic scripts for simplistic users. I will also offer an Async scripting ability for more advanced users.
I understand the event driven methodology of node.js, and have a solution working based on Async callbacks. Now I am trying to simplify this for our more basic scripting requirements.
.:The Question:.
Is there a way to run a script (using its own sandbox) where execution can be paused while a result is being delivered.
E.g.
var util = require('util'),
vm = require('vm'),
user = require('user');
vm.runInThisContext('user.WaitForProceed(); console.log('Waiting over');');
console.log('Finished!');
User is my own module that will do the waiting. Basically I want it to sit there and block on that line in the vm object until it has received a result back. After which it will continue onto the console.log line.
The output of this example unimportant as it is also achievable through callbacks. The narrative nature of the example script is more important for this solution.
J

There is no way to pause the execution in Node. At least they tell us so :)
There are some libraries which support an advanced flow control, like Invoke, maybe this could help, but I understands, that's not what you asked for :)
Also you could implement a busy-loop using nextTick()
Or you could implement a blocking call in C(++) and provide it as a library. I never did this.
One last way is to readFileSync() to a namedpipe which closes on a certian event.
As you already mentioned, it's against the language principes, therefor these solutions are all hacky.

If you really want to sleep the execution of the Node process, you can. However, as you state, it seems that you're fully aware of the implications.
Here is an NPM module to do this:
https://github.com/ErikDubbelboer/node-sleep
You can use it like so:
var sleep = require('sleep');
sleep.sleep(1); //sleep for 1 sec
sleep.usleep(2000000); //sleep for 2 sec
I partially write this for future visitors that arrive by Google search: You should not use the aforementioned technique. If you decide that you must, be aware that this will block your Node process and it won't be able to do any additional work until the sleep period is over. Additionally, you will violate every expectation of any user who is aware that it's Node.js programs, as Node.js programs are suppose to be non-blocking.
Good luck.

If you realy want to pause execution while waiting for result you may try to work with node-sync. It build on node-fibers. Your application needs to be executed with node-fibers script instead of node. node-sync adds sync method to Function.prototype that allows to run it syncroniously.
Also you need to wrap your call in the fiber (thread) so as not to block the event-loop.
var Sync = require('sync');
var someAsyncFunction = function(a, b, callback) {
setTimeout(function() {
var result = a + b;
callback(undefined, result);
}, 1000);
};
// Run in a fiber
Sync(function(){
// This code runs in separate fiber and does not block the event loop
// First argument is 'this' context
var result = someAsyncFunction.sync(null, 2, 3);
// Waiting one second
console.log(result); // Will output 5
});
// Event loop here
Please be careful with it. You need to understand that is not the node way.