Develop Reference

Mechanics of Javascript Asynchronous execution

Can someone please explain the mechanics behind the asynchronous behavior of javascript, specifically when it comes to a .subscribe(), or provide a link that would help me understand what's really going on under the hood?
Everything appears to be single-threaded, then I run into situations that make it either multi-threaded, or seem multi-threaded, because I still can't shake this feeling I'm being fooled.
Everything seems to run perfectly synchronously, one instruction at a time, until I call .subscribe(). Then, suddenly, something happens - it returns immediately, and somehow the next outside function runs even though the subscription code has not yet. This feels asynchronous to me, how does the browser keep things straight - if not threads, can someone please explain the mechanics in play that is allowing this to occur?
I also notice something happens with debug stepping, I can't step in to the callback, I can only put a breakpoint within it. In what order should I really expect things to occur?

Your Javascript itself runs single threaded unless you're talking about WebWorkers (in the browser) or WorkerThreads (in nodejs).
But, many functions you call or operations you initiate in Javascript can run separate from that single threaded nature of your Javascript. For example, networking is all non-blocking and asynchronous. That means that when you make a call to fetch() in the browser of http.get() in nodejs, you are initiating an http request and then the mechanics of making that request and receiving the response are done independent of your Javascript. Here are some example steps:
You make a call to fetch() in the browser to make an http request to some other host.
The code behind fetch, sends out the http request and then immediately returns.
Your Javascript continues to execute (whatever you had on the lines of code immediately after the fetch() call will execute.
Sometime later, the response from the fetch() call arrives back on the network interface. Some code (internal to the Javascript environment) and independent from your single thread of Javascript sees that there's incoming data to read from the http response. When that code has gathered the http response, it inserts an event into the Javascript event queue.
When the single thread of Javascript has finished executing whatever event it was executing, it checks the event queue to see if there's anything else to do. At some point, it will find the event that signified the completion of the previous fetch() call and there will be a callback function associated with that event. It will call that callback which will resolve the promise associated with the fetch() call which will cause your own Javascript in the .then() handler for that promise to run and your Javascript will be presented the http response you were waiting for.
Because these non-blocking, asynchronous operations (such as networking, timers, disk I/O, etc...) all work in this same manner and can proceed independent of your Javascript execution, they can run in parallel with your Javascript execution and many of them can be in flight at the same time while your Javascript is doing other things. This may sometimes give the appearance of multiple threads and there likely are some native code threads participating in some of this, though there is still only one thread of your Javascript (as long as we're not talking about WebWorkers or WorkerThreads).
The completion of these asynchronous operations is then synchronized with the single thread of Javascript via the event queue. When they finish, they place an event in the event queue and when the single thread of Javascript finishes what it was last doing, it grabs the next event from the event queue and processes it (calling a Javascript callback).
I also notice something happens with debug stepping, I can't step in to the callback, I can only put a breakpoint within it. In what order should I really expect things to occur?
This is because the callback isn't called synchronously. When you step over the function that you passed a callback to, it executes that function and it returns. Your callback is not yet called. So, the debugger did exactly what you asked it to do. It executed that function and that function returned. The callback has not yet executed. Your are correct that if you want to see the asynchronous callback execute, you need to place a breakpoint in it so when it does get called some time in the future, the debugger will then stop there and let you step further. This is a complication of debugging asynchronous things, but you get used to it after awhile.
Everything appears to be single-threaded, then I run into situations that make it either multi-threaded, or seem multi-threaded, because I still can't shake this feeling I'm being fooled.
This is because there can be many asynchronous operations in flight at the same time as they aren't running your Javascript while they are doing their thing. So, there's still only one thread of Javascript, even though there are multiple asynchronous operations in flight. Here's a code snippet:
console.log("starting...");
fetch(host1).then(result1 => { console.log(result1)});
fetch(host2).then(result2 => { console.log(result2)});
fetch(host3).then(result3 => { console.log(result3)});
fetch(host4).then(result4 => { console.log(result4)});
console.log("everything running...");
If you run this, you will see something like this in the console:
starting...
everything running...
result1
result2
result3
result4
Actually, the 4 results may be in any order relative to one another, but they will be after the first two lines of debug output. In this case, you started four separate fetch() operations and they are running independent of your Javascript. In fact, your Javascript is free to do other things while they run. When they finish and when your Javascript isn't doing anything else, your four .then() handlers will each get called at the appropriate time.
When the networking behind each fetch() operation finishes, it will insert an event in the Javascript event queue and if your Javascript isn't doing anything at the time, the insertion of that event will wake it up and cause it to process that completion event. If it was doing something at the time, then when it finishes that particular piece of Javascript and control returns back to the system, it will see there's an event waiting in the event queue and will process it, calling your Javascript callback associated with that event.

Search for articles or videos on the “JavaScript Event Loop”. You’ll find plenty of them. Probably go through a few of them and it will start to make sense (particularly the single-threaded aspect of it)
I found this one with a quick search, and it does a good, very high-level walkthrough.
The JavaScript Event Loop explained
The main impact of JavaScript single threading is that your code is never pre-emptively interrupted to execute code on another thread.
It may seem that some code is being skipped, but close attention to the syntax will show that what is being “skipped” is just a callback function or closure to be called at a time in the future (when it will be placed into the event queue and processed single-threadedly, itself).
Careful, it can be tricky to follow nested callbacks.
Probably also look into Promises and async /await (‘just syntactic convenience on the event loop, but can really help code readability)

Why do callbacks functions allow us to do things asynchronously in Javascript?

I've read that callbacks make JavaScript asynchronously. But I'm not sure if I understood the explanation. This is what I get
Callback functions allow us to do things asynchronously, since they ensure
that the lines prior to the callback are completely finished before loading
the next line.
Is that true? Thanks

First off, it isn't the callback that enables anything. A given operation in node.js or even browser-based javascript either is or isn't asynchronous. It really has nothing to do with the callback, though a callback is typically used to communicate results of an asynchronous operation.
For example, Javascript's array.forEach() uses a callback, but it is not asynchronous. So, async operations are async because their underlying implementation is non-blocking. You start the operation by making a function call, the operation proceeds in the background and the rest of your code continues to run. Meanwhile, when the asynchronous operation completes, it usually then needs to tell your code that it is done and perhaps communicate some results. A callback function is the chosen mechanism for communicating the completion of the async operation.
I've read that callbacks make JavaScript asynchronously.
No, that is not really true. Callbacks can be used with synchronous operations too. Just because one uses a callback does not make anything asynchronous. The underlying native code implementation of an operation must be asynchronous (such as an Ajax call or other networking operation). Callbacks are used to communicate the results of asynchronous operations. They are also have many other non-asynchronous uses too. So, a callback is just one tool used in an asynchronous operation and a callback is a tool with many other uses too. You cannot say callback === asynchronous.
Callback functions allow us to do things asynchronously, since they
ensure that the lines prior to the callback are completely finished
before loading the next line.
It is hard to tell exactly what you mean by this, but it sounds wrong to me. When using asynchronous operations, code is typically not executed in the order laid out in the file. For example, if you did this:
console.log("async start");
callSomeAsyncOperation(function(result) {
console.log("async done");
});
console.log("I'm here now");
You would see this in the log:
async start
I'm here now
async done
Callbacks and Event Queues Explained
It may also be useful to understand how an asynchronous operation works. Javascript works off an event queue. A given sequence of Javascript code runs to its completion. When that completes, the engine looks in the event queue to see if there are any more events to process. If so, the first event in the queue is pulled out and a callback that was registered for that event is called. This starts a new sequence of Javascript code running. That code continues to run until it finishes. When it finishes, the engine checks for another event. If there is one, it is then processed by calling a callback associated with that event. When there are no more events to process, the engine goes to sleep waiting for the next event. When an event occurs (outside the main Javascript thead), it is then added to the queue and that process of adding it to the queue causes the JS engine to wake up and service that event.
When writing your Javascript, you will often register and event handler for an event. The way this works in Javascript is that you say what event you're interested in (which may also include specifying some other information like what object you're looking for events on) and then you pass it is a function reference. You are, in essence, telling the Javascript engine that you want it to call your function reference when this event occurs. This type of function reference is referred to as a "callback". It's just a normal function, but the context in which it is being used is called a "callback" because some other code will "call you back" at some time in the future by executing your function. You can then place appropriate code in that function reference (inside that callback function) that can respond to that event. Depending upon the type of event, you may only get called once or it may call you every time that event occurs.
You can read more about how this event queue and callbacks work in these references:
Run Arbitrary Code While Waiting For Callback in Node?
blocking code in non-blocking http server
Hidden threads in Javascript/Node that never execute user code: is it possible, and if so could it lead to an arcane possibility for a race condition?
How does JavaScript handle AJAX responses in the background? (written about the browser, but concept is the same)

First of all, let's understand what callback is (by definition):
In computer programming, a callback is a piece of executable code that
is passed as an argument to other code, which is expected to call back
(execute) the argument at some convenient time. The invocation may be
immediate as in a synchronous callback, or it might happen at later
time as in an asynchronous callback. In all cases, the intention is to
specify a function or subroutine as an entity that is, depending on
the language, more or less similar to a variable.
Now, talking about Javascript, not always a callback is asynchronous. For example:
function syncOperation(callback) {
callback();
}
syncOperation(function() {
console.log('a');
console.log('b');
});
console.log('c');
That callback is synchronous, since it doesn't make any asynchronous operation. If you open your console and run the code above, you'll see that it will log a, then b, then c.
Now, let's see an asynchronous example:
function asyncOperation(callback) {
setTimeout(callback, 0);
}
asyncOperation(function() {
console.log('a');
console.log('b');
});
console.log('c');
You'll see firstly c, then a, and b. That's because the setTimeout function runs asynchronously.
Some built-in Javascript functions are asynchronous, and they will run, and at some time, they will call the function you passed as parameter back. That's why if you do:
var a = 'text';
setTimeout(function() { a = 'new text'; }, 0);
console.log(a);
You'll see in your console text, because it will run before the variable has been changed.

Is Javascript synchronous(blocking) or Asynchronous(nonblocking) by default

I am trying to grasp on Javascript Asynchronous functions and callbacks.
I got stuck on the concept of callback functions, where I am reading on some places: they are use to have sequential execution of code (mostly in context of jquery e.g animate)and some places specially in the context of Nodejs; they are use to have a parallel execution Asynchronous and avoid blocking of code.
So can some expert in this topic please shed light on this and clear this fuzz in my mind (examples??).
so I could make my mind for the usage of callback function
or that is solely depends on the place of where you are calling/placing a callback function in your code? .
Thanks,
P.S: I am scared that this question would be close as subjective but still I could expect concrete answer for this (perhaps some examples)
Edit: actually this is the example from internet which makes me ambigous:
function do_a(){
// simulate a time consuming function
setTimeout( function(){
console.log( '`do_a`: this takes longer than `do_b`' );
}, 1000 );
}
function do_b(){
console.log( '`do_b`: this is supposed to come out after `do_a` but it comes out before `do_a`' );
}
do_a();
do_b();
Result
`do_b`: this is supposed to come out after `do_a` but it comes out before `do_a`
`do_a`: this takes longer than `do_b`
when JS is sequential then do_b should always come after do_a according to my understanding.

The core of JavaScript is largely synchronous, in that functions complete their task fully, before completing. Prior to the advent of AJAX, it was really only setTimeout and setInterval that provided asynchronous behavior.
However, it's easy to forget that event handlers are, effectively async code. Attaching a handler does not invoke the handler code and that code isn't executed until some unknowable time in the future.
Then came AJAX, with its calls to the server. These calls could be configured to be synchronous, but developers generally preferred async calls and used callback methods to implement them.
Then, we saw the proliferation of JS libraries and toolkits. These strove to homogenize different browsers' implementations of things and built on the callback approach to async code. You also started to see a lot more synchronous callbacks for things like array iteration or CSS query result handling.
Now, we are seeing Deferreds and Promises in the mix. These are objects that represent the value of a long running operation and provide an API for handling that value when it arrives.
NodeJS leans towards an async approach to many things; that much is true. However this is more a design decision on their part, rather than any inherent async nature of JS.

Javascript is always a synchronous(blocking) single thread language but we can make Javascript act Asynchronous through programming.
Synchronous code:
console.log('a');
console.log('b');
Asynchronous code:
console.log('a');
setTimeout(function() {
console.log('b');
}, 1000);
setTimeout(function() {
console.log('c');
}, 1000);
setTimeout(function() {
console.log('d');
}, 1000);
console.log('e');
This outputs: a e b c d

In node long running processes use process.nextTick() to queue up the functions/callbacks. This is usually done in the API of node and unless your programming(outside the api) with something that is blocking or code that is long running then it doesn't really effect you much. The link below should explain it better then I can.
howtonode process.nextTick()
jQuery AJAX also takes callbacks and such as it its coded not to wait for server responses before moving on to the next block of code. It just rememebers the function to run when the server responds. This is based on XMLHTTPRequest object that the browsers expose. The XHR object will remember the function to call back when the response returns.
setTimeout(fn, 0) of javascript will run a function once the call stack is empty (next available free tick) which can be used to create async like features.setTimeout(fn, 0) question on stackoverflow
To summerise the async abilities of javascript is as much to do with the environments they are programmed in as javascript itself. You do not gain any magic by just using lots of function calls and callbacks unless your using some API/script.
Jquery Deferred Object Is another good link for async capabilities of jQuery. Googling might find you information on how jQuery Deferred works also for more insight.

In JavaScript the term "asynchronous" typically refers to code that gets executed when the call stack is empty and the engine picks a job from one of its job queues for execution.
Once code is being executed, it represents a synchronous sequence of execution, which continues until the call stack is empty again. This sequence of execution will not be interrupted by events in order to execute some other JavaScript code (when we discard Web Workers). In other words, a single JavaScript environment has no preemptive concurrency.
While synchronous execution is ongoing, events might be registered as jobs in some job queues, but the engine will not process those before first properly execution what is on the call stack. Only when the call stack is empty will the engine take notice of the job queues, pick one according to priority, and execute it (and that is called asynchronous).
Callbacks
Callbacks can be synchronous or asynchronous -- this really depends on how they are called.
For instance, here the callback is executed synchronously:
new Promise(function (resolve) { /* .... */ });
And here the callback is executed asynchronously:
setTimeout(function () { /* ... */ });
It really depends on the function that takes the callback as argument; how it deals with eventually calling that callback.
Ways to get code to execute asynchronously
The core ECMAScript language does not offer a lot of ways to do this. The well-known ones are offered via other APIs, such as the Web API, which are not part of the core language (setTimeout, setInterval, requestAnimationFrame, fetch, queueMicrotask, addEventListener, ...).
Core ECMAScript offers Promise.prototype.then and (depending on that) await. The callback passed to then is guaranteed to execute asynchronously. await will ensure that the next statements in the same function will be executed asynchronously: await makes the current function return, and this function's execution context will be restored and resumed by a job.
It also offers listening to when the garbage collector is about to garbage collect an object, with FinalizationRegistry.
Web Workers
Web Workers will execute in a separate execution environment, with their own call stack. Here preemptive concurrency is possible. When the term asynchronous is used in the JavaScript world, it typically does not refer to this kind of parallelism, although the communication with a Web Worker happens via asynchronous callback functions.

Javascript by default is "synchronous", it's the web APIs that handle "asynchronous" behaviour.
As for the setTimeout example,
console.log(...), in the global scope works straight away, while those inside functions wrapped inside setTimeout, wait inside the callback queue only to be pushed back on the call stack once ready. Thus they take time. Also, the time specified is not exact but the minimum time after which that piece of code can run anytime.
Thanks !

How JS determine when to call Callback function?

I am wondering how JavaScript callbacks work. I specifically can't understand how asynchronous XmlHttpRequest works. How can JS determine that server returned some value (in one thread), and when to call callback method? Is it build on timers?

A very similar question was answered here in more detail.
The basic answer is that the underlying networking is happening at the OS level where there can be threads or some type of notifications when incoming networking packets arrive. When the result has completed, an event is added to the javascript event queue. When that event gets to the top of the event queue and javascript is ready to act on it, the proper javascript ajax event will be triggered which starts the chain of javascript that results in calling your callback.
There may some timers involved for timeouts, but timers are not used to know when the ajax response has arrived. That's based on the OS level networking support that the browser uses.

You wouldn't use timers since you couldn't tell when they should trigger.
I'd imagine it is using some sort of stack/queue [list or table] to keep track of calls and popping the response function off depending on the returned info telling you which request it was responding to.

Remember, javascript runs in an environment. That environment is not single threaded. When an xhr returns the browser (the environment) notifies the javascript engine that the xhr returned, and it in turn will execute the callback. Also remember that even though the programming model of javascript for the user is single threaded, javascript itself does not have to be.

Basically..
Whenever there's no javascript executing at the moment, events triggering from settimeout and responses to XmlHttpRequest are checked. They are indeed added to an event queue.
The interesting effect of this, is that as long as javascript is currently executing, events like these will never be triggered.

The asynchronous nature of the XmlHttpRequest is provided at lower level than javascript (by the browser). The callback function is initiated by the browser signalling to the javascript engine that the request is complete.

How do you determine if an asynchronous function is completed in before executing another one in jQuery (on really slow computers)?

For example, let's say I use the .addClass() method. You can't append a callback to it (e.g. you can't do $('...').addClass('...').load('...') ).
In most cases, you can just call your functions back to back before using .load() and you're fine.
But what if I have a super slow computer where using .addClass() takes 5 seconds? How do I execute another asynchronous function only after .addClass() has finished, or for that matter, after any JavaScript asynchronous function is finished?
EDIT: I was wrong, .addClass is not asynchronous.
So, then, just to be sure:
$('foo').addClass('');
/*this code here will not execute UNLESS the previous line is absolutely finished?*/

addClass is not asynchronous, and there is no environment in which it will take any significant time to perform its job. jQuery's various operations that are asynchronous (ajax, effects, that kind of thing) accept callbacks as part of their API.
I think (without going back and re-reading the API) that if an API function doesn't have a callback, you can assume the operation is synchronous.
The converse is not true; jQuery has several API functions (each, for instance) that accept callbacks that aren't asynchronous, it's just that the callback is handy for other reasons.
In any case, any asynchronous operation should be clearly identified as such in the documentation.
Update: Addressing your updated question: Correct, the code after addClass will not execute until addClass is complete. Even if it took a second or two (which of course it doesn't, but let's pretend), addClass will plod through doing its thing, and return back to your code. The subsequent code won't run until it does, and no other code can run while that's happening (because JavaScript on web browsers is single-threaded unless you use web workers, and even then coordination between threads is explicit).
There are some operations where you have to give the browser a moment to process what you've told it to do. I'm trying to think of a specific example, something around adding form fields in IE. Definitely not simple things. In those rare situations, you can ensure you've given the browser an opportunity to do its thing by explicitly yielding to the browser with a setTimeout:
doSomethingTheBrowserHasToThinkAbout();
setTimeout(function() {
// Now you know the browser's had time to think about it
doSomethingElse();
}, 0); // <== Won't really be 0, most browsers clamp this to 5-10ms minimum
But you don't have to do that (and don't want to do that) except for certain specific operations. I'm sorry I'm not immediately remembering an example, but we're talking edge cases here.

addClass is not completing for me until i put subsequent ajax/rendering in a setTimeout. It must be asynchronous to some degree. I'm trying to add a class to the body (for loading gif) and it just adds the "class" attribute without a value. When the code after is put into a setTimeout, it completes. This is in Chrome and almost 4 years later...

addClass is not asynchronous. In jQuery the asynchronous methods allow you to specify a callback function which will be called when the method completes. If there's no method signature that accepts a callback then chances are that method is synchronous.