Develop Reference

Node Async practical benefit?

I am new to node async/sync concept. these day i am confused by the sequence of async function executions.
I am kinda hard to get the benefit of the async calls of single-process node:
Here is an example:
1.Sequence of async/await
Assume an async function:
async readPathInFile(filePath) {
//open filePath, read the content and return
return file path in filePath
}
We have several calls like:
var a = 'a.txt';
var b = await readPathInFile(a); // call_a
var c = await readPathInFile(b); // call_b
var d = await readPathInFile(c); // call_c
As we know the readPathInFile is defined as asynchronous, but the calls
call_a, call_b, call_c relies on the previous ones sequentially.
For these calls, the calls have no differences with synchronous calls.
So, What is the benefit of the asynchronous definition ?
Callbacks concept has the same concerns:
For example:
readPathInFile(filePath, callback) {
//open filePath, read the content and return
callback(file path in filePath)
}
var a = 'a.txt';
readPathInFile(a, (b)=>{
//call_a
readPathInFile(b, (c)=>{
//call_b
readPathInFile(c, (d)=>{
//
});
});
}); // call_a
call_z();
//only if the call_z does not rely on a,b,c, z finishes without caring about a,b,c. This is a only benefit.
Please correct me if my assumption is wrong.

As the term implies, asynchronous means that functions run out of order. In this respect your understanding is correct. What this implies however is not at all obvious.
To answer your question is a few words, the async/await keywords are syntax sugar, meaning they're not so much intrinsic qualities of the language, rather are shortcuts for saying something else. Specifically, the await keyword translates the expression into the equivalent Promise syntax.
function someFunction(): Promise<string> {
return new Promise(function(resolve) {
resolve('Hello World');
});
}
console.log('Step 1');
console.log(await someFunction());
console.log('Step 3');
is the same as saying
...
console.log('Step 1');
someFunction().then(function (res) {
console.log(res);
console.log('Step 3');
});
Note the 3rd console.log is inside the then callback.
When you redefine someFunction using the async syntax, you'd get the following:
async function someFunction() {
return 'Hello World';
}
The two are equivalent.
To answer your question
I'd like to reiterate that async/await are syntactical sugar. their effect can be replicated by existing functionality
Their purpose therefore is to combine the worlds of asynchronous and synchronous code. If you've ever seen an older, larger JS file, you'll quickly notice how many callbacks are used throughout the script. await is the mechanism to use asynchronous functions while avoiding the callback hell, but fundamentally, a function marked async will always be async regardless of how many awaits you put in front of it.
If you're struggling to understand how an await call works, just remember that the callback you pass to .then(...) can be called at any time. This is the essence of asynchronicity. await just adds a clean way to use it.
A visual example
I like analogies. Here's my favourite;
Picture yourself at a cafe. There are 3 people waiting in line. The barista takes your order and informs you that your coffee will be ready in 5 minutes. So this means you can sit down at a table and read a magazine or what ever, instead of waiting in line for the barista to make your coffee and accept payment etc.
The advantage of that is that the people behind you in the queue can get their own things done, instead of waiting in line for the coffees to be ready.
This is fundamentally how asynchronous code works. The barista can only do one thing at a time, but their time can be optimised by allowing everyone in the queue to do other things while they wait
Hope that helps
This article (linked below in the comments) has an awesome illustration: Here it is

What is the benefit of the asynchronous definition ?
To enable standard logical-flow code to run asynchronously. Contrast your examples:
var a = 'a.txt';
var b = await readPathInFile(a); // call_a
var c = await readPathInFile(b); // call_b
var d = await readPathInFile(c); // call_c
vs.
var a = 'a.txt';
readPathInFile(a, (b)=>{
//call_a
readPathInFile(b, (c)=>{
//call_b
readPathInFile(c, (d)=>{
//
});
});
}); // call_a
The latter is often called "callback hell." It's much harder to reason about even in that case, but as soon as you add branching and such, it gets extremely hard to reason about. In contrast, branching with async functions and await is just like branching in synchronous code.
Now, you could write readPathInFileSync (a synchronous version of readPathInFile), which would not be an async function and would let you do this:
var a = 'a.txt';
var b = readPathInFileSync(a); // call_a
var c = readPathInFileSync(b); // call_b
var d = readPathInFileSync(c); // call_c
So what's the difference between that and the async version at the beginning of the answer? The async version lets other things happen on the one main JavaScript thread while waiting for those I/O calls to complete. The synchronous version doesn't, it blocks the main thread, waiting for I/O to complete, preventing anything else from being handled.
Here's an example, using setTimeout to stand in for I/O:
const logElement = document.getElementById("log");
function log(msg) {
const entry = document.createElement("pre");
entry.textContent = Date.now() + ": " + msg;
logElement.appendChild(entry);
entry.scrollIntoView();
}
// Set up "other things" to happen
const things = [
"this", "that", "the other", "something else", "yet another thing", "yada yada"
];
let starting = false; // We need this flag so we can show
// the "Doing X work" messsage before
// we actually start doing it, without
// giving the false impression in
// rare cases that other things happened
// during sync work
otherThing();
function otherThing() {
if (!starting) {
log(things[Math.floor(Math.random() * things.length)]);
}
setTimeout(otherThing, 250);
}
// Stand-in for a synchronous task that takes n milliseconds
function doSomethingSync(n) {
const done = Date.now() + n;
while (Date.now() < done) {
// Busy-wait. NEVER DO THIS IN REAL CODE,
// THIS IS SIMULATING BLOCKING I/O.
}
}
// Stand-in for an asynchronous task that takes n milliseconds
function doSomething(n) {
return new Promise(resolve => {
setTimeout(resolve, n);
});
}
function doWorkSync() {
doSomethingSync(200);
doSomethingSync(150);
doSomethingSync(50);
doSomethingSync(400);
}
async function doWorkAsync() {
await doSomething(200);
await doSomething(150);
await doSomething(50);
await doSomething(400);
}
// Do the work synchronously
document.getElementById("run-sync").addEventListener("click", (e) => {
const btn = e.currentTarget;
btn.disabled = true;
log(">>>> Doing sync work");
starting = true;
setTimeout(() => {
starting = false;
doWorkSync();
log("<<<< Done with sync work");
btn.disabled = false;
}, 50);
});
// Do the work asynchronously
document.getElementById("run-async").addEventListener("click", (e) => {
const btn = e.currentTarget;
btn.disabled = true;
log(">>>> Doing async work");
starting = true;
setTimeout(() => {
starting = false;
doWorkAsync().then(() => {
log("<<<< Done with async work");
btn.disabled = false;
});
}, 50);
});
html {
font-family: sans-serif;
}
#log {
max-height: 5em;
min-height: 5em;
overflow: auto;
border: 1px solid grey;
}
#log * {
margin: 0;
padding: 0;
}
<div>Things happening:</div>
<div id="log"></div>
<input type="button" id="run-sync" value="Run Sync">
<input type="button" id="run-async" value="Run Async">
There are times you don't care about that blocking, which is what the xyzSync functions in the Node.js API are for. For instance, if you're writing a shell script and you just need to do things in order, it's perfectly reasonable to write synchronous code to do that.
In constrast, if you're running a Node.js-based server of any kind, it's crucial not to allow the one main JavaScript thread to be blocked on I/O, unable to handle any other requests that are coming in.
So, in summary:
It's fine to write synchronous I/O code if you don't care about the JavaScript thread doing other things (for instance, a shell script).
If you do care about blocking the JavaScript thread, writing code with async/await makes it much easier (vs. callback functions) to avoid doing that with simple, straight-forward code following the logic of your operation rather than its temporality.

Javascript Promises lets asynchronous functions return values like synchronous methods instead of immediately returning the final value the asynchronous function returns a promise to provide data in the future.
Actually Promises solves the callback hell issue.

ES6 generators: transforming callbacks to iterators

I'm experimenting with ES6 generators with the help of babel, and I have trouble understand how (or if!) I can effectively use callback based async function to output an iterator.
Let's say I want be able to write a function that takes a number of urls, asynchronously download them and returns them as soon as they are downloaded.
I would like to be able to write something like the following:
let urls = ['http://www.google.com', 'http://www.stackoverflow.com' ];
for ( {url, data} of downloadUrls(urls) ) {
console.log("Content of url", url, "is");
console.log(data);
}
How can I implement downloadUrls ?
Ideally I would like to be able to write the following:
var downloadUrls = function*(urls) {
for( let url of urls ) {
$.ajax(url).done( function(data) {
yield data;
});
}
};
This of course doesn't work, since ``yield'' is being invoked inside a callback and not directly inside the generator.
I can find many examples online of people trying the same, they are either not much transparent), require enabling browser/node flags, or use node-specific features/libraries.
The library closest to what I need seems to be task.js, but I'm unable to have even the simplest example run on current Chrome.
Is there a way to get the intended behaviour using standard and current features , (With current I mean usable with transpilers like babel, but without the need to enable extra flags on the browser) or do I have to wait for async/await ?

2019 update
Yielding via callbacks is actually pretty simple. Since you can only call yield directly from the generator function* where it appears (and not from callbacks), you need to yield a Promise instead, which will be resolved from the callback:
async function* fetchUrls(urls) {
for (const url of urls)
yield new Promise((resolve, reject) => {
fetch(url, { mode: 'no-cors' }).then(response => resolve(response.status));
});
}
(async function main() {
const urls = ['https://www.ietf.org/rfc/rfc2616.txt', 'https://www.w3.org/TR/PNG/iso_8859-1.txt'];
// for-await-of syntax
for await (const status of fetchUrls(urls))
console.log(status);
}());
If the example doesn't work in the browser (it my return 0 instead of 200 due to Cross Origin Read Blocking), try it live on repl.it.

Is there a way to get the intended behaviour using standard and current features
Yes, use promises and generators. Many promise libraries, and some standalone ones, feature the use of generator "coroutines".
But notice that you cannot mix iteration with asynchrony, you can use generators for either only. Your example seems to confuse them a bit - it looks like you expect that for ( {url, data} of downloadUrls(urls) ) { loop to work synchronously, which cannot work.
do I have to wait for async/await?
No, you don't have to wait, Babel already supports them!

Here is a clean way to use a generator / iterator to flatten asynchronous code which works for me in node.js:
var asyncProcedureGenerator1 = function*() {
var it = yield(0); //get a reference to the iterator
try {
var a = yield (asyncPart1.bind(it))(0); //call the function, set this = it
var b = yield (asyncPart2.bind(it))(a);
var c = yield (asyncPart3.bind(it))(b);
console.log("c = ", c);
}
catch(err)
{
console.log("Something went wrong: ", err);
}
};
var runAsyncGenerator = function(generator) {
var asyncProcedureIterator = generator(); //create an iterator
asyncProcedureIterator.next(); //start the iterator
asyncProcedureIterator.next(asyncProcedureIterator); //pass a reference of the iterator to itself
}
var asyncPart1 = function(param1) {
var it = this; //the iterator will be equal to this.
console.log("Starting asyncPart1 with param1 = ", param1);
setTimeout(function() {
console.log("Done with asyncPart1");
var returnValue = 42 + param1;
console.log("asyncPart1 returned ", returnValue);
it.next(returnValue); //when we are done, resume the iterator which has yielded to us.
},2000);
};
var asyncPart2 = function(param1) {
var it = this; //the iterator will be equal to this.
console.log("Starting asyncPart2 with param1 = ", param1);
setTimeout(function() {
console.log("Done with asyncPart2");
var returnValue = param1 / 2;
console.log("asyncPart2 returned ", returnValue);
//it.throw("Uh oh.");
it.next(returnValue);
},2000);
};
var asyncPart3 = function(param1) {
var it = this; //the iterator will be equal to this.
console.log("Starting asyncPart3 with param1 = ", param1);
setTimeout(function() {
console.log("Done with asyncPart3");
var returnValue = param1 / 3;
console.log("asyncPart3 returned ", returnValue);
it.next(returnValue);
},2000);
};
runAsyncGenerator(asyncProcedureGenerator1);
The idea is to run the generator, creator an iterator, and then pass a reference of that iterator to itself.
Then the iterator can call asynchronous functions (with yield) and pass them a reference to itself which allows those functions to either return success and resume the execution by calling iterator.next(result) or failure by calling iterator.throw(error).
I just came up with this pattern, so there may be some gotchas I haven't found yet, but it seems to work and allows very flat code with minimal additions.

Understanding code flow with yield/generators

I've read over several examples of code using JavaScript generators such as this one. The simplest generator-using block I can come up with is something like:
function read(path) {
return function (done) {
fs.readFile(path, "file", done);
}
}
co(function *() {
console.log( yield read("file") );
})();
This does indeed print out the contents of file, but my hangup is where done is called. Seemingly, yield is syntactic sugar for wrapping what it returns to in a callback and assigning the result value appropriately (and at least in the case of co, throwing the error argument to the callback). Is my understanding of the syntax correct?
What does done look like when yield is used?

Seemingly, yield is syntactic sugar for wrapping what it returns to in a callback and assigning the result value appropriately (and at least in the case of co, throwing the error argument to the callback)
No, yield is no syntactic sugar. It's the core syntax element of generators. When that generator is instantiated, you can run it (by calling .next() on it), and that will return the value that was returned or yielded. When the generator was yielded, you can continue it later by calling .next() again. The arguments to next will be the value that the yield expresion returns inside the generator.
Only in case of co, those async callback things (and other things) are handled "appropriately" for what you would consider natural in an async control flow library.
What does done look like when yield is used?
The thread function example from the article that you read gives you a good impression of this:
function thread(fn) {
var gen = fn();
function next(err, res) {
var ret = gen.next(res);
if (ret.done) return;
ret.value(next);
}
next();
}
In your code, yield does yield the value of the expression read("file") from the generator when it is ran. This becomes the ret.val, the result of gen.next(). To this, the next function is passed - a callback that will continue the generator with the result that was passed to it. In your generator code, it looks as if the yield expression returned this value.
An "unrolled" version of what happens could be written like this:
function fn*() {
console.log( yield function (done) {
fs.readFile("filepath", "file", done);
} );
}
var gen = fn();
var ret1 = gen.next();
var callasync = ret1.value;
callasync(function next(err, res) {
var ret2 = gen.next(res); // this now does log the value
ret2.done; // true now
});

I posted a detailed explanation of how generators work here.
In a simplified form, your code might look like this without co (untested):
function workAsync(fileName)
{
// async logic
var worker = (function* () {
function read(path) {
return function (done) {
fs.readFile(path, "file", done);
}
}
console.log(yield read(fileName));
})();
// driver
function nextStep(err, result) {
try {
var item = err?
worker.throw(err):
worker.next(result);
if (item.done)
return;
item.value(nextStep);
}
catch(ex) {
console.log(ex.message);
return;
}
}
// first step
nextStep();
}
workAsync("file");
The driver part of workAsync asynchronously iterates through the generator object, by calling nextStep().

How to wrap async function calls into a sync function in Node.js or Javascript?

Suppose you maintain a library that exposes a function getData. Your users call it to get actual data:
var output = getData();
Under the hood data is saved in a file so you implemented getData using Node.js built-in fs.readFileSync. It's obvious both getData and fs.readFileSync are sync functions. One day you were told to switch the underlying data source to a repo such as MongoDB which can only be accessed asynchronously. You were also told to avoid pissing off your users, getData API cannot be changed to return merely a promise or demand a callback parameter. How do you meet both requirements?
Asynchronous function using callback/promise is the DNA of JavasSript and Node.js. Any non-trivial JS app is probably permeated with this coding style. But this practice can easily lead to so called callback pyramid of doom. Even worse, if any code in any caller in the call chain depends on the result of the async function, those code has to be wrapped in callback function as well, imposing a coding style constraint on caller. From time to time I find the need to encapsulate an async function (often provided in a 3rd party library) into a sync function in order to avoid massive global re-factoring. Searching for a solution on this subject usually ended up with Node Fibers or npm packages derived from it. But Fibers just cannot solve the problem I am facing. Even the example provided by Fibers' author illustrated the deficiency:
...
Fiber(function() {
console.log('wait... ' + new Date);
sleep(1000);
console.log('ok... ' + new Date);
}).run();
console.log('back in main');
Actual output:
wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
back in main
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
If function Fiber really turns async function sleep into sync, the output should be:
wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
back in main
I have created another simple example in JSFiddle and looking for code to yield expected output. I'll accept a solution that only works in Node.js so you are free to require any npm package despite not working in JSFiddle.

deasync turns async function into sync, implemented with a blocking mechanism by calling Node.js event loop at JavaScript layer. As a result, deasync only blocks subsequent code from running without blocking entire thread, nor incuring busy wait. With this module, here is the answer to the jsFiddle challenge:
function AnticipatedSyncFunction(){
var ret;
setTimeout(function(){
ret = "hello";
},3000);
while(ret === undefined) {
require('deasync').runLoopOnce();
}
return ret;
}
var output = AnticipatedSyncFunction();
//expected: output=hello (after waiting for 3 sec)
console.log("output="+output);
//actual: output=hello (after waiting for 3 sec)
(disclaimer: I am the co-author of deasync. The module was created after posting this question and found no workable proposal.)

You've got to use promises:
const asyncOperation = () => {
return new Promise((resolve, reject) => {
setTimeout(()=>{resolve("hi")}, 3000)
})
}
const asyncFunction = async () => {
return await asyncOperation();
}
const topDog = () => {
asyncFunction().then((res) => {
console.log(res);
});
}
I like arrow function definitions more. But any string of the form "() => {...}" could also be written as "function () {...}"
So topDog is not async despite calling an async function.
EDIT: I realize a lot of the times you need to wrap an async function inside a sync function is inside a controller. For those situations, here's a party trick:
const getDemSweetDataz = (req, res) => {
(async () => {
try{
res.status(200).json(
await asyncOperation()
);
}
catch(e){
res.status(500).json(serviceResponse); //or whatever
}
})() //So we defined and immediately called this async function.
}
Utilizing this with callbacks, you can do a wrap that doesn't use promises:
const asyncOperation = () => {
return new Promise((resolve, reject) => {
setTimeout(()=>{resolve("hi")}, 3000)
})
}
const asyncFunction = async (callback) => {
let res = await asyncOperation();
callback(res);
}
const topDog = () => {
let callback = (res) => {
console.log(res);
};
(async () => {
await asyncFunction(callback)
})()
}
By applying this trick to an EventEmitter, you can get the same results. Define the EventEmitter's listener where I've defined the callback, and emit the event where I called the callback.

If function Fiber really turns async function sleep into sync
Yes. Inside the fiber, the function waits before logging ok. Fibers do not make async functions synchronous, but allow to write synchronous-looking code that uses async functions and then will run asynchronously inside a Fiber.
From time to time I find the need to encapsulate an async function into a sync function in order to avoid massive global re-factoring.
You cannot. It is impossible to make asynchronous code synchronous. You will need to anticipate that in your global code, and write it in async style from the beginning. Whether you wrap the global code in a fiber, use promises, promise generators, or simple callbacks depends on your preferences.
My objective is to minimize impact on the caller when data acquisition method is changed from sync to async
Both promises and fibers can do that.

There is a npm sync module also. which is used for synchronize the process of executing the query.
When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.
Sample code
/*require sync module*/
var Sync = require('sync');
app.get('/',function(req,res,next){
story.find().exec(function(err,data){
var sync_function_data = find_user.sync(null, {name: "sanjeev"});
res.send({story:data,user:sync_function_data});
});
});
/*****sync function defined here *******/
function find_user(req_json, callback) {
process.nextTick(function () {
users.find(req_json,function (err,data)
{
if (!err) {
callback(null, data);
} else {
callback(null, err);
}
});
});
}
reference link: https://www.npmjs.com/package/sync

Nowadays this generator pattern can be a solution in many situations.
Here an example of sequential console prompts in nodejs using async readline.question function:
var main = (function* () {
// just import and initialize 'readline' in nodejs
var r = require('readline')
var rl = r.createInterface({input: process.stdin, output: process.stdout })
// magic here, the callback is the iterator.next
var answerA = yield rl.question('do you want this? ', r=>main.next(r))
// and again, in a sync fashion
var answerB = yield rl.question('are you sure? ', r=>main.next(r))
// readline boilerplate
rl.close()
console.log(answerA, answerB)
})() // <-- executed: iterator created from generator
main.next() // kick off the iterator,
// runs until the first 'yield', including rightmost code
// and waits until another main.next() happens

I can't find a scenario that cannot be solved using node-fibers. The example you provided using node-fibers behaves as expected. The key is to run all the relevant code inside a fiber, so you don't have to start a new fiber in random positions.
Lets see an example: Say you use some framework, which is the entry point of your application (you cannot modify this framework). This framework loads nodejs modules as plugins, and calls some methods on the plugins. Lets say this framework only accepts synchronous functions, and does not use fibers by itself.
There is a library that you want to use in one of your plugins, but this library is async, and you don't want to modify it either.
The main thread cannot be yielded when no fiber is running, but you still can create plugins using fibers! Just create a wrapper entry that starts the whole framework inside a fiber, so you can yield the execution from the plugins.
Downside: If the framework uses setTimeout or Promises internally, then it will escape the fiber context. This can be worked around by mocking setTimeout, Promise.then, and all event handlers.
So this is how you can yield a fiber until a Promise is resolved. This code takes an async (Promise returning) function and resumes the fiber when the promise is resolved:
framework-entry.js
console.log(require("./my-plugin").run());
async-lib.js
exports.getValueAsync = () => {
return new Promise(resolve => {
setTimeout(() => {
resolve("Async Value");
}, 100);
});
};
my-plugin.js
const Fiber = require("fibers");
function fiberWaitFor(promiseOrValue) {
var fiber = Fiber.current, error, value;
Promise.resolve(promiseOrValue).then(v => {
error = false;
value = v;
fiber.run();
}, e => {
error = true;
value = e;
fiber.run();
});
Fiber.yield();
if (error) {
throw value;
} else {
return value;
}
}
const asyncLib = require("./async-lib");
exports.run = () => {
return fiberWaitFor(asyncLib.getValueAsync());
};
my-entry.js
require("fibers")(() => {
require("./framework-entry");
}).run();
When you run node framework-entry.js it will throw an error: Error: yield() called with no fiber running. If you run node my-entry.js it works as expected.

You shouldn't be looking at what happens around the call that creates the fiber but rather at what happens inside the fiber. Once you are inside the fiber you can program in sync style. For example:
function f1() {
console.log('wait... ' + new Date);
sleep(1000);
console.log('ok... ' + new Date);
}
function f2() {
f1();
f1();
}
Fiber(function() {
f2();
}).run();
Inside the fiber you call f1, f2 and sleep as if they were sync.
In a typical web application, you will create the Fiber in your HTTP request dispatcher. Once you've done that you can write all your request handling logic in sync style, even if it calls async functions (fs, databases, etc.).

I struggled with this at first with node.js and async.js is the best library I have found to help you deal with this. If you want to write synchronous code with node, approach is this way.
var async = require('async');
console.log('in main');
doABunchOfThings(function() {
console.log('back in main');
});
function doABunchOfThings(fnCallback) {
async.series([
function(callback) {
console.log('step 1');
callback();
},
function(callback) {
setTimeout(callback, 1000);
},
function(callback) {
console.log('step 2');
callback();
},
function(callback) {
setTimeout(callback, 2000);
},
function(callback) {
console.log('step 3');
callback();
},
], function(err, results) {
console.log('done with things');
fnCallback();
});
}
this program will ALWAYS produce the following...
in main
step 1
step 2
step 3
done with things
back in main

Making Node.js code sync is essential in few aspects such as database. But actual advantage of Node.js lies in async code. As it is single thread non-blocking.
we can sync it using important functionality Fiber()
Use await() and defer ()
we call all methods using await(). then replace the callback functions with defer().
Normal Async code.This uses CallBack functions.
function add (var a, var b, function(err,res){
console.log(res);
});
function sub (var res2, var b, function(err,res1){
console.log(res);
});
function div (var res2, var b, function(err,res3){
console.log(res3);
});
Sync the above code using Fiber(), await() and defer()
fiber(function(){
var obj1 = await(function add(var a, var b,defer()));
var obj2 = await(function sub(var obj1, var b, defer()));
var obj3 = await(function sub(var obj2, var b, defer()));
});
I hope this will help. Thank You

What is a good approach to develop a synchronous/blocking and an asynchrounous/non-blocking library-api in parallel? (JavaScript)

I rewrote this question, because the old version was obviously misleading.
Please read the text and make shure you understood what I'm asking for. If
there is still anything left in the dark I'll modify this question for clarity.
Just inform me.
One of my projects is to port a library from Python to JavaScript.
The Python library is entirely blocking/synchronous when it comes to I/O
and such. This is of course perfectly normal for Python code.
I plan to port the synchronous/blocking methods as they are to JavaScript.
This has several reasons and whether or not it's worth the effort is a
good but different question.
Additionally I wan't to add an asynchronous/non-blocking api.
Think of it like the fs module in node where there are i.e. fs.open and
fs.openSync coexisting.
The library is pure JavaScript and will run in Node and in the Browser.
The question is what a good/the best approach for the development of these two coexisting APIs would be.
I believe its good to have the same thing happening in one place only.
Hence an approach where some parts of the implementation could be shared would be preferable.
Not at any price of course, that's why I'm asking.
I had a proposal for an approach in here, but I'm going to post it as a
possible answer. However, I'm waiting for some serious discussion to happen
before I decide what I accept as an answer.
So far approaches are:
implement both apis separately and definetly use promises for the asynchronous functions.
use something like the obtain api proposal - beeing a more integrated approach

If you're talking I/O in node.js then most I/O methods have a synchronous version.
There is no direct conversion from Asynchronicity To Synchronicity. I can think of two approaches:
Have each asynchronous method run a polling loop waiting for the async task to complete before returning.
Drop the idea of mimicking synchronous code and instead invest in better coding patterns (such as promises)
To illustrate I will assume option 2 is a better choice. The following example uses Q promises (easily installed with npm install q.
The idea behind promises is that although they are asynchronous the return object is a promise for a value as if it was a normal function.
// Normal function
function foo(input) {
return "output";
}
// With promises
function promisedFoo(input) {
// Does stuff asynchronously
return promise;
}
The first function takes an input and returns a result. The second example takes an input and immediately returns a promise which will eventually resolve to a value when the async task finishes. You then manage this promise as follows:
var promised_value = promisedFoo(input);
promised_value.then(function(value) {
// Yeah, we now have a value!
})
.fail(function(reason) {
// Oh nos.. something went wrong. It passed in a reason
});
Using promises you no longer have to worry when something will happen. You can easily chain promises so things happen synchronously without insane nested callbacks or 100 named functions.
It well worth learning about. Remember promises are meant to make async code behave like sync code even though it isn't blocking.

Write lower level API using promises that takes async/sync flag.
Higher level async API returns these promises directly (while also working with async callbacks like it's 1970).
Higher level sync API unwraps the value synchronously from the promise and returns the value or throws the error.
(Examples use bluebird which is orders of magnitude faster and has more features at the cost of file size compared to Q, although that might not be ideal for browsers.)
Low level api that is not exposed:
//lowLevelOp calculates 1+1 and returns the result
//There is a 20% chance of throwing an error
LowLevelClass.prototype.lowLevelOp = function(async, arg1, arg2) {
return new Promise(function(resolve, reject) {
if (Math.random() < 0.2) {
throw new Error("random error");
}
if (!async) resolve(1+1);
else {
//Async
setTimeout(function(){
resolve(1+1);
}, 50);
}
});
};
High level exposed API that works synchronously, using promises or callbacks:
HighLevelClass.prototype.opSync = function(arg1, arg2) {
var inspection =
this.lowLevel.lowLevelOp(false, arg1, arg2).inspect();
if (inspection.isFulfilled()) {
return inspection.value();
}
else {
throw inspection.error();
}
};
HighLevelClass.prototype.opAsync = function(arg1, arg2, callback) {
//returns a promise as well as accepts callback.
return this.lowLevel.lowLevelOp(true, arg1, arg2).nodeify(callback);
};
You can automatically generate the high level api for synchronous methods:
var LowLevelProto = LowLevelClass.prototype;
Object.keys(LowLevelProto).filter(function(v) {
return typeof LowLevelProto[v] === "function";
}).forEach(function(methodName) {
//If perf is at all a concern you really must do this with a
//new Function instead of closure and reflection
var method = function() {
var inspection = this.lowLevel[methodName].apply(this.lowLevel, arguments);
if (inspection.isFulfilled()) {
return inspection.value();
}
else {
throw inspection.error();
}
};
HighLevelClass.prototype[methodName + "Sync" ] = method;
});

I implemented a library that does what I'm asking for ObtainJS.
(Yes, the Library uses Promises BUT not as others proposed in their ansewers here)
Reposting the Readme.md:
ObtainJS
ObtainJS is a micro framework to bring together asynchronous and
synchronous JavaScript code. It helps you to Don't Repeat Yourself
(DRY) if you are developing a library with interfaces for both
blocking/synchronous and non-blocking/asynchronous execution models.
As a USER
of a library that was implemented with ObtainJS you won't have to learn
a lot. Typically a function defined using ObtainJS has as first argument
the switch, that lets you choose the execution path, followed by its normal
arguments:
// readFile has an obtainJS API:
function readFile(obtainAsyncExecutionSwitch, path) { /* ... */ }
execute synchronously
If the obtainSwitch is a falsy value readFile will execute synchronously
and return the result directly.
var asyncExecution = false, result;
try {
result = readFile(asyncExecution, './file-to-read.js');
} catch(error) {
// handle the error
}
// do something with result
execute asynchronously
If the obtainSwitch is a truthy value readFile will execute asynchronously
and always return a Promise.
See Promises at MDN
var asyncExecution = true, promise;
promise = readFile(asyncExecution, './file-to-read.js');
promise.then(
function(result) {
// do something with result
},
function(error){
// handle the error
}
)
// Alternatively, use the returned promise directly:
readFile(asyncExecution, './file-to-read.js')
.then(
function(result) {
// do something with result
},
function(error){
// handle the error
}
)
You can use a callback based api, too. Note that the Promise is returned anyways.
var asyncExecution;
function unifiedCallback(error, result){
if(error)
// handle the error
else
// do something with result
}
asyncExecution = {unified: unifiedCallback}
readfile(asyncExecution, './file-to-read.js');
or with a separate callback and errback
var asyncExecution;
function callback(result) {
// do something with result
}
function errback(error) {
// handle the error
}
var asyncExecution = {callback: callback, errback: errback}
readfile(asyncExecution, './file-to-read.js');
```
As a smart ;-) LIBRARY AUTHOR
who's going to implement a API using with ObtainJS the work is a bit more.
Stay with me.
The behavior above is achieved by defining a twofold dependency tree: one
for the actions of the synchronous execution path and one for the actions
of the asynchronous execution path.
Actions are small functions with dependencies on the results of other
actions. The asynchronous execution path will fallback to synchronous
actions if there is no asynchronous action defined for a dependency.
You wouldn't define an asynchronous action if its synchronous
equivalent is non-blocking. This is where you DRY!
So, what you do, for example, is splitting your synchronous and blocking
method in small function-junks. These junks depend on the results of each
other. Then you define a non-blocking AND asynchronous junk for each
synchronous AND blocking junk. The rest does obtainJS for you. Namely:
creating a switch for synchronous or asynchronous execution
resolving the dependency tree
executing the junks in the right order
providing you with the results via:
return value when using the synchronous path
promises OR callbacks (your choice!) when using the asynchronous path
Here is the readFile function
from above, taken directly from working code at
ufoJS
define(['ufojs/obtainJS/lib/obtain'], function(obtain) {
// obtain.factory creates our final function
var readFile = obtain.factory(
// this is the synchronous dependency definition
{
// this action is NOT in the async tree, the async execution
// path will fall back to this method
uri: ['path', function _path2uri(path) {
return path.split('/').map(encodeURIComponent).join('/')
}]
// synchronous AJAX request
, readFile:['uri', function(path) {
var request = new XMLHttpRequest();
request.open('GET', path, false);
request.send(null);
if(request.status !== 200)
throw _errorFromRequest(request);
return request.responseText;
}]
}
,
// this is the asynchronous dependency definition
{
// aynchronous AJAX request
readFile:['uri', '_callback', function(path, callback) {
var request = new XMLHttpRequest()
, result
, error
;
request.open('GET', path, true);
request.onreadystatechange = function (aEvt) {
if (request.readyState != 4 /*DONE*/)
return;
if (request.status !== 200)
error = _errorFromRequest(request);
else
result = request.responseText
callback(error, result)
}
request.send(null);
}]
}
// this are the "regular" function arguments
, ['path']
// this is the "job", a driver function that receives as first
// argument the obtain api. A method that the name of an action or
// of an argument as input and returns its result
// Note that job is potentially called multiple times during
// asynchronoys execution
, function(obtain, path){ return obtain('readFile'); }
);
})
a skeleton
var myFunction = obtain.factory(
// sync actions
{},
// async actions
{},
// arguments
[],
//job
function(obtain){}
);
action/getter definition
// To define a getter we give it a name provide a definition array.
{
// sync
sum: ['arg1', 'arg2',
// the last item in the definition array is always the action/getter itself.
// it is called when all dependencies are resolved
function(arg1, arg2) {
// function body.
var value = arg1 + arg2
return value
}]
}
// For asynchronous getters you have different options:
{
// async
// the special name "_callback" will inject a callback function
sample1: ['arg1', '_callback', function(arg1, callback) {
// callback(error, result)
}],
// you can order separate callback and errback when using both special
// names "_callback" and "_errback"
sample2: ['arg1', '_callback', '_errback', function(arg1, callback, errback) {
// errback(error)
// callback(result)
}],
// return a promise
sample3: ['arg1', function(arg1) {
var promise = new Promise(/* do what you have to*/);
return promise
}]
}
The items in the definition array before the action are the dependencies
their values are going to be injected into the call to action, when
available.
If the type of an dependency is not a string: It's injected as a value
directly. This way you can effectively do currying.
If the type of the value is a string: It's looked up in the dependency
tree for the current execution path(sync or async).
If its name is defined as an caller-argument (in the third argument of obtain.factory) the value
is taken from the invoking call.
If its name is defined as the name of another action, that action is
executed and its return value is used as a parameter. An action will
executed only once per run, later invocations will return a cached value.
If the execution path is asynchronous obtain will first look for a
asynchronous action definition. If that is not found it falls back
to a synchronous definition.
If you wish to pass a String as value to your getter you must define it as
an instance of obtain.Argument: new obtain.Argument('mystring argument is not a getter')
A more complete example
from ufoLib/glifLib/GlyphSet.js
Note that: obtainJS is aware of the host object and propagates this
correctly to all actions.
/**
* Read the glif from I/O and cache it. Return a reference to the
* cache object: [text, mtime, glifDocument(if alredy build by this.getGLIFDocument)]
*
* Has the obtainJS sync/async api.
*/
GlypSet.prototype._getGLIFcache = obtain.factory(
{ //sync
fileName: ['glyphName', function fileName(glyphName) {
var name = this.contents[glyphName];
if(!(glyphName in this.contents) || this.contents[glyphName] === undefined)
throw new KeyError(glyphName);
return this.contents[glyphName]
}]
, glyphNameInCache: ['glyphName', function(glyphName) {
return glyphName in this._glifCache;
}]
, path: ['fileName', function(fileName) {
return [this.dirName, fileName].join('/');
}]
, mtime: ['path', 'glyphName', function(path, glyphName) {
try {
return this._io.getMtime(false, path);
}
catch(error) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
throw error;
}
}]
, text: ['path', 'glyphName', function(path, glyphName) {
try {
return this._io.readFile(false, path);
}
catch(error) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
throw error;
}
}]
, refreshedCache: ['glyphName', 'text', 'mtime',
function(glyphName, text, mtime) {
return (this._glifCache[glyphName] = [text, mtime]);
}]
}
//async getters
, {
mtime: ['path', 'glyphName', '_callback',
function(path, glyphName, callback) {
var _callback = function(error, result){
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
callback(error, result)
}
this._io.getMtime({unified: _callback}, path);
}]
, text: ['path', 'glyphName', '_callback',
function(path, glyphName, callback){
var _callback = function(error, result) {
if(error instanceof IONoEntryError)
error = new KeyError(glyphName, error.stack);
callback(error, result)
}
this._io.readFile({unified: _callback}, path);
}
]
}
, ['glyphName']
, function job(obtain, glyphName) {
if(obtain('glyphNameInCache')) {
if(obtain('mtime').getTime() === this._glifCache[glyphName][1].getTime()) {
// cache is fresh
return this._glifCache[glyphName];
}
}
// still here? need read!
// refreshing the cache:
obtain('refreshedCache')
return this._glifCache[glyphName];
}
)