I'm new to js. I've read many of the prior stack overflow posts on asynchronicity but still don't understand the question below.
I have the following code to upload an image file to an S3 bucket. The key thing the code needs to achieve is to have the image1 variable in the parent store the information in the data variable from the s3.upload call in the child function.
My code using promises below works fine. My understanding is that the same thing can be done using callbacks only, and I've been trying to rewrite the code below with callbacks only as a learning exercise, but it has not worked. How would I change this code to do the same thing with callbacks and no promises?
Parent function:
try {
image1 = await uploadToAWSBucket(file);
}
catch (error) {
return next(error);
}
Child:
const uploadToAWSBucket = async (fileObject) => {
let randomFileName = shortid.generate();
const AWSUploadObject = {
Bucket: BUCKET_NAME,
Key: randomFileName,
Body: fileObject.buffer,
ContentType: fileObject.mimetype,
};
return new Promise((resolve, reject) => {
s3.upload(AWSUploadObject, (err, data) => {
if (err) {
return reject(err);
}
return resolve(data);
});
});
};
At first, you need to add a callback arg to you async function, removing async keyword
const uploadToAWSBucket = (fileObject, callback) => {
Next, you need to handle s3 response in a callback manner replacing Promise with callback usage.
s3.upload(AWSUploadObject, (err, data) => {
if (err) {
callback(err);
return;
}
callback(null, data);
});
Or maybe you can even simplify it to
s3.upload(AWSUploadObject, callback)
You also need to update your usage to a callback manner
uploadToAWSBucket(file, (error, image1) => {
if (error) {
next(error);
return;
}
// your success code here
});
The final result is
const uploadToAWSBucket = (fileObject, callback) => {
let randomFileName = shortid.generate();
const AWSUploadObject = {
Bucket: BUCKET_NAME,
Key: randomFileName,
Body: fileObject.buffer,
ContentType: fileObject.mimetype,
};
s3.upload(AWSUploadObject, callback);
};
That's it. I hope this explanation will help you to understand how to use callbacks.
If my understanding is correct, you want to use image1 after the catch block.
In that case, I suppose, you will be calling some function with image1. It can be done as follows, with some snippets taken from this answer:
const uploadToAWSBucket = (fileObject, callback) => { ... }; // described in the linked answer
uploadToAWSBucket(file, function callback(error, image1) {
if(error) { return next(error); }
someOtherFunction(image1, next); // "next" is passed as callback, with the assumption that nothing else needed to be called after that.
});
If you want to call 2 more functions with the result of someOtherFunction, it can be done as follows:
uploadToAWSBucket(file, function callback(error, image1) {
if(error) { return next(error); }
someOtherFunction(image1, function someOtherFunctionCb(error, someOtherFunctionResult) {
if(error) { return next(error); }
someOtherFunction2(someOtherFunctionResult, function someOtherFunction2Cb(error, someOtherFunction2Result) {
if(error) { return next(error); }
someOtherFunction3(someOtherFunction2Result, function someOtherFunction3Cb(error, someOtherFunction3Result) {
if(error) { return next(error); }
next(null, someOtherFunction3Result);
});
});
});
});
Basically, you cannot have local global variables if you use callbacks. I will try to explain a problem situation.
let image1 = null;
uploadToAWSBucket(file, function uploadToAWSBucketCallback(error, _image1) {
if(error) { return next(error); }
image1 = _image1;
});
someOtherFunction(image1, function someOtherFunctionCb(error, someOtherFunctionResult) {
if(error) { return next(error); }
...
});
In the above snippet, someOtherFunction will be called before uploadToAWSBucketCallback is executed. That means, image1 is not assigned with _image1. Now, you know what will be the value of image1 when someOtherFunction is called.
The second snippet shows how to pass result of one async function to another, by nesting the subsequent calls inside the callbacks. This makes code less readable for many. There are libraries like async, which helps to make things easier & readable.
The second snippet can be rewritten with async library's waterfall function like this:
async.waterfall([
function uploadToAWSBucketStep(callback) {
uploadToAWSBucket(file, callback);
},
function someOtherFunctionStep(image1, callback) {
someOtherFunction(image1, callback);
},
function someOtherFunction2Step(someOtherFunctionResult, callback) {
someOtherFunction2(someOtherFunctionResult, callback);
},
function someOtherFunction3Step(someOtherFunction2Result, callback) {
someOtherFunction3(someOtherFunction2Result, callback);
}
], function lastStep(error, someOtherFunction3Result) {
if(error) { return next(error); };
next(null, someOtherFunction3Result);
});
Promisifcation of a callback-based function is well understood and well documented.
I have never seen a discussion of "de-promisification", but it is pretty simple.
Starting with your uploadToAWSBucket() and assuming you want your callback to be "nodeback" style (signature (err, data)), then you can write:
const uploadToAWSBucketNodeback = (fileObject, nodeback) => {
uploadToAWSBucket(fileObject) // call the promise-returning "async" version.
.then(data => { // success path
nodeback(null, data);
})
.catch(nodeback); // error path
};
Or you could write a generic de-promisifier ...
const depromisify = (asyncFunction) => {
return function(...params) {
let nodeback = params.pop(); // strip nodeback off the end of params
asyncFunction(...params)
.then(data => { // success path
nodeback(null, data);
})
.catch(nodeback); // error path
}
};
... then
const uploadToAWSBucketNodeback = depromisify(uploadToAWSBucket);
Either approach will allow you to write:
uploadToAWSBucketNodeback(fileObject, function(err, data)) {
if(err) {
// handle error
} else {
// handle data
}
}
Notes
we just need to know that the original asyncFunction is thenable/catchable.
the original asyncFunction is completely opaque to the depromisified function.
we don't need to know anything about the internal workings of the original asyncFunction. Thus, the composition of AWSUploadObject doesn't need to be replicated ..... it is still performed by uploadToAWSBucket().
After reading everyone's responses, I came up with the following which I think works and is basically what #Pavlo Zhukov suggested. (Note the function names have changed slightly from my earlier post.)
Code from parent function:
let image1;
uploadToAWSBucketCallbackStyle(file, (err, data) => {
if (err) {
return next(err);
}
image1 = data;
// Do stuff with image1 here or make additional function
// calls using image1.
});
Child function:
const uploadToAWSBucketCallbackStyle = (fileObject, callback) => {
let randomFileName = shortid.generate();
const AWSUploadObject = {
Bucket: BUCKET_NAME,
Key: randomFileName,
Body: fileObject.buffer,
ContentType: fileObject.mimetype,
};
s3.upload(AWSUploadObject, callback);
}
Related
I am faced with some very old JavaScript code that uses YAHOO.util.Connect.asyncRequest() to speak with a REST API. I would like to wait for the response of the asyncRequest and then work with the response-data. The current code structure does not suffice, because I want to "stay" in the current JS method to keep the state.
My thought was to convert the callback to a promise, i.e. "promisify" the method call. However the examples I could find so far did not bring me to a working solution yet.
If there is a better more up to date solution to the YAHOO.util.Connect.asyncRequest(), I might also welcome that.
This is how the code currently looks like:
async function showEditBookingDialogue(bookingForm, courtOnline) {
//ToDo: Consolidate all the prompts into one window.
let bookingAttributes = bookingForm.getAttribute('data').split('_');
let bookingId = bookingAttributes[0];
//I would like to wait for the response here:
YAHOO.util.Connect.asyncRequest('POST', 'com/cms/BookingRequestHandler.php?action=checkpermissions&bid=' + bid, callback);
...
}
...
var callback = {
customevents: {
onStart: handleEvent.start,
onComplete: handleEvent.complete,
onSuccess: handleEvent.success,
onFailure: handleEvent.failure,
onUpload: handleEvent.upload,
onAbort: handleEvent.abort
},
scope: handleEvent,
argument: ["foo", "bar", "baz"]
};
var handleEvent = {
start: function (eventType, args) {
},
complete: function (eventType, args) {
},
success: function (eventType, args) {
handleResponse(eventType, args);
},
failure: function (eventType, args) {
handleResponse(eventType, args);
},
upload: function (eventType, args) {
handleResponse(eventType, args);
},
abort: function (eventType, args) {
}
};
My current attempts look something like that, however nothing is logged in the console:
const promise = (...args) => {
return new Promise((resolve, reject) => {
YAHOO.util.Connect.asyncRequest('POST', 'com/cms/BookingRequestHandler.php?action=checkpermissions&bid=' + bid, (err, data) => {
if (err) return reject(err)
resolve(data)
})
})
}
await promise()
.then(data => {
console.log("works")
})
.catch(err => {
console.log("error")
});
From looking at example code such as this one, and even your original code, shows that the callback should be an object of handlers. Though official docs also state that a function also is accepted, so it's kinda confusing.
Here is an example of how you could build it using the callback object.
const yahooConnectRequest = body =>
new Promise((resolve, reject) => {
const callback = {
success: resolve,
failure: reject,
};
YAHOO.util.Connect.asyncRequest('POST', 'com/cms/BookingRequestHandler.php', callback, body);
});
Although you can use then chaining with async, it doesn't make much sense, unless you have a case that needs it. The example below is how you could do it without then.
// in async context.
try {
const body = 'action=checkpermissions&bid=1234';
const response = await yahooConnectRequest(body);
console.log(response);
} catch(error) {
console.error(error);
}
I'm curious to learn if this worked out.
I have 2 sections of code 1) that is called by 2nd to populate the array and write it into a file.
async function timeSeries(obj) {
data = [
{
original_value: []
}
]
//read file named as passed object's _id
await fs.readFile("./api/assignment_data/" + obj._id + ".json", "utf-8", function read(err, datas) {
if (err) {
throw err;
}
const filedata = JSON.parse(datas)
filedata.map(line => data[0].original_value.push(line.original_value))
})
setTimeout(() => {
try {
fs.writeFileSync("./api/timeseries.json", JSON.stringify(data), { encoding: 'utf8', flag: 'w' })
} catch (error) {
console.log(error)
}
}, 300);
}
The problem is, I can't access the global data array above after using it inside the fs.readfile function ( callback scope hell problem), I had to setTimeout then I am able to write it inside a file using another fs.writeFileSync function ( if I return the array I get a promise, I want data).How do I solve this? instead of writing it into another file and using it inside another route(below) how can I directly return the array in the second route and pass it as a json res?
section 2)
router.route("/api/debug/:num").get((req, res) => {
fs.readFile("./api/assignment_data/metrics.json", "utf8", function read(err, data) {
if (err) {
console.log(err);
}
const objdata = JSON.parse(data)
timeSeries(objdata[req.params.num])
})
fs.readFile("./api/timeseries.json", "utf8", function read(err, data) {
if (err) {
console.log(err);
}
const objdata = JSON.parse(data)
res.json(data)
})
})
If you use fs.readFile and want to do an action after the file has been read, you must do the action (write and read a file in your case) inside the callback function. Also, you can use fs.readFileSync if you can read synchronously.
First off, we need to explain a few things:
fs.readFile() is non-blocking and asynchronous. That means that when you call it, it starts the operation and then returns immediately and starts the execute the code that comes right after it. Then, some time later, it calls its callback.
So, your code is:
Calling fs.readFile()
Then, immediately setting a timer
Then, it's an indeterminate race between the fs.readFile() callback and the timer to see who finishes first. If the timer finishes first, then it will call its callback and you will attempt to access data BEFORE it has been filled in (because the fs.readFile() callback has not yet been called).
You cannot write reliable code this way as you are guessing on the timing of indeterminate, asynchronous operations. Instead, you have to use the asynchronous result from within the callback because that's the only place that you know the timing for when it finished and thus when it's valid. So, one way to solve your problem is by chaining the asynchronous operations so you put the second one INSIDE the callback of the first:
function timeSeries(obj, callback) {
//read file named as passed object's _id
fs.readFile("./api/assignment_data/" + obj._id + ".json", "utf-8", function read(err, datas) {
if (err) {
console.log(err);
// tell caller about our error
callback(err)
return;
} else {
let data = [{original_value: []}];
const filedata = JSON.parse(datas);
for (let line of filedata) {
data[0].original_value.push(line.original_value);
}
fs.writeFile("./api/timeseries.json", JSON.stringify(data), { encoding: 'utf8' }, (err) => {
if (err) {
console.log(err);
callback(err);
return;
} else {
// give our data to the caller
callback(data);
}
});
}
})
}
Then, to call this function, you pass it a callback and in the callback you can either see the error or get the data.
In modern nodejs, it's a bit easier to use async/await and the promise-based interfaces in the fs module:
const fsp = require('fs').promises;
async function timeSeries(obj) {
//read file named as passed object's _id
try {
let datas = await fsp.readFile("./api/assignment_data/" + obj._id + ".json", "utf-8");
const filedata = JSON.parse(datas);
let data = [{original_value: []}];
for (let line of filedata) {
data[0].original_value.push(line.original_value);
}
await fsp.writeFile("./api/timeseries.json", JSON.stringify(data), { encoding: 'utf8' });
return data;
} catch(e) {
console.log(e);
// handle error here or throw back to the caller
throw e;
}
}
For this version, the caller can use await and try/catch to get errors:
try {
let data = await timeSeries(obj);
// do something with data here
} catch(e) {
// handle error here
}
Based on what code you have written , I could just modify it using simple async-await - hope this helps
import fs from 'fs'
async function timeSeries(obj) {
const data = [{
original_value: []
}]
const assData = fs.readFileSync('./api/assignment_data/metrics.json', 'utf8')
const filedata = JSON.parse(assData)
filedata.map(line => data[0].original_value.push(line.original_value))
// no need for timeOut
fs.writeFileSync('./api/timeseries.json', JSON.stringify(data));
//return data if u need
return data
}
router.route("/api/debug/:num").get(async (req, res) => {
try {
const metricData = fs.readFileSync('./api/assignment_data/metrics.json', 'utf8')
const objdata = JSON.parse(data)
const timeSeriesData = await timeSeries(objdata[req.params.num])
// returning TimeSeriesData
res.status(200).json(timeSeriesData)
})
}
catch (error) {
res.status(500).send(error.message)
}
I asked this question earlier and it got associated with this question ( How do I return the response from an asynchronous call? ) But It doesn't resolve my problem
i am very new to node.js and i'm having some error that i can't deal with or find any help that i understand. I used express-generator to initialize the app. I'm trying to send and object i get through an api call to the front end. I wrote a class "StockClass.js" with a function makeApiCall() which makes the call and returns the object. But when i call that function in the router i get "undefined". Heres the code
//==== StockClass.js ===
const yahooFinance = require("yahoo-finance");
class stockClass {
static makeApiCall(symbol) {
yahooFinance.quote(
{
symbols: [symbol],
modules: ["price", "summaryDetail"],
},
function (err, quotes) {
if (err) { console.log(err) }
console.log(quotes)
return quotes;
}
);
}
}
module.exports = stockClass;
//=====index.js======
const StockClass = require("../handlers/StockClass");
router.get("/new", function (req, res) {
let quotes = StockClass.makeApiCall("AAPL");
console.log(quotes);
res.render('path', { quotes });
});
The console.log in the StockClass.js logs the object (quotes) correctly while the console.log in index.js logs "undefined".
Link below explains the yahoo-finance api call. https://www.npmjs.com/package/yahoo-finance
==========================================================================
I also tried using a middleware and attaching the data to the response object like this
//==========middleware=========
const yahooFinance = require("yahoo-finance");
module.exports = {
makeApiCall: (symbol) => {
return function (req, res, next) {
yahooFinance.quote(
{
symbols: [symbol],
modules: ["price", "summaryDetail"],
},
function (err, quotes) {
if (err) { console.log(err) }
res.stockObj = quotes;
console.log(res.stockObj);
}
);
next();
}
}
}
//======= index.js =========
const handler = require("./handlers/stockUtils");
router.get("/new", handler.makeApiCall("AAPL"), function (req, res) {
let quotes = res.stockObj;
console.log(quotes);
res.render('path', { quotes });
});
And the results are the same. The console.log in the middleware function logs the correct object, but the console.log in index.js logs undefined
This is a classic case of asynchronous programming (non blocking) and coming from synchronous programming background (like python, java, c etc) I too faced this problem of understanding the flow of the code in javascript. It took me awhile but once you grasp the way javascript language behaves, you will be able to see how the JS engine interprets your code and how it gets executed. Here are some links you might find useful:
Amazing post by Tyler
Concepts in MDN
Anyways, there are three ways to solve your problem.
Callback (old way)
Promises (somewhat new way)
Async/Await (latest way)
Callback Way
I have simplified your modularised code to wait for async operation finish and then make a callback. I will let you split it again and use it in express or whatever web framework you are using.
const yahooFinance = require("yahoo-finance");
class StockClass {
static makeApiCall(symbol, callback) {
yahooFinance.quote(
{
symbols: [symbol],
modules: ["price", "summaryDetail"],
},
function (err, quotes) {
if (err) {
console.log(err);
}
callback(quotes);
}
);
}
}
function toBeExecutedWhenDone(data) {
console.log(data);
}
// since its a static function we are calling it directly from Class instance
StockClass.makeApiCall("AAPL", toBeExecutedWhenDone); // this function can be your response
In the code above, you execute or return your response only when you have the data/quotes (or you can say when the async operation finishes). I'll let you explore more later.
Promises Way
Promises allow you to return something called Promise! 😝😝
Promises gets resolved when the async operation finishes. Here is how your code looks.
const yahooFinance = require("yahoo-finance");
class StockClass {
static makeApiCall(symbol) {
return new Promise((resolve, reject) => {
yahooFinance.quote(
{
symbols: [symbol],
modules: ["price", "summaryDetail"],
},
function (err, quotes) {
if (err) {
reject(err);
}
resolve(quotes);
}
);
});
}
}
// since its a static function we are calling it directly from Class instance
StockClass.makeApiCall("AAPL")
.then((data) => console.log(data)) // you could add your return your response instead of console log.
.catch((err) => console.error(err));
Async Await way!
Async await requires your function to be declared with async keyword. You can look at express guides as it already supports async functions. Here is how the snippet looks like.
const yahooFinance = require("yahoo-finance");
class StockClass {
static makeApiCall(symbol) {
return new Promise((resolve, reject) => {
yahooFinance.quote(
{
symbols: [symbol],
modules: ["price", "summaryDetail"],
},
function (err, quotes) {
if (err) {
reject(err);
}
resolve(quotes);
}
);
});
}
}
async function giveMeQuotes() {
try {
const stock = await StockClass.makeApiCall("AAPL");
console.log(stock); // this gets executed when above async operation finishes
} catch (error) {
console.error(error);
}
}
// executing to show that it works...
giveMeQuotes();
Hope it helps!!! Good luck!
Problem
I'm trying to get the diskName value back from this.getDiskName('C:')
and assign it to element['name']:
getDisksInfo () {
...
element['name'] = this.getDiskName('C:')
...
},
getDiskName (diskLetter) {
if (process.platform == 'win32') {
var exec = require('child_process').exec
var cmd = `wmic logicaldisk where caption="${diskLetter}" get VolumeName`
exec(cmd, (err, stdout, stderr) => {
if (err) {
console.log(err)
}
let diskName = stdout.split('\n')[1]
return diskName
})
}
}
I tried doing this, but I keep getting different errors:
getDiskName (diskLetter, callback) {
...
exec(cmd, (err, stdout, stderr) => {
if callback(null, () => {
let diskName = stdout.split('\n')[1]
return diskName
})
...
}
Question
Could someone please explain how to return the value properly?
Your problem is that you are missing either a callback coming into getDiskName() or a Promise() coming out.
Since the Promise approach seems to be more popular nowadays, I'll go with that for this answer.
With a Promise approach, you need the function to return a Promise. In most cases, you just wrap all the code up in a Promise and return that:
getDiskName(diskLetter) {
return new Promise((resolve, reject) => {
// rest of your code in the function
});
}
Then, instead of your return, you'll call resolve():
let diskName = stdout.split('\n')[1];
resolve(diskName)
And for your error, you'll call reject:
if (err) {
reject(err);
}
Then, in the function that uses it, you'll have to wait for the then() in your function:
this.getDiskName('C:').then(diskName => console.log(diskName))
The callback method is similar, you just pass in the callback into getDiskName and call it when you're ready.
This is a more idiomatic method to handle a case like this. We'll pass a function in to getDiskName which takes the disk name (which is the return value) as a parameter.
getDisksInfo () {
...
this.getDiskName('C:', function(diskName) {
element['name'] = diskName;
});
// Note that code from here to the end doesn't have access
// to element['name']
...
},
getDiskName (diskLetter, func) {
if (process.platform == 'win32') {
var exec = require('child_process').exec
var cmd = `wmic logicaldisk where caption="${diskLetter}" get VolumeName`
exec(cmd, (err, stdout, stderr) => {
if (err) {
console.log(err)
}
let diskName = stdout.split('\n')[1]
func(diskName);
})
}
}
Now, this still might not work for you since perhaps you have code after the call which relies on knowing the diskName. In that case, you would probably roll that code into your anonymous function. Perhaps getDisksInfo takes a function as a parameter instead.
This is the general pattern, you have to determine how it best fits in your program.
I am trying to call two functions and pass the output of the first function as a parameter into the second.
Function 1:
module.exports.getAllStatisticsByUserId = function(id, callback){
User.findById(id, (err, user) =>{
if(err)
throw err;
if(user)
callback(null, user.statistics);
});
}
Function 2:
module.exports.getGameByStatisticsId = function(id, callback){
Statistics.findById(id, (err, statistics) =>{
if(err)
throw err;
if(statistics)
callback(null, statistics.game);
});
};
I am trying to execute the second method by passing the output of the first method as a parameter but the asynchronous nature of javascript is messing it up. I have tried implementing promises to no avail.
Can anyone suggest some good javascript practices to deal with calling functions asynchronously when they need each other? Any help would be appreciated.
After fixing the issue I mentioned above, you can call them in sequence like this:
module.exports.getAllStatisticsByUserId = function(id, callback){
User.findById(id, (err, user) =>{
if(err) callback(err);
if(user) callback(null, user.statistics);
});
};
module.exports.getGameByStatisticsId = function(id, callback){
Statistics.findById(id, (err, statistics) =>{
if(err) callback(err);
if(statistics) callback(null, statistics.game);
});
};
someService.getAllStatisticsByUserId(id, (err, statistics) => {
if (err || !statistics) {
// handle error
return;
}
someService.getGameByStatisticsId(statistics.id, (err, game) => {
if (err || !game) {
// handle error
return;
}
// handle game
});
});
However, as noted in Mongoose documentation:
When a callback function is not passed, an instance of Query is returned, which provides a special query builder interface.
A Query has a .then() function, and thus can be used as a promise.
So you can simply rewrite the calls like this:
someService.getAllStatisticsByUserId(id).then(statistics =>
someService.getGameByStatisticsId(statistics.id)
).then(game => {
// handle game
}).catch(err => {
// handle error
});
or convert it into an async/await function:
async function getGameByUserId(id) {
try {
const statistics = await someService.getAllStatisticsByUserId(id);
const game = await someService.getGameByStatisticsId(statistics.id);
// handle game
} catch (error) {
// handle error
}
}
Note that an async function always returns a Promise, so you must await it or chain it with a .then() to ensure completion of the query and resolve the returned value, if any.
It looks like you should be able to write:
getAllStatisticsByUserId("me", (err, stats) => {
getGameByStatisticsId(stats.id, (err, game) => {
console.log(game);
});
});
Here's how it would look if these functions returned promises instead.
getAllStatisticsByUserId("me")
.then(stats => getGameByStatisticsId(stats.id))
.then(game => console.log(game))
Even better, if you're able to use a version of Node that supports async/await then you could write.
let stats = await getAllStatisticsByUserId("me");
let game = await getGameByStatisticsId(stats.id);
console.log(game);
This would mean slightly rewriting the original functions (unless User.findById and Statistics.findById already return promises).
module.exports.getAllStatisticsByUserId = function(id, callback){
return new Promise((resolve, reject) => {
User.findById(id, (err, user) =>{
if(err) return reject(err);
return resolve(user.statistics);
});
});
}