I'm having the following code, this all code is implemented in specified function myFunc, I need that the all function will finish (myFunc) i.e. when the file was extracted successfully/or not to return some status (success/ error).
var myFunc = () => {
var DecompressZip = require('decompress-zip');
var unzipper = new DecompressZip(filename)
unzipper.on('error', function (err) {
console.log('Caught an error');
});
unzipper.on('extract', function (log) {
console.log('Finished extracting');
});
unzipper.on('progress', function (fileIndex, fileCount) {
console.log('Extracted file ' + (fileIndex + 1) + ' of ' + fileCount);
});
unzipper.extract({
path: 'some/path',
filter: function (file) {
return file.type !== "SymbolicLink";
}
});
};
Since this open source is working with event this is a problem (to get return status...) my intention is to change it to promise by promisify or
like following:
var myFunc = () => {
return new Promise(function(resolve, reject) {
var DecompressZip = require('decompress-zip');
var unzipper = new DecompressZip(filename)
unzipper.on('error', function (err) {
console.log('Caught an error');
reject();
});
unzipper.on('extract', function (log) {
console.log('Finished extracting');
resolve();
});
unzipper.on('progress', function (fileIndex, fileCount) {
console.log('Extracted file ' + (fileIndex + 1) + ' of ' + fileCount);
});
unzipper.extract({
path: 'some/path',
filter: function (file) {
return file.type !== "SymbolicLink";
}
});
};
My questions are:
Since I'm not too expert in JS Does it make sense to convert the events to promise?
There is other
good solution which I can use for the use-case?
This is the OP
https://github.com/bower/decompress-zip
1) Yes it makes perfect sense to convert such events to a Promise. Currently, different libraries have different ways of expressing events (events, messages, callback functions, then error callback function, exceptions, error codes, etc...). Promises will soon unify all this now they're a Javascript standard. So it a good practice to put a Promise layer around the libraries you're using and use only Promises in your code instead of the old mess.
2) Your solution looks good to me.
Converting events to promises only makes sense when you are absolutely positive that the "end" event will only fire once.
So in this case, yes, what you are suggesting for the implementation should work (assuming I understand your code correctly).
Related
Lets say this is my code (just a sample I wrote up to show the idea)
var extract = require("./postextract.js");
var rescore = require("./standardaddress.js");
RunFunc();
function RunFunc() {
extract.Start();
console.log("Extraction complete");
rescore.Start();
console.log("Scoring complete");
}
And I want to not let the rescore.Start() run until the entire extract.Start() has finished. Both scripts contain a spiderweb of functions inside of them, so having a callback put directly into the Start() function is not appearing viable as the final function won't return it, and I am having a lot of trouble understanding how to use Promises. What are ways I can make this work?
These are the scripts that extract.Start() begins and ends with. OpenWriter() is gotten to through multiple other functions and streams, with the actual fileWrite.write() being in another script that's attached to this (although not needed to detect the end of run. Currently, fileWrite.on('finish') is where I want the script to be determined as done
module.exports = {
Start: function CodeFileRead() {
//this.country = countryIn;
//Read stream of thate address components
fs.createReadStream("Reference\\" + postValid.country + " ADDRESS REF DATA.csv")
//Change separator based on file
.pipe(csv({escape: null, headers: false, separator: delim}))
//Indicate start of reading
.on('resume', (data) => console.log("Reading complete postal code file..."))
//Processes lines of data into storage array for comparison
.on('data', (data) => {
postValid.addProper[data[1]] = JSON.stringify(Object.values(data)).replace(/"/g, '').split(',').join('*');
})
//End of reading file
.on('end', () => {
postValid.complete = true;
console.log("Done reading");
//Launch main script, delayed to here in order to not read ahead of this stream
ThisFunc();
});
},
extractDone
}
function OpenWriter() {
//File stream for writing the processed chunks into a new file
fileWrite = fs.createWriteStream("Processed\\" + fileName.split('.')[0] + "_processed." + fileName.split('.')[1]);
fileWrite.on('open', () => console.log("File write is open"));
fileWrite.on('finish', () => {
console.log("File write is closed");
});
}
EDIT: I do not want to simply add the next script onto the end of the previous one and forego the master file, as I don't know how long it will be and its supposed to be designed to be capable of taking additional scripts past our development period. I cannot just use a package as it stands because approval time in the company takes up to two weeks and I need this more immediately
DOUBLE EDIT: This is all my code, every script and function is all written by me, so I can make the scripts being called do what's needed
You can just wrap your function in Promise and return that.
module.exports = {
Start: function CodeFileRead() {
return new Promise((resolve, reject) => {
fs.createReadStream(
'Reference\\' + postValid.country + ' ADDRESS REF DATA.csv'
)
// .......some code...
.on('end', () => {
postValid.complete = true;
console.log('Done reading');
resolve('success');
});
});
}
};
And Run the RunFunc like this:
async function RunFunc() {
await extract.Start();
console.log("Extraction complete");
await rescore.Start();
console.log("Scoring complete");
}
//or IIFE
RunFunc().then(()=>{
console.log("All Complete");
})
Note: Also you can/should handle error by reject("some error") when some error occurs.
EDIT After knowing about TheFunc():
Making a new Event emitter will probably the easiest solution:
eventEmitter.js
const EventEmitter = require('events').EventEmitter
module.exports = new EventEmitter()
const eventEmitter = require('./eventEmitter');
module.exports = {
Start: function CodeFileRead() {
return new Promise((resolve, reject) => {
//after all of your code
eventEmitter.once('WORK_DONE', ()=>{
resolve("Done");
})
});
}
};
function OpenWriter() {
...
fileWrite.on('finish', () => {
console.log("File write is closed");
eventEmitter.emit("WORK_DONE");
});
}
And Run the RunFunc like as before.
There's no generic way to determine when everything a function call does has finished.
It might accept a callback. It might return a promise. It might not provide any kind of method to determine when it is done. It might have side effects that you could monitor by polling.
You need to read the documentation and/or source code for that particular function.
Use async/await (promises), example:
var extract = require("./postextract.js");
var rescore = require("./standardaddress.js");
RunFunc();
async function extract_start() {
try {
extract.Start()
}
catch(e){
console.log(e)
}
}
async function rescore_start() {
try {
rescore.Start()
}
catch(e){
console.log(e)
}
}
async function RunFunc() {
await extract_start();
console.log("Extraction complete");
await rescore_start();
console.log("Scoring complete");
}
So, I'm having a problem with JavaScript asynchronous execution when making an API call to AWS S3.
I have a sequence of nested callbacks that are working fine up until a specific S3 call that my code is not waiting for. Here's my code:
getThumbUrls(contentIndex, function(data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
});
getThumbUrls() looks like this:
function getThumbUrls(contentIndex, callback) {
console.log('Entering getThumbUrls');
var thumbUrls = [];
JSON.parse(contentIndex).forEach(videoKey => {
// get the thumbnail: bucket-name/thumbnails/<first-key>
console.log('videoKey = ' + videoKey);
getThumbFileName(videoKey, function(thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
});
});
callback(thumbUrls);
}
And getThumbFileName() looks like this:
function getThumbFileName(videoKey, callback) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
var urlKey;
//console.log('listObjects params = ' + JSON.stringify(params, null, 4));
s3.listObjectsV2(params, (err, data) => {
if (err) {
console.log(err, err.stack);
callback(err);
return;
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
});
callback(urlKey);
//callback('20161111-TheWind.jpg');
}
Obviously, what's happening is that execution doesn't wait for the s3.listObjectsV2 call to finish. I've verified that the entire flow works properly when all getThumbFileName() does is callback with the filename.
Would someone kindly show me how to force execution to wait for s3.listObjectsV2 to complete before calling back with undefined?
As discussed, you should avoid callbacks approach when dealing with asynchronous operations over iterations, due their difficulty.
(You can skip this section if you don't want to know motivation behind promises approach).
Just to mention, in a callback approach, you must have to wait for all callbacks to complete in your getThumbUrls(), using a if which will check if all callbacks has been called, then just call callback(thumbUrls); with all responses pushed into your thumbUrls array:
function getThumbUrls(contentIndex, callback) {
const thumbUrls = [];
// counter which will increment by one for every callback
let counter = 0;
JSON.parse(contentIndex).forEach(videoKey => {
getThumbFileName(videoKey, function (thumbFileName) {
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
// for each callback response you must add 1 to a counter and then
counter++;
// check if all callbacks already has been called
if (counter === JSON.parse(contentIndex).length) {
// right here, thumbsUrls are filled with all responses
callback(thumbUrls);
}
});
});
}
So, you can make use of Promises, and a Promise.all will be enough for you to handle all responses from api. You can study over internet and check your code below, which is using a promise approach. I've added some comments to help you understanding what is happening.
// when using promises, no callbacks is needed
getThumbUrls(contentIndex)
.then(function (data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
})
// when using promises, no callbacks is needed
function getThumbUrls(contentIndex) {
console.log('Entering getThumbUrls');
// not needed anymore, Promise.all will return all values
// var thumbUrls = [];
// Promise.all receives an array of promises and returns to next .then() all results
// changing forEach to map to return promises to my Promise.all
return Promise.all(JSON.parse(contentIndex).map(videoKey => {
console.log('videoKey = ' + videoKey);
// returning a promise
return getThumbFileName(videoKey)
.then(function (thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
return CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName;
});
}))
}
// when using promises, no callbacks is needed
function getThumbFileName(videoKey) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
// urlKey not need anymore
// var urlKey;
// most of AWS functions has a .promise() method which returns a promise instead calling callback funcions
return s3.listObjectsV2(params).promise()
.then(function (data) {
var thumbsKey = data.Contents;
//if you want to return only first one thumbsKey:
return thumbsKey[0];
})
.catch(function (err) {
console.log(err, err.stack);
callback(err);
return;
})
}
Hope this helps you out in your study.
Would someone kindly show me how to force execution to wait
That's the wrong question. You are not trying to get execution to "wait," or, at least, you shouldn't be. You just need to call the callback in the right place -- inside the callback from s3.listObjectsV2(), not outside.
function getThumbFileName(videoKey, callback) {
...
s3.listObjectsV2(params, (err, data) => {
if (err) {
...
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
callback(urlKey); // right
});
// wrong // callback(urlKey);
}
The way you wrote it, the callback fires after s3.getObjectsV2() begins to run -- not after it finishes (calls its own callback).
I'm trying to learn about what the promise is and how to convert callback to promise. While I'm converting my code to promise I got very confused about the ref. I would very appreciate it if you show me how to convert this code as a simple example.
database.ref('/users').on("child_added").then(function(snap){
var subData = snap.val();
database.ref('/subs/' + subData.subid + '/pri/' + snap.key).once("value").then(function(userSnap) {
var userData = userSnap.val();
subData.name = userData.name;
subData.age = userData.age;
database.ref('/subs/' + subData.subid).once("value",function(subDSnap) {
var subDData = subDSnap.val();
subData.type = subDData.type;
database_m.ref('/users/' + snap.key).set(subData);
});
});
});
A Promise is not a replacement for every type of callback; rather it's an abstraction around one particular task that will either succeed once or fail once. The code you're converting looks more like an EventEmitter, where an event can occur multiple times, so replacing .on('child_added', ...) with a Promise implementation is not a good fit.
However, later on, you have a .once(...) call. That's a bit closer to a Promise in that it will only complete once. So if you really wanted to convert that, here's what it could look like:
function get(database, url) {
return new Promise(function (resolve, reject) {
database
.ref(url)
.once('value', resolve)
.once('error', reject);
});
}
database.ref('/users').on("child_added", function(snap) {
var subData = snap.val();
get(database, '/subs/' + subData.subid + '/pri/' + snap.key)
.then(function(userSnap) {
var userData = userSnap.val();
subData.name = userData.name;
subData.age = userData.age;
return get(database, '/subs/' + subData.subid);
})
.then(function(subDSnap) {
var subDData = subDSnap.val();
subData.type = subDData.type;
database_m.ref('/users/' + snap.key).set(subData);
})
.catch(function (err) {
// handle errors
});
});
});
I am not sure I understand the question and if your "on" is returning a promise or just listening, but in your code you have nested '.then', which is not the common way to deal with promises and I am not sure that's what you wanted to achieve here.
You could do (assuming that the on function returns a promise, which I doubt)
database.ref('/users').on("child_added")
.then(function(snap){/* do something with the first part of your snap function*/})
.then (results => {/* do something else, such as your second ref call*/})
.catch(error => {/* manage the error*/})
To learn about promises, there are many examples online but what I really liked is the promise tutorial at google, with this nice snippet that explains it
var promise = new Promise(function(resolve, reject) {
// do a thing, possibly async, then…
if (/* everything turned out fine */) {
resolve("Stuff worked!");
}
else {
reject(Error("It broke"));
}
});
then, once you have the function that returns this promise, you can start doing
.then(...)
.then(...)
.then(...)
.catch(...)
My problem is that the code does not seem to be running in order, as seen below.
This code is for my discord.js bot that I am creating.
var Discord = require("discord.js");
var bot = new Discord.Client();
var yt = require("C:/Users/username/Documents/Coding/Discord/youtubetest.js");
var youtubetest = new yt();
var fs = require('fs');
var youtubedl = require('youtube-dl');
var prefix = "!";
var vidid;
var commands = {
play: {
name: "!play ",
fnc: "Gets a Youtube video matching given tags.",
process: function(msg, query) {
youtubetest.respond(query, msg);
var vidid = youtubetest.vidid;
console.log(typeof(vidid) + " + " + vidid);
console.log("3");
}
}
};
bot.on('ready', () => {
console.log('I am ready!');
});
bot.on("message", msg => {
if(!msg.content.startsWith(prefix) || msg.author.bot || (msg.author.id === bot.user.id)) return;
var cmdraw = msg.content.split(" ")[0].substring(1).toLowerCase();
var query = msg.content.split("!")[1];
var cmd = commands[cmdraw];
if (cmd) {
var res = cmd.process(msg, query, bot);
if (res) {
msg.channel.sendMessage(res);
}
} else {
let msgs = [];
msgs.push(msg.content + " is not a valid command.");
msgs.push(" ");
msgs.push("Available commands:");
msgs.push(" ");
msg.channel.sendMessage(msgs);
msg.channel.sendMessage(commands.help.process(msg));
}
});
bot.on('error', e => { console.error(e); });
bot.login("mytoken");
The youtubetest.js file:
var youtube_node = require('youtube-node');
var ConfigFile = require("C:/Users/username/Documents/Coding/Discord/json_config.json");
var mybot = require("C:/Users/username/Documents/Coding/Discord/mybot.js");
function myyt () {
this.youtube = new youtube_node();
this.youtube.setKey(ConfigFile.youtube_api_key);
this.vidid = "";
}
myyt.prototype.respond = function(query, msg) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
msg.channel.sendMessage("There was an error finding requested video.");
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
}
});
console.log("2");
};
module.exports = myyt;
As the code shows, i have an object for the commands that the bot will be able to process, and I have a function to run said commands when a message is received.
Throughout the code you can see that I have put three console.logs with 1, 2 and 3 showing in which order I expect the parts of the code to run. When the code is run and a query is found the output is this:
I am ready!
string +
2
3
1
This shows that the code is running in the wrong order that I expect it to.
All help is very highly appreciated :)
*Update! Thank you all very much to understand why it isn't working. I found a solution where in the main file at vidid = youtubetest.respond(query, msg) when it does that the variable is not assigned until the function is done so it goes onto the rest of my code without the variable. To fix I simply put an if statement checking if the variable if undefined and waiting until it is defined.*
Like is mentioned before, a lot of stuff in javascript runs in async, hence the callback handlers. The reason it runs in async, is to avoid the rest of your code being "blocked" by remote calls. To avoid ending up in callback hell, most of us Javascript developers are moving more and more over to Promises. So your code could then look more like this:
myyt.prototype.respond = function(query, msg) {
return new Promise(function(resolve, reject) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
reject("There was an error finding requested video."); // passed down to the ".catch" statement below
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
resolve(2); // Resolve marks the promises as successfully completed, and passes along to the ".then" method
}
});
}).then(function(two) {
// video is now the same as myyt.vidid as above.
console.log(two);
}).catch(function(err) {
// err contains the error object from above
msg.channel.sendMessage(err);
})
};
This would naturally require a change in anything that uses this process, but creating your own prototypes seems.. odd.
This promise returns the vidid, so you'd then set vidid = youtubetest.response(query, msg);, and whenever that function gets called, you do:
vidid.then(function(id) {
// id is now the vidid.
});
Javascript runs async by design, and trying to hack your way around that leads you to dark places fast. As far as I can tell, you're also targetting nodeJS, which means that once you start running something synchronously, you'll kill off performance for other users, as everyone has to wait for that sync call to finish.
Some suggested reading:
http://callbackhell.com/
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://stackoverflow.com/a/11233849/3646975
I'd also suggest looking up ES6 syntax, as it shortens your code and makes life a hellofalot easier (native promises were only introduced in ES6, which NodeJS 4 and above supports (more or less))
In javascript, please remember that any callback function you pass to some other function is called asynchronously. I.e. the calls to callback function may not happen "in order". "In order" in this case means the order they appear on the source file.
The callback function is simply called on certain event:
When there is data to be processed
on error
in your case for example when the youtube search results are ready,
'ready' event is received or 'message' is received.
etc.
I have a file containing JSON data that I want to read. I found some code for reading a file at neontribe.co.uk, but I can't seem to get the console to wait for the load to complete.
function onDeviceReady(){
window.resolveLocalFileSystemURL(cordova.file.externalDataDirectory, function(fs) {
var directoryReader = fs.createReader();
directoryReader.readEntries(function(entries) {
var i;
for (i=0; i<entries.length; i++) {
document.addEventListener('deviceready', onDeviceReady2, false);
function onDeviceReady2() {
function readFromFile(fileName) {
var pathToFile = cordova.file.externalDataDirectory + fileName;
window.resolveLocalFileSystemURL(pathToFile, function (fileEntry) {
fileEntry.file(function (file) {
var reader = new FileReader();
reader.onloadend = function (e) {
console.log("Inside Load" + JSON.parse(this.result));
return JSON.parse(this.result);
};
reader.readAsText(file);
}, errorHandler.bind(null, fileName));
}, errorHandler.bind(null, fileName));
}
function some_function(callback) {
console.log("inside function1");
var data = readFromFile(entries[i].name);
callback(data);
}
some_function(function(data) {
console.log("data!" + data);
console.log("Data String!" + JSON.stringify(data));
});
// var obj = JSON.parse(data);
// console.log("arresteeFirst!" + data.arresteeFirst);
// console.log("data!" + data);
}
console.log(entries[i].name);
}
}, function (error) {
alert(error.code);
});
}, function (error) {
alert(error.code);
});
}
When I run it I get this output in the console:
inside function1
Data!undefined
Data String!undefined
1452034357845.json
Inside Load[object Object]
So it looks like it goes to some_function and then prints the inside function 1. But then it does not wait for the function to pass the result from the load. It prints the two lines from the callback function straight away. It then prints the filename (at the end of the loop) and then finally prints the console message from within the load function. It looks like I am returning an object according to the console so data should not be undefined.
There are a few bits wrong with the code, mainly because it's not written taking into account the asynchronous calls and just assuming that everything happens in a synchronous fashion.
The callback(data) within some_function(callback) is in fact called after you called var data = readFromFile(entries[i].name);. But two issues arise here.
function readFromFile(fileName) doesn't return any data (actually, it doesn't return anything at all);
reader.readAsText(file); is treated asynchronous. In a nutshell, it means that your code will keep running (in your case print those messages) and call the reader.onloadend callback once the data is fully loaded - which would be too late for you as your message has already been printed.
There are several ways to fix this code, a good way is to use Promises. I like Bluebird for promises personally.
With promises, the solution would look something like this (pseudo-code):
function readFromFile(fileName) {
return new Promise(resolve, reject) {
var pathToFile = cordova.file.externalDataDirectory + fileName;
window.resolveLocalFileSystemURL(pathToFile, function (fileEntry) {
fileEntry.file(function (file) {
var reader = new FileReader();
reader.onloadend = function (e) {
console.log("Share the results... " + JSON.parse(this.result));
resolve(JSON.parse(this.result));
};
reader.readAsText(file);
}, errorHandler.bind(null, fileName));
}, errorHandler.bind(null, fileName));
} // Return the promise
}
function some_function(callback) {
console.log("inside function1");
var promise = readFromFile(entries[i].name);
promise.then(callback); // Will be called when promise is resolved
}
some_function(console.log);
Promises allows you to conceptually return a "contract" that the value will be fetched and all functions waiting for that value (.then) will be called once the data is received and the promise is resolved (you can reject a promise if it fails as well). (Sorry about the bad explanation, but there are loads of better documentation about it all over the internet, I recommend taking a look at it).
I hope it helps, there are other ways of coming around this problem, I believe that a promises might be the most elegant without getting into generators.