Develop Reference

How to use a promise in do-while loop waiting a callback to be completed

I have a callback which waits for a response from a HTTP request which responds with the word "done" if a file is successfully uploaded and I make one request via a callback to upload a single file every time.
What I want is that when the response is "done", I want to upload multiple files with a do-while loop and I'm thinking of doing that with promises, but I don't really know how.
My code now:
var self = this;
let i = 0;
let fileInput = fileCmp.get("v.files");
do {
// my callback
self.uploadHelper(component, event, fileInput[i]);
console.log("Uploading: " + fileInput[i].name);
i++;
} while (i < fileInput.length);
The thing I want is to go to i=1 (second file) only when I get the response "done" or something else from the call.
My callback which is called from uploadHelper():
uploadChunk: function (component, file, fileContents, fromPos, toPos, attachId) {
console.log('uploadChunk');
var action = component.get("c.saveTheChunk");
var chunk = fileContents.substring(fromPos, toPos);
action.setParams({
parentId: component.get("v.recordId"),
fileName: file.name,
base64Data: encodeURIComponent(chunk),
contentType: file.type,
fileId: attachId
});
action.setCallback(this, function (a) {
console.log('uploadChunk: Callback');
attachId = a.getReturnValue();
fromPos = toPos;
toPos = Math.min(fileContents.length, fromPos + this.CHUNK_SIZE);
if (fromPos < toPos) {
this.uploadChunk(component, file, fileContents, fromPos, toPos, attachId);
} else {
console.log('uploadChunk: done');
component.set("v.showLoadingSpinner", false);
// enabling the next button
component.set("v.nextDisabled", false);
component.set("v.uploadDisabled", true);
component.set("v.clearDisabled", true);
component.set("v.showToast", true);
component.set("v.toastType", 'success');
component.set("v.fileName", '');
component.set("v.toastMessage", 'Upload Successful.');
}
});
$A.getCallback(function () {
$A.enqueueAction(action);
})();
}

You have to make your uploadHelper method not aync to achieve what you want.
Try to use the async, await and Promise objects to create a promis in your function (instead of a callback) and force it to happened synchronously.
It may look somthing like this:
// uploadHelper definiton, fit its to your code
const uploadHelper = (component, event, file) => {
// Create the promise
return new Promise(function(component, event, file) {
// Do what you want to do
})
}
// Use it
var self = this;
let i = 0;
let fileInput = fileCmp.get("v.files");
do{
await self.uploadHelper(component, event, fileInput[i]).then(function() {
console.log("Uploading: "+fileInput[i].name);
i++;
});
}
while(i< fileInput.length);
For further info try this links:
https://developer.mozilla.org/he/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

As your uploadChunk does an asynchronous task, it would make sense for the function to take a callback, that gets called back when the task is done (your code does not contain any callback yet). As callbacks are difficult to handle (especially with loops), it makes sense to wrap the callback into a Promise (that resolves when the callback gets called):
uploadChunk: function (component, file, fileContents, fromPos, toPos, attachId) {
return new Promise(resolve => { // the promise gets returned immeadiately, the "resolve" callback can be called to resolve it
// ....
action.setCallback(this, function (a) {
//...
resolve();
});
});
},
Your uploadHelper could now return the promise returned back to the loop:
uploadHelper(component, event, file) {
//...
return this.uploadChunk(component, file, /*...*/);
}
Now that we have that, we can simply await that promise inside of an async function:
(async () => { // arrow function to preserve "this"
for(const file of files) { // Why do...while if you can use a for..of?
await this.uploadChunk(component, event, file);
}
})();

Get rid of the while loop, and simply pop an item off your fileInput array every time you use it. Cancel everything when the array becomes empty.
(You need to have a working callback function though...)

Hey you can do it with an auto run function :
var self = this;
let i = 0;
let fileInput = fileCmp.get("v.files");
do{
// my callback
(()=>{
self.uploadHelper(component, event, fileInput[i]);
console.log("Uploading: "+fileInput[i].name);
i++;
})();
}
while(i< fileInput.length);

AWS S3 / Javascript callback issue

So, I'm having a problem with JavaScript asynchronous execution when making an API call to AWS S3.
I have a sequence of nested callbacks that are working fine up until a specific S3 call that my code is not waiting for. Here's my code:
getThumbUrls(contentIndex, function(data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
});
getThumbUrls() looks like this:
function getThumbUrls(contentIndex, callback) {
console.log('Entering getThumbUrls');
var thumbUrls = [];
JSON.parse(contentIndex).forEach(videoKey => {
// get the thumbnail: bucket-name/thumbnails/<first-key>
console.log('videoKey = ' + videoKey);
getThumbFileName(videoKey, function(thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
});
});
callback(thumbUrls);
}
And getThumbFileName() looks like this:
function getThumbFileName(videoKey, callback) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
var urlKey;
//console.log('listObjects params = ' + JSON.stringify(params, null, 4));
s3.listObjectsV2(params, (err, data) => {
if (err) {
console.log(err, err.stack);
callback(err);
return;
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
});
callback(urlKey);
//callback('20161111-TheWind.jpg');
}
Obviously, what's happening is that execution doesn't wait for the s3.listObjectsV2 call to finish. I've verified that the entire flow works properly when all getThumbFileName() does is callback with the filename.
Would someone kindly show me how to force execution to wait for s3.listObjectsV2 to complete before calling back with undefined?

As discussed, you should avoid callbacks approach when dealing with asynchronous operations over iterations, due their difficulty.
(You can skip this section if you don't want to know motivation behind promises approach).
Just to mention, in a callback approach, you must have to wait for all callbacks to complete in your getThumbUrls(), using a if which will check if all callbacks has been called, then just call callback(thumbUrls); with all responses pushed into your thumbUrls array:
function getThumbUrls(contentIndex, callback) {
const thumbUrls = [];
// counter which will increment by one for every callback
let counter = 0;
JSON.parse(contentIndex).forEach(videoKey => {
getThumbFileName(videoKey, function (thumbFileName) {
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
// for each callback response you must add 1 to a counter and then
counter++;
// check if all callbacks already has been called
if (counter === JSON.parse(contentIndex).length) {
// right here, thumbsUrls are filled with all responses
callback(thumbUrls);
}
});
});
}
So, you can make use of Promises, and a Promise.all will be enough for you to handle all responses from api. You can study over internet and check your code below, which is using a promise approach. I've added some comments to help you understanding what is happening.
// when using promises, no callbacks is needed
getThumbUrls(contentIndex)
.then(function (data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
})
// when using promises, no callbacks is needed
function getThumbUrls(contentIndex) {
console.log('Entering getThumbUrls');
// not needed anymore, Promise.all will return all values
// var thumbUrls = [];
// Promise.all receives an array of promises and returns to next .then() all results
// changing forEach to map to return promises to my Promise.all
return Promise.all(JSON.parse(contentIndex).map(videoKey => {
console.log('videoKey = ' + videoKey);
// returning a promise
return getThumbFileName(videoKey)
.then(function (thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
return CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName;
});
}))
}
// when using promises, no callbacks is needed
function getThumbFileName(videoKey) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
// urlKey not need anymore
// var urlKey;
// most of AWS functions has a .promise() method which returns a promise instead calling callback funcions
return s3.listObjectsV2(params).promise()
.then(function (data) {
var thumbsKey = data.Contents;
//if you want to return only first one thumbsKey:
return thumbsKey[0];
})
.catch(function (err) {
console.log(err, err.stack);
callback(err);
return;
})
}
Hope this helps you out in your study.

Would someone kindly show me how to force execution to wait
That's the wrong question. You are not trying to get execution to "wait," or, at least, you shouldn't be. You just need to call the callback in the right place -- inside the callback from s3.listObjectsV2(), not outside.
function getThumbFileName(videoKey, callback) {
...
s3.listObjectsV2(params, (err, data) => {
if (err) {
...
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
callback(urlKey); // right
});
// wrong // callback(urlKey);
}
The way you wrote it, the callback fires after s3.getObjectsV2() begins to run -- not after it finishes (calls its own callback).

JavaScript and Promise.all()

I'm trying to load a bunch of resources async using promises with Jimp.js. The loading code is a disaster trying to chain it all, and I need a cleaner solution.
What I came up with was below. This obviously doesn't do anything, because it's junk code, but I need to know if there was a failure loading any of the resources, and I need to know when they completed.
function doSomething(asdf) {
return new Promise((resolve, reject) => {
//console.log("It is done.");
// Succeed half of the time.
var x = Math.random();
if (x > .5) {
resolve(["SUCCESS",asdf,x])
} else {
reject(["Failure",asdf,x])
}
});
}
func();
function func() {
//Imagine a is actually an image loaded by doSomething
var a=null; doSomething("1").then(function (data) {a = data;},
(err) => {throw new err;});
//Imagine b is a font resource.
var b=null; doSomething("2").then(function (data) {b = data;},
(err) => {throw new err;});
Promise.all([a, b]).then(function() {
console.log(a);
console.log(b);
//Then here I expect everything to be correct, and continue on with the next function.
},
(err) => {console.log('Oops:' + err);}).
catch( (err) => {console.log('Oops:' + err);});
}
For some reason, this never outputs "Oops".
Here is a fail output:
[ 'SUCCESS', '1', 0.756461151774289 ]
null
What am I missing here?
Update
I took part of an answer I received and changed it so that it behaves exactly as I wanted:
function func() {
var a=doSomething("1").then(function (data) {a = data;});
var b=doSomething("2").then(function (data) {b = data;});
Promise.all([a, b]).then(function() {
console.log(a);
console.log(b);
},
(err) => {console.log('Reject:' + err);});
}
Update
Here is the actual code I'm using that's working great now:
LoadResources() {
var ps = [];
console.log("Loading now");
ps.push(jimp.read(this.ipath+"c4box.png").then(function (image) {obj.imBox = image;}));
ps.push(jimp.read(this.ipath+"red.png").then(function (image) {obj.imRed = image;}));
ps.push(jimp.read(this.ipath+"green.png").then(function (image) {obj.imGreen = image;}));
ps.push(jimp.read(this.ipath+"top.png").then(function (image) {obj.imTop = image;}));
ps.push(jimp.read(this.ipath+"bot.png").then(function (image) {obj.imBot = image;}));
ps.push(jimp.loadFont(jimp.FONT_SANS_32_WHITE).then(function (font) {obj.imFont = font;}));
Promise.all(ps).then( () => {
obj.loaded = true;
obj.imBg = new jimp(512, 576, function (err, image) { });
console.log("Actually loaded now.");
obj.StartGame();
});
console.log("Loading commands complete");
}

You cannot use those a and b variables for the images. (See here for the values that would be passed into Promise.all). You need to use variables for the promise objects that doSomething() returns. The images will only be available inside the then callback - Promise.all creates a promise that fulfills with an array of the results:
function func() {
// aPromise is a promise for an image loaded by doSomething
var aPromise = doSomething("1");
// bPromise is a promise for a font resource.
var bPromise = doSomething("2");
Promise.all([aPromise, bPromise]).then(function([a, b]) {
// ^^^^^^
console.log(a);
console.log(b);
// Then here I expect everything to be correct, and continue on with the next function.
}, (err) => {
console.log('Oops:' + err);})
});
}

Promise.all([a, b])
Cause a and b are null as you set them to null. Therefore Promise.all won't wait at all, it will resolve one tick afterwards, and as a and b get resolved / rejected very fast, that might has happened before and a / b gets set before it reaches
console.log(a)
which will log the right results sometimes, but thats based on chance.

Promise.all returns a promise and this promise contains the result of the previous promises
Promise.all([ doSomething('1'), doSomething('2')])
.then(results => {
// results is an array which contains the result of the previous promises
const [a, b] = results
}).catch(err => console.log('Oops:' + err))

JS Chunked file upload with promises

I'm building a chunked file uploader. I understood that promises are the best way to handle the results but still, can't wrap my mind of how I should capture the promises from the outside.
caller.js
let fileUploader = new ChunkedUpload()
fileUploader.setFile(file)
fileUploader.upload().then(result => {
// Here I'd like to catch the current result of the file
// If it's not uploaded, I need the percentage and continue
// If it is I need the response from the API
console.log(result)
})
uploader.js
upload() {
let totalChunks = Math.ceil(this._file.size/this._sliceSize)
return this._chunk(0, 0, totalChunks)
}
_chunk(start, chunk, totalChunks) {
let lastChunk = false
let end = start + this._sliceSize
if (this._file.size - end < 0) {
end = this._file.size
lastChunk = true
}
let slice = this._sliceFile(this._file, start, end)
let formData = new FormData()
formData.append('chunk', chunk)
formData.append('chunks', totalChunks)
formData.append('file', slice)
return new Promise((resolve, reject) => {
axios.post(this._url, formData).then(response => {
if (lastChunk) {
// This part is okay, it returns the result
resolve({
uploaded: true,
url: response.data,
uploadPercentage: 100,
uploadSize: this.formatBytes(file.size)
});
} else {
// Here's the issue. Next chunk upload is called,
// however I cannot track the process in caller.js
resolve(this._chunk(
start += this._sliceSize,
chunk += 1,
totalChunks
));
}
})
})
}
I accept any comments and whatsoever. Maybe my approach is wrong, please let me know!

Promise should be used to ONE-TIME callback - success or error.
What you want is "progress" information.
That way you should:
Use a callback function instead to get details about progress; or
Listen and emit events;
BUT, if you really want to use promise and not use callbacks or events, I would suggest:
return promise with details AND a method inside called continue() that HAS to be called so the process may continue.
Below I give you the code for that:
caller.js
let fileUploader = new ChunkedUpload()
fileUploader.setFile(file)
var callback = result => {
// Here I'd like to catch the current result of the file
// If it's not uploaded, I need the percentage and continue
// If it is I need the response from the API
console.log(result);
if (result.uploaded) {
console.log('DONE');
} else {
console.log('STILL UPLOADING...');
result.continue()
.then(callback);
}
}
fileUploader.upload().then(callback);
uploader.js
upload() {
let totalChunks = Math.ceil(this._file.size/this._sliceSize)
return this._chunk(0, 0, totalChunks)
}
_chunk(start, chunk, totalChunks) {
let lastChunk = false
let end = start + this._sliceSize
if (this._file.size - end < 0) {
end = this._file.size
lastChunk = true
}
let slice = this._sliceFile(this._file, start, end)
let formData = new FormData()
formData.append('chunk', chunk)
formData.append('chunks', totalChunks)
formData.append('file', slice)
return new Promise((resolve, reject) => {
axios.post(this._url, formData).then(response => {
if (lastChunk) {
// This part is okay, it returns the result
resolve({
uploaded: true,
url: response.data,
uploadPercentage: 100,
uploadSize: this.formatBytes(file.size)
});
} else {
// Here's the issue. Next chunk upload is called,
// however I cannot track the process in caller.js
// you may include in the object below the metrics you need
resolve({
uploaded: false,
continue: () => {
return this._chunk(
start += this._sliceSize,
chunk += 1,
totalChunks
}
});
}
})
})
}

Thanks #Rafel! Your answer led me to events.
Apparently I got caught into the Promise construct antipattern as #Bergi mentioned and catching a reject would've resulted into another variable definition.
Thanks everyone!

node.js looping through GETs with promise

I'm new to promises and I'm sure there's an answer/pattern out there but I just couldn't find one that was obvious enough to me to be the right one. I'm using node.js v4.2.4 and https://www.promisejs.org/
This should be pretty easy I think...I need to do multiple blocks of async in a specific order, and one of the middle blocks will be looping through an array of HTTP GETs.
//New Promise = asyncblock1 - FTP List, resolve the returned list array
//.then(asynchblock2(list)) - loop through list array and HTTP GET needed files
//.then(asynchblock3(list)) - update local log
I tried creating a new Promise, resolving it, passing the list to the .then, doing the GET loop, then the file update. I tried using a nested promise.all inside asynchblock2, but it's actually going in reverse order, 3, 2, and 1 due to the timing of those events. Thanks for any help.
EDIT: Ok, this is the pattern that I'm using which works, I just need a GET loop in the middle one now.
var p = new Promise((resolve, reject) => {
setTimeout(() => {
console.log('2 sec');
resolve(1);
},
2000);
}).then(() => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('1.5 sec');
// instead of this section, here I'd like to do something like:
// for(var i = 0; i < dynamicarray.length; i++){
// globalvar[i] = ftpclient.getfile(dynamicarray[i])
// }
// after this loop is done, resolve
resolve(1);
},
1500);
});
}).then(() => {
return new Promise((resolve) => {
setTimeout(() => {
console.log('1 sec');
resolve(1);
},
1000);
});
});
EDIT Here is the almost working code!
var pORecAlert = (function(){
var pa;
var newans = [];
var anstodownload = [];
var anfound = false;//anfound in log file
var nexttab;
var lastchar;
var po;
var fnar = [];
var antext = '';
//-->> This section works fine; it's just creating a JSON object from a local file
try{
console.log('trying');
porfile = fs.readFileSync('an_record_files.json', 'utf8');
if(porfile == null || porfile == ''){
console.log('No data in log file - uploaded_files_data.json being initialized!');
plogObj = [];
}
else{
plogObj = JSON.parse(porfile);
}
}
catch(jpfp){
console.log('Error parsing log file for PO Receiving Alert: ' + jpfp);
return endPORecAlertProgram();
};
if((typeof plogObj) === 'object'){
console.log('an_record_files.json log file found and parsed for PO Receiving Alert!');
}
else{
return mkError(ferror, 'pORecAlert');
};
//finish creating JSON Object
pa = new Client();
pa.connect(ftpoptions);
console.log('FTP Connection for FTP Check Acknowledgement begun...');
pa.on('greeting', function(msg){
console.log('FTP Received Greeting from Server for ftpCheckAcknowledgement: ' + msg);
});
pa.on('ready', function(){
console.log('on ready');
//START PROMISE LIST
var listpromise = new Promise((reslp, rejlp) => {
pa.list('/public_html/test/out', false, (cerr, clist) => {
if(cerr){
return mkError(ferror, 'pORecAlert');
}
else{
console.log('Resolving clist');
reslp(clist);
}
});
});
listpromise.then((reclist) => {
ftpplist:
for(var pcl = 0; pcl < reclist.length; pcl++){
console.log('reclist iteration: ' + pcl);
console.log('checking name: ', reclist[pcl].name);
if(reclist[pcl].name.substring(0, 2) !== 'AN'){
console.log('Not AN - skipping');
continue ftpplist;
}
else{//found an AN
for(var plc = 0; plc < plogObj.length; plc++){
if(reclist[pcl].name === plogObj[plc].anname){
//console.log('Found reclist[pcl].name in local log');
anfound = true;
};
};
if(anfound === false){
console.log('Found AN file to download: ', reclist[pcl].name);
anstodownload.push(reclist[pcl].name);
};
};
};
console.log('anstodownload array:');
console.dir(anstodownload);
return anstodownload;
}).then((fnar) => {
//for simplicity/transparency, here is the array being overwritten
fnar = new Array('AN_17650_37411.699.txt', 'AN_17650_37411.700', 'AN_17650_37411.701', 'AN_17650_37411.702.txt', 'AN_17650_37411.801', 'AN_17650_37411.802.txt');
return Promise.all(fnar.map((gfname) => {
var nsalertnames = [];
console.log('Getting: ', gfname);
debugger;
pa.get(('/public_html/test/out/' + gfname), function(err, anstream){//THE PROBLEM IS THAT THIS GET GETS TRIGGERED AN EXTRA TIME FOR EVERY OTHER FILE!!!
antext = '';
console.log('Get begun for: ', gfname);
debugger;
if(err){
ferror.nsrest_trace = 'Error - could not download new AN file!';
ferror.details = err;
console.log('Error - could not download new AN file!');
console.log('************************* Exiting *************************')
logError(ferror, gfname);
}
else{
// anstream.on('data', (anchunk) => {
// console.log('Receiving data for: ', gfname);
// antext += anchunk;
// });
// anstream.on('end', () => {
// console.log('GET end for: ', gfname);
// //console.log('path to update - gfname ', gfname, '|| end text.');
// fs.appendFileSync(path.resolve('test/from', gfname), antext);
// console.log('Appended file');
// return antext;
// });//end end
};
});//get end
}));//end Promise.all and map
}).then((res99) => {
// pa.end();
// return Promise(() => {
console.log('end all. res99: ', res99);
// //res4(1);
// return 1;
// });
});
});
})();
-->> What happens here:
So I added the almost working code. What is happening is that for every other file, an additional Get request gets made (I don't know how it's being triggered), which fails with an "Unable to make data connection".
So for my iteration over this array of 6, there ends up being 9 Get requests. Element 1 gets requested (works and expected), then 2 (works and expected), then 2 again (fails and unexpected/don't know why it was triggered). Then 3 (works and expected), then 4 (works and expected), then 4 again (fails and unexpected) etc

what you need is Promise.all(), sample code for your app:
...
}).then(() => {
return Promise.all(arry.map(item => ftpclient.getFile(item)))
}).then((resultArray) => {
...

So thanks for the help (and the negative votes with no useful direction!)
I actually reached out to a good nodejs programmer and he said that there seemed to be a bug in the ftp module I was using, and even when trying to use a blackbird .map, the quick succession of requests somehow kicked off an error. I ended up using promise-ftp, blackbird, and promiseTaksQueue - the kicker was that I needed interval. Without it the ftp would end up causing a strange illogical error in the ftp module.

You need the async library. Use the async.eachSeries in situations where you need to use asynchronous operations within a loop, then execute a function when all of those are complete. There are many variations depending on the flow you want but this library does it all.
https://github.com/caolan/async
async.each(theArrayToLoop, function(item, callback) {
// Perform async operation on item here.
doSomethingAsync(item).then(function(){
callback();
})
}, function(err){
//All your async calls are finished continue along here
});