My problem is depending on the result of an asynchronous operation during a map. I know similar questions have been asked, but cannot quite figure out how to apply this to my case.
I have figured out how to solve my problem using synchronous methods, my code is as follows,
if (store.store !== {}) {
const files = Object.values(store.store).map((file, index) => {
return fs.existsSync(file.fileName) ? file : FILES[index];
});
}
I want to try writing this using asynchronous callbacks. I am new to javascript and just want to get the hang of these callbacks, this is what I tried, but it obviously does not work.
if (store.store !== {}) {
const exists = (path: string) => {
return fs.access(path, fs.constants.F_OK, (e) => {
return e ? false : true;
});
};
const files = Object.values(store.store).map((file, index) => {
return exists(file.fileName)? file : FILES[index];
});
I assume during execution the code does not wait for the result of the call back. However, using async/await with fs.promises.access does not work because I think the map function doesn't work asynchronously. Anyways, I want to be able to do this without async/await, is there any way to do this?
UPDATED SOLUTION:
Easy way: wrap the below solution in a Promise:
const getAuthorizedFiles => storeObj => new Promise((resolve, reject) => {
const doneCount = Object.keys(storeObj).length;
if (doneCount === 0) {
reject();
} else {
const files = [];
let count = 0;
Object.values(storeObj).forEach((file, index) => {
fs.access(file.fileName, fs.constants.F_OK, err => {
files[index] = !err ? file : null;
count += 1;
if (count === doneCount) {
resolve(files);
}
});
});
}
});
// Implementation:
getAuthorizedFiles(store.store).then(files => { console.log(files[0]); });
If OP just really doesn't want to deal with Promises for some reason, (though they are now a part of the spec) then they can also just make their own fun callback pattern function:
const getAuthorizedFiles = (storeObj, cb) => {
if (typeof cb !== 'function') { cb = x => x; }
const doneCount = Object.keys(storeObj).length;
if (doneCount === 0) {
cb(new Error(`I don't want to deal with an empty object`));
} else {
const files = [];
let count = 0;
Object.values(storeObj).forEach((file, index) => {
fs.access(file.fileName, fs.constants.F_OK, err => {
files[index] = !err ? file : null;
count += 1;
if (count === doneCount) {
cb(null, files);
}
});
});
}
};
// Implementation:
getAuthorizedFiles(store.store, files => { console.log(files[0]); });
ORIGINAL SOLUTION:
This is a simple way to do it without async/await. Set the files value as an empty array. Swap out the .map for a .forEach so you are no longer concerned with trying to return some value from an asynchronous function. Sounds like order is important, so when the access callback resolves, assign the file from the store.store Array into the new files Array.
if (store.store !== {}) {
const files = [];
Object.values(store.store).forEach((file, index) => {
fs.access(file.fileName, fs.constants.F_OK, err => {
files[index] = !err ? file : null;
});
});
}
fs.access as a third argument supports callback which means it is not returned back when fs.access is called, but you can make promise with primisify util
const access = util.promisify(fs.access);
then you can create async exists method utilizing then/catch to determine if you have access permission to the file
const asyncExists = (file) => {
return access(file.fileName, fs.constants.F_OK).then(() => file).catch(() => null);
};
and at the end you can map files to the array of promises and wrap it into Promise.all() to wait until all promises are resolved
Promise.all(Object.values(store.store).map(file => asyncExists(file))).filter(f => f != null)
.then(files => {
// Now you have array of files that you have access to it
});
Related
I've designed a function diversify() to take some expensive function f() and run it in parallel on all the cores of the machine it's being run on. I've also set this up so that if f() returns a value on one core, all the other threads are automatically killed. The code for that is below:
let diversify = f => {
// Split a function `f` into a thread for each core of the machine.
// The first thread to finish will return its result and end all others.
if ( cluster.isMaster ) {
let children = []
for ( let i = 0; i < os.cpus().length; i++) {
children.push(cluster.fork())
}
cluster.on('message', (_, msg) => {
if ( msg.cmd == 'stop' ) { children.forEach(child => {
child.process.kill()
})}
return msg.out
})
} else {
let out = f()
process.send({ cmd: 'stop', out })
}
}
The problem is that after one process tells the master process to stop through msg.cmd = 'stop', there is no way to return the output msg.out. The way that it is being implemented in the code above, the return msg.out statement is inside an anonymous function nested inside the larger diversify function. Therefore, this return value is not exposed when you run something like diversify(() => { return true }), resulting in undefined instead. Is there any way to return what's being sent in msg.out?
Sending messages to workers is by design asynchronous.
There is technically no way to return a value from a callback that is an event handler, since the event may occur at any time.
The canonical solution to this problem is via promises. Simply wrap the function body in a promise and resolve the value instead of returning it.
const diversify = (f) =>
new Promise((resolve) => {
// Split a function `f` into a thread for each core of the machine.
// The first thread to finish will return its result and end all others.
if (cluster.isMaster) {
let children = [];
for (let i = 0; i < os.cpus().length; i++) {
children.push(cluster.fork());
}
cluster.on('message', (_, msg) => {
if (msg.cmd == 'stop') {
children.forEach((child) => {
child.process.kill();
});
}
resolve(msg.out);
});
} else {
let out = f();
process.send({ cmd: 'stop', out });
}
});
The result is then accessible in out:
const getPid = () => {
return process.pid;
};
diversify(() => {
console.log(getPid());
return true;
}).then((out) => {
console.log(out); // true
});
// or using async/await
const diversifyAsync = async () => {
const out = await diversify(() => {
console.log(getPid());
return true;
});
console.log(out); // true
};
diversifyAsync();
I am having trouble getting promises to work the way I need. I have tried many different ways to resolve the promise but nothing I have done will work as I need. I am trying to get drag and drop of file working on a web page. I need a list of all the files in a Set (this.files) that is passed to a call to upload the files. The problem is that the Promise.all is being run before the promises are completed.
I am still struggling to wrap my mind around promises so maybe I have it all wrong but it seems from all my research this should work. Any help would be appreciated.
async dndDropFiles(event) {
event.preventDefault();
let ptable = [];
this.files = new Set();
if (event.dataTransfer.types[0] === "Files") {
var items = event.dataTransfer.items;
for (var i=0; i<items.length; i++) {
// webkitGetAsEntry is where the magic happens
var item = await items[i].webkitGetAsEntry();
if (item) {
ptable.push(new Promise(async resolve => {
resolve(await this.dndTraverseFileTree(item, ""));
}))
}
}
Promise.all(ptable)
.then( (results) => {
if (this.files.size > 0) {
this.progress = this.uploadService.upload(this.files, this.currentDir, this.currentProject);
}
})
}
}
async dndTraverseFileTree(item, path) {
if (item.isFile) {
// Get file
item.file((file) => {
this.files.add(file);
});
} else if (item.isDirectory) {
// Get folder contents
let ptable = [];
var dirReader = await item.createReader();
dirReader.readEntries((entries) => {
for (var i=0; i<entries.length; i++) {
ptable.push(new Promise( async resolve => {
resolve(await this.dndTraverseFileTree(entries[i], path + item.name + "/"));
}));
}
Promise.all(ptable)
.then (results => {});
});
}
}
It feels like you are making things a bit too difficult, and I suggest another good read into coding standards and the way Promises work :) The async/await construct has been introduced to increase code readability.
Anyways, I have some untested code here. But it should do the trick. Also I strongly advise you to add typings. You are using angular, so I can only assume you are using TypeScript. With typings you will make less errors, and the compiler helps you along the way.
Before I give the code, this webkitGetAsEntry is non standard. And should only be used if you really don't want to target old browsers or safari/ios:
Non-standard
This feature is non-standard and is not on a standards track. Do not use it on production sites facing the Web: it will not work for every user. There may also be large incompatibilities between implementations and the behavior may change in the future.
But, you could go about it like this. First function to process the event. The second one to traverse the tree:
async dndDropFiles(event: DragEvent): Promise<void> {
if (event.dataTransfer.types[0] !== "Files" || !event.dataTransfer.items) {
return;
}
const entries = [...(event.dataTransfer.items as any)].map(
item => item.webkitGetAsEntry()
);
const allEntries = await this.dndTraverseFileTree(entries);
const files = await Promise.all(
allEntries.map(
(entry) => new Promise((resolve, reject) => entry.file(resolve, reject))
)
);
this.files = new Set(files);
if (this.files.size > 0) {
this.progress = this.uploadService.upload(
this.files, this.currentDir, this.currentProject
);
}
}
async dndTraverseFileTree(entries: any[]): Promise<any[]> {
const dirs = entries.filter(entry => !!entry && entry.isDirectory);
const files = entries.filter(entry => !!entry && entry.isFile);
if (dirs.length) {
const childEntries = (
await Promise.all(
dirs.map(dir => new Promise(
(resolve, reject) => dir.createReader().readEntries(resolve, reject))
)
)
).flat();
return this.dndTraverseFileTree(childEntries);
}
return [ ...files ];
}
I have some code that does this: First scrape this array of webpages. After that, scrape another array of webpages.
The following code does what I expect:
let bays=[];
let promises=promisesN=[];
for (let y=2019;y>=2015;y--)
promises.push(new Promise(resolve=>
curl.get(`/*url*/${y}.html`,null, (error,resp,body)=>
resp.statusCode==200? resolve(parse(body)):reject(error)
)));
Promise.all(promises).then(()=>{
bays.forEach(bay=>{
if (bay.no.match(/\d+/)<=103) return;
promisesN.push(new Promise(resolve=>
curl.get(`/*url*/${bay.code}/`,null, (error,resp,body)=>
resp.statusCode==200? resolve(image(bey,body)):reject(error)
)))});
Promise.all(promisesN).then(()=>{
bays.sort((a,b)=>{return parseInt(a.no.match(/\d+/))<parseInt(b.no.match(/\d+/))? -1:1});
console.log(bays);
});
}).catch(error=>console.log(error));`
So I've read you can write a simplier nesting-free syntax:
doSomething()
.then(function(result) {
return doSomethingElse(result);
})
.then(function(newResult) {
return doThirdThing(newResult);
})
.then(function(finalResult) {
console.log('Got the final result: ' + finalResult);
})
.catch(failureCallback);
How to apply this to the code above?
correctness
let promises=promisesN=[];
This is really incorrect. It makes both variables reference the same array, and makes promisesN an implicit global. The fact that it appears to work means you aren’t in strict mode. Always use strict mode. The correct version of what you intended is:
let promises = [];
let promisesN = [];
cleanliness
new Promise(resolve=>
curl.get(`/*url*/${y}.html`,null, (error,resp,body)=>
resp.statusCode==200? resolve(parse(body)):reject(error)
))
You’re repeating this pattern, so make it into a function, or use a package that does the job for you, like request-promise[-native] or axios. (Also, please show your real code. reject isn’t defined here.)
const getAsync = url => new Promise((resolve, reject) => {
curl.get(url, null, (error, resp, body) => {
if (resp.statusCode === 200) {
resolve(body);
} else {
reject(error);
}
});
});
Notice how you’re free to make the function more readable when it isn’t repeated, and to extend it later.
let promises = [];
let promisesN = [];
for (let y = 2019; y >= 2015; y--) {
promises.push(getAsync(`/*url*/${y}.html`).then(parse));
}
Promise.all(promises).then(bays => {
bays.forEach(bay => {
if (bay.no.match(/\d+/) <= 103) return;
promisesN.push(getAsync(`/*url*/${bay.code}/`).then(body => image(bay, body)));
});
Promise.all(promisesN).then(() => {
bays.sort((a, b) => {return parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1;});
console.log(bays);
});
}).catch(error => console.log(error));
I had to take a few guesses at what your real code looks like again, because you’re surely doing something with the resolved value of Promise.all(promises). It doesn’t have any easily-accessible side-effects. bey also seemed likely enough to be bay.
Now you can give promisesN a more appropriate scope:
let promises = [];
for (let y = 2019; y >= 2015; y--) {
promises.push(getAsync(`/*url*/${y}.html`).then(parse));
}
Promise.all(promises).then(bays => {
let promisesN = bays
.filter(bay => bay.no.match(/\d+/) > 103)
.map(bay => getAsync(`/*url*/${bay.code}/`).then(body => image(bay, body)));
Promise.all(promisesN).then(() => {
bays.sort((a, b) => {return parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1;});
console.log(bays);
});
}).catch(error => console.log(error));
and use an expression-bodied arrow function where appropriate, since you’re already using them whenever they aren’t appropriate:
bays.sort((a, b) => parseInt(a.no.match(/\d+/)) < parseInt(b.no.match(/\d+/)) ? -1 : 1);
Now, if my guess about bays is right, then you can’t unnest. If it comes from somewhere else then you can. Normally I would leave a comment about that but I already wrote all this, so… please clarify that for further cleanup.
If you're looking to simplify your code, you might consider the use of async/await instead of promises.
The async/await syntax will greatly simplify the presentation and ease comprehension of the code, especially given that your logic relies on asynchronous iteration of arrays.
Consider the following code revision of your code:
/* Define local helper that wraps curl() in async function declaration */
function async doRequest(url) {
return (await new Promise(resolve=> curl.get(url, null, (error,resp,body) =>
resp.statusCode==200 ? resolve(res) : reject(error))))
}
/* Re-define simplified scrape logic using await/async */
function async doScrape() {
try {
var bays = []
/* Iterate date range asynchronously */
for (let y=2019; y>=2015; y--) {
/* Use doRequest helper function to fetch html */
const response = await doRequest(`/*url*/${y}.html`)
const bay = parse(response)
bays.push(bay)
}
/* Iterate bays array that was obtained */
for(const bay of bays) {
/* Use doRequest helper again to fetch data */
const response = await doRequest(`/*url*/${bay.code}/`)
/* Await may not be needed here */
await image(bay, response)
}
/* Perform your sort (which is non asynchronous) */
bays.sort((a,b)=> parseInt(a.no.match(/\d+/))<parseInt(b.no.match(/\d+/))? -1:1);
console.log("Result", bays);
}
catch(err) {
/* If something goes wrong we arrive here - this is
essentially equivalent to your catch() block */
console.error('Scrape failed', err);
}
}
/* Usage */
doScrape()
Hope that helps!
Not entirely sure if this is what you want, but I've separated your code out a bit because I found it easier for me to read.
let bays = [];
let promises = [];
let promisesN = [];
for (let y = 2019; y >= 2015; y--) {
const promiseOne = new Promise((resolve, reject) => {
return curl.get(`/*url*/${y}.html`, null, (error, resp, body) => {
resp.statusCode === 200 ? resolve(parse(body)) : reject(error);
});
});
promises.push(promiseOne);
}
Promise.all(promises)
.then(() => {
bays.forEach((bay) => {
if (bay.no.match(/\d+/) <= 103) {
return;
}
const promiseTwo = new Promise((resolve, reject) => {
return curl.get(`/*url*/${bay.code}/`, null, (error, resp, body) => {
resp.statusCode === 200 ? resolve(image(bay, body)) : reject(error);
});
});
promisesN.push(promiseTwo);
});
return Promise.all(promisesN);
})
.then(() => {
bays.sort((a, b) => {
return parseInt(a.no.match(/\d+/), 10) < parseInt(b.no.match(/\d+/), 10) ? -1 : 1;
});
console.log(bays);
})
.catch((error) => {
console.log(error);
});
I am wondering though, you are firing the promises instantly on each iteration of your for loop. This might be intentional, but it means if those promises resolve before the code gets to execute Promise.all you may run into issues. I personally would do something like, e.g. const promiseOne = () => somePromise, that way you can create a bunch of promises, and then once they're all created, map over that array and fire them at once. Same thing goes for the second promises.
Not sure if this is helpful, let me know if it is. Feel free to ask more questions too.
I'm tackling a project that requires me to use JavaScript with an API method call. I'm a Java programmer who has never done web development before so I'm having some trouble with it.
This API method is asynchronous and it's in a while loop. If it returns an empty array, the while loop finishes. Otherwise, it loops. Code:
var done = true;
do
{
async_api_call(
"method.name",
{
// Do stuff.
},
function(result)
{
if(result.error())
{
console.error(result.error());
}
else
{
// Sets the boolean to true if the returned array is empty, or false otherwise.
done = (result.data().length === 0) ? true : false;
}
}
);
} while (!done);
This doesn't work. The loop ends before the value of "done" is updated. I've done some reading up on the subject and it appears I need to use promises or callbacks because the API call is asynchronous, but I can't understand how to apply them to the code I have above.
Help would be appreciated!
edit: see the bottom, there is the real answer.
I encourage you yo use the Promise API. Your problem can be solved using a Promise.all call:
let promises = [];
while(something){
promises.push(new Promise((r, j) => {
YourAsyncCall(() => r());
});
}
//Then this returns a promise that will resolve when ALL are so.
Promise.all(promises).then(() => {
//All operations done
});
The syntax is in es6, here is the es5 equivalent (Promise API may be included externally):
var promises = [];
while(something){
promises.push(new Promise(function(r, j){
YourAsyncCall(function(){ r(); });
});
}
//Then this returns a promise that will resolve when ALL are so.
Promise.all(promises).then(function(){
//All operations done
});
You can also make your api call return the promise and push it directly to the promise array.
If you don't want to edit the api_call_method you can always wrap your code in a new promise and call the method resolve when it finishes.
edit: I have seen now the point of your code, sorry. I've just realized that Promise.all will not solve the problem.
You shall put what you posted (excluding the while loop and the control value) inside a function, and depending on the condition calling it again.
Then, all can be wraped inside a promise in order to make the external code aware of this asynchronous execution. I'll post some sample code later with my PC.
So the good answer
You can use a promise to control the flow of your application and use recursion instead of the while loop:
function asyncOp(resolve, reject) {
//If you're using NodeJS you can use Es6 syntax:
async_api_call("method.name", {}, (result) => {
if(result.error()) {
console.error(result.error());
reject(result.error()); //You can reject the promise, this is optional.
} else {
//If your operation succeeds, resolve the promise and don't call again.
if (result.data().length === 0) {
asyncOp(resolve); //Try again
} else {
resolve(result); //Resolve the promise, pass the result.
}
}
});
}
new Promise((r, j) => {
asyncOp(r, j);
}).then((result) => {
//This will call if your algorithm succeeds!
});
/*
* Please note that "(...) => {}" equivals to "function(...){}"
*/
sigmasoldier's solution is correct, just wanted to share the ES6 version with async / await:
const asyncFunction = (t) => new Promise(resolve => setTimeout(resolve, t));
const getData = async (resolve, reject, count) => {
console.log('waiting');
await asyncFunction(3000);
console.log('finshed waiting');
count++;
if (count < 2) {
getData(resolve, reject, count);
} else {
return resolve();
}
}
const runScript = async () => {
await new Promise((r, j) => getData(r, j, 0));
console.log('finished');
};
runScript();
If you don't want to use recursion you can change your while loop into a for of loop and use a generator function for maintaining done state. Here's a simple example where the for of loop will wait for the async function until we've had 5 iterations and then done is flipped to true. You should be able to update this concept to set your done variable to true when your webservice calls have buffered all of your data rows.
let done = false;
let count = 0;
const whileGenerator = function* () {
while (!done) {
yield count;
}
};
const asyncFunction = async function(){
await new Promise(resolve => { setTimeout(resolve); });
};
const main = new Promise(async (resolve)=>{
for (let i of whileGenerator()){
console.log(i);
await asyncFunction();
count++;
if (count === 5){
done = true;
}
}
resolve();
});
main.then(()=>{
console.log('all done!');
});
Also you may try recursion solution.
function asyncCall(cb) {
// Some async operation
}
function responseHandler(result) {
if (result.error()) {
console.error(result.error());
} else if(result.data() && result.data().length) {
asyncCall(responseHandler);
}
}
asyncCall(responseHandler);
Here is a solution I came up with. Place this in an async function.
let finished = false;
const loop = async () => {
return new Promise(async (resolve, reject) => {
const inner = async () => {
if (!finished) {
//insert loop code here
if (xxx is done) { //insert this in your loop code after task is complete
finshed = true;
resolve();
} else {
return inner();
}
}
}
await inner();
})
}
await loop();
If you don't want to use Promises you can restructure your code like so:
var tasks = [];
var index = 0;
function processNextTask()
{
if(++index == tasks.length)
{
// no more tasks
return;
}
async_api_call(
"method.name",
{
// Do stuff.
},
function(result)
{
if(result.error())
{
console.error(result.error());
}
else
{
// process data
setTimeout(processNextTask);
}
}
);
}
Your loop won't work, because it is sync, your async task is async, so the loop will finish before the async task can even respond. I'd reccomend you to use Promises to manage async tasks:
//first wrapping your API into a promise
var async_api_call_promise = function(methodName, someObject){
return new Promise((resolve, reject) => {
async_api_call(methodName, someObject, function(result){
if(result.error()){
reject( result.error() )
}else{
resolve( result.data() )
}
});
})
}
now to your polling code:
//a local utility because I don't want to repeat myself
var poll = () => async_api_call_promise("method.name", {/*Do stuff.*/});
//your pulling operation
poll().then(
data => data.length === 0 || poll(), //true || tryAgain
err => {
console.error(err);
return poll();
}
).then((done) => {
//done === true
//here you put the code that has to wait for your "loop" to finish
});
Why Promises? Because they do state-management of async operations. Why implement that yourself?
let taskPool = new Promise(function(resolve, reject) {
resolve("Success!");
});
let that = this;
while (index < this.totalPieces) {
end = start + thisPartSize;
if (end > filesize) {
end = filesize;
thisPartSize = filesize - start;
}
taskPool.then(() => {
that.worker(start, end, index, thisPartSize);
});
index++;
start = end;
}
I'm attempting to stop multiple requests from hitting the disk at once by caching requests and storing promises into an array. When the initial request finishes it should resolve all of the promises. Here's what I have, but unfortunately it doesn't look like new Promise() can be used this way, and deffered is no longer part of the spec. Note: some ES6 syntax such as const and the rocket operator are present in this example
This is a NodeJS application and I would prefer to not bring in any external libraries, however I will if necessary.
var observers = {}
function resolveObservers(link, value) {
for(var i = observers[link].length - 1; i >= 0; i--) {
if(observers[link][i] != null) {
observers[link][i].resolve(value)
observers[link].splice(i, 1)
}
}
}
function get(link) {
const b64link = base64.encode(link)
const promise = new Promise()
var handle = false
if(observers[b64link] == null) {
observers[b64link] = []
handle = true
} else if(observers[b64link].length == 0) {
handle = true
}
observers[b64link].push(promise)
if(handle) {
doAsyncOne.then(() => {
doAsyncTwo.then(() => {
doAsyncThree.then(data => {
resolveObservers(b64link, data)
})
})
})
}
}
The idea is that the Async code will only execute one time, and once it finishes all promises created by parallel requests will be resolved.
EDIT: I'm aware of how Promises in JS are normally used, I guess I'm looking for how Promises are used in other languages, usually called deferring.
EDIT2: You should be able to chain this event, for example:
get('...').then(data => {
// ...
})
You still can use the new Promise constructor in that way, even if you don't have deferreds any more:
var observers = {}
function get(link) {
const b64link = base64.encode(link)
return new Promise(resolve => {
if (observers[b64link] == undefined) {
observers[b64link] = [];
}
observers[b64link].push(resolve);
if (observers[b64link].length == 1) {
doAsyncOne
.then(() => doAsyncTwo)
.then(() => doAsyncThree)
.then(data => {
for (resolve of resolveObservers[b64link])
resolve(data)
}, err => {
err = Promise.rejct(err)
for (resolve of resolveObservers[b64link])
resolve(err)
})
}
});
}
But as you can see, error handling is not especially pretty (you'd even forgotten it completely), this is basically the deferred antipattern. There's a much simpler solution - just cache the promise objects themselves; they're values like every other and can be memoised! You don't even need to construct a new promise on every call:
var promises = {}
function get(link) {
const b64link = base64.encode(link)
if (promises[b64link] == undefined) {
promises[b64link] = doAsyncOne
.then(() => doAsyncTwo)
.then(() => doAsyncThree);
}
return promises[b64link];
}
That's it!