I'm trying to make multiple database queries using bluebird's Promise.each(). The part where I'm stuck is that I'm not able to handle all the rejections (if multiple promises fail). If I do the same thing using Promise.all() it works fine(It will! because in Promise.all() if 1 promise fails the result is rejected too). My question is:
How should I handle rejections in Promise.each()?
function foo(bar){
return new Promise(resolve, reject){
var query = "elect count(*) from students Where 1=1";//DELIBRATE MISTAKE
connection.query(query, function(err, result){
if(err){reject(err)}resolve(result);
})
}
}
function api(req, res){
var tasks = [];
for(var i = 0; i < 10; i++){
tasks.push(foo(bar));
}
Promise.each(tasks).catch(err=>{return err;});
res.send('message')
}
Response:
Unhandled rejection Error: ER_PARSE_ERROR
You're using Bluebird#each method incorrectly. This method do the following:
Iterate over an array, or a promise of an array, which contains promises (or a mix of promises and values) with the given iterator function with the signature (value, index, length) where value is the resolved value of a respective promise in the input array.
So the first parameter must be an array of promises/values, and the second is a callback which accepts three parameters: value, index, length.
Working example:
let queryAsync = Promise.promisify(connection.query, { context: connection });
function foo(bar) {
var query = 'elect count(*) from students Where 1=1'; // DELIBRATE MISTAKE
return queryAsync(query);
}
function api(req, res){
var tasks = [/* TODO: fill array with taskIds or something else*/];
Promise
.each(tasks, task => foo(task))
.then(() => res.send('message'))
.catch(err => {
console.log(err);
res.status(500).send(err);
});
}
In the example above I use Bluebird#promisify method to promisify callback-style connection.query function. Bluebird already presents the promisification functionality, you shouldn't create your own.
Related
I have a function which returns a list of objects in Javascript, and I'm calling this function from another and attempting to use some of the values from it, but whenever I try to access said values, they come back undefined.
This is my function which generates the list - the idea is that it creates a sqlite3 database if it does not exist, and returns an array containing every event.
function listAllEvents() {
const sqlite3 = require('sqlite3').verbose();
const db = new sqlite3.Database('schedule.db');
const selectionArray = [];
db.serialize(() => {
db.run(`
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
`);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
});
});
return selectionArray;
}
I call this function from another, but when I try to access values from the array, they don't seem to be working and I can't quite figure it out.
function displayUpcomingEvents() {
const events = listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
For example, if I were to create two events in the database, through the console,
events is an Array(2) with indices
- 0: {name: "my_event", date: "2019-06-04", id: "c017c392d446d4b2"}
- 1: {name: "my_event_2", date: "2019-06-04", id: "6d655ac8dd02e3fd"},
events.length returns 0,
and events[0] returns undefined.
Why is this, and what can I do to fix it?
The possible reason why this is happening, is because of the async nature of JS, that means all the console.log statements are getting executed before the successful execution of the listAllEvents() function,
So my suggestion is to try using the promises, and perform all the actions mentioned after the listAllEvents() function only when that function returns a promise.
You can also try making the function async and using await to wait for its successful execution. (Much Smarter Choice will be using async)
Link to ASYNC Functions and Usage
Link to Promises
Also you can check the validity of answer by doing console.log(row) where you are pushing rows to the array. You will observer that the console.log(row) will be executed at the last, after printing events and other log statements.
The problem is that your function is returning the variable before a value is set. The db.serialize function will run asynchronously (outside the normal flow of the program) and the return statement will run immediately after. One thing you can do is use async/await in conjunction with Promise. In this case the the variable results will wait for the promise to be resolved before continuing to the next line.
async function listAllEvents() {
const selectionArray = [];
let promise = new Promise( function (resolve, reject) {
db.serialize(() => {
db.run(
CREATE TABLE IF NOT EXISTS todo (
name text,
date text,
id text primary key
)
);
db.all('SELECT * FROM todo ORDER BY date', [], (err, rows) => {
if (err) {
// add code here to reject promise
throw err;
}
rows.forEach((row) => {
selectionArray.push(row);
});
resolve(selectionArray);// resolve the promise
});
});
});
let results = await promise;
return results;
};
async function displayUpcomingEvents() {
const events = await listAllEvents();
// console.log(events); <-- This line here! In the console, it correctly states the length of the array
// console.log(events.length) <-- This line, however, returns 0. Why?
// console.log(events[0]) <-- This doesn't work either, it just returns "undefined".
for (let i = 0; i < events.length; i += 1) {
$('#upcomingEvents').after('<li>asdf</li>');
}
}
Note here that the displayUpcomingEvents function will also need to be async or you cannot use the await keyword.
Additional reading for Promise keyword MDN: Promise
Additional reading for Async/Await MDN: Asyn/Await
I am trying to use the new async/await in conjunction with mapping over an array of values.
However I am a bit confused, which should have the await keyword infront of it, in this specific scenario.
There isnt much information regarding this topic yet.
I am trying to simply map() over and array of values, which are used in an Axios get() request, which then returns a then() value, which gets added to the array returned by the map() function.
This is my code right now:
async function get_prices (arrayOfDates, crypto, FIAT) {
// I tried this way (return value can be found below)
let arry_of_prices = arrayOfDates.map(async function(date){
let a = await fetch_data(date, crypto, FIAT)
return a
});
// And this way (return value can be found below)
let arry_of_prices = await arrayOfDates.map(date => fetch_data(date, crypto, FIAT));
}
const fetch_data = (timestamp, crypto, FIAT) => {
axios.get(`https://min-api.cryptocompare.com/data/pricehistorical?fsym=${crypto}&tsyms=${FIAT}&ts=${timestamp}`)
.then(function (response) {
return Object.values(response.data).map(key => Object.values(key))[0][0]
})
.catch(function (error) {
console.log(error);
});
}
let arr = [1468965600, 1469052000, 1469138400,1469484000]
get_prices(arr, "BTC", "USD").then(arr => console.log(arr))
The first try returned 4 resolved promises, but the values were undefined.
Whereas the second return an array with 4 undefined.
Does anyone have an idea of how this could be structured, so that the map() waits until the function is resolved and then adds the value to the array?
map doesn't know anything about promises, so when you use an async callback with map, you'll get an array of promises, not values. If you then want a promise that will resolve when all of those promises have resolved, you can use Promise.all.
However, in your case, you don't need an async function, since you're using fetch_data in the callback. You need to modify fetch_data so it returns the promise from axios.get (and remove the catch handler):
const fetch_data = (timestamp, crypto, FIAT) => {
return axios.get(`https://min-api.cryptocompare.com/data/pricehistorical?fsym=${crypto}&tsyms=${FIAT}&ts=${timestamp}`)
.then(function (response) {
return Object.values(response.data).map(key => Object.values(key))[0][0]
});
}
(This is one of the rules of promises: If you're returning the promise, you don't handle rejections from it [unless you want to convert them into different rejections, or if you can usefully convert them to resolutions instead]. If you don't return the promise, you do handle rejections.)
Now you already have a promise, so you use that in the map (no need for async):
async function get_prices (arrayOfDates, crypto, FIAT) {
let arr_of_price_promises = arrayOfDates.map(function(date){
let a = fetch_data(date, crypto, FIAT)
return a
});
return Promise.all(arr_of_price_promises);
}
Notice I removed the await from in front of fetch_data. We return fetch_data's promise directly, then wait for them all.
Or with an arrow function:
async function get_prices (arrayOfDates, crypto, FIAT) {
let arr_of_price_promises = arrayOfDates.map(date => fetch_data(date, crypto, FIAT));
return Promise.all(arr_of_price_promises);
}
I'm new to node and I'm having an issue with resolving an async Promise. My promise isn't resolving and I'm not sure what I did wrong. I'm still having troubles understanding promises and callbacks so any feedback is helpful.
var filterFiles = function(){
return new Promise(function(resolve, reject){
fs.readdir(rootDir, function(err, files){
if(err) return console.log(err);
var task = function(file){
return new Promise(function(resolve, reject){
if(! /^\..*/.test(file)){
fs.stat(rootDir + '/' + file, function(err, stats){
if(stats.isDirectory()){
dirArray.push(file);
console.log(dirArray.length);
resolve(file);
}
if(stats.isFile()){
fileArray.push(file);
console.log(fileArray.length);
resolve(file);
}
})
}
})
};
var actions = files.map(task);
return Promise.all(actions).then(function(resolve, reject){
resolve({dirArray: dirArray, fileArray: fileArray});
});
})
})
}
filterFiles().then(function(data){
console.log(data);
var obj = {
fileArray: fileArray,
dirArray: dirArray
};
res.send(obj);
})
It see at least three errors:
When you hit this if statement if(! /^\..*/.test(file)){ and it does not execute the if block, then the parent promise is never settled.
There is no error handling on fs.stat() so if you get an error on that call, you are ignoring that and will be attempting to use a bad value.
The error handling on your call to fs.readdir() is incomplete and will leave you with a promise that is never settled (when it should be rejected).
For a robust solution, you really don't want to be mixing promises and callbacks in the same code. It leads to the opportunity for lots of mistakes, particularly with error handling (as you can see you had at least three errors - two of which were in error handling).
If you're going to use Promises, then promisify the async operations you are using at the lowest level and use only promises to control your async code flow. The simplest way I know of to promisify the relevant fs operations is to use the Bluebird promise library with its Promise.promisifyAll(). You don't have to use that library. You could instead manually write promise wrappers for the async operations you're using.
Here's a version of your code using the Bluebird promise library:
const Promise = require('bluebird');
const fs = Promise.promisifyAll(require('fs'));
function filterFiles() {
return fs.readdirAsync(rootDir).then(function(files) {
let fileArray = [];
let dirArray = [];
// filter out entries that start with .
files = files.filter(function(f) {
return !f.startsWith(".");
});
return Promise.map(files, function(f) {
return fs.statAsync(f).then(function(stats) {
if (stats.isDirectory()) {
dirArray.push(f);
} else {
fileArray.push(f);
}
});
}).then(function() {
// make the resolved value be an object with two properties containing the arrays
return {dirArray, fileArray};
});
});
}
filterFiles().then(function(data) {
res.json(data);
}).catch(function(err) {
// put whatever is appropriate here
res.status(500).end();
});
This was rewritten/restructured with these changes:
Use promises for all async operations
Fix all error handling to reject the returned promise
Filter out files starting with a . synchronously before processing any files (simplifies async processing).
Use Promise.map() to process an array of values in parallel.
In the filterFiles().then() handler, handle errors
You can't res.send() a Javascript object so I used res.json(data) instead (though I'm not sure what exactly you really want to send).
Replace regex comparison with more efficient and simpler to understand .startsWith().
If you don't want to use the Bluebird promise library, you can make your own promise wrappers for the fs methods you use like this:
fs.readdirAsync = function(dir) {
return new Promise(function(resolve, reject) {
fs.readdir(dir, function(err, data) {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
fs.statAsync = function(f) {
return new Promise(function(resolve, reject) {
fs.stat(f, function(err, data) {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
function filterFiles() {
return fs.readdirAsync(rootDir).then(function(files) {
let fileArray = [];
let dirArray = [];
// filter out entries that start with .
files = files.filter(function(f) {
return !f.startsWith(".");
});
return Promise.all(files.map(function(f) {
return fs.statAsync(f).then(function(stats) {
if (stats.isDirectory()) {
dirArray.push(f);
} else {
fileArray.push(f);
}
});
})).then(function() {
// make the resolved value be an object with two properties containing the arrays
return {dirArray, fileArray};
});
});
}
filterFiles().then(function(data) {
res.json(data);
}).catch(function(err) {
res.status(500).end();
});
The main issue you are having is that outer-most Promise is not resolved or rejected. You can fix this by resolving your Promise.all instead of returning it.
resolve(
Promise.all(actions)
.then(function(resolvedTasks){
// ... next potential issue is here
return {dirArray: dirArray, fileArray: fileArray}
})
);
(I know, kind of awkward-looking right?)
Next, your return value after the Promise.all resolves is a little weird. In the task function, you're pushing items onto dirArray and fileArray, but they are not declared or assigned in your snippet. I will assume that they are in-scope for this code. In this case, you just need to return your desired object.
Additionally, to make your async code more readable, here are some tips:
try not to mix callbacks with Promises
use a Promise library to promisify any code limited to callbacks. Example: bluebird's promisifyAll
avoid nesting callbacks/promises when possible
I am having a problem trying to make my for loop behave synchronously when I am making an asynchronous call in each iteration of the loop.
Here is my code:
pipeline: function(userContext, options, done){
var orderData = [];
options.data.forEach(function(store, index){
store.orders.forEach(function(order){
shopify.getPipeline(userContext, {'order':order,'storeId':index}, function(result){
var snapshotId = "30775bf1bb854c5d84c9c2af37bc8fb0";
var resourceToQuery = config.structrUrls.getUrl("ConfigurationSnapshot") + '/' + snapshotId ;
var requestOptions = {
method: "GET",
timeout: 8000
};
requestify.request(resourceToQuery, requestOptions)
.then(function(snapshotResult){
result.snapshots = snapshotResult.getBody().result;
result.snapshots.configurationPayload = JSON.parse(snapshotResult.getBody().result.configurationPayload);
orderData.push(result);
})
.catch(function(err){
console.log (err);
done(err);
});
});
});
});
done(null, orderData);
}
I understand the problem here, but do not know how to remedy it. Let me explain the function:
options.data contains an array of stores, and each store contains an array of orders. For each order, I am calling shopify.getPipeline() for pipeline data (this is a synchronous operation), and in the callback I make a requestify request (a node module used for making http requests) for snapshot data, which I want to append to the result before pushing it onto my "orderData" array. When this all completes, I am calling "done", a callback function, with my orderData. As you can see, since requestify calls are asynchronous, done is called before any data is added to the orderData array.
I think I need to use some kind of promise in order to guarantee the result before calling done, but I have been unsuccessful in implementing a promise into this function. In the documentation for q, it seems like the function I would want to use is promise.all(), which 'Returns a promise that is fulfilled with an array containing the fulfillment value of each promise, or is rejected with the same rejection reason as the first promise to be rejected'. I'm failing to see how to translate my forEach loop into an array of promises. I was also looking at the async parallel function and ran into the same problem regarding the array of functions.
Any help is greatly appreciated. Thanks!
To construct an array of Promises for use in Promise.all, you can map across the array of stores, and again across the array of orders, returning a Promise for each order which will be resolved or rejected based on the result of requestify.request. Merging the resulting nested array gives you a single array of promises which can then be passed to Promise.all.
Using your example:
pipeline: function(userContext, options, done){
var nestedPromises = options.data.map.forEach(function(store, index){
return store.orders.map(function(order){
return new Promise(function(resolve, reject){
shopify.getPipeline(userContext, {'order':order,'storeId':index}, function(result){
var snapshotId = "30775bf1bb854c5d84c9c2af37bc8fb0";
var resourceToQuery = config.structrUrls.getUrl("ConfigurationSnapshot") + '/' + snapshotId ;
var requestOptions = {
method: "GET",
timeout: 8000
};
requestify.request(resourceToQuery, requestOptions)
.then(function(snapshotResult){
result.snapshots = snapshotResult.getBody().result;
result.snapshots.configurationPayload = JSON.parse(snapshotResult.getBody().result.configurationPayload);
resolve(result);
})
.catch(function(err){
reject(err);
});
});
});
});
});
// Flatten nested array.
var promises = Array.prototype.concat.apply([], nestedPromises);
Promise.all(promises).then(function(orderData){
done(null, orderData);
}).catch(function(err){
done(err);
});
}
I have been reading many posts on how to chain asynchronous functions but I just can't seem to get it right!
As the title indicates. I am trying to chain mongoskin database calls together, so that i can gather all the information in chunks and then finally send the accumulated result in the response.
I have the object user as :
var User = {
username: 'someusername',
accounts: [{name: 'account_1'}, {name: 'account_2'}]
}
For each of the accounts I need to gather data and then send the accumulated data in the response. So i am using the following for loop to iterate over the accounts:
var promise = require('bluebird');
var db = require('mongoskin').db('mongodb://localhost/someDB');
for(var x in user.accounts){
//Fetch account data
user.accounts[x].accountData = fetchAccountData(user.accounts[x].name);
}
//Finally send the collected response
response.send(user);
And the function fetchAccountData looks like the following:
function fetchAccountData(screen_id){
db.collection('master')
.aggregate([
{$match: {screen_id: screen_id}}
], function(err, res){
if(err)
return null;
else{
console.log('Done', screen_id);
return res;
}
});
}
How can i chain this to have the following algorithm:
start:
for each account:
fetchDataForAccount
Finally:
Send Response
Your algorithm can be achieved using the following code:
var Promise = require('bluebird');
var mongo = require('mongoskin'), db;
Promise.promisifyAll(mongo.Collection.prototype);
db = mongo.db('mongodb://localhost/someDB');
Promise.all(user.accounts.map(function(acct) {
return fetchAccountData(acct.name).then(function(data) {
acct.accountData = data;
});
}))
.then(function() {
response.send(user);
})
.catch(function(err) {
// handle error
});
function fetchAccountData(screen_id){
return db
.collection('master')
.aggregateAsync([
{$match: {screen_id: screen_id}}
]);
}
EDIT: Here's a breakdown of the code
The first thing you need to do is ensure that aggregate returns a Promise instead of using a continuation (e.g. callback). You can do this by using bluebird's amazing promisification abilities :) Here we use it on mongo.Collection.prototype so that when collection() is called it will return a promise-capable collection instance. Then we have fetchAccountData return the promise returned by aggregateAsync so the client has a way of knowing when that promise is resolved.
Next, we map over each account in accounts and return a promise which will be fulfilled once the account data is fetched and it has been assigned to the account object. We then use Promise.all which will return a promise that is fulfilled "when all the items in the array are fulfilled" (from the docs).
Finally, we have to use then() to "wait" until the promise returned from all has resolved, and the finally send back the response with the complete user object.