Develop Reference

Is there a way to use async/await with this algolia example code?

This code i am about to put here does not work in the respect that my graphQL server always receives an empty array back... i know this is because the call completes before the data is available.. i don't have much experience with promises so i am wondering how i can get it to work, i played with a bunch of different combinations to attempt to await the response so that i could just pass back an array of id's to the playground... the browsewr.on(end) part works fine and the proper data shows in the console? Anyone care to shed some light for me?
const browser = await lookbookIndex.browseAll();
let hits = [];
let returnArray = [];
browser.on('result', (content) => {
hits = hits.concat(content.hits);
});
browser.on('end', () => {
console.log('Finished!');
console.log('We got %d hits', hits.length);
returnArray = hits ? hits.map(a => a.objectID) : [];
});
browser.on('error', err => err);
return returnArray;

Async syntax automatically wraps everything to Promise instance.
But you also can return Promise instance manually. It accepts callback with resolve & reject arguments. And you can call resolve to resolve the promise:
async function search() {
const browser = await lookbookIndex.browseAll();
return new Promise((resolve, reject) => {
let hits = [];
browser.on('result', (content) => {
hits = hits.concat(content.hits);
});
browser.on('end', () => {
console.log('Finished!');
console.log('We got %d hits', hits.length);
resolve(hits ? hits.map(a => a.objectID) : [])
});
browser.on('error', err => reject(err));
})
}
And you can await the promise:
await new Promise(...)
Similar questions:
Return data from function

Using NodeJS and MySQL, await/async functions sometimes not waiting for query to finish

I have two async functions,
async function sqlPushSteamappid(appId){
let tmp;
let sql='INSERT INTO cleetusbot.tmp (appId) VALUES ('+appId+');';
tmp = new Promise((res, rej) => {
global.pool.query(sql, function (err, results, fields) {
if(err){
console.log(err);
}
});
});
return await tmp;
}
and
async function sqlGetSteamNames(){
let tmp;
let sql='SELECT * FROM cleetusbot.steamGames INNER JOIN tmp ON cleetusbot.steamGames.appId = cleetusbot.tmp.appId;';
tmp = new Promise((res, rej) => {
global.pool.query(sql,function (err, results, fields) {
if(err){
console.log(err);
}
res(results);
});
});
await tmp;
return tmp;
}
Both return what I need, however most of the time when they are called the MySQL queries either don't fully return completely, or don't return an answer at in within the promise. Am I missing something in my code or do I have to make the MySQL timeout longer?
Here is how im calling them in my code:
for(let i = 0; i < gameList.length; i++){
sqlPushSteamappid(gameList[i]);
}
//sometimes does not return anything
let steamNameObj = await sqlGetSteamNames();

First of all you should understand why things happen the way they do in your snippet.
Don't think of your mysql at all, those asynchronous calls act like any other; also take into account that a for iteration is asynchronous itself which means that all the function calls inside for will be called without waiting for the previous to be finished (even if you await them). sqlGetSteamNames() will also be called right after all your sqlPushSteamappid() are called (again, without waiting for them to finish).
You need to get rid of the uncertainty of execution order and you can do that having your sqlPushSteamappid() return a promise, and use Promise.all (docs here) to coordinate your requests. So first feed all your sqlPushSteamappid() in Promise.all and then after it returns, you can call await sqlGetSteamNames() as you do in your code.
Working example:
const promises = [
new Promise((resolve, reject) => {
setTimeout(() => resolve("3 sec passed"), 3000);
}),
new Promise((resolve, reject) => {
setTimeout(() => resolve("2 sec passed"), 2000);
}),
new Promise((resolve, reject) => {
setTimeout(() => resolve("4 sec passed"), 4000);
})
];
const ending = async () => { setTimeout(() => { console.log("2 sec to finish") }, 1000); }
const start = async () => {
const results = await Promise.all(promises);
console.log(results);
await ending();
}
start();

Javascript API Queue in forEach loop

Summary of Objective
I need to implement an API queue in my Node.js backend for API calls. The API rate limit I need to adhear to is 1 request every 2 seconds and it's a hard limit. I'm making my API call inside of a forEach loop since I need to do one API call for each user.
I've found a lot of articles online about how to create a queue but they mostly involve adding API calls to an array so I'm not sure how to implement a queue in this situation.
Any help would be greatly apprecaited and I can share more code if it's help.
Code
async function refreshStats() {
try {
// get list of all fortnite users
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
fnUserList.forEach(async fnUser => {
//make API call. apiCall is a function I created to make the API call and format the response
const { lifeStats, statsEqual } = await apiCall(
fnUser.fnUserPlatform
);
//execute other functions with apiCall response
});
} catch (err) {
console.error("error in refreshStats", err);
}
}

If i get it correctly. You can take advantage of generator functions and combine it with setInterval. You can make make a queue function that enqueues its items in specified intervals.
Create a generator function basically makes an apiCall and pauses
async function* queueGenerator(userList) {
for (let fnUser of userList) {
const result = {lifeStats, statsEqual} = await apiCall(fnUser.fnUserPlatform);
yield result;
}
}
Then in your method create a queue and enqueue items with setInterval
async function refreshStats() {
try {
// get list of all fortnite users
let handle;
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
const queue = queueGenerator(fnUserList);
const results = [];
handle = setInterval(async () => {
const result = await queue.next();
results.push(result.value);
if (results.length === users.length) clearInterval(handle);
}, 2000);
} catch (err) {
console.error("error in refreshStats", err);
}
}
Also there is another way which is to use setTimeout combined with Promises. which involves creating promises that resolves in setTimeOut with enough of delays.
async function refreshStatsV2() {
const fnUserList = await Users.find({}, "_id fnUserPlatform");
const promises = fnUserList.map((fnUser, ix) => (
new Promise(resolve =>
setTimeout(async() => {
const result = {
lifeStats,
statsEqual
} = await apiCall(ix.fnUserPlatform);
resolve(result);
}, ix * 2000) // delay every next item 2sec
)));
const result = await Promise.all(promises); // wait all
console.log(result);
}

Error: deadline-exceeded - when calling a firebase callable cloud function (onCall). onRequest works fine

I'm getting a "deadline-exceeded" error on the frontend when calling a firebase callable cloud function (onCall).
I know that I have to return a Promise so the function knows when to clean itself, but it is still not working.
After 60 seconds, "deadline-exceeded" is throw to the frontend but the function keeps running on the server and finish with success. All batch operations are written to the firestore.
10:37:14.782 AM
syncExchangeOperations
Function execution took 319445 ms, finished with status code: 200
10:36:57.323 AM
syncExchangeOperations
Function execution started
10:36:57.124 AM
syncExchangeOperations
Function execution took 170 ms, finished with status code: 204
10:36:56.955 AM
syncExchangeOperations
Function execution started
async function syncBinanceOperations(
userId,
userExchange,
userExchangeLastOperations,
systemExchange
) {
try {
const client = Binance({
apiKey: userExchange.apiKey,
apiSecret: userExchange.privateKey
});
const batch = admin.firestore().batch();
const lastOperations = userExchangeLastOperations
? userExchangeLastOperations
: false;
const promises = [];
promises.push(
syncBinanceTrades(client, lastOperations, userId, systemExchange, batch)
);
promises.push(
syncBinanceDeposits(client, lastOperations, userId, systemExchange, batch)
);
promises.push(
syncBinanceWhitdraws(
client,
lastOperations,
userId,
systemExchange,
batch
)
);
promises.push(
updateUserExchange(userId, userExchange.id, {
lastSync: moment().format('x')
})
);
await Promise.all(promises);
return batch.commit();
} catch (error) {
return handleErrors(error);
}
}
exports.syncExchangeOperations = functions.https.onCall(
async (data, context) => {
try {
userAuthenthication(data.userId, context.auth);
let user = await getUser(data.userId);
if (!user.plan.benefits.syncExchanges) {
throw 'Operação não autorizada para o plano contratado';
}
let userExchange = await getUserExchange(data.userId, data.exchangeId);
let response = await Promise.all([
getUserLastOperations(data.userId, userExchange.exchangeId),
getSystemExchange(userExchange.exchangeId)
]);
let userExchangeLastOperations = response[0];
let systemExchange = response[1];
switch (systemExchange.id) {
case 'binance':
return syncBinanceOperations(
user.id,
userExchange,
userExchangeLastOperations,
systemExchange
);
}
} catch (error) {
return handleErrors(error);
}
}
);
It works fine if I change this function to a HTTP request. It waits the function to finish and returns.
exports.syncExchangeOperations = functions
.runWith(runtimeOpts)
.https.onRequest((req, res) => {
return cors(req, res, async () => {
try {
let auth = await admin.auth().verifyIdToken(req.get('Authorization').split('Bearer ')[1]);
let userExchange = await getUserExchange(
auth.uid,
req.query.exchangeId
);
let response = await Promise.all([
getUserLastOperations(auth.uid, userExchange.exchangeId),
getSystemExchange(userExchange.exchangeId)
]);
let userExchangeLastOperations = response[0];
let systemExchange = response[1];
switch (systemExchange.id) {
case 'binance':
await syncBinanceOperations(
auth.uid,
userExchange,
userExchangeLastOperations,
systemExchange
);
}
res.status(200).send();
} catch (error) {
res.status(401).send(handleErrors(error));
}
});
});

The "deadline-exeeded" that you encountered is an error thrown by the Firebase Javascript library on the client (not the function itself). The Firebase docs are lacking documentation o how to use functions.runWithOptions() on a callable function. For some reason the functions().httpsCallable() has a built in timeout on the client side.
So if you use this on your Node.js function:
exports.testFunction = functions.runWith({ timeoutSeconds: 180 }).https.onCall(async (data, ctx) => {
// Your Function Code that takes more than 60second to run
});
You need to override the buit in Javascript Library timeout on the client like this:
let testFunction = firebase.functions().httpsCallable("testFunction", {timeout: 180000});
I don't know what is the purpose of the built in timeout on the client, for me it has no purpose since it doesn't even stop the execution of the function on the server. But it must be there for some internal reasons.
Notice the Node.js timeoutSeconds is in seconds and the timeout option on the client library is in milliseconds.

"Deadline exceeded" means that the function invocation timed out from the perspective of the client. The default is 60 seconds.
Try increasing the timeout on both the client and function so that it has time to complete before the client timeout is reached. You can do this by specifying it in an HttpsCallableOptions object.
Also try returning something other than batch.commit(). Whatever that function return will be serialized and sent to the client, which could cause problems. Instead, just await batch.commit() then return something predictable, like a plain JavaScript object.
See the API documentation for information on setting the timeout:
https://firebase.google.com/docs/reference/js/firebase.functions.Functions#https-callable
https://firebase.google.com/docs/reference/js/firebase.functions.HttpsCallableOptions.html#timeout

How can I wait until the whole batch of requests are made and the promises are solved before I send the data on the callback function?

I have an object with 5 items, each of them will send an http request 3 times.
I save that in a
var promise
var promise1
var promise2
In the end, I am resolving (trying) the promises using
Promise.all([promise, promise1, promise2]]
And then I send the data to a callback function.
I am using array.map() to do my task on that array, all the requests and Promise.all are happening inside that.
How can I wait until the whole batch of requests are made and the promises are solved before I send the data on the callback function?
async function requestJahrStatistic(jahreStatistic, callback){
Promise.all(
jahreStatistic.map(async (item) => {
var periods = getReportingPeriod(item.period);
connection.statistic_id = item.statistic_id;
connection.reporting_period = periods.prevYear;
var promise = new Promise(function(resolve, reject) {
sendHTTPRequest(item, function(result) {
resolve(result);
});
});
connection.reporting_period = periods.actualYear;
var promise1 = new Promise(function(resolve, reject) {
sendHTTPRequest(item, function(result) {
resolve(result);
});
});
connection.reporting_period = periods.nextYear;
var promise2 = new Promise(function(resolve, reject) {
sendHTTPRequest(item, function(result) {
resolve(result);
});
});
Promise.all([promise, promise1, promise2]).then(async resolved => {
var res = await resolved
return res
});
})
).then(async resolved =>{
var resp = await resolved;
callback(resp)
});
}
This was the last thing I tried before writing the question

There are several problems with that code:
requestJahrStatistic shouldn't be async if it reports its results by calling a callback
You use this pattern in a couple of places:
.then(async resolved => {
var res = await resolved
return res
});
That serves no purpose (unless... see #5) and can be completely removed.
There's no reason for the map callback to be async, as you're not using await within it.
You're repeating your logic wrapping sendHTTPRequest in a promise, and failing to handle errors in it. Don't repeat yourself, make a function for that.
It looks like connection.statistic_id and connection.reporting_period are used by the HTTP requests somehow. They shouldn't be, that's spooky action at a distance. :-) But if they are, then none of this can be in parallel since you have to wait for a request using a given statistic_id and reporting_period to complete before you can start the next.
You're not handling errors.
If I assume connection.reporting_period is used by the HTTP requests, that means they can't overlap, so none of this can be in parallel and you can't use Promise.all for it. You'd need something like:
If connection.reporting_period isn't used by the HTTP requests, this can all be parallel:
function sendHTTPRequestP(item) {
return new Promise((resolve, reject) => {
sendHTTPRequest(item, result => {
if (/*an error occurred*/) {
reject(new Error(/*...*/));
} else {
resolve(result);
}
});
})
}
// Completely serial because of spooky action at a distance with
// `connection.statistic_id` and `connection.reporting_period`
function requestJahrStatistic(jahreStatistic, callback) {
Promise.resolve(async () => {
const results = [];
for (const item of jahreStatistic) {
const periods = getReportingPeriod(item.period);
connection.statistic_id = item.statistic_id;
connection.reporting_period = periods.prevYear;
const result1 = await sendHTTPRequestP(item);
connection.reporting_period = periods.actualYear;
const result2 = await sendHTTPRequestP(item);
connection.reporting_period = periods.nextYear;
const result3 = await sendHTTPRequestP(item);
results.push([result1, result2, result3]);
}
return results;
})
.then(callback)
.catch(error => {
// Handle/report error, call `callback` with the appropriate error flag
});
}
Or something along those lines. Note that what callback will receive is an array of arrays. The outer array will have as many entries as jahreStatistic; each of those entries will be an array of the results of the three HTTP calls.
If you can change things so that each operation takes arguments rather than spooky action at a distance (I see that sendHTTPRequest already has the item so can presumably get statistic_id from it, so we just have to pass period), you can make things parallel:
function sendHTTPRequestP(item, reporting_period) {
return new Promise((resolve, reject) => {
sendHTTPRequest(item, reporting_period, result => {
if (/*an error occurred*/) {
reject(new Error(/*...*/));
} else {
resolve(result);
}
});
})
}
function requestJahrStatistic(jahreStatistic, callback){
Promise.all(
jahreStatistic.map((item) => {
const periods = getReportingPeriod(item.period);
return Promise.all([
sendHTTPRequestP(item, periods.prevYear),
sendHTTPRequestP(item, periods.actualYear),
sendHTTPRequestP(item, periods.nextYear)
]);
})
)
.then(callback)
.catch(error => {
// Handle/report error, call `callback` with the appropriate error flag
});
}
Or something along those lines.