Summary of Objective
I need to implement an API queue in my Node.js backend for API calls. The API rate limit I need to adhear to is 1 request every 2 seconds and it's a hard limit. I'm making my API call inside of a forEach loop since I need to do one API call for each user.
I've found a lot of articles online about how to create a queue but they mostly involve adding API calls to an array so I'm not sure how to implement a queue in this situation.
Any help would be greatly apprecaited and I can share more code if it's help.
Code
async function refreshStats() {
try {
// get list of all fortnite users
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
fnUserList.forEach(async fnUser => {
//make API call. apiCall is a function I created to make the API call and format the response
const { lifeStats, statsEqual } = await apiCall(
fnUser.fnUserPlatform
);
//execute other functions with apiCall response
});
} catch (err) {
console.error("error in refreshStats", err);
}
}
If i get it correctly. You can take advantage of generator functions and combine it with setInterval. You can make make a queue function that enqueues its items in specified intervals.
Create a generator function basically makes an apiCall and pauses
async function* queueGenerator(userList) {
for (let fnUser of userList) {
const result = {lifeStats, statsEqual} = await apiCall(fnUser.fnUserPlatform);
yield result;
}
}
Then in your method create a queue and enqueue items with setInterval
async function refreshStats() {
try {
// get list of all fortnite users
let handle;
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
const queue = queueGenerator(fnUserList);
const results = [];
handle = setInterval(async () => {
const result = await queue.next();
results.push(result.value);
if (results.length === users.length) clearInterval(handle);
}, 2000);
} catch (err) {
console.error("error in refreshStats", err);
}
}
Also there is another way which is to use setTimeout combined with Promises. which involves creating promises that resolves in setTimeOut with enough of delays.
async function refreshStatsV2() {
const fnUserList = await Users.find({}, "_id fnUserPlatform");
const promises = fnUserList.map((fnUser, ix) => (
new Promise(resolve =>
setTimeout(async() => {
const result = {
lifeStats,
statsEqual
} = await apiCall(ix.fnUserPlatform);
resolve(result);
}, ix * 2000) // delay every next item 2sec
)));
const result = await Promise.all(promises); // wait all
console.log(result);
}
Related
I have a Node.js AWS Lambda function created via the serverless framework. I have multiple helper functions inside it. I am having an issue with one of them due to being async. The function runs and logs out all parts I put comments next to however it doesn't update callDuration. I think that the code is having an issue due to async where it finishes in the wrong order. My goal is to be able to return the callDuration to my main function for further processing. How can I get all code to process/run and be able to meet my goal and have the code run in the right order
Here is the function:
const callAggregate = async (billingData, billingDB) => {
const accountSid = process.env.TWILIO_ACCOUNT_SID
const authToken = process.env.TWILIO_AUTH_TOKEN
const client = require('twilio')(accountSid, authToken)
// Setup model
const Billing = billingDB.model('Billing')
await Billing.findOne({_id: billingData._id}).exec().then(bill => {
const callArray = bill.callSid
console.log(bill) // This logs out
let callDuration = 0
for (const call of callArray) {
console.log(call) // This logs out
client.calls(call)
.fetch()
.then(callDetails => {
console.log(callDetails) // This logs out
callDuration += callDetails.duration
})
}
console.log(`Billing for ${callDuration} minutes of voice calling for ${billingData._id}`) // This logs out
Billing.findOneAndUpdate(
{_id: billingData._id},
{ $inc: { call_duration: callDuration }, callSid: []},
(err, doc) => {
if(err) {
console.log(err)
}
}
)
return callDuration
})
}
This is a case of mixing and matching promises with plain callbacks and mixing await with .then(), both of which make proper flow-control and error handling management difficult.
Inside your function which is async and uses await in some places, you also have a promise you are not awaiting (which means it runs open loop and nothing waits for it) and you have a database function that is using a plain callback, not the promise interface so nothing waits for it either.
More specifically, nothing is waiting for this:
client.calls(call).fetch()
So, because of not waiting for the .fetch() to finish, you were attempting to use the variable callDuration before the code was done modifying that variable (giving you the wrong value for it).
Similarly, nothing is waiting for Billing.findOneAndUpdate(...) to complete either.
A clean solution is to switch everything over to promises and await. This involves, using only promises with your database (no plain callbacks) and converting the .then() handlers into await.
async function callAggregate(billingData, billingDB) {
const accountSid = process.env.TWILIO_ACCOUNT_SID
const authToken = process.env.TWILIO_AUTH_TOKEN
const client = require('twilio')(accountSid, authToken)
// Setup model
const Billing = billingDB.model('Billing')
let bill = await Billing.findOne({ _id: billingData._id }).exec();
const callArray = bill.callSid
console.log(bill) // This logs out
let callDuration = 0
for (const call of callArray) {
console.log(call) // This logs out
let callDetails = await client.calls(call).fetch();
console.log(callDetails) // This logs out
callDuration += callDetails.duration
}
console.log(`Billing for ${callDuration} minutes of voice calling for ${billingData._id}`) // This logs out
let doc = await Billing.findOneAndUpdate({ _id: billingData._id }, { $inc: { call_duration: callDuration }, callSid: [] }).exec();
return callDuration
}
I'm trying to limit the number of requests I send to an API.
I'm using Limiter and it's working just like I need, the only issue is that I can't find a way to use it with await (I need all the responses before rendering my page)
Can someone give me a hand with it?
Btw the Log returns a boolean.
const RateLimiter = require('limiter').RateLimiter;
const limiter = new RateLimiter(50, 5000)
for (let i = 0; i < arrayOfOrders.length; i++) {
const response = limiter.removeTokens(1, async (err, remainingRequests) => {
console.log('request')
return await CoreServices.load('updateOrder', {
"OrderNumber": arrayOfOrders[i],
"WorkFlowID": status
})
})
console.log('response', response)
}
console.log('needs to log after all the request');
this is loggin:
response true
response true
response false
needs to log after all the request
request
request
request
...
Promisifying .removeTokens will help, see if this code works
const RateLimiter = require('limiter').RateLimiter;
const limiter = new RateLimiter(50, 5000);
const tokenPromise = n => new Promise((resolve, reject) => {
limiter.removeTokens(n, (err, remainingRequests) => {
if (err) {
reject(err);
} else {
resolve(remainingRequests);
}
});
});
(async() => { // this line required only if this code is top level, otherwise use in an `async function`
const results = await Promise.all(arrayOfOrders.map(async (order) => {
await tokenPromise(1);
console.log('request');
return CoreServices.load('updateOrder', {
"OrderNumber": order,
"WorkFlowID": status
});
}));
console.log('needs to log after all the request');
})(); // this line required only if this code is top level, otherwise use in an `async function`
explanation
Firstly:
const tokenPromise = n => new Promise((resolve, reject) => {
limiter.removeTokens(n, (err, remainingRequests) => {
if (err) {
reject(err);
} else {
resolve(remainingRequests);
}
});
});
promisifies the limiter.removeTokens to use in async/await - in nodejs you could use the built in promisifier, however lately I've had too many instances where that fails - so a manual promisification (I'm making up a lot of words here!) works just as well
Now the code is easy - you can use arrayOfOrders.map rather than a for loop to create an array of promises that all run parallel as much as the rate limiting allows, (the rate limiting is done inside the callback)
await Promise.all(... will wait until all the CoreServices.load have completed (or one has failed - you could use await Promise.allSettled(... instead if you want)
The code in the map callback is tagged async so:
await tokenPromise(1);
will wait until the removeTokens callback is called - and then the request
return CoreServices.load
is made
Note, this was originally return await CoreServices.load but the await is redundant, as return await somepromise in an async function is just the same as return somepromise - so, adjust your code too
The following code is for my FCM function where I am listening to firestore the getting tokens before constructing payload to send. Every time its sent the system logs that the tokens are empty. How can I make sure its not empty when sending the fcm?
let functions = require('firebase-functions');
let admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.sendNotification =functions.firestore.document('chatrooms/{chatRoomId}/messages/{messageId}')
.onWrite((snap, context) => {
let message = snap.after.data().messageBody;
let messageSender = snap.after.data().senderName;
let messageUserId = snap.after.data().userId;
let chatRoomId = context.params.chatRoomId;
let tokens = [];
let chatRoomRef = admin.firestore().collection("chatrooms").doc(chatRoomId);
return admin.firestore().runTransaction(t => {
return t.get(chatRoomRef)
.then(chatroom => {
let usersArray = chatroom.data().chatMembers;
usersArray.forEach(user_id => {
let userIdRef = admin.firestore().collection("tokens").doc(user_id);
return t.get(userIdRef).then(doc => {
if (doc.exists) {
let user_token = doc.data().token;
functions.logger.log('token: ', token);
tokens.push(user_token);
}
}).catch(err => {
functions.logger.error(err);
})
});
});
}).then(() => {
//The transaction has run successfully, we expect tokens array to not be empty
functions.logger.log("Construction the notification message.");
const payload = {
data: {
data_type: "data_type_chat_message",
title: "Tuchat",
message: message,
sender_id: messageUserId,
sender_name: messageSender,
chatRoom_id: chatRoomId
}
};
const options = {
priority: "high",
timeToLive: 60 * 60 * 24
};
return admin.messaging().sendToDevice(tokens, payload).catch(err => {
functions.logger.error(err);
});
}).catch(err => {
functions.logger.error('Transaction error: ', err);
})
});
Also before trying transactions it was returning empty tokens.
The problem is because of the way you're dealing with promises inside the forEach loop. The loop will not wait for promises to be resolved for the code inside it. It is currently just iterating as fast as possible over each user ID and not waiting for the queries to complete inside it. That means the code will continue before tokens can get populated.
You should instead collect each of the promises inside the loop and use Promise.all() to wait for the entire batch, then return the promise from that to indicate when all the work is complete. That will ensure that tokens contains everything you want before the next then in the chain is executed.
The general form of the code will be like this:
.then(() => {
const promises = []
collection.forEach(item => {
const promise = doSomeWork()
promises.push(promise)
})
return Promise.all(promises)
})
.then(() => {
// this will continue only after all promises are resolved
})
See also:
How to use promise in forEach loop of array to populate an object
Node JS Promise.all and forEach
I'm trying to understand how to chain two different API calls including one with a for loop in a 'notes' Vue component. I have a really basic experience of promises and I'm looking to improve.
I'm making a first API call to get all the notes and pushing them into an array using a Vuex mutation. During that first API call I'm also mapping the different users emails into an Object.
Using this mapped object, I'm making a second API call inside a for loop to get all the users avatars.
Here's what the first API call looks like :
getAllNotesAPI(entity) {
noteService.getNotes(entity)
.then((response) => {
if (response.data.length === '0') {
// Set hasData to false if the response is 0
this.hasData = false;
} else {
// Push data into the note array using a store mutation
this.setAllNotes(response.data);
}
// Mapping all users emails into 'userEmails'
this.userEmails = [...new Set(response.data.map(x => x.userEmail))];
// Calling my second API call here to get all the avatars associated with these emails
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
})
.catch((error) => {
console.log(error);
})
.finally(() => {
this.endLoader('notes');
});
},
this.getAvatarAPI is the second API call which looks like this :
getAvatarAPI(login) {
userService.getAvatar(login)
.then((response) => {
let newAvatar = {
userEmail: login,
picture: response.data.picture
};
// Push the response into a userAvatar Object using a store mutation
this.setUserAvatar(newAvatar);
}).catch((error) => {
console.log(error)
})
},
I've tried using async / await but couldn't figure out how to bind this inside of an async function (this.getAvatarAPI(this.userEmails)) was undefined, I've tried chaining using multiples then but couldn't figure out how to : get all my notes then all my avatars then end the 'note' loader once both those API calls are done.
If any of you could give me some pointers or the beginning of an answer that would be truly appreciated !
First whilst not related to your problem, avoid for loop when non necessary:
Do you need the i index?
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
no. You need the userMail. Then
this.userEmails.forEach(userMail => {
this.getAvatarAPI(userEmail)
})
Now, to synchronize promises, you need to return a promise (let's not talk about async yet)
make getAvatarAPI return a promise
getAvatarAPI(login) {
return userService.getAvatar(login).then(blabla) // notice the return here
retrieve the promises of getAvatar API
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return after all promises have fulfilled
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return Promise.all(promises)
On a side note with async/await
If you use it you are not forced anymore to write return, you need to write async/await though
The underlying idea stay the same. Specifying the async keywords says that your function will return a promise-like.
e.g
async function p () {
return 5
}
p.then(x => console.log(x)) // does print 5 even though we didn't explicitely write return Promise.resolve(5)
Now you have to ensure you await the async function when you call it:
getAvatarAPI: async login => {
return userService.getAvatar(login).then(blabla)
}
// DO NOT do it
this.userEmails.forEach(userMail => {
return await this.getAvatarAPI(userEmail)
})
In forEach loop above, you will do your getAvatarAPI call in sequence because await "stops" iterating as long as getAvatarAPI has not resolved.
The proper way would be
getAllNotesAPI: async entity => {
try { // notice the necesary try-catch block
const response = await noteService.getNotes(entity)
blabla
let promises = this.userEmails.map(userMail => {
return this.getA...
})
let result = await Promise.all(promises)
// eventually return result, or simply await Promise... without lefthand-side assignment
} catch (error) {
console.log(error);
}
console.log(this.end('loader'))
}
I'm getting a "deadline-exceeded" error on the frontend when calling a firebase callable cloud function (onCall).
I know that I have to return a Promise so the function knows when to clean itself, but it is still not working.
After 60 seconds, "deadline-exceeded" is throw to the frontend but the function keeps running on the server and finish with success. All batch operations are written to the firestore.
10:37:14.782 AM
syncExchangeOperations
Function execution took 319445 ms, finished with status code: 200
10:36:57.323 AM
syncExchangeOperations
Function execution started
10:36:57.124 AM
syncExchangeOperations
Function execution took 170 ms, finished with status code: 204
10:36:56.955 AM
syncExchangeOperations
Function execution started
async function syncBinanceOperations(
userId,
userExchange,
userExchangeLastOperations,
systemExchange
) {
try {
const client = Binance({
apiKey: userExchange.apiKey,
apiSecret: userExchange.privateKey
});
const batch = admin.firestore().batch();
const lastOperations = userExchangeLastOperations
? userExchangeLastOperations
: false;
const promises = [];
promises.push(
syncBinanceTrades(client, lastOperations, userId, systemExchange, batch)
);
promises.push(
syncBinanceDeposits(client, lastOperations, userId, systemExchange, batch)
);
promises.push(
syncBinanceWhitdraws(
client,
lastOperations,
userId,
systemExchange,
batch
)
);
promises.push(
updateUserExchange(userId, userExchange.id, {
lastSync: moment().format('x')
})
);
await Promise.all(promises);
return batch.commit();
} catch (error) {
return handleErrors(error);
}
}
exports.syncExchangeOperations = functions.https.onCall(
async (data, context) => {
try {
userAuthenthication(data.userId, context.auth);
let user = await getUser(data.userId);
if (!user.plan.benefits.syncExchanges) {
throw 'Operação não autorizada para o plano contratado';
}
let userExchange = await getUserExchange(data.userId, data.exchangeId);
let response = await Promise.all([
getUserLastOperations(data.userId, userExchange.exchangeId),
getSystemExchange(userExchange.exchangeId)
]);
let userExchangeLastOperations = response[0];
let systemExchange = response[1];
switch (systemExchange.id) {
case 'binance':
return syncBinanceOperations(
user.id,
userExchange,
userExchangeLastOperations,
systemExchange
);
}
} catch (error) {
return handleErrors(error);
}
}
);
It works fine if I change this function to a HTTP request. It waits the function to finish and returns.
exports.syncExchangeOperations = functions
.runWith(runtimeOpts)
.https.onRequest((req, res) => {
return cors(req, res, async () => {
try {
let auth = await admin.auth().verifyIdToken(req.get('Authorization').split('Bearer ')[1]);
let userExchange = await getUserExchange(
auth.uid,
req.query.exchangeId
);
let response = await Promise.all([
getUserLastOperations(auth.uid, userExchange.exchangeId),
getSystemExchange(userExchange.exchangeId)
]);
let userExchangeLastOperations = response[0];
let systemExchange = response[1];
switch (systemExchange.id) {
case 'binance':
await syncBinanceOperations(
auth.uid,
userExchange,
userExchangeLastOperations,
systemExchange
);
}
res.status(200).send();
} catch (error) {
res.status(401).send(handleErrors(error));
}
});
});
The "deadline-exeeded" that you encountered is an error thrown by the Firebase Javascript library on the client (not the function itself). The Firebase docs are lacking documentation o how to use functions.runWithOptions() on a callable function. For some reason the functions().httpsCallable() has a built in timeout on the client side.
So if you use this on your Node.js function:
exports.testFunction = functions.runWith({ timeoutSeconds: 180 }).https.onCall(async (data, ctx) => {
// Your Function Code that takes more than 60second to run
});
You need to override the buit in Javascript Library timeout on the client like this:
let testFunction = firebase.functions().httpsCallable("testFunction", {timeout: 180000});
I don't know what is the purpose of the built in timeout on the client, for me it has no purpose since it doesn't even stop the execution of the function on the server. But it must be there for some internal reasons.
Notice the Node.js timeoutSeconds is in seconds and the timeout option on the client library is in milliseconds.
"Deadline exceeded" means that the function invocation timed out from the perspective of the client. The default is 60 seconds.
Try increasing the timeout on both the client and function so that it has time to complete before the client timeout is reached. You can do this by specifying it in an HttpsCallableOptions object.
Also try returning something other than batch.commit(). Whatever that function return will be serialized and sent to the client, which could cause problems. Instead, just await batch.commit() then return something predictable, like a plain JavaScript object.
See the API documentation for information on setting the timeout:
https://firebase.google.com/docs/reference/js/firebase.functions.Functions#https-callable
https://firebase.google.com/docs/reference/js/firebase.functions.HttpsCallableOptions.html#timeout