Develop Reference

Sleep / delay inside promise.all

I am building a backend to handle pulling data from a third party API.
There are three large steps to this, which are:
Delete the existing db data (before any new data is inserted)
Get a new dataset from the API
Insert that data.
Each of these three steps must happen for a variety of datasets - i.e. clients, appointments, products etc.
To handle this, I have three Promise.all functions, and each of these are being passed individual async functions for handling the deleting, getting, and finally inserting of the data. I have this code working just for clients so far.
What I'm now trying to do is limit the API calls, as the API I am pulling data from can only accept up to 200 calls per minute. To quickly test the rate limiting functionality in code I have set it to a max of 5 api calls per 10 seconds, so I can see if it's working properly.
This is the code I have so far - note I have replaced the name of the system in the code with 'System'. I have not included all code as there's a lot of data that is being iterated through further down.
let patientsCombinedData = [];
let overallAPICallCount = 0;
let maxAPICallsPerMinute = 5;
let startTime, endTime, timeDiff, secondsElapsed;
const queryString = `UPDATE System SET ${migration_status_column} = 'In Progress' WHERE uid = '${uid}'`;
migrationDB.query(queryString, (err, res) => {
async function deleteSystemData() {
async function deleteSystemPatients() {
return (result = await migrationDB.query("DELETE FROM System_patients WHERE id_System_account = ($1) AND migration_type = ($2)", [
System_account_id,
migrationType,
]));
}
await Promise.all([deleteSystemPatients()]).then(() => {
startTime = new Date(); // Initialise timer before kicking off API calls
async function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function getSystemAPIData() {
async function getSystemPatients() {
endTime = new Date();
timeDiff = endTime - startTime;
timeDiff /= 1000;
secondsElapsed = Math.round(timeDiff);
if (secondsElapsed < 10) {
if (overallAPICallCount > maxAPICallsPerMinute) {
// Here I want to sleep for one second, then check again as the timer may have passed 10 seconds
getSystemPatients();
} else {
// Proceed with calls
dataInstance = await axios.get(`${patientsPage}`, {
headers: {
Authorization: completeBase64String,
Accept: "application/json",
"User-Agent": "TEST_API (email#email.com)",
},
});
dataInstance.data.patients.forEach((data) => {
patientsCombinedData.push(data);
});
overallAPICallCount++;
console.log(`Count is: ${overallAPICallCount}. Seconds are: ${secondsElapsed}. URL is: ${dataInstance.data.links.self}`);
if (dataInstance.data.links.next) {
patientsPage = dataInstance.data.links.next;
await getSystemPatients();
} else {
console.log("Finished Getting Clients.");
return;
}
}
} else {
console.log(`Timer reset! Now proceed with API calls`);
startTime = new Date();
overallAPICallCount = 0;
getSystemPatients();
}
}
await Promise.all([getSystemPatients()]).then((response) => {
async function insertSystemData() {
async function insertClinkoPatients() {
const SystemPatients = patientsCombinedData;
Just under where it says ' if (secondsElapsed < 10) ' is where I want to check the code every second to see if the timer has passed 10 seconds, in which case the timer and the count will be reset, so I can then start counting again over the next 10 seconds. Currently the recursive function is running so often that an error displayed related to the call stack.
I have tried to add a variety of async timer functions here but every time the function is returned it causes the parent promise to finish executing.
Hope that makes sense

I ended up using the Bottleneck library, which made it very easy to implement rate limiting.
const Bottleneck = require("bottleneck/es5");
const limiter = new Bottleneck({
minTime: 350
});
await limiter.schedule(() => getSystemPatients());

How to prevent event loop freeze on Nestjs?

I'm trying to create asynchronous task in NestJs. I want that after one request to some controller was made, my async task start begin your long job while loop, and then the node/nest process steel able to answer other requests of other controllers.
But when the async loop begin, all the other endpoints of Nest being freeze until de async task are done.
I already tryed to use setTimeout(), setInterval(), setIntermmediate(), queueMicrotask() and Promise.resolve()
My code:
// Async task:
private checkTime(baseTime: number, startTimeStamp: number){
while(true){
const elapsedTime = Date.now() - startTimeStamp
console.log(`Elapsed time: ${elapsedTime}`)
if(elapsedTime > baseTime){
console.log(`Time excced!`)
break;
}
}
}
And I already try something like that:
queueMicrotask(() => this.checkTime(edge.baseTime, startedTimeStamp))
setTimeout(() => this.checkTime(edge.baseTime, startedTimeStamp), 0)

Use javascript generators
Code snippet:
function* longTask(time = 1000) {
const startTime = Date.now();
// divides the long task into smaller chunks
yield startTime;
while (Date.now() - startTime < time) {
// process(): excute a small chunk of the task
// and yields the control back to the caller
yield Date.now() - startTime;
}
}
function completeLongTask(lt) {
const val = lt.next();
if (!val.done) {
setTimeout(() => {
completeLongTask(lt);
}, 0);
}
}
function triggerTask() {
const lt = longTask();
completeLongTask(lt);
}
triggerTask();
Although the solution above works, I think you needn't use generators for your use case. See if this simpler solution works for you.
function checkTime(baseTime, startTimeStamp) {
const currentTimeStamp = Date.now();
const elapsedTime = currentTimeStamp - startTimeStamp;
console.log(`Elapsed time: ${elapsedTime}`);
if (elapsedTime > baseTime) {
console.log(`Time excced!`);
return;
} else {
setTimeout(() => {
checkTime(baseTime, startTimeStamp);
});
}
}

Calendar.events.lists callback parameter doesn't get called

I'm building chatbot using Dialogflow. I wanted to integrate it with Google Calendar, I followed the Google official tutorial on youtube. My code looks as follows:
function makeAppointment (agent) {
// Calculate appointment start and end datetimes (end = +1hr from start)
//console.log("Parameters", agent.parameters.date);
const appointment_type = agent.parameters.AppointmentType;
const dateTimeStart = new Date(Date.parse(agent.parameters.date.split('T')[0] + 'T' + agent.parameters.time.split('T')[1].split('-')[0] + timeZoneOffset));
const dateTimeEnd = new Date(new Date(dateTimeStart).setHours(dateTimeStart.getHours() + 1));
const appointmentTimeString = dateTimeStart.toLocaleString(
'en-US',
{ month: 'long', day: 'numeric', hour: 'numeric', timeZone: timeZone }
);
// Check the availibility of the time, and make an appointment if there is time on the calendar
var result = undefined;
var not_needed = createCalendarEvent(dateTimeStart, dateTimeEnd, appointment_type).then(() => {
agent.add(Ok, let me see if we can fit you in. ${appointmentTimeString} is fine!.);
result = 1;
}).catch(() => {
agent.add(I'm sorry, there are no slots available for ${appointmentTimeString}.);
result = 1
});
while(result == undefined) continue;
return not_needed;
}
function createCalendarEvent (dateTimeStart, dateTimeEnd, appointment_type) {
return new Promise((resolve, reject) => {
calendar.events.list({
auth: serviceAccountAuth, // List events for time period
calendarId: calendarId,
timeMin: dateTimeStart.toISOString(),
timeMax: dateTimeEnd.toISOString()
}, (err, calendarResponse) => {
// Check if there is a event already on the Calendar
if (err || calendarResponse.data.items.length > 0) {
reject(err || new Error('Requested time conflicts with another appointment'));
} else {
// Create event for the requested time period
calendar.events.insert({ auth: serviceAccountAuth,
calendarId: calendarId,
resource: {summary: appointment_type +' Appointment', description: appointment_type,
start: {dateTime: dateTimeStart},
end: {dateTime: dateTimeEnd}}
}, (err, event) => {
err ? reject(err) : resolve(event);
}
);
}
});
});
}
There is nothing in the response so I created an infinite while loop which is waiting for the promise to get resolved or rejected. Since than cloud function is getting timed out after 60 seconds because the while loop never breaks. Why is the callback passed to calendar.events.lists never called?
Thank you

The Dialogflow library is aware of promises. In fact, it is so aware of them that you must return a Promise if you are doing any asynchronous operations in your Handler function. It will wait for the Promise to be resolved before it sends anything back.
Since createCalendarEvent() returns a Promise already, you can return this Promise from makeAppointment().
You don't need the while loop - in fact, that is probably a big part of what isn't working. Since node is single-threaded, the thread never leaves the while loop to process the Promise, so result will never actually be set to 1.
Additionally, you probably don't need to wrap the calls to calendar.events.list() and calendar.events.insert() inside a new Promise(). If you don't provide a callback function for these, they will return a Promise, and you can handle them with standard then/catch blocks or await (if you're using a sufficiently modern version of node).

Time delay in Truffle Tests

I am writing test cases for my contract and have to delay the assertion check because it is time sensitive. getCompletedCampaigns() will have the address of the Campaign whose deadline has passed.
it("Time sensitive check", async () => {
var deadLine = Math.round(Date.now() / 1000) + 3;
let eventDetails = await contract.createCampaign("Campaign name",deadLine,
{
from: accounts[0]
});
addressFromEvent = eventDetails['logs'][1]['args']['campaignAddress'];
async function checker() {
let deployedCampaigns = await factory.getCompletedCampaigns();
assert.equal(addressFromEvent, deployedCampaigns[0]);
}
function timeout(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
async function sleep() {
await timeout(5000);
return checker();
}
sleep();
});
The test passes even if the assertion is supposed to fail. The assertion happens after the test suite has finished executing all tests and forces the prompt to come out of truffle develop console because if it had failed. In the below testing I ve failed the test on purpose.
Contract: Testing CrowdCoin
✓ CampaignFactory deployment
✓ Create a new Campaign (168ms)
✓ Get ongoing Campaigns (246ms)
✓ Get completed Campaigns (189ms)
4 passing (1s)
truffle(develop)>
/home/vagrant/code/test/crowdcoin.test.js:82
assert.equal(0, deployedCampaigns[1]);
^
AssertionError: expected 0 to equal '0x806ea81c279b6000b9fd9f14d2845dec87fc3544'
at checker (/home/vagrant/code/test/crowdcoin.test.js:82:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
How do I make sure the test check happens along with time delay?

it("Time sensitive check", async () => {
var deadLine = Math.round(Date.now() / 1000) + 3;
let eventDetails = await contract.createCampaign("Campaign name",deadLine,
{
from: accounts[0]
});
function timeout(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
await timeout(5000);
addressFromEvent = eventDetails['logs'][1]['args']['campaignAddress'];
let deployedCampaigns = await factory.getCompletedCampaigns();
await assert.equal(addressFromEvent, deployedCampaigns[0]);
});
By moving the entire delay logic to a different method and all logical statements preceded by await, the execution of the said logical statements and test completion is not done unless timeout() has completed.

Actually you can try using time in #openzeppelin/test-helper to speed up your test case as you don't necessary wait for 3 secs running on this test case.
const { time } = require("#openzeppelin/test-helpers");
it("Time sensitive check", async () => {
let duration = time.duration.seconds(3);
let eventDetails = await contract.createCampaign("Campaign name",duration,
{
from: accounts[0]
});
await time.increase(duration);
addressFromEvent = eventDetails['logs'][1]['args']['campaignAddress'];
let deployedCampaigns = await factory.getCompletedCampaigns();
await assert.equal(addressFromEvent, deployedCampaigns[0]);
});
which make more sense on testing timestamp in blockchain.
Doc: https://docs.openzeppelin.com/test-helpers/0.5/api#time
An example of how #openzeppelin team work on test case that you can look into:
https://github.com/OpenZeppelin/openzeppelin-contracts/blob/24a0bc23cfe3fbc76f8f2510b78af1e948ae6651/test/token/ERC20/utils/TokenTimelock.test.js

Trying to block-sleep in some js/node code using setTimeout

Given the following code which I run in the command line with node fileName.js it seems to run all the items in the loop and THEN sleep at the end ... sorta like it's all running parallel or something.
I would like the code to block/pause during the setTimeout instead of just running the function AFTER the setTimeout is complete. Or, use a different method if setTimeout is the incorrect one, in this use case.
const removeUsers = async () => {
const users = await db.getUsers(); // returns 32 users.
// Split the users up into an array, with 2 users in each 'slot'.
var arrays = [], size = 2;
while (users.length > 0) {
arrays.push(users.splice(0, size));
}
// Now, for each slot, delete the 2 users then pause for 1 sec.
arrays.forEach(a => {
console.log(++counter;);
// Delete 2x users.
a.forEach(async u => {
console.log('Deleting User: ' + u.id);
await 3rdPartyApi.deleteUser({id: u.id});
});
// Now pause for a second.
// Why? 3rd party api has a 2 hits/sec rate throttling.
setTimeout(function () { console.log('Sleeping for 1 sec'); }, 1000);
});
}
and the logs are like this..
1.
Deleting User: 1
Deleting User: 2
2.
Deleting User: 3
Deleting User: 4
3.
...
(sleep for 1 sec)
(sleep for 1 sec)
(sleep for 1 sec)
...
end.
See how the sleep doesn't feel like it blocks.. it just fires off a sleep command which then gets handled after a sec...
This is what I'm really after...
1.
Deleting User: 1
Deleting User: 2
(sleep for 1 sec)
2.
Deleting User: 3
Deleting User: 4
(sleep for 1 sec).
3.
...
end.

This calls a bunch of async functions. They each return a promise (async functions always return promises), and those promises are discarded, because Array#forEach doesn’t do anything with the return value of the function it’s passed.
a.forEach(async u => {
console.log('Deleting User: ' + u.id);
await 3rdPartyApi.deleteUser({id: u.id});
});
This starts a timer and doesn’t even attempt to wait for it.
setTimeout(function () { console.log('Sleeping for 1 sec'); }, 1000);
Split off the timer into a function that returns a promise resolving in the appropriate amount of time (available as Promise.delay if you’re using Bluebird, which you should be):
const delay = ms =>
new Promise(resolve => {
setTimeout(resolve, ms);
});
and keep everything in one async function so you’re not discarding any promises:
function* chunk(array, size) {
for (let i = 0; i < array.length;) {
yield array.slice(i, i += size);
}
}
const removeUsers = async () => {
const users = await db.getUsers(); // returns 32 users.
for (const a of chunk(users, 2)) {
console.log(++counter);
// Delete 2x users.
for (const u of a) {
console.log('Deleting User: ' + u.id);
await ThirdPartyApi.deleteUser({id: u.id});
}
console.log('Sleeping for 1 sec');
await delay(1000);
}
};