I'm using socket.io and mongoose in my express server.
My socket is listening for events using the following code:
socket.on('do something', async () => {
try {
await doA();
doX();
await doB();
doY();
await doC();
} catch (error) {
console.log(error);
}
});
doA, doB and doC are async operations that writes on database using mongoose, but in general they can be any method returning a promise.
I want that 'do something' runs synchronously.
If the event queue processes more events at the same time I have consistency problems in my mongodb.
In other words if the server receives two 'do something' events, I want that the second event received is processed only when the first event is fully processed (after the await doC). Unfortunately the 'do something' callback is async.
How to handle this?
It's possible to implement a queue by adding the functions you want to run to an array, and then running them one by one. I've created an example below.
let queue = [];
let running = false;
const delay = (t, v) => {
return new Promise((resolve) => {
setTimeout(resolve.bind(null, "Returned value from Promise"), t)
});
}
const onSocketEvent = async () => {
console.log("Got event");
if (!running) {
console.log("Nothing in queue, fire right away");
return doStuff();
}
// There's something in the queue, so add it to it
console.log("Queuing item")
queue.push(doStuff);
}
const doStuff = async () => {
running = true;
const promiseResult = await delay(2000);
console.log(promiseResult);
if (queue.length > 0) {
console.log("There's more in the queue, run the next one now")
queue.shift()();
} else {
console.log("Queue empty!")
running = false;
}
}
onSocketEvent();
setTimeout(() => onSocketEvent(), 1000);
setTimeout(() => onSocketEvent(), 1500);
setTimeout(() => onSocketEvent(), 2000);
setTimeout(() => onSocketEvent(), 2500);
I would suggest adding a delay between each await. This will prevent deadlocks from occurring and fix your issue. For such things, I would suggest using the Caolan's async library.
Task delay example:
setTimeout(function() { your_function(); }, 5000); // 5 seconds
If your function has no parameters and no explicit receiver, you can call directly setTimeout(func, 5000)
Useful jQuery timers plugin
Related
I was wondering if there is any way to break the current process of a UseEffect and have it start on the next render, like this
...
useEffect(() => {
SlowFunction(update);
}, [update]);
setUpdate(1)
// a bit of time passes but not long enough for the SlowFunction(1) to be done
setUpdate(2)
//when this is called and the useEffect runs, stop the SlowFunction(1) and run SlowFunction(2)
my updated personal function is called in the use effect like so,
const [update, setUpdate] = useState(0);
const [thisConst, setThisConst] = useState(0);
async function SlowFunction(firstParam, paramEtc, { signal } = {}) {
while (true) {
//wait two seconds between each
await new Promise((r) => setTimeout(r, 2000));
// Before starting every individual "task" in the function,
// first throw if the signal has been aborted. This will stop the function
// if cancellation occurs:
signal?.throwIfAborted();
// else continue working...
console.log('working on another iteration');
}
console.log('Completed!');
return 'some value';
}
useEffect(() => {
const controller = new AbortController();
const { signal } = controller;
(async () => {
try {
const result = await SlowFunction(update, 'some other value', {
signal,
});
setConst(result);
} catch (ex) {
console.log('EXCEPTION THROWN: ', ex);
}
})();
return () => controller.abort(new Error('Starting next render'));
}, [update]);
The AbortSignal API is the standard method for handling cancellation.
I'll provide an example of how to use it with a function like your SlowFunction. You'll need to accept an abort signal as an optional parameter so that when the next render occurs, the function can be cancelled.
Here's an example cancellable function:
async function SlowFunction (firstParam, paramEtc, {signal} = {}) {
for (let i = 0; i < 1_000_000; i += 1) {
// Before starting every individual "task" in the function,
// first throw if the signal has been aborted. This will stop the function
// if cancellation occurs:
signal?.throwIfAborted();
// else continue working...
console.log('working on another iteration');
}
return 'some value';
}
You can use it in an effect hook like this: returning a cleanup function which invokes the abort method on the controller:
useEffect(() => {
const controller = new AbortController();
const {signal} = controller;
(async () => {
try {
const result = await SlowFunction(update, 'some other value', {signal});
setConst(result);
}
catch (ex) {
// Catch the exception thrown when the next render starts
// and the function hasn't completed yet.
// Handle the exception if you need to,
// or do nothing in this block if you don't.
}
})();
return () => controller.abort(new Error('Starting next render'));
}, [update]);
If the function completes before the next render occurs, then the abort operation will have no effect, but if it hasn't yet, then the next time that the statement signal?.throwIfAborted(); is reached, the function will throw an exception and terminate.
Update in response to your comment:
If your JavaScript runtime is too old to support the AbortSignal.throwIfAborted() method, you can work around that by replacing that line:
signal?.throwIfAborted();
with:
if (signal?.aborted) {
throw signal?.reason ?? new Error('Operation was aborted');
}
The actual Problem
The problem I am trying to solve is that a user may not provide enough information on a message (discord). To then get all of the necessary data the user is prompted to react to a bot message.
There are 3 stages of that bot message, "InitialPrompt", "TimeoutWarning" and "TimeoutSuccess(TicketFailure)".
What I wanted the solution to be
With the way I've written my code there is no way to abort the timeout after it has been initialized. I thought that throwing an error would stop the execution of a function. I guess that doesn't happen because async calls get queued up instead of being ran line by line.
Is there a way to do this without adding a boolean and checking infront of each function call?
The solution that I could come up with
const interPtr = {interrupted : false};
interruptSignal(interPtr);
if(interPtr.interrupted) return;
console.log("...");
...
The actual code
JS Fiddle
(async () => {
const sleep = async time => new Promise(resolve => setTimeout(resolve, time));
const interruptSignal = () => new Promise(async (res, rej) => {
console.log("interruptStarted")
await sleep(2000);
console.log("interruptRan");
throw "Interrupted"
});
const timeOutHandler = async () => {
interruptSignal();
console.log("TimeoutStarted")
await sleep(5000);
console.log("TimeoutWarning")
await sleep(5000);
console.log("TimeoutSuccess->TicketFailure")
};
try {
await timeOutHandler();
} catch (e) {
console.log(e);
}
})()
I have a main function which calls two async functions with sleep function in between. This is a basic example:
index.js
const func1 = async() => {
setTimeout(()=>{console.log('func 1...')}, 1000);
}
const func2 = async() => {
setTimeout(()=>{console.log('func 2...')}, 1000);
}
const sleep = ms => {
console.log(`Sleeping for ${ms/1000} seconds`);
return new Promise(resolve => {
setTimeout(resolve, ms);
})
}
const main = async() => {
try {
await func1();
// Sleeping for a long long time
console.log('Before Sleep');
await sleep(2000000);
console.log('After Sleep')
await func2();
return 'success';
} catch(err) {
console.log(err);
return 'error'
}
}
And this is my test code:
index.test.js
const index = require('./index');
jest.useFakeTimers();
describe('Testing index.js...', () => {
test('Should return success', async() => {
const promise = index();
jest.advanceTimersByTime(2000000);
promise.then(response => {
expect(response).toBe('success');
})
});
})
The test passes, but the console shows the following:
func 1...
Before Sleep
Sleeping for 2000 seconds
I tried the same this, but with func1() and func2() being synchronous functions:
const func1 = () => {
console.log('func 1...');
}
const func2 = () => {
console.log('func 2...');
}
const sleep = ms => {
// Sleeping for a long time
console.log(`Sleeping for ${ms/1000} seconds`);
return new Promise(resolve => {
setTimeout(resolve, ms);
})
}
const main = async() => {
try {
func1();
// Sleeping for a long long time
console.log('Before Sleep');
await sleep(2000000);
console.log('After Sleep')
func2();
return 'success';
} catch(err) {
console.log(err);
return 'error'
}
}
In that case, the test passes and the logs are also as expected:
func 1...
Before Sleep
Sleeping for 2000 seconds
After Sleep
func 2...
In the same synchronous code, if I make func1 async (keeping func2 synchronous), the problem reappears.
If func1 is synchronous and func2 is async, everything works as expected.
I have also tried using jest.runAllTimers() and jest.runOnlyPendingTimers(). I have also tried using async-await in the test file, but that (understandably) gives a timeout error:
index.test.js using async-await
const index = require('./index');
jest.useFakeTimers();
describe('Testing index.js...', () => {
test('Should return success', async() => {
const promise = index();
jest.advanceTimersByTime(3000000);
const response = await promise;
expect(response).toBe('success');
});
})
Console:
func 1...
Before Sleep
Sleeping for 2000 seconds
Error:
Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.Timeout
How can I make this work?
I have gone through and tried a solutions on lot of Github issues in jest, and also a lot of questions on stack overflow, none of the solutions seem to work.
I am using jest 25.5.4
Edit: I also tried increasing the value in jest.advanceTimersBytTime() to a day. And also tried making the function in describe async.
I've had a similar issue recently, what worked for me is to advance the timers from within an async-call. Seems like jest does not support setting the timers within a promise (see https://github.com/facebook/jest/pull/5171#issuecomment-528752754). Try doing:
describe('Testing index.js...', () => {
it('Should return success', () => {
const promise = main();
Promise.resolve().then(() => jest.advanceTimersByTime(2000005));
return promise.then((res) => {
expect(res).toBe('success');
});
});
});
async, raw promises and done callback shouldn't be used together in tests. This is a common sign that a developer isn't fully comfortable with Jest asynchronous testing, this leads to error-prone tests.
The problem with original is that promise.then(...) promise is ignored because it's not chained. Asynchronous test should return a promise in order for it to be chained.
func1 and func2 return promises that resolve immediately and produce one-tick delay rather than 1 second delay. This should be taken into account because otherwise there's a race condition with setTimeout being called after advanceTimersByTime.
It should be:
test('Should return success', async() => {
const promise = index();
await null; // match delay from await func1()
jest.advanceTimersByTime(2000000);
const response = await promise;
expect(response).toBe('success');
});
Summary of Objective
I need to implement an API queue in my Node.js backend for API calls. The API rate limit I need to adhear to is 1 request every 2 seconds and it's a hard limit. I'm making my API call inside of a forEach loop since I need to do one API call for each user.
I've found a lot of articles online about how to create a queue but they mostly involve adding API calls to an array so I'm not sure how to implement a queue in this situation.
Any help would be greatly apprecaited and I can share more code if it's help.
Code
async function refreshStats() {
try {
// get list of all fortnite users
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
fnUserList.forEach(async fnUser => {
//make API call. apiCall is a function I created to make the API call and format the response
const { lifeStats, statsEqual } = await apiCall(
fnUser.fnUserPlatform
);
//execute other functions with apiCall response
});
} catch (err) {
console.error("error in refreshStats", err);
}
}
If i get it correctly. You can take advantage of generator functions and combine it with setInterval. You can make make a queue function that enqueues its items in specified intervals.
Create a generator function basically makes an apiCall and pauses
async function* queueGenerator(userList) {
for (let fnUser of userList) {
const result = {lifeStats, statsEqual} = await apiCall(fnUser.fnUserPlatform);
yield result;
}
}
Then in your method create a queue and enqueue items with setInterval
async function refreshStats() {
try {
// get list of all fortnite users
let handle;
const fnUserList = await Users.find({}, "_id fnUserPlatform"); // my fnUser _id 5cca01ea8f52f40117b2ff51
const queue = queueGenerator(fnUserList);
const results = [];
handle = setInterval(async () => {
const result = await queue.next();
results.push(result.value);
if (results.length === users.length) clearInterval(handle);
}, 2000);
} catch (err) {
console.error("error in refreshStats", err);
}
}
Also there is another way which is to use setTimeout combined with Promises. which involves creating promises that resolves in setTimeOut with enough of delays.
async function refreshStatsV2() {
const fnUserList = await Users.find({}, "_id fnUserPlatform");
const promises = fnUserList.map((fnUser, ix) => (
new Promise(resolve =>
setTimeout(async() => {
const result = {
lifeStats,
statsEqual
} = await apiCall(ix.fnUserPlatform);
resolve(result);
}, ix * 2000) // delay every next item 2sec
)));
const result = await Promise.all(promises); // wait all
console.log(result);
}
So I have a synchronous function (client.functionOne) that is creating a Discord.js message collector, which deletes messages sent to a channel. This function also calls an asynchronous function which creates a setTimeout loop.
The issue is, for some reason, the "collect" function gets held up whenever the code in client.functionTwo is run and it does not run as fast as it would if it wasn't running.
I am unsure why it is doing this. Could someone please help? Thanks in advance.
client.functionTwo = async (channel) => {
let timeout;
const interval = async () => {
// Logic here with several awaits
// Schedule a new timeout
timeout = setTimeout(interval, 2000);
}
interval();
}
client.functionOne = (channel) => {
setTimeout(() => {
const collector = channel.createMessageCollector(() => true, { time: 150000 });
client.functionTwo(channel);
collector.on("collect", (message) => {
if (message.author.bot) return;
message.delete();
});
}, 1000);
}
// client.functionTwo = async (channel) => {
// let timeout = await setTimeout(interval, 2000);
// }
client.functionOne = async (channel) => {
const collector = await channel.createMessageCollector(() => => true,
{ time: 150000 });
// await client.functionOne(channel);
collector.on("collect", async (message) => {
if (message.author.bot) return;
await message.delete();
});
}
After extensive testing I was able to determine that my issue was caused by rate limiting with Discord and the JavaScript library I am using.