I am using kue for delayed jobs in my node.js application.
I have some problems to figure out how I can restart a job using the API of kue without having to move the id of a job manually from the the list of failed jobs to the list of inactive jobs using redis commands.
Is this possible using kue?
I don't want to set a fixed number of retry attempts - I just want to retry specific jobs.
Suggestions for a well maintained alternative to kue are also welcome.
i dont know if this is working but you could try to reset the state of the job to active, and save the job again:
job.on('failed', function() {
job.state('inactive').save();
Edit: setting state to inactive will correctly re-enqueue the task.
This can also be done using queue level events.
queue.on('job failed', function(id, result) {
kue.Job.get(id, function(err, job) {
if (!err && shouldRetry(job))
job.state('inactive').save();
});
});
Thus you don't need to do for every job that you wish to retry. Instead you can filter it in the queue level event.
see Failure Attempts in the official docs
By default jobs only have one attempt, that is when they fail, they
are marked as a failure, and remain that way until you intervene.
However, Kue allows you to specify this, which is important for jobs
such as transferring an email, which upon failure, may usually retry
without issue. To do this invoke the .attempts() method with a number.
queue.create('email', {
title: 'welcome email for tj'
, to: 'tj#learnboost.com'
, template: 'welcome-email'
}).priority('high').attempts(5).save();
reference: failure attempts
Related
So I need to implement an "expensive" API endpoint. Basically, the user/client would need to be able to create a "group" of existing users.
So this "create group" API would need to check that each users fulfill the criteria, i.e. all users in the same group would need to be from the same region, same gender, within an age group etc. This operation can be quite expensive, especially since there are no limit on how many users in one group, so its possible that the client requests group of 1000 users for example.
My idea is that the endpoint will just create entry in database and mark the "group" as pending, while the checking process is still happening, then after its completed, it will update the group status to "completed" or "error" with error message, then the client would need to periodically fetch the status if its still pending.
My implementation idea is something along this line
const createGroup = async (req, res) => {
const { ownerUserId, userIds } = req.body;
// This will create database entry of group with "pending" status and return the primary key
const groupId = await insertGroup(ownerUserId, 'pending');
// This is an expensive function which will do checking over the network, and would take 0.5s per user id for example
// I would like this to keep running after this API endpoint send the response to client
checkUser(userIds)
.then((isUserIdsValid) => {
if (isUserIdsValid) {
updateGroup(groupId, 'success');
} else {
updateGroup(groupId, 'error');
}
})
.catch((err) => {
console.error(err);
updateGroup(groupId, 'error');
});
// The client will receive a groupId to check periodically whether its ready via separate API
res.status(200).json({ groupId });
};
My question is, is it a good idea to do this? Do I missing something important that I should consider?
Yes, this is the standard approach to long-running operations. Instead of offering a createGroup API that creates and returns a group, think of it as having an addGroupCreationJob API that creates and returns a job.
Instead of polling (periodically fetching the status to check whether it's still pending), you can use a notification API (events via websocket, SSE, webhooks etc) and even subscribe to the progress of processing. But sure, a check-status API (via GET request on the job identifier) is the lowest common denominator that all kinds of clients will be able to use.
Did I not consider something important?
Failure handling is getting much more complicated. Since you no longer create the group in a single transaction, you might find your application left in some intermediate state, e.g. when the service crashed (due to unrelated things) during the checkUser() call. You'll need something to ensure that there are no pending groups in your database for which no actual creation process is running. You'll need to give users the ability to retry a job - will insertGroup work if there already is a group with the same identifier in the error state? If you separate the group and the jobs into independent entities, do you need to ensure that no two pending jobs are trying to create the same group? Last but not least you might want to allow users to cancel a currently running job.
First of all, I know there is already an answer to a similar question here, but I am still not sure this is because of RabbitMQ's impossibility or because I haven't researched enough.
I come from JS/Node background where event pub/sub pattern works like this: when many consumers subscribe to the same topic with pub/sub pattern, all of them should get the same message whenever some producer publishes it.
I expect to implement the same pattern with a message broker.
For example:
Consumer 1 listens to 'request.user.#'
Consumer 2 listens to 'request.user.#'
Consumer 3 listens to 'request.#.#'
Producer 1 publishes to topic 'request.user.add'
Producer 2 publishes to topic 'request.user.detail'
What RabbitMQ actually does (according to this RabbitMQ example about Topics)
Consumer 3 gets both messages, while either Consumer 1 or Consumer 2 gets the first message, and only either of them gets the second message.
What I expect to implement
Three of them gets both messages.
Do you have any idea to implement this with a message broker (RabbitMQ in top priority)? Please point out if I miss something or am not clear somewhere in my question.
Thanks in advance.
EDIT with SOLUTION:
Thanks to #cantSleepNow, here is the code (NodeJS) for Consumer 1 and Consumer 2 that I have come up with after his hint:
var amqp = require('amqplib/callback_api');
amqp.connect('amqp://localhost', (err, conn) => {
conn.createChannel((err, ch) => {
var ex = 'topic_exchange'; // Exchange name
ch.assertExchange(ex, 'topic'); // Exchange with 'topic' type
ch.assertQueue('', {exclusive: true}, (err, q) => {
ch.bindQueue(q.queue, ex, 'request.user.#'); // Topic pattern
ch.consume(q.queue, (msg) => {
// Process message
});
});
});
});
Well this is possible using the fanout method. You can use RabbitMQ as a broadcast mechanism.
This will have one Sender and multiple subscribers and the link is here and you can find it under the Tutorial three: Publish/Subscribe senction
What I expect to implement
Three of them gets both messages.
Well simply use the topic exchange and have each consumer declare it's own queue with appropriate routing key.
When you're publishing, you are publishing to a topic exchange (let's call it E1) with (for example) 'request.user.add' and all the queues bound to E1 with the matching routing keys(since we are talking topics here) will get the message.
Or maybe to put it like this: one message is consumed from one queue once, and from one exchange as many times as there are queues bound to it (with appropriate routing keys).
EDIT after #hirikarate added the solution to question
Well, I don't use javascript, but I'll try to help :) With exclusive you're saying that only one consumer can connect to the queue, which is ok. Also if you create a queue with a unique name in each consumer you'd achieve almost the same (obviously the difference is that other consumers would be allowed to consume from it) Again, don't know in JS, but there should be a way that you create a queue with the given name, or have the server return the name to you. Other then that looks ok, you just need to test it.
I have been trying to implement a RESTFul API with NodeJS and I use Mongoose (MongoDB) as the database backend.
The following example code registers multiple users with the same username when requests are sent at the same time, which is not what I desire. Although I tried to add a check!
I know this happens because of the asynchronous nature of NodeJS, but I could not find a method to do this properly. It looks like "findOne" method immediately returns, causing registerUser to return and then another request is processed.
By the way, I don't want to check for existing users with a separate API function, I need to check at the registration stage. Is there any way to do this?
Controller.prototype.registerUser = function (req, res) {
Users.findOne({'user_name': req.body.user_name}, function(err, user) {
if(!user) {
new User({user_name: req.body.user_name}).save(function(err) {
if(!err) {
res.send("User saved");
} else {
res.send("DB Error: Could not save user!");
}
});
} else {
res.send("User exists");
}
});
}
You should consider setting the user_name to be unique in the Schema. That would ensure that the user_name stays unique even if simultaneous requests are made to set an identical user name.
Yes, the reason this is happening is as you suspected because multiple requests can execute the code simultaneously and therefore the User.fineOne can return false multiple times. Incidentally this can happen with other stacks as well, even ones that use one thread per request.
To solve this, you need a way to somehow either control that just one user is being worked on at the time, you can accomplish this by adding all registerUser requests to a queue and then pulling them off the queue one by one and calling res.Send only after it's processed form the queue.
Alternatively, maybe you can keep a local array of user names, and each time a new request comes in and check the array if it's already there. If it isn't add it to the array and work on it. If it is in the array, send the response "User exists". Then, once the user has been successfully created, you can remove it from that array. (I haven't thought this one through 100% but I think it should work as well.)
I would like that the jobs.create fails if an identical job is already in the system. Is there any way to acomplish this?
I need to run the same job every 24 hours, but some jobs could take even more than 24 hours, so I need to be sure that the job isn't already in the system (active, queued o failed) before adding it.
UPDATED:
Ok, I going to simplify the problem to be able to explain it here.
Lest say I have an analytics service and I have to send a report to my users once a day. Completing these reports some times(just a few cases but it is a possibility) take several hours even more than a day.
I need a way to know which are the currently running jobs to avoid duplicated jobs. I couldn't find anything in the ´´´´kue´´´´ API to know which jobs are currently running. Also I need some kind of event fired when more jobs are needed and then call my getMoreJobs producer.
Maybe my approach is wrong, if so please let me know a better way to solve my problem.
This is my simplified code:
var kue = require('kue'),
cluster = require('cluster'),
numCPUs = require('os').cpus().length;
numCPUs = CONFIG.sync.workers || numCPUs;
var jobs = kue.createQueue();
if (cluster.isMaster) {
console.log('Starting master pid:' + process.pid);
jobs.on('job complete', function(id){
kue.Job.get(id, function(err, job){
if (err || !job) return;
job.remove(function(err){
if (err) throw err;
console.log('removed completed job #%d', job.id);
});
});
function getMoreJobs() {
console.log('looking for more jobs...');
getOutdateReports(function (err, reports) {
if (err) return setTimeout(getMoreJobs, 5 * 60 * 60 * 1000);
reports.forEach(function(report) {
jobs.create('reports', {
id: report.id,
title: report.name,
params: report.params
}).attempts(5).save();
});
setTimeout(getMoreJobs, 60 * 60 * 1000);
});
}
//Create the jobs
getMoreJobs();
console.log('Starting ', numCPUs, ' workers');
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
cluster.on('death', function(worker) {
console.log('worker pid:' + worker.pid + ' died!'.bold.red);
});
} else {
//Process the jobs
console.log('Starting worker pid:' + process.pid);
jobs.process('reports', 20, function(job, done){
//completing my work here
veryHardWorkGeneratingReports(function(err) {
if (err) return done(err);
return done();
});
});
}
The answer to one of your questions is that Kue puts the jobs that it pops off of the redis queue into "active", and you'll never get them unless you look for them.
The answer to the other question is that your distributed work queue is the consumer, not the producer of tasks. Mingling them like you have is okay, but, it's a muddy paradigm. What I've done with Kue is to make a wrapper for kue's json api, so that a job can be put into the queue from anywhere in the system. Since you seem to have a need to shovel jobs in, I suggesting writing a separate producer application that does nothing but get external jobs and stick them into your Kue work queue. It can monitor the work queue for when jobs are running low and load a batch in, or, what I would do, is make it shovel jobs in as fast as it can, and spool up multiple instances of your consumer application to process the load more quickly.
To re-iterate: Your separation of concerns isn't very good here. You should have a producer of tasks that's completely separate from your task consumer app. This gives you more flexibility, ease of scale (Just fire up another consumer on another machine and you're scaled!) and overall ease of code management. You should also allow, if possible, whomever is giving you these tasks that you "go looking for" access to your Kue server's JSON api instead of going out and finding them. The job producer can schedule its own tasks with Kue.
Look at https//github.com/LearnBoost/kue.
In json.js script check rows 64-112. There you'll find methods which return an object containing jobs, also filtered with type, state or id-range. (jobRange(), jobStateRange(), jobTypeRange().)
Scrolling down the main page to JSON API -section, you'll find the examples of the returned objects.
That how to call and use those methods you know much better than I do.
jobs.create() will fail, if you pass an unknown keyword. I would created a function to check the current job in forEach-loop, and returns a keyword. Then just call this function instead of literal keyword in jobs.create() -parameters.
Information got through those methods in json.js, may help you create that "moreJobToDo"-event too.
My situation ...
I have a set of workers that are scheduled to run periodically, each at different intervals, and would like to find a good implementation to manage their execution.
Example: Let's say I have a worker that goes to the store and buys me milk once a week. I would like to store this job and it's configuration in a mysql table. But, it seems like a really bad idea to poll the table (every second?) and see which jobs are ready to be put into the execution pipeline.
All of my workers are written in javascript, so I'm using node.js for execution and beanstalkd as a pipeline.
If new jobs (ie. scheduling a worker to run at a given time) are being created asynchronously and I need to store the job result and configuration persistently, how do I avoid polling a table?
Thanks!
I agree that it seems inelegant, but given the way that computers work something *somewhere* is going to have to do polling of some kind in order to figure out which jobs to execute when. So, let's go over some of your options:
Poll the database table. This isn't a bad idea at all - it's probably the simplest option if you're storing the jobs in MySQL anyway. A rate of one query per second is nothing - give it a try and you'll notice that your system doesn't even feel it.
Some ideas to help you scale this to possibly hundreds of queries per second, or just keep system resource requirements down:
Create a second table, 'job_pending', where you put the jobs that need to be executed within the next X seconds/minutes/hours.
Run queries on your big table of all jobs only once in a longer while, then populate the small table which you query every shorter while.
Remove jobs that were executed from the small table in order to keep it small.
Use an index on your 'execute_time' (or whatever you call it) column.
If you have to scale even further, keep the main jobs table in the database, and use the second, smaller table I suggest, just put that table in RAM: either as a memory table in the DB engine, or in a Queue of some kind in your program. Query the queue at extremely short intervals if you have too - it'll take some extreme use cases to cause any performance issues here.
The main issue with this option is that you'll have to keep track of jobs that were in memory but didn't execute, e.g. due to a system crash - more coding for you...
Create a thread for each of a bunch of jobs (say, all jobs that need to execute in the next minute), and call thread.sleep(millis_until_execution_time) (or whatever, I'm not that familiar with node.js).
This option has the same problem as no. 2 - where you have to keep track job execution for crash recovery. It's also the most wasteful imo - every sleeping job thread still takes system resources.
There may be additional options of course - I hope that others answer with more ideas.
Just realize that polling the DB every second isn't a bad idea at all. It's the most straightforward way imo (remember KISS), and at this rate you shouldn't have performance issues so avoid premature optimizations.
Why not have a Job object in node.js that's saved to the database.
var Job = {
id: long,
task: String,
configuration: JSON,
dueDate: Date,
finished: bit
};
I would suggest you only store the id in RAM and leave all the other Job data in the database. When your timeout function finally runs it only needs to know the .id to get the other data.
var job = createJob(...); // create from async data somewhere.
job.save(); // save the job.
var id = job.id // only store the id in RAM
// ask the job to be run in the future.
setTimeout(Date.now - job.dueDate, function() {
// load the job when you want to run it
db.load(id, function(job) {
// run it.
run(job);
// mark as finished
job.finished = true;
// save your finished = true state
job.save();
});
});
// remove job from RAM now.
job = null;
If the server ever crashes all you have to is query all jobs that have [finished=false], load them into RAM and start the setTimeouts again.
If anything goes wrong you should be able to restart cleanly like such:
db.find("job", { finished: false }, function(jobs) {
each(jobs, function(job) {
var id = job.id;
setTimeout(Date.now - job.dueDate, function() {
// load the job when you want to run it
db.load(id, function(job) {
// run it.
run(job);
// mark as finished
job.finished = true;
// save your finished = true state
job.save();
});
});
job = null;
});
});