Sponsors
If you find Bull valuable, please consider sponsoring its development by using the Taskforce front-end .
Besides helping Bull’s development, you will also benefit from a constantly-improving UI for managing all of your queues and jobs.
UIs
There are a few third-party UIs that you can use for monitoring:
Bull v3
Bull <= v2
Monitoring & Alerting
- With Prometheus Bull Queue Exporter
Feature Comparison
Since there are a few job queue solutions, here is a table comparing them:
Feature | Bull | Kue | Bee | Agenda |
---|---|---|---|---|
Backend | redis | redis | redis | mongo |
Priorities | ✓ | ✓ | ✓ | |
Concurrency | ✓ | ✓ | ✓ | ✓ |
Delayed jobs | ✓ | ✓ | ✓ | |
Global events | ✓ | ✓ | ||
Rate Limiter | ✓ | |||
Pause/Resume | ✓ | ✓ | ||
Sandboxed worker | ✓ | |||
Repeatable jobs | ✓ | ✓ | ||
Atomic ops | ✓ | ✓ | ||
Persistence | ✓ | ✓ | ✓ | ✓ |
UI | ✓ | ✓ | ✓ | |
Optimized for | Jobs / Messages | Jobs | Messages | Jobs |
Install
npm install bull --save
or
yarn add bull
Requirements: Bull requires a Redis version greater than or equal to 2.8.18
.
Typescript Definitions
npm install @types/bull --save-dev
yarn add --dev @types/bull
Definitions are currently maintained in the DefinitelyTyped repo.
Quick Guide
Using promises
Alternatively, you can use return promises instead of using the done
callback:
videoQueue.process(function(job){ // don't forget to remove the done callback!
// Simply return a promise
return fetchVideo(job.data.url).then(transcodeVideo);
// Handles promise rejection
return Promise.reject(new Error('error transcoding'));
// Passes the value the promise is resolved with to the "completed" event
return Promise.resolve({ framerate: 29.5 /* etc... */ });
// If the job throws an unhandled exception it is also handled correctly
throw new Error('some unexpected error');
// same as
return Promise.reject(new Error('some unexpected error'));
});
Separate processes
The process function can also be run in a separate process. This has several advantages:
- The process is sandboxed so if it crashes it does not affect the worker.
- You can run blocking code without affecting the queue (jobs will not stall).
- Much better utilization of multi-core CPUs.
- Less connections to redis.
In order to use this feature just create a separate file with the processor:
// processor.js
module.exports = function(job){
// Do some heavy work
return Promise.resolve(result);
}
And define the processor like this:
// Single process:
queue.process('/path/to/my/processor.js');
// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');
// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');
Repeated jobs
A job can be added to a queue and processed repeatedly according to a cron specification:
paymentsQueue.process(function(job){
// Check payments
});
// Repeat payment job once every day at 3:15 (am)
paymentsQueue.add(paymentsData, {repeat: {cron: '15 3 * * *'}});
As a tip, check your expressions here to verify they are correct: cron expression descriptor
Pause / Resume
A queue can be paused and resumed globally (pass true
to pause processing for
just this worker):
queue.pause().then(function(){
// queue is paused now
});
queue.resume().then(function(){
// queue is resumed now
})
Events
A queue emits some useful events, for example…
.on('completed', function(job, result){
// Job completed with output result!
})
For more information on events, including the full list of events that are fired, check out the Events reference
Queues performance
Queues are cheap, so if you need many of them just create new ones with different names:
var userJohn = new Queue('john');
var userLisa = new Queue('lisa');
.
.
.
However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result.
Important Notes
The queue aims for an “at least once” working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.
When a worker is processing a job it will keep the job “locked” so other workers can’t process it.
It’s important to understand how locking works to prevent your jobs from losing their lock - becoming stalled -
and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration
on interval
lockRenewTime
(which is usually half lockDuration
). If lockDuration
elapses before the lock can be renewed,
the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:
- The Node process running your job processor unexpectedly terminates.
- Your job processor was too CPU-intensive and stalled the Node event loop, and as a result, Bull couldn’t renew the job lock (see #488 for how we might better detect this). You can fix this by breaking your job processor into smaller parts so that no single part can block the Node event loop. Alternatively, you can pass a larger value for the
lockDuration
setting (with the tradeoff being that it will take longer to recognize a real stalled job).
As such, you should always listen for the stalled
event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.
As a safeguard so problematic jobs won’t get restarted indefinitely (e.g. if the job processor aways crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount
times (default: 1
).