Motivation

Dramatiq’s primary reason for being is the fact that I wanted a distributed task queueing library that is simple and has sane defaults for most SaaS workloads. In that sense, it draws a lot of inspiration from GAE Push Queues and Sidekiq.

Dramatiq’s driving principles are as follows:

  • high reliability and performance
  • simple and easy to understand core
  • convention over configuration

If you’ve ever had to use Celery in anger, Dramatiq could be the tool for you.

Compared to *

I’ve used Celery professionally for years and my growing frustration with it is one of the reasons why I developed dramatiq. Here are some of the main differences between Dramatiq, Celery and RQ:

  Dramatiq Celery RQ
At-least-once delivery Yes No [1] No
Automatic retries Yes No No
Code auto-reload Yes No No
Delayed tasks Yes Yes [2] No
Locks and rate limiting Yes No No
Result storage No Yes Yes
Simple implementation Yes No [3] Yes
Task prioritization Yes No [4] No [4]
Redis support Yes Yes Yes
RabbitMQ support Yes Yes No
In-memory broker support Yes No No
[1]Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.
[2]Celery has poor support for delayed tasks. Delayed tasks are put on the same queue that is used for normal tasks and they’re simply pulled into worker memory until they can be executed, making it hard to autoscale workers by queue size. Dramatiq enqueues delayed tasks on a separate queue and moves them back when they’re ready to be executed.
[3]Celery’s source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.
[4](1, 2) Celery and RQ don’t support task prioritization. You have to deploy multiple sets of workers in order to prioritize queues. Dramatiq lets you prioritize down to the individual actor level.