Dramatiq’s primary reason for being is the fact that I wanted a distributed task queueing library that is simple and has sane defaults for most SaaS workloads. In that sense, it draws a lot of inspiration from GAE Push Queues and Sidekiq.

Dramatiq’s driving principles are as follows:

  • high reliability and performance
  • simple and easy to understand core
  • convention over configuration

If you’ve ever had to use Celery in anger, Dramatiq could be the tool for you.

Compared to *

I’ve used Celery professionally for years and my growing frustration with it is one of the reasons why I developed dramatiq. Here are some of the main differences between Dramatiq, Celery and RQ:

  Dramatiq Celery Huey RQ
Python 2 support No Yes Yes Yes
Windows support Yes No Yes No
Simple implementation Yes No [3] Yes Yes
Automatic retries Yes No Yes No
Reliable delivery Yes Optional [1] No No
Locks and rate limiting Yes No Yes No
Task prioritization Yes No [4] Yes No [4]
Delayed tasks Yes Yes [2] Yes No
Cronlike scheduling No [5] Yes Yes No
Chaining / Pipelining Yes Yes Yes No
Result storage Yes Yes Yes Yes
Code auto-reload Yes No No No
RabbitMQ support Yes Yes Yes No
Redis support Yes Yes Yes Yes
In-memory broker support Yes No Yes No
Greenlet support Yes Yes Yes No
[1]Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.
[2]Celery has poor support for delayed tasks. Delayed tasks are put on the same queue that is used for normal tasks and they’re simply pulled into worker memory until they can be executed, making it hard to autoscale workers by queue size. Dramatiq enqueues delayed tasks on a separate queue and moves them back when they’re ready to be executed.
[3]Celery’s source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.
[4](1, 2) Celery and RQ don’t support task prioritization. You have to deploy multiple sets of workers in order to prioritize queues. Dramatiq lets you prioritize down to the individual actor level.
[5]For cron-like scheduling functionality, you can combine Dramatiq with APScheduler or Periodiq.