Motivation

Dramatiq’s primary reason for being is the fact that I wanted a distributed task queueing library that is simple and has sane defaults for most SaaS workloads. In that sense, it draws a lot of inspiration from GAE Push Queues and Sidekiq.

Dramatiq’s driving principles are as follows:

• high reliability and performance

• simple and easy to understand core

• convention over configuration

If you’ve ever had to use Celery in anger, Dramatiq could be the tool for you.

Compared to *

Note

This section was last updated in 2019. It’s possible that various bits of the table below are now outdated. Clarifications in PR form are always welcome!

I’ve used Celery professionally for years and my growing frustration with it is one of the reasons why I developed dramatiq. Here are some of the main differences between Dramatiq, Celery, Huey and RQ:

	Dramatiq	Celery	Huey	RQ
Python 2 support	No	Yes	Yes	Yes
Windows support	Yes	No	Yes	No
Simple implementation	Yes	No [3]	Yes	Yes
Automatic retries	Yes	No	Yes	No
Reliable delivery	Yes	Optional [1]	No	No
Locks and rate limiting	Yes	No	Yes	No
Task prioritization	Yes [4]	No [4]	Yes	Yes
Delayed tasks	Yes	Yes [2]	Yes	No
Cronlike scheduling	No [5]	Yes	Yes	No
Chaining / Pipelining	Yes	Yes	Yes	No
Result storage	Yes	Yes	Yes	Yes
Code auto-reload	Yes	No	No	No
RabbitMQ support	Yes	Yes	Yes	No
Redis support	Yes	Yes	Yes	Yes
In-memory broker support	Yes	No	Yes	No
Greenlet support	Yes	Yes	Yes	No

[1]

Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.

[2]

Celery has poor support for delayed tasks. Delayed tasks are put on the same queue that is used for normal tasks and they’re simply pulled into worker memory until they can be executed, making it hard to autoscale workers by queue size. Dramatiq enqueues delayed tasks on a separate queue and moves them back when they’re ready to be executed.

[3]

Celery’s source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.

[4] (1,2)

Celery doesn’t support task prioritization. Dramatiq supports global prioritization under RabbitMQ via the broker_priority. It also provides worker-local prioritization of prefetched messages.

[5]

For cron-like scheduling functionality, you can combine Dramatiq with APScheduler or Periodiq. 3rd-party packages exist which implement this approach for various frameworks, like: Dramatiq-Crontab