Skip to main content

Celery: Distributed Task Queue

A simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

Overview​

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation but supports scheduling as well. It allows you to run time-consuming, blocking, or periodic work outside of your application's main execution flow, ensuring a responsive user experience and enabling scalable system architecture.

At its core, Celery helps you offload work to a pool of background "worker" processes. You define tasks as simple Python functions, and Celery handles the details of sending these tasks to a message broker (like RabbitMQ or Redis), distributing them to available workers, and optionally storing the results in a backend. This architecture makes it easy to build robust, scalable, and maintainable applications that can handle everything from sending emails to processing large datasets.

Key Concepts​

  • Tasks Defining and Registering Tasks: The fundamental units of work in Celery. A task is a Python function that can be executed asynchronously by a worker. You can define tasks with simple decorators, configure retries, set rate limits, and more.

  • Workers Worker Architecture: The processes that execute tasks. Workers listen for jobs on a message queue and perform the work defined in your tasks. You can run multiple workers across many machines to scale your processing power horizontally.

  • Broker & Backend Result Backends & Persistence: The two essential external components. The Broker (e.g., RabbitMQ, Redis) is the message transport that passes tasks from your application to the workers. The Result Backend is a data store (e.g., Redis, a SQL database, or a NoSQL store) used to save the state and return values of your tasks.

  • Canvas (Workflows) Workflows & Canvas: A powerful set of primitives for creating complex workflows. You can arrange tasks into simple sequences (chains), run them in parallel (groups), or create complex dependency graphs with callbacks (chords).

  • Beat (Scheduler) Scheduling & Periodic Tasks: Celery's built-in periodic task scheduler. Beat runs as a separate service that sends tasks to the queue at regular intervals, defined by simple time deltas or crontab-style schedules.

  • Concurrency Execution Pools and Concurrency: Celery workers can execute multiple tasks concurrently using different execution pools, including multiprocessing (prefork), cooperative multitasking (eventlet, gevent), or simple in-line execution (solo).

Common Use Cases​

  • Running background jobs for web applications, such as sending confirmation emails or processing uploaded images.
  • Scheduling periodic tasks, like generating nightly reports, cleaning up old data, or syncing with external APIs.
  • Distributing long-running computations or data processing jobs across a cluster of machines.
  • Building real-time data processing pipelines and event-driven systems.
  • Creating complex, multi-step workflows with dependencies and error handling.

Getting Started​

New to Celery? The best place to start is the Getting Started guide, which will walk you through setting up your first Celery application. From there, learn how to define and call tasks in the Defining and Registering Tasks section. To understand how to build powerful, multi-step jobs, check out the guide on Workflows & Canvas.