Architecture Overview

This section contains architecture diagrams and documentation for celery.

Available Diagrams

Celery Ecosystem Context Diagram

This system context diagram illustrates the Celery ecosystem and its interactions with external entities.

At the core is the Celery Framework, which consists of several key components:

• Worker Architecture: The primary execution engine that consumes tasks from the broker and executes them.
• Introduction to Celery Beat: A scheduler that sends periodic tasks to the message broker based on a defined schedule.
• CLI Architecture: A command-line interface used by developers and operators to manage workers, inspect status, and control the cluster.

External to the core framework are:

• Celery Ecosystem Context Diagram: The user's application that uses the Celery SDK to enqueue tasks and retrieve results.
• Celery Ecosystem Context Diagram: An external service (like RabbitMQ, Redis, or Amazon SQS) that acts as a transport for tasks and control messages.
• Result Backends & Persistence: An external storage service (like Redis, a SQL database, or Memcached) where task results and states are persisted.
• Monitoring & Events: External applications (like Flower) that subscribe to events published by workers to provide real-time monitoring and analytics.

The diagram shows the flow of tasks from producers (Client App, Beat) to consumers (Worker) via the Broker, and the storage of results in the Backend. It also highlights the control path from the CLI to the Workers through the Broker.

Key Architectural Findings:

• Celery relies on Kombu to interface with various Message Brokers (RabbitMQ, Redis, SQS, etc.) for task distribution.
• Result Backends (Redis, SQLAlchemy, MongoDB, etc.) are used to store task outcomes and metadata.
• Celery Workers publish events to the broker, which can be consumed by monitoring tools like Flower.
• The Celery CLI uses a broadcast 'mailbox' system via the broker to send remote control commands to workers.
• Celery Beat maintains a local or remote schedule database to trigger periodic tasks.

Celery Internal Component Architecture

The Celery component architecture is centered around the celery.app instance, which serves as the primary entry point for configuration, task registration, and component orchestration.

The architecture is divided into several key subsystems:

• CLI & Application Core: The celery.bin.celery module provides the command-line interface that initializes the celery.app. The app manages core services like celery.app.amqp for messaging, celery.app.control for remote commands, and celery.app.events for monitoring.
• Worker Subsystem: The celery.apps.worker (managed by WorkController) is the execution engine. It uses celery.bootsteps to manage a dependency graph of internal components such as the networking Hub, the execution Pool, and the Consumer. The Consumer itself is a complex component that handles the actual message lifecycle, including connection management and task dispatching.
• Scheduling: celery.beat is a dedicated service for periodic task triggering, which can run as a standalone process or be embedded within a worker.
• Orchestration & Results: celery.canvas provides the "Canvas" system for defining complex task workflows like chains, groups, and chords. Task outcomes are persisted and retrieved via pluggable celery.backends.base.

The system relies heavily on celery.bootsteps for its modular and extensible initialization process, allowing different worker components to be started and stopped in the correct order based on their dependencies.

Key Architectural Findings:

• The Celery class in celery.app.base is the central hub, lazily instantiating components like amqp, backend, control, and events.
• The worker architecture is built on celery.bootsteps, which uses a directed acyclic graph (DAG) to manage the lifecycle of components like the Pool, Hub, and Consumer.
• The Consumer is a high-level bootstep that manages its own internal blueprint of sub-steps (Connection, Heart, Gossip, etc.) for broker interaction.
• celery.canvas primitives (signatures) are decoupled from execution, allowing workflows to be defined and then sent to the broker via the app's producer pool.
• The CLI in celery.bin acts as a bootstrap layer, discovering the application instance and then handing off control to the worker or beat services.

Celery Asynchronous Task Execution Flow

This sequence diagram traces the lifecycle of a Celery task from its invocation on the client side to its execution in a worker and the storage of its result.

1. Task Invocation: The client calls delay() or apply_async() on a task instance. This triggers the Celery method.
2. Message Preparation: The Celery app uses its AMQP component to create a task message and then publishes it to the message broker (e.g., RabbitMQ or Redis) using a kombu.Producer.
3. Task Receipt: The worker's Consumer receives the message from the broker. It identifies the task's execution strategy and creates a Request object.
4. Execution Scheduling: The consumer passes the request to the WorkController, which dispatches it to the execution pool (e.g., prefork, eventlet).
5. Task Execution: Inside the pool worker, the celery.app.trace logic wraps the actual task function. It handles pre-run signals, executes the function, and manages post-run activities.
6. Result Storage: Upon successful completion, the tracer calls the BaseBackend method of the configured result backend to persist the task's return value.

Key Architectural Findings:

• Tasks are initiated via apply_async which delegates to Celery.send_task.
• The AMQP component handles the low-level message creation and publishing via kombu.
• The worker's Consumer uses a 'strategy' pattern to handle incoming messages based on task type.
• Task execution is decoupled from message consumption using an execution pool (e.g., prefork).
• The trace_task function in celery.app.trace is the core wrapper that executes the task and interacts with the result backend.

Celery Domain Entities and Canvas Primitives

The data model for Celery's domain entities and Canvas primitives centers around the Task and its execution context, as well as the Introduction to Task Signatures which allows for complex workflow compositions.

Key Entities:

• Task: The base class for all executable units. It defines execution parameters like retries, rate limits, and time limits.
• Context: Represents the runtime state of a task execution (request). it tracks the task ID, arguments, and relationships to other tasks in a workflow (parent, root, chain, chord).
• AsyncResult: A handle to a task's result and state. It maintains a recursive relationship with its parent (for chains) and children (for subtasks).
• Signature: A "lazy" task invocation that wraps a task name with arguments and options. It is the building block for Canvas primitives.
• Canvas Primitives:
  • Group: Executes a list of signatures in parallel.
  • Chain: Executes a list of signatures sequentially, passing the result of one to the next.
  • Chord: A group with a callback (body) that executes after all tasks in the group (header) complete.
• Schedules: Define when periodic tasks should run, with implementations for fixed intervals (schedule), cron-like patterns (crontab), and solar events.

Relationships:

• A Task is invoked via a Signature.
• Executing a Signature or Task returns an AsyncResult.
• Canvas Primitives (Group, Chain, Chord) are specialized Signatures that compose other signatures.
• GroupResult aggregates multiple AsyncResult objects from a group execution.
• Context links a running task to its position in a workflow via parent_id, root_id, and chord.

Key Architectural Findings:

• Signature is the base class for all Canvas primitives (Group, Chain, Chord) and inherits from Python's dict.
• AsyncResult maintains a tree structure via 'parent' and 'children' fields, enabling result tracking across complex workflows.
• Task execution state is encapsulated in a Context object, which is pushed onto a thread-local request_stack.
• Chords are composed of a 'header' (a Group) and a 'body' (a Signature).
• Schedules (schedule, crontab, solar) all inherit from BaseSchedule and provide logic for 'is_due' and 'remaining_estimate'.

Typical Celery Production Deployment Architecture

This deployment architecture diagram illustrates a typical production setup for Celery, based on the infrastructure configurations found in the codebase (Docker Compose, Helm charts, and systemd units).

The architecture is centered around a Message Broker (typically RabbitMQ or celery.backends.redis), which acts as the communication hub. Client Applications (Producers) submit tasks to the broker. The Celery Cluster consists of multiple Worker Nodes that pull and execute these tasks asynchronously, and a Celery Beat node that handles scheduling for periodic tasks.

Task results are stored in a Result Backend (such as Redis, a SQL database, or cloud storage like DynamoDB/Azure Blob), where they can be retrieved by the client. For monitoring and management, Flower provides a web-based dashboard that interacts with the broker to track real-time events and issue remote control commands to workers. The diagram also reflects the use of the Celery CLI for manual inspection and cluster management, as seen in the project's documentation and health check implementations.

Key Architectural Findings:

• Docker Compose configuration defines a multi-container environment with RabbitMQ (broker), Redis (backend), and specialized backends like DynamoDB and Azurite.
• Helm charts implement a scalable Deployment for workers with a configurable replica count and health probes using 'celery inspect'.
• Systemd service files distinguish between the worker process (using 'celery multi' for process management) and the scheduler ('celery beat').
• Monitoring is primarily handled by Flower, which connects to the broker to capture worker events and provide a management API.
• The project supports a wide array of result backends, including filesystem, database, cache, and various cloud-native storage services.

Celery Task Lifecycle States

The Celery Task Lifecycle diagram illustrates the various states a task transitions through from its initial submission to its final resolution.

Key components and transitions:

• celery.states: The initial state of a task. It is the default state when a task is first submitted and before it is picked up by a worker.
• celery.states: A state used primarily in events to indicate that a worker has received the task message from the broker.
• celery.states: Indicates that a worker has begun executing the task. This state is only recorded if the task_track_started setting is enabled.
• celery.states: A terminal state indicating the task completed successfully and returned a result.
• celery.states: A terminal state indicating the task failed with an exception or exceeded its retry limit.
• celery.states: A state indicating the task failed but is being rescheduled for another attempt. This leads back to the PENDING state for the next execution.
• celery.states: A terminal state indicating the task was cancelled, either before it started or while it was running (terminated).
• celery.states: A state (mostly for events) indicating the worker rejected the task, for example, if it was malformed or expired.
• celery.states: A state indicating the task was intentionally ignored by the user (e.g., via an Ignore exception).

The diagram shows the flow from submission through the worker's handling logic, including the possibility of retries and revocations at different stages.

Key Architectural Findings:

• States are defined as constants in celery/states.py.
• PENDING is the default state in the result backend when no record exists for a task ID.
• RECEIVED and REJECTED are primarily event-driven states and are not typically persisted in result backends.
• STARTED state tracking is optional and depends on the task_track_started configuration.
• RETRY triggers the creation of a new task message, effectively restarting the lifecycle for that task ID.
• REVOKED can be reached from PENDING, RECEIVED, or STARTED states if a revocation signal is received or the task is found in the revoked list.
• IGNORED is a specific state reached when a task raises the celery.exceptions.Ignore exception.