Skip to main content

Core Application & Configuration

The core of the Celery framework is the application object and its configuration system. This implementation relies on a lazy-loading mechanism that allows developers to define tasks and set configuration parameters before the application is fully initialized.

The Celery Application Object​

The Celery class, defined in celery/app/base.py, serves as the central entry point for the framework. It manages the task registry, configuration, and broker connections.

When instantiating a Celery app, the main argument is used as the namespace for auto-generated task names. This is particularly important when running standalone scripts.

from celery import Celery

# 'myapp' is the 'main' name for this application
app = Celery('myapp', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

In the example above (from examples/app/myapp.py), the add task will be named myapp.add. If main were omitted, Celery would attempt to determine the module name automatically.

Lazy Configuration Management​

Configuration in this codebase is managed lazily through the app.conf attribute. During initialization in Celery.__init__, app.conf is set to a Settings object wrapping a PendingConfiguration proxy.

PendingConfiguration​

The PendingConfiguration class (in celery/app/base.py) allows you to set configuration values on the app before it is finalized. However, as soon as any key is accessed, it triggers the finalization of the configuration.

# In celery/app/base.py
class PendingConfiguration(UserDict, AttributeDictMixin):
    # ...
    @cached_property
    def data(self):
        return self.callback()  # Triggers _finalize_pending_conf

Settings Object​

Once finalized, app.conf becomes a Settings object (defined in celery/app/utils.py). This object is a specialized dictionary that provides:

  • Dual Key Support: It handles both the old uppercase (e.g., CELERY_BROKER_URL) and new lowercase (e.g., broker_url) setting names using _old_key_to_new and _new_key_to_old mapping functions.
  • Property Access: Common settings like broker_url and result_backend are available as properties that also check environment variables.

Loading Configuration​

There are several ways to provide configuration to the application.

config_from_object​

The most common method is config_from_object, which can take a module name or a configuration object.

# examples/django/proj/celery.py
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')

When a namespace is provided, Celery will only look for settings starting with that prefix (e.g., CELERY_BROKER_URL).

config_from_envvar​

The config_from_envvar method allows loading configuration from a module specified in an environment variable:

# In celery/app/base.py
def config_from_envvar(self, variable_name, silent=False, force=False):
    module_name = os.environ.get(variable_name)
    # ...
    return self.config_from_object(module_name, silent=silent, force=force)

detect_settings​

The detect_settings function in celery/app/utils.py is responsible for merging different configuration sources. It enforces a strict rule: you cannot mix old and new setting formats in the same source. If a mixture is detected, it raises an ImproperlyConfigured error with suggestions for renaming.

Loaders​

Loaders are responsible for the actual mechanics of reading configuration and importing modules.

  • AppLoader: The default loader used when a custom Celery instance is created.
  • DefaultLoader: Used for the "default app" (when no explicit app is created). It looks for a celeryconfig.py module by default.

Loaders also manage the lifecycle of the worker, providing hooks like on_worker_init and on_worker_shutdown (defined in celery/loaders/base.py).

Application Finalization​

Finalization is the process where the application transitions from a pending state to a ready state. This happens automatically when app.conf is accessed or when tasks are registered, provided autofinalize is set to True (the default).

The finalize method in celery/app/base.py performs several critical steps:

  1. Loads built-in tasks: Imports tasks required for Celery's internal operations.
  2. Triggers Fixups: Executes "fixups" like the Django integration (celery.fixups.django:fixup).
  3. Evaluates Pending Tasks: Any tasks defined using the @app.task decorator before finalization are now fully bound to the app.
  4. Signals: Dispatches the on_after_finalize signal.

If autofinalize is disabled, attempting to use the task registry before calling app.finalize() manually will result in a RuntimeError.