The Configuration Lifecycle

Celery uses a lazy configuration strategy to provide flexibility during application startup. This allows developers to define settings, load them from objects, or modify them directly on the application instance before the configuration is "locked in" for use by workers or tasks.

The lifecycle transitions through three distinct states: Initialization, Pending, and Finalized.

Initialization

When a Celery instance is created in celery/app/base.py, it initializes its configuration attribute (self._conf) as a Settings object. However, this Settings object is initially backed by a PendingConfiguration proxy rather than a concrete dictionary of values.

# celery/app/base.py - Celery.__init__
self._conf = Settings(
    PendingConfiguration(
        self._preconf, self._finalize_pending_conf),
    prefix=self.namespace,
    keys=(_old_key_to_new, _new_key_to_old),
)

At this stage:

• self._preconf stores any settings passed directly to the Celery constructor (like broker or backend).
• self._finalize_pending_conf is registered as the callback to trigger the transition to the finalized state.

The Pending Phase

The PendingConfiguration class (found in celery/app/base.py) acts as a buffer. It allows you to set configuration values directly on app.conf without triggering the full loading process (which might involve expensive imports or environment lookups).

While in this phase, app.configured remains False. You can update settings using standard dictionary methods or attribute access:

app = Celery(broker='amqp://')
app.conf.task_always_eager = True  # Stored in PendingConfiguration._data
app.conf.update(worker_prefetch_multiplier=10)
print(app.configured)  # False

The PendingConfiguration stores these changes in its internal _data dictionary (which is a reference to the app's _preconf). It remains in this state until a value is read from the configuration.

The Finalization Process

Finalization is the transition from a proxy to a concrete Settings object. It is triggered automatically the first time any key is accessed on app.conf.

Triggering Finalization

The PendingConfiguration class implements a data property decorated with @cached_property. Accessing any key via __getitem__ or iterating over the config triggers this property, which in turn executes the callback (the app's _finalize_pending_conf method).

# celery/app/base.py - PendingConfiguration
@cached_property
def data(self):
    return self.callback()

The _load_config Sequence

When finalization is triggered, the Celery._load_config method performs the following steps:

1. Signal Dispatch: Sends the on_configure signal.
2. Source Loading: If app.config_from_object() was called previously, the loader now imports and reads that object.
3. Setting Detection: Merges the loaded configuration with the _preconf values.
4. Default Application: Iterates through _pending_defaults (promises added via app.add_defaults()) and applies them.
5. State Update: Sets self.configured = True.
6. Post-Configure Signal: Sends the on_after_configure signal with the finalized settings.

The Finalized State

Once finalized, app.conf is a fully populated Settings object (defined in celery/app/utils.py). This class inherits from ConfigurationView, which provides a unified view over multiple layers of configuration:

1. Changes: User-provided overrides.
2. Defaults: The standard Celery default settings.

Key Mapping and Compatibility

The Settings object handles the translation between old-style (uppercase, e.g., CELERY_BROKER_URL) and new-style (lowercase, e.g., broker_url) setting names. It uses the _old_key_to_new and _new_key_to_old mapping functions passed during initialization.

It also provides specialized properties for critical settings that check environment variables before falling back to the configuration:

# celery/app/utils.py - Settings
@property
def broker_url(self):
    return (
        os.environ.get('CELERY_BROKER_URL') or
        self.first('broker_url', 'broker_host')
    )

Important Considerations

Early Finalization "Gotcha"

Because finalization is triggered by key access, inspecting app.conf too early can lock the configuration before you have finished setting it up.

app = Celery()
print(app.conf.broker_url)  # Triggers finalization with defaults
app.config_from_object('myconfig')  # May require force=True now

If you need to load a configuration object after the app has already been finalized, you must use the force=True parameter in config_from_object.

Task Auto-Finalization

Accessing the app.tasks registry also triggers a broader app finalization via app.finalize(auto=True). This ensures that all tasks are bound to the app and the configuration is stable before any task execution logic is invoked.

# celery/app/base.py - Celery.tasks
@cached_property
def tasks(self):
    self.finalize(auto=True)
    return self._tasks