Worker-Specific CLI Options

The Celery worker command provides several specialized command-line options to control execution strategies, resource allocation, and node identification. These options are implemented as custom Click parameter types in celery.bin.worker, ensuring that inputs are validated and transformed into the internal structures required by the worker bootsteps.

Execution Pool Selection

The --pool (or -P) option determines the execution mechanism used by the worker. This is managed by the WorkersPool class, which dynamically resolves pool names to their implementation classes.

Pool Resolution Logic

The WorkersPool class inherits from click.Choice and initializes itself with the names of all available pools found in the environment via concurrency.get_available_pool_names().

When a pool is selected via the CLI, the convert method performs the following resolution:

1. It checks if the value is already a pool class.
2. It attempts to get the implementation class using concurrency.get_implementation().
3. It handles configuration overrides: if the user specifies prefork (the default) on the CLI, but the application configuration (app.conf.worker_pool) specifies a different pool, the configured pool takes precedence.

# From celery/bin/worker.py
class WorkersPool(click.Choice):
    def convert(self, value, param, ctx):
        # ... validation logic ...
        value = super().convert(value, param, ctx)
        worker_pool = ctx.obj.app.conf.worker_pool
        if value == 'prefork' and worker_pool:
            # Respect app configuration over the default CLI 'prefork'
            value = concurrency.get_implementation(worker_pool)
        else:
            value = concurrency.get_implementation(value)
            # ... fallback logic ...
        return value

Common pool choices include prefork, eventlet, gevent, solo, and threads.

Dynamic Concurrency (Autoscaling)

The --autoscale option allows a worker to dynamically adjust its pool size based on load. The Autoscale parameter type parses a comma-separated string into a tuple representing the maximum and minimum number of worker processes/threads.

Parsing and Validation

The Autoscale.convert method ensures that the input is correctly formatted and that the resulting tuple is ordered as (max, min).

• Format: max,min (e.g., --autoscale=10,3).
• Single Value: If only one integer is provided (e.g., --autoscale=10), it is interpreted as the maximum, and the minimum is set to 0.
• Sorting: The implementation automatically sorts the values in descending order, so 3,10 and 10,3 both result in (10, 3).

# From celery/bin/worker.py
class Autoscale(ParamType):
    name = "<min workers>, <max workers>"

    def convert(self, value, param, ctx):
        value = value.split(',')
        # ... length validation ...
        if len(value) == 1:
            try:
                value = (int(value[0]), 0)
            except ValueError:
                self.fail(f"Expected an integer. Got {value} instead.")

        try:
            # Ensures the result is always (max, min)
            return tuple(reversed(sorted(map(int, value))))
        except ValueError:
            self.fail("Expected two comma separated integers.")

Worker Node Identification

The --hostname (or -n) option sets the unique identifier for the worker node. The Hostname class processes this input to support dynamic variable expansion.

Variable Expansion

The Hostname class uses celery.utils.nodenames.host_format to replace specific tokens in the hostname string. This allows developers to use the same configuration across multiple machines or processes.

Supported expansion variables defined in celery/utils/nodenames.py include:

• %h: Full hostname (including domain).
• %n: Hostname part only (up to the first dot).
• %d: Domain name part only.
• %i: Process index (useful when running multiple instances on one host).
• %I: Process index with a prefix.

The implementation first ensures a default node name is present using default_nodename before applying the format:

# From celery/bin/worker.py
class Hostname(StringParamType):
    def convert(self, value, param, ctx):
        return host_format(default_nodename(value))

Embedded Scheduler Constraints

The --beat (or -B) flag allows running the Celery Beat scheduler inside the worker process. However, the CeleryBeat parameter type enforces platform-specific restrictions.

Windows Limitation

Because of how process management works on Windows, Celery does not support running the embedded beat scheduler within the worker on that platform. The convert method explicitly checks the application's environment and fails if -B is used on a Windows system.

# From celery/bin/worker.py
class CeleryBeat(ParamType):
    def convert(self, value, param, ctx):
        if ctx.obj.app.IS_WINDOWS and value:
            self.fail('-B option does not work on Windows. '
                      'Please run celery beat as a separate service.')
        return value

On non-Windows systems, this flag simply enables the Beat bootstep within the worker, allowing it to consume periodic task definitions and trigger them.