Embedded Beat Service

To run the Celery Beat scheduler directly within a worker process, you can enable the embedded Beat service by initializing the worker with the beat argument.

from celery.worker.worker import WorkController

# Initialize a worker with the embedded Beat service enabled
worker = app.WorkController(
    concurrency=4,
    beat=True,
    schedule_filename='celerybeat-schedule',
    scheduler='celery.beat.PersistentScheduler'
)
worker.start()

Architecture of the Embedded Service

The embedded Beat service is implemented as a worker bootstep in celery.worker.components.Beat. When the worker starts, this bootstep initializes an EmbeddedService which manages the periodic task scheduler loop.

The Beat Bootstep

The Beat class is a conditional bootstep. It only activates if the beat argument is passed to the worker.

# celery/worker/components.py

class Beat(bootsteps.StartStopStep):
    """Step used to embed a beat process.

    Enabled when the ``beat`` argument is set.
    """

    label = 'Beat'
    conditional = True

    def __init__(self, w, beat=False, **kwargs):
        self.enabled = w.beat = beat
        w.beat = None
        super().__init__(w, beat=beat, **kwargs)

    def create(self, w):
        from celery.beat import EmbeddedService
        # ... validation logic ...
        b = w.beat = EmbeddedService(w.app,
                                     schedule_filename=w.schedule_filename,
                                     scheduler_cls=w.scheduler)
        return b

Threaded vs. Process Execution

The EmbeddedService factory in celery.beat determines how the scheduler runs based on the environment:

1. Process-based (_Process): By default, if multiprocessing is available, Beat runs as a child process of the worker.
2. Thread-based (_Threaded): If multiprocessing is unavailable or if explicitly requested, Beat runs in a background thread. When running as a thread, the max_interval is forced to 1 second to ensure the worker can shut down promptly.

Configuration Options

You can customize the behavior of the embedded service using the following parameters passed to the WorkController or defined in the Celery configuration:

Parameter	Description	Configuration Key
beat	Enables the embedded service.	N/A (CLI -B)
schedule_filename	Path to the schedule database file.	beat_schedule_filename
scheduler	The scheduler class to use.	beat_scheduler
max_interval	Max seconds to sleep between ticks.	beat_max_loop_interval

Example: Custom Scheduler and Filename

worker = app.WorkController(
    beat=True,
    schedule_filename='/var/run/celery/beat-schedule',
    scheduler='my_project.schedulers.CustomScheduler'
)

Troubleshooting and Constraints

Greenlet Incompatibility

The embedded Beat service is not compatible with greenlet-based execution pools like gevent or eventlet. The Beat bootstep performs a defensive check during initialization and will raise an ImproperlyConfigured error if a greenlet pool is detected.

# celery/worker/components.py

def create(self, w):
    pool_module = w.pool_cls if isinstance(w.pool_cls, str) else w.pool_cls.__module__
    if pool_module.endswith(('gevent', 'eventlet')):
        raise ImproperlyConfigured(
            'Embedded beat does not work with gevent/eventlet.'
        )
    # ...

Lifecycle Dependency

Because the service is embedded, its lifecycle is tied directly to the worker process:

• If the worker process is terminated, the Beat scheduler stops immediately.
• In process-based mode, the worker sends a termination signal to the Beat child process during shutdown.
• In threaded mode, the worker waits for the Beat thread to finish its current tick (hence the 1-second max_interval limit).

Database Locking

When using the default PersistentScheduler, the schedule_filename creates a local database file (usually using shelve). Ensure that only one worker process is running with the embedded Beat service against the same schedule file to avoid database corruption or locking issues.