Skip to main content

Message Security and Signed Serialization

The message security system in this codebase provides a mechanism for ensuring the integrity and authenticity of tasks sent over the wire. By using the auth serializer, Celery can sign message bodies using RSA private keys and verify them using X.509 certificates.

Core Cryptographic Components​

The security implementation relies on two primary classes for handling cryptographic material, both located in the celery.security package.

Private Keys​

The PrivateKey class in celery.security.key wraps an RSA private key. It is responsible for signing the serialized message body.

from celery.security.key import PrivateKey

# Loading a private key from a PEM string
key = PrivateKey(pem_data, password=b'secret')

# Signing data using a digest algorithm (e.g., SHA256)
signature = key.sign(b"message body", digest_algorithm)

Note: This implementation strictly supports RSA keys. If a non-RSA key is provided, the constructor raises a ValueError.

Certificates​

The Certificate class in celery.security.certificate wraps an X.509 certificate. It is used by the receiver to verify the signature attached to a message.

from celery.security.certificate import Certificate

cert = Certificate(cert_pem_data)

# Verify a signature
cert.verify(body, signature, digest_algorithm)

Like PrivateKey, the Certificate class only supports RSA public keys. It also provides utility methods like has_expired() to check the certificate's validity and get_id() to generate a unique identifier based on the issuer and serial number.

Trust Management with Certificate Stores​

To verify incoming messages, a worker must have access to the public certificates of the clients or other workers that sent them. This is managed by certificate stores.

CertStore​

The base CertStore class is an in-memory repository of trusted certificates. Certificates are indexed by their unique ID (returned by Certificate.get_id()).

FSCertStore​

The FSCertStore class is a file-system-based implementation that automatically loads certificates from a directory or a glob pattern.

from celery.security.certificate import FSCertStore

# Load all .pem files from a directory
store = FSCertStore('/var/ssl/trusted/*.pem')

During initialization, FSCertStore iterates through the files and adds them to the store. If it encounters an expired certificate, it raises a SecurityError.

The Secure Serialization Flow​

The SecureSerializer in celery.security.serialization orchestrates the signing and verification process. It acts as a wrapper around a standard serializer (like JSON).

Serialization (Signing)​

When a message is serialized:

  1. The data is first serialized using the base serializer (e.g., json.dumps).
  2. The PrivateKey signs the resulting byte string.
  3. The SecureSerializer packs the body, signature, and the signer's certificate ID into a single payload.

The packed format uses a specific separator (\x00\x01) and Base64 encoding for the metadata: base64(signer_id) + \x00\x01 + base64(signature) + \x00\x01 + content_type + \x00\x01 + content_encoding + \x00\x01 + body

Deserialization (Verification)​

When a message is received:

  1. The payload is unpacked into its constituent parts.
  2. The signer_id is used to look up the corresponding Certificate in the CertStore.
  3. The certificate verifies the signature against the message body.
  4. If verification succeeds, the body is passed to the base serializer (e.g., json.loads) to reconstruct the original data.

Integration and Configuration​

The security system is typically initialized via the setup_security() method on the Celery application instance. This method configures the auth serializer and registers it with the Kombu registry.

Application Setup Example​

from celery import Celery

app = Celery('myapp')

app.conf.update(
    task_serializer='auth',
    accept_content=['auth'],
    security_key='/path/to/worker.key',
    security_certificate='/path/to/worker.pem',
    security_cert_store='/path/to/certs/*.pem',
)

app.setup_security()

The setup_security function (found in celery/security/__init__.py) performs several critical checks:

  • It ensures task_serializer is set to 'auth'.
  • It ensures accept_content is strictly limited to ['auth'].
  • It disables all other insecure serializers to prevent downgrade attacks.

Security Considerations​

RSA Requirement​

The implementation specifically uses RSA with PSS padding (padding.PSS) and MGF1. This is hardcoded in both PrivateKey.sign and Certificate.verify. Attempting to use other algorithms like ECDSA will result in errors during initialization.

Content Acceptance​

A critical security requirement is setting accept_content = ['auth']. If other serializers are allowed, an attacker could bypass the signature verification by sending a message serialized with a standard, unsigned serializer like json or pickle. The setup_security() function enforces this by raising an ImproperlyConfigured exception if the settings are not restrictive enough.