Skip to main content

Configuring Redis for Results

To use Redis as a result store in this project, you configure the result_backend setting with a Redis URL. The RedisBackend class manages the connection lifecycle, result serialization, and real-time updates via Pub/Sub.

Basic Configuration​

Set the result_backend to a standard Redis URL. The backend supports both standard connections and Unix domain sockets.

from celery import Celery

app = Celery('tasks')

# Standard TCP connection
app.conf.result_backend = 'redis://:password@localhost:6379/0'

# Unix domain socket connection
app.conf.result_backend = 'socket:///path/to/redis.sock?virtual_host=0'

# Optional: Configure connection pool limits
app.conf.redis_max_connections = 20
app.conf.redis_socket_timeout = 5.0

The RedisBackend uses these settings to populate self.connparams, which are then passed to the underlying redis-py client.

Configuring SSL/TLS​

For secure connections, use the rediss:// scheme or provide an SSL configuration dictionary. If you use the rediss:// scheme, you can pass SSL parameters as query strings.

Approach 1: URL Query Parameters​

app.conf.result_backend = (
    'rediss://:password@vandelay.com:6379/1'
    '?ssl_cert_reqs=CERT_REQUIRED'
    '&ssl_ca_certs=/var/ssl/ca.pem'
    '&ssl_certfile=/var/ssl/client-cert.pem'
    '&ssl_keyfile=/var/ssl/client-key.pem'
)

Approach 2: Configuration Dictionary​

Alternatively, use the redis_backend_use_ssl setting. This is often cleaner for complex certificate paths.

app.conf.result_backend = 'rediss://localhost:6379/0'
app.conf.redis_backend_use_ssl = {
    'ssl_cert_reqs': 'CERT_REQUIRED',
    'ssl_ca_certs': '/path/to/ca.crt',
    'ssl_certfile': '/path/to/client.crt',
    'ssl_keyfile': '/path/to/client.key',
}

When SSL is detected, RedisBackend automatically sets the connection_class to redis.SSLConnection.

High Availability with Redis Sentinel​

To use Redis Sentinel for high availability, use the sentinel:// scheme. This triggers the use of SentinelBackend, which can handle multiple sentinel nodes.

# Multiple sentinels separated by semicolons
app.conf.result_backend = (
    'sentinel://host1:26379/0;'
    'sentinel://host2:26379/0;'
    'sentinel://host3:26379/0'
)

app.conf.result_backend_transport_options = {
    'master_name': 'my_master_group',
    'min_other_sentinels': 1,
    'sentinel_kwargs': {
        'socket_timeout': 0.1,
    }
}

If SSL is configured alongside Sentinel, the backend uses SentinelManagedSSLConnection to ensure the connection to the discovered master is encrypted.

Real-time Result Updates​

The RedisBackend utilizes a ResultConsumer to provide real-time task updates via Redis Pub/Sub. This avoids polling the database for results.

When a task is called, the ResultConsumer subscribes to a channel named after the task ID.

  • Subscription: ResultConsumer.consume_from(task_id)
  • Event Loop: ResultConsumer.drain_events() listens for messages on the subscribed channels.
  • Cleanup: Once a result is received (reaching a state like SUCCESS or FAILURE), ResultConsumer.cancel_for(task_id) unsubscribes from the channel.

Troubleshooting and Limits​

Value Size Limit​

Redis has a hard limit of 512MB for string values. RedisBackend enforces this via _MAX_STR_VALUE_SIZE. If your task result exceeds this, a BackendStoreError is raised.

# Found in celery/backends/redis.py
if isinstance(value, str) and len(value) > self._MAX_STR_VALUE_SIZE:
    raise BackendStoreError('value too large for Redis backend')

SSL Scheme Mismatch​

You must use the rediss:// scheme if you provide SSL parameters. Using redis:// with SSL parameters will result in a ValueError.

Sentinel URI Separator​

Ensure multiple sentinel nodes are separated by a semicolon (;). Using commas or other separators will prevent SentinelBackend from correctly parsing the host list in _params_from_url.

Security Warnings​

If ssl_cert_reqs is set to none or optional, the backend will log a warning (W_REDIS_SSL_CERT_NONE or W_REDIS_SSL_CERT_OPTIONAL) because these settings make the connection vulnerable to man-in-the-middle attacks. For production, always use CERT_REQUIRED.