[PYTHON] Use SSL with Celery + Redis

Introduction

Celery is a framework for processing Python tasks in a queue. You can use Redis as a broker of Celery, but when using Azure ʻAzure Redis Cache`, I was addicted to the part that uses SSL, so I will leave the usage here.

environment

• Python: 3.5.2
• Celery: 4.0.0 --Redis: Use Azure Redis Cache. Since it is a Standard plan, it cannot be placed in the Virtual Network and accesses the Host associated with the Global IP, so SSL must be used for security. With the Premium plan, it can be placed in the Private Subnet. How to configure Virtual Network support for Premium Azure Redis Cache (https://docs.microsoft.com/en-us/azure/redis-cache/cache-how-to-premium-vnet)

Assuming that it is used in Flask, a web application, I think it is the same in other environments.

To use SSL

According to What's new in Celery 4.0 (latent call), Celery will be able to use SSL connections with Redis starting with version 4.0. That is.

It is said that it can be used by setting broker_use_ssl, but in the case of Redis, this information It doesn't work even if I set it by swallowing.

Error while reading from socket: (104, 'Connection reset by peer')

You will be plagued by the error. This error is due to using Redis's redis.connection.Connection instead of redis.connection.SSLConnection to connect to Redis inside Celery.

Fix Redis SSL support If you do not set this pull request as a reference, you will not be able to use SSL.

from celery import Celery
from redis.connection import SSLConnection
from .config import broker_use_ssl


def make_celery(app):
    celery = Celery(app.import_name, backend=app.config['CELERY_BACKEND'],
                    broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    celery.conf.update(broker_use_ssl=broker_use_ssl)

    # URL: https://github.com/celery/kombu/pull/634
    if celery.conf.broker_use_ssl:
        celery.backend.connparams.update(celery.conf.broker_use_ssl) # <-here
        celery.backend.connparams['connection_class'] = SSLConnection # <-here

    TaskBase = celery.Task

    class ContextTask(TaskBase):
        abstract = True

        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)

    celery.Task = ContextTask
    return celery

The ʻapp passed as an argument to make_celeryis the Flask application context. After settingbroker_use_ssl, you need to set celery.backend.connparams ['connection_class'] = SSLConnection`.

Also, the documentation says that broker_use_ssl can be set to True or a dictionary, but setting ** True does not work. ** You need to set the dictionary. Unlike Ruby, the dictionary will be judged as False if it is empty, so

broker_use_ssl = {'ssl_cert_reqs': ssl.CERT_NONE}

I think it's a good idea to keep it.

password

In Celery, set a password

CELERY_BROKER_URL = "redis://:password@hostname:port/db_number"

Must be set in the format of. I wasted my knowledge of HTTP, so that? Can I include the password in the URI? I thought, but when I looked at the internal implementation, the URI was properly parsed internally.

Finally

The internal implementation of Celery was unexpectedly complicated and hard to read. Also, in the Azure documentation, How to use Azure Redis Cache with Python

By default, non-SSL ports are disabled for new Azure Redis Cache instances because some Redis clients do not support SSL. At the time of writing this article, the redis-py client does not support SSL.

However, as far as the commit log is concerned, redis-py supports SSL as of 2014. I would like you to update the old description. It's rather confusing.

Reference URL

• github - Fix Redis SSL support
• What’s new in Celery 4.0 (latentcall)