[PYTHON] Asynchronous processing implementation in Django (Celery, Redis)

About this article

When I'm making a web application with Django, I think that asynchronous processing is indispensable, so I'd like to introduce its introduction with examples. The software used is Celery (celery), -Asynchronous processing by django + Celery -How to implement asynchronous processing in Django I implemented it by referring to the articles in this area.

environment

・ Python 3.6.8 ・ Django 2.2.7 ・ CentOS7

What is asynchronous processing?

Actually, I didn't know the word asynchronous until I actually faced the problem. Or rather, it feels like I didn't even notice that this kind of processing itself is no longer the default. For example, let's say you have an app that puts a value in a form and returns the result via an API or something. There is no need to worry because there is an immediate response if there is only one process. But what if you run a lot of processing at once and it takes 30 minutes to respond? The user must keep the browser open until the result is returned. If you move to another page or close the browser, the process will stop. Introducing here is ** asynchronous processing **. By throwing a task that takes a long time to respond to asynchronous processing, it will continue the processing behind the scenes and return the result regardless of whether the browser is closed or not.

Introduction of Celery / Redis

More detailed explanations are carefully written on the site introduced above, so here. Now, I would like to actually install Celery on the server and work with Django. First of all, the installation and execution environment is Centos7

CentOS

pip3 install celery
pip3 install django-celery-results
yum -y install epel-release
yum search redis
yum install -y redis

The last one I put in is called Redis (remote dictionary server). It seems that it can be used for various purposes when I investigate it, but this time I will ask him to work as a broker. The flow is that a task is generated from the front ⇒ Redis passes the task to Celery ⇒ Celery executes the task. By the way, it seems that RabbitMQ can also be used as a Broker except for Redis. You don't need to mess with the config file to use Celery and Redis. Redis

CentOS

redis-server

If you type, it will start. I think it's okay to play around with it depending on the application, but I didn't need to set anything for my own use.

Django / Celery integration

The Django project file created for testing is below

Django

Project
├── manage.py
├── db.sqlite3
├── Tool
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── forms.py
│   ├── models.py
│   ├── templates
│   │   └── main
│   │       └── celery.html
│   ├── urls.py
│   └── views.py
└── Site
    ├── __init__.py
    ├── celery.py
    ├── settings.py
    ├── tasks.py
    ├── urls.py
    └── wsgi.py

First, describe the Celery settings in settings.py.

Project/Site/settings.py

INSTALLED_APPS = [
・ ・ ・
・ ・ ・
    'django_celery_results',
]
・ ・ ・
#Celery settings
CELERY_BROKER_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379/1')
CELERY_RESULT_BACKEND = "django-db"

I made Redis available as a broker and set it to save the job execution result in the DB. Next, create celery.py in the same hierarchy as settings.py and write the following.

Project/Site/celery.py

import os
from celery import Celery

#Django config file for celery(settings.py)Specify
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Site.settings')★

app = Celery('Site')★

#You can also create a declaration that uses the Django config file as the celery config, and a config file for the celery.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

★ The place I added is the one I edited for this time. Other than that, quoted from Official In addition, add the following to __init__.py in the same hierarchy.

Project/Site/__init__.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

Now you're ready to go. I'm using Celery pounding in an actual project, but the settings are the same as above. I am very grateful to be able to use the official one as it is.

task / screen creation

Next, create the task you want to execute in celery. Create tasks.py in the same hierarchy as setting.py and write the process you want to execute asynchronously there. I can't think of a good example this time, so I will write the task according to the formula.

Project/Site/tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task
import time

@shared_task
def add(x, y):
    print("processing")
    z = x + y
    time.sleep(10)
    print("Processing completed")
    return z

The important thing is the decorator @ shared_task, which you can prefix the function to recognize as a celery task. After that, edit the screen to execute this and the connection (urls.py) to set up the test environment.

First, create a screen.

Project/Tool/templates/main/celery.html

<html lang="ja">
    <head>
	  <title>{% block title %}celery test{% endblock %}</title>
	  <meta http-equiv="refresh" content="10; URL="> 
	</head>        
	<body>           
	  <h1>Even beginners want to do asynchronous processing</h1>                
			<p>Enter a value</p>
			<form method="post" action="celery">
				{% csrf_token %}
			  <tbody>
				<td><input type="text" name="input_a"></td>
				<td>+</td>
				<td><input type="text" name="input_b"></td>
			  </tbody>
			  <input type="submit" name="add_button" value="Calculation!">
			</form>    
	  <h1>result!</h1>
	        <tbody>        
				<td>Output value:</td>
				<td>{{ result }}</td>
			</tbody>
	</body>
</html>

It is like this. You can design it with bootstrap, but I don't care about the design this time so that you can use it as it is even if you copy it.

Then update views.py.

Project/Tool/views.py

from django.shortcuts import render
from django.http import HttpResponse
from django_celery_results.models import TaskResult
from YMD_Site.tasks import add

def celery(requests):
    template = loader.get_template('main/celery.html')
    if 'add_button' in requests.POST:
        x = int(requests.POST['input_a'])
        y = int(requests.POST["input_b"])
        task_id = add.delay(x,y)★
    
    result = list(TaskResult.objects.all().values_list("result",flat=True))
    if len(result) == 0:
        resilt[0] = 0
    context = {'result': result[0]}
    return HttpResponse(template.render(context, requests))

I will explain a little. Although it is marked with a star, add .delay when executing a task (function) for asynchronous processing.

Project/Tool/views.py

    result = list(TaskResult.objects.all().values_list("result",flat=True))
    if len(result) == 0:
        resilt[0] = 0
    context = {'result': result[0]}

In this part, the result processed by Celery is saved in a dedicated table (django_celery_results_taskresult). This time I try to get the result from there.

Finally edit the connection part,

Project/Tool/urls.py

urlpatterns = [
    path('celery', views.celery),
]

This completes the code editing. After that, various executions are performed on the server!

Run

When executing, migrate the DB. Celery's table will be created automatically by migrating.

CentOS

[root@test1 Project]Python3 manage.py makemigrations
[root@test1 Project]Python3 manage.py migarate

OK if you have a Celery table

Next, start Redis,

CentOS

[root@test1 Project]# redis-server
8066:C 02 Dec 15:17:51.448 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
8066:M 02 Dec 15:17:51.449 * Increased maximum number of open files to 10032 (it was originally set to 4096).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 8066
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

8066:M 02 Dec 15:17:51.451 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
8066:M 02 Dec 15:17:51.451 # Server started, Redis version 3.2.12

OK if something like this comes out

Then start Celery. Run the Celery command in the folder where manage.py is located.

CentOS

[root@test1 Project]# celery -A Site worker -l info
=====FINISH=====
/usr/local/lib/python3.6/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,
 
 -------------- celery@test1 v4.3.0 (rhubarb)
---- **** ----- 
--- * ***  * -- Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.6.1810-Core 2019-12-03 07:07:54
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         Site:0x7f7a57d825f8
- ** ---------- .> transport:   redis://localhost:6379/1
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . YMD_Site.tasks.add

[2019-12-03 07:07:54,586: INFO/MainProcess] Connected to redis://localhost:6379/1
[2019-12-03 07:07:54,597: INFO/MainProcess] mingle: searching for neighbors
[2019-12-03 07:07:55,625: INFO/MainProcess] mingle: all alone
[2019-12-03 07:07:55,644: WARNING/MainProcess] /usr/local/lib/python3.6/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-12-03 07:07:55,645: INFO/MainProcess] celery@test1 ready.

The command celery -A Site worker -l info, but the last -l info allows you to specify the log level to output. (ERROR etc.) Change the Site part of the command to your own project name as appropriate.

Finally (many of them start ...) run server and try to enter the page.

CentOS

[root@test1 Project]# python3 manage.py runserver 0.0.0.0:8000

After confirming that it started successfully, go to http: // server IP address: 8000 / celery!

It will be displayed like this.

Enter a value and press Calc.

This is the point, but when you press the button, the page reloads. This process sleeps for 10 seconds after the calculation, so the result is not returned immediately, but the page does not go into a standby state and there is no problem even if you close the browser. After pressing the button, you can see that the server continues processing behind the scenes. If you wait for a while The result is back!

By the way, on the Celery screen on the server side

CentOS

[2019-12-03 07:24:02,046: INFO/MainProcess] Received task: Site.tasks.add[52948e50-4803-4c11-87ab-8eab088e1d7d]  
[2019-12-03 07:24:02,053: WARNING/ForkPoolWorker-1]processing
[2019-12-03 07:24:12,064: WARNING/ForkPoolWorker-1]Processing completed
[2019-12-03 07:24:12,135: INFO/ForkPoolWorker-1] Task Site.tasks.add[52948e50-4803-4c11-87ab-8eab088e1d7d] succeeded in 10.082466656996985s: 2

You can see that the process is working properly. Also, when you enter the Django management screen, a page is automatically created and you can check the processing contents.

Summary

This time, the purpose was to try asynchronous processing, so I made the screen etc. relatively appropriately. The html is processed to automatically refresh every 10 seconds. Because if you update it manually to reflect the result on the screen, the form will be resubmitted ... It seems that you can take measures around here, but this time it is not relevant, so I omitted it.

Actually, I wanted to include serial processing, parallel processing, and multi-starting of workers, but it has already become quite long, so I will write it in another article. Also, there are many issues to be solved when using it, so I hope I can write about that as well. Thank you to everyone who has read this far.