Let Heroku do background processing with Python

Japanese background processing articles are mainly about Ruby, probably because many people use Rails on Heroku. I also wanted a Python article, so I'll keep a record of it, including my own notes.

Why do you process it in the background in the first place?

On Heroku, web dyno handles normal HTTP access (Reference Dynos and the Dyno Manager), web dyno will time out in 30 seconds. If you think about it normally, it takes 30 seconds to process the web, which is a very heavy process or a special process that takes time.

However, there may be times when you want to do that.

Now, implementation

On Heroku's devcenter, properly for languages other than Ruby It was written in Worker Dynos, Background Jobs and Queueing. Here, I will write about Python.

I should say that it remains almost here.

Preparation

1. Install Redis

$ brew install redis #If you don't develop locally, you don't have to enter
$ pip install redis

2. Install RQ (Redis Queue)

$ pip install rq

3. List the required libraries

$ pip freeze > requirements.txt

4. Make a guy who will call the background process.

It seems that this guy will call up the jobs accumulated in redis one by one.

`worker.py`


import os

import redis
from rq import Worker, Queue, Connection

listen = ['high', 'default', 'low']

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

5. Write the process you want to do in the background

`main.py`


from rq import Queue
from worker import conn
from bottle import route, run


q = Queue(connection=conn)

@route('/index')
def index():
	result = q.enqueue(background_process, 'Argument 1')
	return result

def background_process(name):
	#Write a time-consuming process here
	return name * 10
	
	
run(host="0.0.0.0", port=int(os.environ.get("PORT", 5000)))

6. Write worker.py in Procfile

`Procfile`


web:python main.py
worker:python worker.py

7. Add redistogo to heroku add-on

$ heroku addons:create redistogo

8. Deploy

$ git add .
$ git commit -m "add worker"
$ git push heroku master

9. Start the worker

$  heroku scale worker=1

Thank you, the procedure is complete

At this point, I think the background processing will work properly. Also, if you put the redis server properly in the local environment, you can check that the background processing works if you start main.py and worker.py properly.