[PYTHON] celery

What is celery

Celery. python job queue Processing framework. Make the worker daemon wait to create a sync processing mechanism, or start the beat daemon to create a scheduled batch process. The documentation is well maintained.

• More efficient than RDB
• backend uses rabbitmq cluster or redis cluster. RDB can be used but not used best practice
• Job flow control is thought to be easy to change
• Start worker daemon. You can start multiple units on one unit. You can start by distributing the cluster.
• job If both the requester and the receiver use a common library in python, don't miss it.

In terms of processing flow, it is close to thinking that a task scheduler is attached to a processing part such as AWS lambda or IBM bluemix OpenWhisk. I am grateful that I can proceed with processing division without worrying about API definition.

worker

If you want to use async job, start the worker daemon.

I start the daemon with celery -A proj worker, but it's hard to understand how to specify proj!

The rough rules are like this.

• Search for Celery instance by checking the current directory and module search path.
• modname: varname must be a Celery instance if there is a: in the proj string
• If proj.app is not a module, it must be a Celery instance (hey)
• If proj.celery is not a module, it must be a Celery instance (hmm)
• If proj.celery is a module
• If proj.celery.app is not a module, it must be a Celery instance (hey)
• If proj.celery.celery is not a module, it must be a Celery instance (hmm)
• Check for variables in proj.celery for subclass instances of Celery
• Check for variables in proj for subclass instances of Celery

In other words, to give an example ...

• In the somename.py file in the current directory If the variable x for the Celery instance exists
• Start with celery -A somename: x worker. Or
• You can also start it with celery -A somename worker.
• In the somename / __ init__.py file as seen from the current directory If the variable x of the Celery instance exists
• Start with celery -A somename worker: x. Or
• You can also start it with celery -A somename worker.
• somename / __ init__.py exists when viewed from the current directory, If the variable x for Celery instance exists in the somename / celery.py file
• Start with celery -A somename.celery: x worker. Or
• Start with celery -A somename.celery worker. Or
• Start with celery -A somename worker.
• Use from __future__ import absolute_import (python 2)
• somename / __ init__.py exists when viewed from the current directory, If the Celery instance variable x exists in the somename / subname.py file
• Start with celery -A somename.subname: x worker. Or
• Start with celery -A somename.subname worker.
• If there is a ʻapp variable in the somename` module in the module search path, assume it is a Celery instance.
• Start with celery -A somename: app worker.
• Start with celery -A somename worker.
• If the Celery instance variable x exists in the somename module in the module search path
• Start with celery -A somename: x worker.
• Start with celery -A somename worker.
• If the module search path somename.celery can be loaded and the Celery instance variable x exists in it
• Start with celery -A somename.celery: x worker
• Start with celery -A somename.celery worker.
• Start with celery -A somename worker.
• Use from __future__ import absolute_import (python 2)

If you look carefully, the appearance of the app will change slightly depending on the pattern.

celery -A somename worker
...
- ** ---------- [config]
- ** ---------- .> app:         __main__:0x106a950

celery -A somename worker
...
- ** ---------- [config]
- ** ---------- .> app:         somename:0x1585410

The confusing cause is the special rules proj.app and proj.celery.

Extremely confusing cases are not explained in the Flask documentation. Flask generally creates a Flask instance ʻapp variable. However, celery also thinks that it must be a Celery instance if it finds the ʻapp variable, so there is a discrepancy.

A workaround would be, for example:

• Change the variable name of Flask instance to something other than ʻapp`
• Specify with a string with : celery -A your_application: celery worker

beat

If you want to create a scheduled batch, start a beat daemon.

celery -A somename beat

Start it separately from the worker daemon. This is basically not scaled out, so you have to be careful when using it.

Note that schedule settings cannot be embedded in the program and will be listed in the basic configuration file.

by the way

I don't really understand the relationship between job queues and celery ... Is that the image ...?

Postscript: Now you have the option of kafka + faust.