[PYTHON] Create a deploy script with fabric and cuisine and reuse it


This article is the second day of Python Advent Calendar 2016. http://qiita.com/advent-calendar/2016/python

Create a deploy script with fabric and cuisine and reuse it

My name is Matsunaga from LOGICA Co., Ltd. We want to make it easier for you to travel, and we are developing a simpler hotel cross-search service.

Currently, we are developing two products (crawler and media) in-house with Django, but since we are currently developing by one person and there are many servers, we would like to finish the deployment work with one command. Was there.

However, although I wanted to automate the deployment, I did not touch Ansible and Chef did not understand well after doing the tutorial, so I heard that the learning cost seems to be low I decided to make a deployment script using fabric. Did. cuisine is to ensure the sameness (should not be perfect) And since I used Pyenv, Django, Nginx, and gunicorn for both crawlers and media, I decided to make something like Chef's recipe and use it.

What are fabric and cuisine?

I think the following article is easy to understand for a brief explanation of fabric and cuisine. http://qiita.com/pika_shi/items/802e9de8cb1401745caa

The link to the documentation is below. fabric documentation Cooking documentation

Directory structure

The directories are each project (project1, project2), recipes, ssh_keys. Under the templates directory of each project, put the files you want to mix the endpoints and settings for production, such as django's settings.py and Nginx configuration files. In each of these template files, the variables described in secrets.yml are put in using Jinja2 and then uploaded to the server. Put the files and binaries you want to upload in project1 / files. The recipes contain scripts to reuse. ssh_keys is for remotely pulling the contents of the repository on github.

These are managed on github. Of course, add the secrets.yml and ssh_keys directories to gitignore.

├── project1
│   ├── fabfile.py
│   ├── files
│   │   └── phantomjs-2.1.1-linux-x86_64.tar.bz2
│   ├── secrets.yml
│   ├── secrets.yml.example
│   └── templates
│       ├── gunicorn_conf.py
│       ├── nginx.conf
│       └── settings.py
├── project2
│   ├── fabfile.py
│   ├── secrets.yml
│   ├── secrets.yml.example
│   └── templates
│       ├── gunicorn_conf.py
│       ├── nginx.conf
│       └── settings.py
├── recipes
│   ├── __init__.py
│   ├── django.py
│   ├── git.py
│   ├── gunicorn.py
│   ├── httpd_tools.py
│   ├── nginx.py
│   ├── phantomjs.py
│   ├── pyenv.py
│   ├── redis.py
│   ├── service_base.py
├── requirements.txt
└── ssh_keys
    └── github
        └── id_rsa

Examples of recipes to reuse

Since I am using amazon linux, I am using yum for package management, but for anything that starts or stops like sudo service ◯◯ start, make the following script the parent class. The installation itself can be surely done by using package_ensure of cuisine, but I wanted to keep it with a descriptive name as a method.


# -*- coding: utf-8 -*-

from fabric.api import sudo
from fabric.utils import puts
from fabric.colors import green
from cuisine import package_ensure, select_package


class ServiceBase(object):
    def __init__(self, package_name, service_name):
        self.package_name = package_name
        self.service_name = service_name

    def install(self):

    def start(self):
        puts(green('Starting {}'.format(self.package_name)))
        sudo('service {} start'.format(self.service_name))

    def stop(self):
        puts(green('Stopping {}'.format(self.package_name)))
        sudo('service {} stop'.format(self.service_name))

    def restart(self):
        puts(green('Restarting {}'.format(self.package_name)))
        sudo('service {} restart'.format(self.service_name))

Using this, create the installation / start / stop script of nginx as follows. Nginx


from service_base import ServiceBase

class Nginx(ServiceBase):

    def __init__(self):
        super(Nginx, self).__init__('nginx', 'nginx')
        self.remote_nginx_conf_path = '/etc/nginx/nginx.conf'

I will write the upload of the Nginx configuration file later.

Other than the ones managed by yum, for example, Pyenv, Django (because the command is executed by python manage.py ~~) and celery also make common scripts. I will put only pyenv.



class Pyenv(object):
    def __init__(self):

    def install(self):
        """Install pyenv and related tools"""
        pyenv_dir = '~/.pyenv'
        #Confirmation of pyenv installation
        if not dir_exists(pyenv_dir):
            run('curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash')
            text = """
            # settings for pyenv
            export PATH="$HOME/.pyenv/bin:$PATH"
            eval "$(pyenv init -)"
            eval "$(pyenv virtualenv-init -)"
            files.append('~/.bashrc', text)
            run('source ~/.bashrc')

    def install_python(self, py_version):
        """Install the version of python specified on Pyenv"""

        #If pyenv is not installed, install it.
        if not dir_exists('~/.pyenv'):

        #Make sure you have the packages needed to build Python installed
        packages = ['gcc', 'python-devel', 'bzip2-devel', 'zlib-devel', 'openssl-devel', 'sqlite-devel', 'readline-devel', 'patch']
        for package in packages:

        if not dir_exists('~/.pyenv/versions/{}'.format(py_version)):
            run('pyenv install {}'.format(py_version))
            run('pyenv rehash')

    def make_virtualenv(self, py_version, env_name):
        """Create an environment with the specified name"""

        if not dir_exists('~/.pyenv/versions/{}'.format(env_name)):
            run('pyenv virtualenv {} {}'.format(py_version, env_name))
            run('pyenv rehash')
            run('pyenv global {}'.format(env_name))
            run('pyenv global {}'.format(env_name))

    def change_env(self, env_name):
        run('pyenv global {}'.format(env_name))

Upload the configuration file with variables embedded using Jinja2

I use the following function. Since fabric and cuisine do not support Python3, I will switch the remote Python version before and after uploading with the recipes / pyenv.py I wrote earlier w (that is, I have two versions of Python installed on the remote I will.) If I didn't upload the file, I could go with the remote set to 3, but when I did file_write, the remote also fell with an error unless it was 2, so I'm doing this kind of trouble.

def upload_template(remote_path, local_template_path, variables={}, sudo=None):
Upload by putting variables in the template of jinja2
    #Change remote Python to a 2 system environment
    pyenv = Pyenv()

    local_template_name = local_template_path.split('/')[-1]
    local_template_dir = local_template_path.replace(local_template_name, '')
    jinja2_env = Environment(loader=FileSystemLoader(local_template_dir))
    content = jinja2_env.get_template(local_template_name).render(variables)
    file_write(remote_path, content.encode('utf-8'), sudo=sudo)

    #Return to the original Python environment

Use this to upload Nginx configuration files etc. variables contains the data read from secrets.yml.

upload_template(nginx.remote_nginx_conf_path, 'templates/nginx.conf', variables, sudo=sudo)

For example, write the following in the server name of nginx.conf.

server_name  {{ end_point }};

It is OK if you put the endpoint you want to specify in server_name in variables ["end_point "]. I think it's a familiar description for those who usually use Jinja or Django.

The database settings in Django's settings.py are as follows.


      secret_key:Secret key
          engine: django.db.backends.mysql
          name:DB name
          user:DB user name
          password:DB password
          host:DB endpoint
          port:DB port


    'default': {
        'ENGINE': '{{ databases.default.engine }}',
        'NAME': '{{ databases.default.name }}',
        'USER': '{{ databases.default.user }}',
        'PASSWORD': '{{ databases.default.password }}',
        'HOST': '{{ databases.default.host }}',
        'PORT': '{{ databases.default.port }}',


variables = secrets['django']['settings']['production']
upload_template(settings_file_path, 'templates/settings.py', variables)

Actual deployment script

Since it is dangerous to put the raw one, I made a script that only installs nginx and builds the Python environment with Pyenv (I have not confirmed the operation because it is a partial copy from the one actually used)


# -*- coding: utf-8 -*-

import os
import sys

import yaml
from jinja2 import Environment, FileSystemLoader
from fabric.api import env, run, sudo, settings, cd
from fabric.decorators import task
from cuisine import package_ensure, select_package, file_write

from recipes.nginx import Nginx
from recipes.pyenv import Pyenv

#Python information
PYTHON_VERSION = "The version you want to use in production"
VIRTUALENV_NAME = "Environment name used in production"

#Remote Python environment when uploading files
VIRTUALENV_NAME_FOR_FABRIC = "Environment name for remote fabric"

#Selection of package management method

#Load information to embed in template
secrets = yaml.load(file('secrets.yml'))

#env settings Information used to log in to the server to deploy to
env.user = "username"
env.group = "group name"
env.key_filename = "Key path used to log in to the server"
env.use_ssh_config = True

def upload_template(remote_path, local_template_path, variables={}, sudo=None):
    pyenv = Pyenv()

    local_template_name = local_template_path.split('/')[-1]
    local_template_dir = local_template_path.replace(local_template_name, '')
    jinja2_env = Environment(loader=FileSystemLoader(local_template_dir))
    content = jinja2_env.get_template(local_template_name).render(variables)
    file_write(remote_path, content.encode('utf-8'), sudo=sudo)

    #Return to the original Python environment

def deploy():
    #Python environment construction for template upload (remote is not 2 system)
    pyenv = Pyenv()

    #Building a Python environment for production
    pyenv.make_virtualenv(PYTHON_VERSION, VIRTUALENV_NAME)

    #nginx environment construction
    nginx = Nginx()
    variables = {
        'end_point': END_POINT,
    upload_template(nginx.remote_nginx_conf_path, 'templates/nginx.conf', variables, sudo=sudo)

Summary / impression

As an impression of actually using it, there is almost no learning cost (because it is just a shell wrapper) However, when it comes to reusing and uploading template files, I feel that Ansible is fine. (I don't know because I haven't touched it)

Since we are a startup, I was initially worried that writing a deployment script would be an extra effort, but as a result of weighing the cost and ease of creating these deployment scripts, I am now very satisfied. Since we have a large number of servers for the scale, it was good to have a deployment script ready. It's really easy.

If there is a better way, or if we are doing something like this, please let us know in the comments section mm

Recommended Posts

Create a deploy script with fabric and cuisine and reuse it
Quickly create a Python data analysis dashboard with Streamlit and deploy it to AWS
Create a temporary file with django as a zip file and return it
Create a star system with Blender 2.80 script
Create a decision tree from 0 with Python and understand it (5. Information Entropy)
Create a native GUI app with Py2app and Tkinter
Create a batch of images and inflate with ImageDataGenerator
Create a 3D model viewer with PyQt5 and PyQtGraph
[Linux] Create a self-signed certificate with Docker and apache
Until you create a machine learning environment with Python on Windows 7 and run it
Get the matched string with a regular expression and reuse it when replacing on Python3
Temporarily save a Python object and reuse it in another Python
Create a web surveillance camera with Raspberry Pi and OpenCV
Associate Python Enum with a function and make it Callable
[Azure] Create, deploy, and relearn a model [ML Studio classic]
Let's create a script that registers with Ideone.com in Python.
Create a homepage with django
Create applications, register data, and share with a single email
A script that makes it easy to create rich menus with the LINE Messaging API
When I deploy a Django app with Apache2 and it no longer reads static files
Steps to create a Job that pulls a Docker image and tests it with Github Actions
Create a heatmap with pyqtgraph
Steps to set up Pipenv, create a CRUD app with Flask, and containerize it with Docker
Create a directory with python
Let's create a tic-tac-toe AI with Pylearn 2-Save and load models-
Create a striped illusion with gamma correction for Python3 and openCV3
Make a thermometer with Raspberry Pi and make it viewable with a browser Part 4
Create a private DMP with zero initial cost and zero development with BigQuery
I tried to create Bulls and Cows with a shell program
I want to create a pipfile and reflect it in docker
Create a C ++ and Python execution environment with WSL2 + Docker + VSCode
Create a simple Python development environment with VS Code and Docker
Create and return a CP932 CSV file for Excel with Chalice
I made a chatbot with Tensor2Tensor and this time it worked
Deploy script to jboss using fabric
Deploy a Django application with Docker
Create a virtual environment with Python!
Giving idempotence to fabric with cuisine
Create a poisson stepper with numpy.random
Write a batch script with Python3.5 ~
Create a file uploader with Django
I made a POST script to create an issue on Github and register it in the Project
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
2. Make a decision tree from 0 with Python and understand it (2. Python program basics)
[AWS] Create a Python Lambda environment with CodeStar and do Hello World
Set up a Lambda function and let it work with S3 events!
Create a stack with a queue and a queue with a stack (from LetCode / Implement Stack using Queues, Implement Queue using Stacks)
Create a Todo app with Django ④ Implement folder and task creation functions
Create a Python3 environment with pyenv on Mac and display a NetworkX graph
Make a decision tree from 0 with Python and understand it (4. Data structure)
You can do it in 5 minutes !? Create a face detection API with FastAPI and OpenCV and publish it on Heroku
Story of making a virtual planetarium [Until a beginner makes a model with a script and manages to put it together]
Create a Python function decorator with Class
Build a blockchain with Python ① Create a class
Create a dummy image with Python + PIL.
[Python] Create a virtual environment with Anaconda
Let's create a free group with Python
Create a GUI app with Python's Tkinter
A memo with Python2.7 and Python3 on CentOS
Create a large text file with shellscript
Create a virtual environment with Python_Mac version