This article is the second day of Python Advent Calendar 2016. http://qiita.com/advent-calendar/2016/python
My name is Matsunaga from LOGICA Co., Ltd. We want to make it easier for you to travel, and we are developing a simpler hotel cross-search service.
Currently, we are developing two products (crawler and media) in-house with Django, but since we are currently developing by one person and there are many servers, we would like to finish the deployment work with one command. Was there.
However, although I wanted to automate the deployment, I did not touch Ansible and Chef did not understand well after doing the tutorial, so I heard that the learning cost seems to be low I decided to make a deployment script using fabric. Did. cuisine is to ensure the sameness (should not be perfect) And since I used Pyenv, Django, Nginx, and gunicorn for both crawlers and media, I decided to make something like Chef's recipe and use it.
I think the following article is easy to understand for a brief explanation of fabric and cuisine. http://qiita.com/pika_shi/items/802e9de8cb1401745caa
The link to the documentation is below. fabric documentation Cooking documentation
The directories are each project (project1, project2), recipes, ssh_keys.
Under the templates directory of each project, put the files you want to mix the endpoints and settings for production, such as django's settings.py and Nginx configuration files. In each of these template files, the variables described in secrets.yml are put in using Jinja2 and then uploaded to the server.
Put the files and binaries you want to upload in project1 / files.
The recipes contain scripts to reuse.
ssh_keys is for remotely pulling the contents of the repository on github.
These are managed on github. Of course, add the secrets.yml and ssh_keys directories to gitignore.
├── project1
│ ├── fabfile.py
│ ├── files
│ │ └── phantomjs-2.1.1-linux-x86_64.tar.bz2
│ ├── secrets.yml
│ ├── secrets.yml.example
│ └── templates
│ ├── gunicorn_conf.py
│ ├── nginx.conf
│ └── settings.py
├── project2
│ ├── fabfile.py
│ ├── secrets.yml
│ ├── secrets.yml.example
│ └── templates
│ ├── gunicorn_conf.py
│ ├── nginx.conf
│ └── settings.py
├── recipes
│ ├── __init__.py
│ ├── django.py
│ ├── git.py
│ ├── gunicorn.py
│ ├── httpd_tools.py
│ ├── nginx.py
│ ├── phantomjs.py
│ ├── pyenv.py
│ ├── redis.py
│ ├── service_base.py
├── requirements.txt
└── ssh_keys
└── github
└── id_rsa
Since I am using amazon linux, I am using yum for package management, but for anything that starts or stops like sudo service ◯◯ start, make the following script the parent class. The installation itself can be surely done by using package_ensure of cuisine, but I wanted to keep it with a descriptive name as a method.
recipes/service_base.py
# -*- coding: utf-8 -*-
from fabric.api import sudo
from fabric.utils import puts
from fabric.colors import green
from cuisine import package_ensure, select_package
select_package('yum')
class ServiceBase(object):
def __init__(self, package_name, service_name):
self.package_name = package_name
self.service_name = service_name
def install(self):
package_ensure(self.package_name)
def start(self):
puts(green('Starting {}'.format(self.package_name)))
sudo('service {} start'.format(self.service_name))
def stop(self):
puts(green('Stopping {}'.format(self.package_name)))
sudo('service {} stop'.format(self.service_name))
def restart(self):
puts(green('Restarting {}'.format(self.package_name)))
sudo('service {} restart'.format(self.service_name))
Using this, create the installation / start / stop script of nginx as follows. Nginx
recipes/nginx.py
from service_base import ServiceBase
class Nginx(ServiceBase):
def __init__(self):
super(Nginx, self).__init__('nginx', 'nginx')
self.remote_nginx_conf_path = '/etc/nginx/nginx.conf'
I will write the upload of the Nginx configuration file later.
Other than the ones managed by yum, for example, Pyenv, Django (because the command is executed by python manage.py ~~) and celery also make common scripts. I will put only pyenv.
Pyenv
recipes/pyenv.py
class Pyenv(object):
def __init__(self):
pass
def install(self):
"""Install pyenv and related tools"""
pyenv_dir = '~/.pyenv'
#Confirmation of pyenv installation
if not dir_exists(pyenv_dir):
run('curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash')
text = """
# settings for pyenv
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
"""
files.append('~/.bashrc', text)
run('source ~/.bashrc')
def install_python(self, py_version):
"""Install the version of python specified on Pyenv"""
#If pyenv is not installed, install it.
if not dir_exists('~/.pyenv'):
self.install()
#Make sure you have the packages needed to build Python installed
packages = ['gcc', 'python-devel', 'bzip2-devel', 'zlib-devel', 'openssl-devel', 'sqlite-devel', 'readline-devel', 'patch']
for package in packages:
package_ensure(package)
if not dir_exists('~/.pyenv/versions/{}'.format(py_version)):
run('pyenv install {}'.format(py_version))
run('pyenv rehash')
def make_virtualenv(self, py_version, env_name):
"""Create an environment with the specified name"""
self.install_python(py_version)
if not dir_exists('~/.pyenv/versions/{}'.format(env_name)):
run('pyenv virtualenv {} {}'.format(py_version, env_name))
run('pyenv rehash')
run('pyenv global {}'.format(env_name))
else:
run('pyenv global {}'.format(env_name))
def change_env(self, env_name):
run('pyenv global {}'.format(env_name))
I use the following function. Since fabric and cuisine do not support Python3, I will switch the remote Python version before and after uploading with the recipes / pyenv.py I wrote earlier w (that is, I have two versions of Python installed on the remote I will.)
If I didn't upload the file, I could go with the remote set to 3, but when I did file_write, the remote also fell with an error unless it was 2, so I'm doing this kind of trouble.
def upload_template(remote_path, local_template_path, variables={}, sudo=None):
"""
Upload by putting variables in the template of jinja2
"""
#Change remote Python to a 2 system environment
pyenv = Pyenv()
pyenv.change_env(VIRTUALENV_NAME_FOR_FABRIC)
local_template_name = local_template_path.split('/')[-1]
local_template_dir = local_template_path.replace(local_template_name, '')
jinja2_env = Environment(loader=FileSystemLoader(local_template_dir))
content = jinja2_env.get_template(local_template_name).render(variables)
file_write(remote_path, content.encode('utf-8'), sudo=sudo)
#Return to the original Python environment
pyenv.change_env(VIRTUALENV_NAME)
Use this to upload Nginx configuration files etc. variables contains the data read from secrets.yml.
upload_template(nginx.remote_nginx_conf_path, 'templates/nginx.conf', variables, sudo=sudo)
For example, write the following in the server name of nginx.conf.
server_name {{ end_point }};
It is OK if you put the endpoint you want to specify in server_name in variables ["end_point "]. I think it's a familiar description for those who usually use Jinja or Django.
The database settings in Django's settings.py are as follows.
secrets.yml
django:
settings:
production:
secret_key:Secret key
databases:
default:
engine: django.db.backends.mysql
name:DB name
user:DB user name
password:DB password
host:DB endpoint
port:DB port
project1/templates/settings.py
DATABASES = {
'default': {
'ENGINE': '{{ databases.default.engine }}',
'NAME': '{{ databases.default.name }}',
'USER': '{{ databases.default.user }}',
'PASSWORD': '{{ databases.default.password }}',
'HOST': '{{ databases.default.host }}',
'PORT': '{{ databases.default.port }}',
},
}
project1/fabfile.py
variables = secrets['django']['settings']['production']
upload_template(settings_file_path, 'templates/settings.py', variables)
Since it is dangerous to put the raw one, I made a script that only installs nginx and builds the Python environment with Pyenv (I have not confirmed the operation because it is a partial copy from the one actually used)
project1/fabfile.py
# -*- coding: utf-8 -*-
import os
import sys
sys.path.append(os.pardir)
import yaml
from jinja2 import Environment, FileSystemLoader
from fabric.api import env, run, sudo, settings, cd
from fabric.decorators import task
from cuisine import package_ensure, select_package, file_write
from recipes.nginx import Nginx
from recipes.pyenv import Pyenv
#Python information
PYTHON_VERSION = "The version you want to use in production"
VIRTUALENV_NAME = "Environment name used in production"
#Remote Python environment when uploading files
PYTHON_VERSION_FOR_FABRIC = "In 2 system"
VIRTUALENV_NAME_FOR_FABRIC = "Environment name for remote fabric"
#Selection of package management method
select_package('yum')
#Load information to embed in template
secrets = yaml.load(file('secrets.yml'))
#env settings Information used to log in to the server to deploy to
env.user = "username"
env.group = "group name"
env.key_filename = "Key path used to log in to the server"
env.use_ssh_config = True
def upload_template(remote_path, local_template_path, variables={}, sudo=None):
pyenv = Pyenv()
pyenv.change_env(VIRTUALENV_NAME_FOR_FABRIC)
local_template_name = local_template_path.split('/')[-1]
local_template_dir = local_template_path.replace(local_template_name, '')
jinja2_env = Environment(loader=FileSystemLoader(local_template_dir))
content = jinja2_env.get_template(local_template_name).render(variables)
file_write(remote_path, content.encode('utf-8'), sudo=sudo)
#Return to the original Python environment
pyenv.change_env(VIRTUALENV_NAME)
@task
def deploy():
#Python environment construction for template upload (remote is not 2 system)
pyenv = Pyenv()
pyenv.install_python(PYTHON_VERSION_FOR_FABRIC)
pyenv.make_virtualenv(PYTHON_VERSION_FOR_FABRIC, VIRTUALENV_NAME_FOR_FABRIC)
#Building a Python environment for production
pyenv.install_python(PYTHON_VERSION)
pyenv.make_virtualenv(PYTHON_VERSION, VIRTUALENV_NAME)
#nginx environment construction
nginx = Nginx()
nginx.install()
variables = {
'end_point': END_POINT,
}
upload_template(nginx.remote_nginx_conf_path, 'templates/nginx.conf', variables, sudo=sudo)
nginx.stop()
nginx.start()
As an impression of actually using it, there is almost no learning cost (because it is just a shell wrapper) However, when it comes to reusing and uploading template files, I feel that Ansible is fine. (I don't know because I haven't touched it)
Since we are a startup, I was initially worried that writing a deployment script would be an extra effort, but as a result of weighing the cost and ease of creating these deployment scripts, I am now very satisfied. Since we have a large number of servers for the scale, it was good to have a deployment script ready. It's really easy.
If there is a better way, or if we are doing something like this, please let us know in the comments section mm
Recommended Posts