[Python/Django] Summary of frequently used commands (4) -Part 2- <Production operation: Amazon EC2 (Amazon Linux 2)>
background
Here are some frequently used commands when developing with Python/Django.
I would like to take this opportunity to thank you for solving the problem by borrowing the wisdom of our predecessors, and I am very sorry to say that I will summarize it here as my own memo.
environment
(Production environment)
- AWS EC2 (Amazon Linux 2) --Python 3.7.9 * Default on Amazon Linux 2 as of 12/10/2020
- Django 3.1.3 --PostgreSQL 11.5 * Same as above --Nginx 1.12 * Same as above
- Gunicorn
- Putty 0.74
(Development environment)
- Windwos 10 Pro
- Python 3.9.0
- Django 3.1.3
- PostgreSQL 13.1
- Nginx 1.19.5
- Gunicorn
- Putty 0.74
Premise
Build a WEB server, AP server, and DB server on Amazon EC2 (Elastic Computing Cloud): Amazon Linux 2.
(Diagram)
--Execute the following commands with the SSH connection to the EC2 instance.
- When connecting to Amazon EC2 (Elastic Compute Cloud) using PuTTY, refer to "Connecting from Windows to Linux instance using PuTTY".
Please also refer to [Python/Django] Summary of frequently used commands (4) -Part 1- <Production operation: Amazon EC2 (Amazon Linux 2)>.
9. Set environment variables.
9-1. Open the file for setting environment variables.
Terminal
$ vi ~/.bash_profile
9-2. Set environment variables.
You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)
~/.bash_profile
(...Abbreviation...)
export DJANGO_SECRET_KEY='<SECRET given when creating a Djnago project_KEY(50 digits)＞'
export DJANGO_SETTINGS_MODULE=<Project name>.settings
export DB_USER=<Arbitrary user name>
export DB_PASSWORD=<Arbitrary password>
export ALLOWED_HOSTS=<Elastic IP> (or <unique domain name>) *
export AWS_SES_ACCESS_KEY_ID=<Access key ID obtained when setting up Amazon SES usage>
export AWS_SES_SECRET_ACCESS_KEY=<Secret Access key obtained when setting up Amazon SES usage>
9-3. To make the edited contents valid, execute the following.
Terminal
$ source ~/.bash_profile
9-4. Check the contents of the environment variables.
Terminal
$ env
(...Abbreviation...)
XDG_SESSION_ID=514
HOSTNAME=ip-XXX-XXX-XXXX-XXX.ap-northeast-1.compute.internal
AWS_SES_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=XXX.XXX.XXX.XXX 57002 22
DJANGO_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SSH_TTY=/dev/pts/0
USER=<Arbitrary user name>
AWS_SES_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
(...Omission...)
ALLOWED_HOSTS=<Original domain name>
MAIL=/var/spool/mail/<Arbitrary user name>
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/<Arbitrary user name>/.local/bin:/home/<Arbitrary user name>/bin
PWD=/home/<Arbitrary user name>
DJANGO_SETTINGS_MODULE=<Project name>.settings
DB_PASSWORD=<Arbitrary password>
LANG=ja_JP.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/<Arbitrary user name>
LOGNAME=<Arbitrary user name>
DB_USER=<Arbitrary user name>
SSH_CONNECTION=XXX.XXX.XXX.XXX 57002 XXX.XXX.XXX.XXX 22
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/1001
_=/usr/bin/env
10. Place the static file in the distribution directory.
To reduce the load on the AP server, static files are returned from the WEB server (Nginx).
10-1. Create a static file placement directory and change the owner to .
Terminal
$ sudo mkdir -p /usr/share/nginx/html/static For static files
$sudo chown <any user>/usr/share/nginx/html/static
$ sudo mkdir /usr/share/nginx/html/media for media files
$sudo chown <any user>/usr/share/nginx/html/media
10-2. Enter the Python virtual environment and execute the following in the directory where manage.py exists.
(Premise):
settings.py
STATIC_ROOT = 'usr/share/nginx/html/static'
By executing the following, static files such as css and jpg will be placed in the / usr/share/nginx/static directory.
Terminal
(venv_<Project name>)$ python manage.py collectstatic
236 static files copied to '/usr/share/nginx/html/static'.
11. Perform DB migration.
Enter the Python virtual environment and execute the following in the directory where manage.py exists.
Terminal
(venv_<Project name>)$ python manage.py migrate
12. Install Nignx and configure the settings.
12-1. Installing Nginx
Terminal
$ amazon-linux-extras list | grep nginx
38 nginx1=latest enabled [ =stable ]
We have found that nginx1 is the highest version that can be installed. Execute the following to install.
Terminal
$ sugo amazon-linux-extras install nginx1
12-2. Nginx settings
Edit the configuration file for Nginx and configure the environment settings.
You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)
Terminal
$ sudo vi /etc/nginx/nginx.conf
nginx.conf
(...Abbreviation...)
include /etc/nginx/conf.d/*.conf;
server {
listen 80;
listen [::]:80;
server_name <Elastic IP or your own domain name>;
return 301 https://$host/$request_uri;
}
# Setting for a TLS enabled server.
server {
listen 443;
listen [::]:443;
server_name <Elastic IP or your own domain name>;
root /usr/share/nginx/html;
ssl_certificate "/etc/letsencrypt/live/<Own domain>/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/<Own domain>/privkey.pem";
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location /static {
alias /usr/share/nginx/html/static;
}
location /media {
alias /usr/share/nginx/html/media;
}
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-proto $scheme;
proxy_pass http://127.0.0.1:8000;
}
location /.well-known/acme-challenge {
root /usr/share/nginx/html;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
(...Abbreviation...)
12-3. Nginx autostart settings
Terminal
$ sudo systemctl enable nginx.service
12-4. Start Nginx
Terminal
$ sudo systemctl start nginx.service
$ systemctl status nginx.service
● nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running)since tree 2020-12-10 08:20:46 JST; 2 days ago
Process: 13919 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Process: 13916 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 13915 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Main PID: 13922 (nginx)
CGroup: /system.slice/nginx.service
├─13922 nginx: master process /usr/sbin/nginx
└─13923 nginx: worker process
12-5. Restart Nginx
If you want to restart Nginx, do the following:
Terminal
$ sudo systemctl reload nginx.service
12-6. Stop Nginx
If you want to stop Nginx, do the following:
Terminal
$ sudo systemctl stop nginx.service
13. Operate Gunicorn.
Enter the Python virtual environment and execute the following in the directory where manage.py exists.
13-1. Start Gunicorn
Terminal
(venv_Project name)$ gunicorn --bind 127.0.0.1:8000 ＜Project name＞.wsgi -D
(venv_Project name)$ ps ax | grep gunicorn
671 pts/0 S+ 0:00 grep --color=auto gunicorn
13937 ? S 0:26 /home/<Arbitrary user name>/venv_<Project name>/bin/python3 /home/<Arbitrary user name>/venv_<Project name>/bin/gunicorn --bind 127.0.0.1:8000 <Project name>.wsgi -D
13940 ? S 0:03 /home/<Arbitrary user name>/venv_<Project name>/bin/python3 /home/<Arbitrary user name>/venv_<Project name>/bin/gunicorn --bind 127.0.0.1:8000 <Project name>.wsgi -D
13-2. Stop Gunicorn
If you want to stop Gunicorn, do the following:
Terminal
(venv_Project name)$ pkill gunicorn
14. Make HTTPS
Please refer to [Nginx] What to do if you cannot get the SSL certificate from Let ’s Encrypt with certbot-auto.
15. Set up scheduled batch execution.
15-1. (Example) Back up the database.
backup_db.py
import csv
import datetime
import os
from django.conf import settings
from django.core.management.base import BaseCommand
from ...models import Post
class Command(BaseCommand):
help = "Backup Database"
def handle(self, *args, **options):
date = datetime.date.today().strftime("%Y%m%d")
file_path = settings.BACKUP_PATH + 'database_' + date + '.csv'
os.makedirs(settings.BACKUP_PATH, exist_ok=True)
with open(file_path, 'w') as file:
writer = csv.writer(file)
header = [field.name for field in Post._meta.fields]
writer.writerow(header)
blogs = Post.objects.all()
for customer in customers:
writer.writerow([str(customer.id,
customer.name,
customer.address,
str(customer.phone_number),
str(customer.created_at),
str(customer.updated_at)])
files = os.listdir(settings.BACKUP_PATH)
if len(files) >= settings.NUM_SAVED_BACKUP:
files.sort()
os.remove(settings.BACKUP_PATH + files[0])
settings.py
BACKUP_PATH ="backup/"
NUM_SAVED_BACKUP = 30
15-2. Use the cron function to set periodic execution.
Cron is a program that automatically executes registered commands on a regular basis. A feature built into Amazon Linux 2 from the beginning.
You can edit it by pressing the "i" key. After editing, press the "esc" key and enter ": wq" to save the contents and finish editing. (Enter ": q!" To exit without saving.)
Terminal
$ crontab -e
Contents added by the crontab command
0 4 * * * source ~/venv_<Project name>/bin/activate; cd ~/venv_<Project name>/<Project name>; python manage.py backup_db > ~/cron.log 2>&1
** Basic cron format **
<minute> <hour> <day> <month> <day of the week> <command>
The above example is an example of executing a command at 4:00 AM (UTC) every day.
Notes
The specification in cron format cannot be changed because the time zone is UTC only.
To set in Japan Standard Time (JST), specify a value of "-9 hours" (minus 9 hours) for UTC.
Therefore, the above example is an example of executing the command at 1 pm (13:00) (JST) every day.
- In addition to backing up the DB, it is advisable to set automatic renewal of SSL certificate (Let's Encrypt), issuance of Google ping (sending sitemap.xml), etc. as necessary.
16. Perform a Django security check.
It will be done after HTTPS conversion is completed. Enter the Python virtual environment and execute the following in the directory where manage.py exists.
settings.py
# security.W004
SECURE_HSTS_SECONDS = 31536000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
# security.W006
SECURE_CONTENT_TYPE_NOSNIFF = True
# security.W007
SECURE_BROWSER_XSS_FILTER = True
# security.W008
SECURE_SSL_REDIRECT = True
# security.W012
SESSION_COOKIE_SECURE = True
# security.W016
CSRF_COOKIE_SECURE = True
# security.W019
X_FRAME_OPTIONS = 'DENY'
# security.W021
SECURE_HSTS_PRELOAD = True
Terminal
(venv_<Project name>)$ python manage.py check --deploy
If the WARNING message is displayed, it means that some security measures are insufficient. Normally, WARNING is not displayed in the above settings in the project settings file (settings.py).
Reference: Security in Django
(Relation)
[Python/Django] Summary of frequently used commands (1) <Creating virtual environment, project, application>
[Python/Django] Summary of frequently used commands (2)
(Editor's note)
You can use EC2 as a development environment as well as during production operation, so I would like to consider using Docker as well.
Recommended Posts