[PYTHON] [Flask] I tried to summarize the "docker-compose configuration" that can be created quickly for web applications

The other day, I built a Flask app using docker-compose and introduced it on Qiita.

■ I made a game WebApp with Flask + Docker + Vue.js + AWS ...

■ Source code and game rules are available on Github

This time, I delved into the docker-compose configuration in Flask application development and summarized it. I hope it will be helpful for those who are considering developing web applications with multiple containers: relaxed:

Introduction

Containerization technology that is difficult to get at first. But once you get used to it, you can't let go.

Docker is actually easy

--Reproduce the environment with one command line --Easy to create an environment Just fetch the corresponding image from Docker-Hub --Developed in the same way in any environment Since the environment is confined in the container, it works in the same way on win and mac and can be developed immediately --You can code the system construction process ――You can try various environments and easily erase them. You can experiment with the environment and easily install plugins. After trying it, you can erase the entire container, so there is no stress.

docker-compose is even easier

--Multiple containers can be managed in conjunction. --You can easily network containers.

Why multiple containers?

Why not pack it in one container? It's not a bad thing, and depending on the scale and content, one container may be enough.

There are many advantages to simply dividing it into multiple containers. We will divide the roles into basic ** 1 container 1 task **.

--Easy to identify errors --Load can be distributed --Processing can be parallelized --Can be scaled individually --It is possible to prevent another process from stopping at the companion in conjunction with an error.

Even a quick export has such advantages. Rather, I think that there are many cases where multiple containers work together when building a system with Docker.

Main commands

Execute the command in the same directory as the docker-compose.yml file. Build and control the Docker container based on the yaml file.

up Create / start multiple containers docker-compose up -d

-d Start in background in detach mode

If you take a container name as an argument, only that container will start.

docker-compose up -d hoge
#Start only hoge container

kill Forcibly stop the running container docker-compose kill

down Stop the container and delete the created container network. docker-compose down

build Build the service. Simply put, build ʻimage. docker-compose build`

Extra: Frequently used commands

Attach to the started container docker exec -it ID_OR_NAME bash

The ʻID_OR_NAMEpart can be attached by simply entering thecontainer_name` defined in yaml. I use it a lot for DB initialization, checking output contents, checking simple files, etc. in a container.

docker-compose.yml From here, we'll touch on the docker-compose settings in the main Flask app.

docker-compose.yml

version: "3"

services:
 nginx:
  build: nginx
  container_name: nginx
  ports:
   - "80:80"
  volumes:
   - ./nginx/nginx.conf:/etc/nginx/nginx.conf
   - ./docker_data/log/nginx:/var/log/nginx
  depends_on:
   - flask
  networks:
   - front

 flask:
  build: .
  container_name: flask
  environment:
   # PROD, DEV, TEST
   FLASK_ENV: PROD
  volumes:
   - .:/introdon
   - ./docker_data/log/flask/gunicorn_access.log:/var/log/gunicorn_access.log
   - ./docker_data/log/flask/gunicorn_error.log:/var/log/gunicorn_error.log
  depends_on:
   - db
   - db_test
  networks:
   - front
   - back
   - test
  expose:
   - 5000

 db:
  build: db
  container_name: mariadb
  ports:
   - "53306:3306"
  volumes:
   - ./db/my.conf:/etc/mysql/conf.d/my.cnf
   - ./docker_data/mysql:/var/lib/mysql
   - ./docker_data/log/mysql:/var/log/mysql
  networks:
   - back
  environment:
   MYSQL_DATABASE: introdon
  env_file: ./db/envfile #ignore because password is stated

 db_test:
  build: db
  container_name: mariadb_test
  ports:
   - "63306:3306"
  volumes:
   - ./db/my.conf:/etc/mysql/conf.d/my.cnf
  networks:
   - test
  environment:
   MYSQL_DATABASE: introdon_test
  env_file: ./db/envfile #ignore because password is stated

networks:
 front:
  driver: bridge
 back:
  driver: bridge
 test:
  driver: bridge

The following is a brief review of the basics.

services: Write the container to be launched with an arbitrary name. This time, there are four containers: nginx, flask, db and db_test.

build: Write the build source of the image.

build: .

Build based on the Dockerfile in the corresponding directory. In this case, it will be a Dockerfile in the same directory as the .yml file.

ports: Port mapping with local. The left side is local and the right side is the port in the container.

ports:
- "63306:3306"

In the above case, if you connect to the local number 63306, you will be connected to the number 3306 in the container.

Enter the port mapping as a character string. Since the yml specification recognizes the xx: yy format as a 60-ary number, it may be recognized as a time, so make it a character string.

You need to use a port that is not used locally, and if it is already used, you will get an error with ʻup`. If you get an error, check the port you are listening to.

sudo lsof -i -P | grep "LISTEN"

volumes: Bind the local file and the file in the container.

--Data can be stored locally and persisted. --Often used to load locally saved config files into a container.

depends_on: Specify the dependent container. In other words, It means "Please start this container after the specified container is started".

networks: Create a network. If you specify a network with any name, you can connect to a container that defines the same network.

networks driver: The default setting is bridge. The driver is as follows.

--bridge: Used for communication on the same host --'overlay': Used for communication between different hosts

Next, I will explain each container individually. : arrow_down:

flask contaner (App server)

Flask container settings that are responsible for the application server role.

#Excerpt

 flask:
  build: .
  container_name: flask
  environment:
   # PROD, DEV, TEST
   FLASK_ENV: PROD
  volumes:
   - .:/introdon
   - ./docker_data/log/flask/gunicorn_access.log:/var/log/gunicorn_access.log
   - ./docker_data/log/flask/gunicorn_error.log:/var/log/gunicorn_error.log
  depends_on:
   - db
   - db_test
  networks:
   - front
   - back
   - test
  expose:
   - 5000

The container is built from the Dockerfile in the same directory. The container name is flask.

The mode of the app is changed by the environment variable FLASK_ENV.

--PROD: Production mode, using Gunicorn with WSGI --DEV: Development mode. Errors and debug contents are displayed, and auto reload is also performed. --TEST: pytest is executed. Use a DB container for testing

volumes:

--flask Volume the app code into a container. --Create a docker_data directory on the host and make the Gunicorn access log and error log persistent.

depends_on Start the app container after the db and db_test containers are started. It only starts the container and does not monitor the internal startup. It is common to run tests and get errors before the DB in the dependent container is up. In that case, just wait a few seconds and then start it up and it will be tested successfully.

networks --front: Connection with Nginx that is responsible for the web server --back: It will be connected to the production DB container. --test: It will be a connection with the DB container for testing.

expose Expose ports only to linked services. In this case, only the front web server, the back DB server, and the test DB test server can be accessed with PORT: 5000. It cannot be accessed from the host environment with PORT: 5000.

Dockerfile A Dockerfile for the flask container that runs the App server.

FROM python:3.8

WORKDIR /introdon

COPY Pipfile ./
COPY Pipfile.lock ./

RUN pip install pipenv && \
pipenv install --system

ENV PYTHONPATH /introdon

CMD ["./flask_env.sh"]

FROM python:3.8 Build using the official python version 3.8 specified image.

Use Pipfile to install the python package with pipenv. pipenv install --system Install directly on system with --system without using python virtual environment. At the time of build, a warning is displayed saying "There is no virtual environment", but there is no problem because it is installed directly. If you are curious, you can create requirements.txt and install it with pip. By using pipenv, you can strictly manage and reproduce dependent packages and their versions.

CMD ["./flask_env.sh"] When the container is up, I'm running flask_env.sh. In this shell file, the mode of "production, development mode, test" is separated according to the environment variable and the application is started.

nginx container (web server)

A container that plays the role of a web server.

#Excerpt

 nginx:
  build: nginx
  container_name: nginx
  ports:
   - "80:80"
  volumes:
   - ./nginx/nginx.conf:/etc/nginx/nginx.conf
   - ./docker_data/log/nginx:/var/log/nginx
  depends_on:
   - flask
  networks:
   - front

ports: It is set to 80:80 for production release. If the host environment hits 80 at the development stage, you may change the left side of the host side for development.

volumes: The configuration file saved on the host is loaded on the container side. I saved the Nginx log data on the host and made it persistent.

later, Start the nginx container after the flask container starts I'm in the front network and connecting to the flask container.

Dockerfile It's short, but below is the Dockerfile.

FROM nginx:1.17.9
CMD ["nginx", "-g", "daemon off;", "-c", "/etc/nginx/nginx.conf"]

The volume /etc/nginx/nginx.conf is started as a configuration file.

mariadb container (DB server)

A container that plays the role of a DB server.

#Excerpt

 db:
  build: db
  container_name: mariadb
  ports:
   - "53306:3306"
  volumes:
   - ./db/my.conf:/etc/mysql/conf.d/my.cnf
   - ./docker_data/mysql:/var/lib/mysql
   - ./docker_data/log/mysql:/var/log/mysql
  networks:
   - back
  environment:
   MYSQL_DATABASE: introdon
  env_file: ./db/envfile #ignore because password is stated

Mapping PORT with 53306: 3306.

volumes: The configuration file saved on the host is being read. In addition, the data of mysql is made persistent by saving it on the host side. The log is also separated into directories and saved on the host side.

networks: I am connected to the flask container via a back network.

environment: The database name and other things that anyone can see are described in yaml as they are. Items that you want to keep secret, such as user names and passwords, are read as separate files. env_file: ./db/envfile

Dockerfile Below is the Dockerfile.

FROM mariadb

RUN apt-get update && \
  apt-get install -y locales && \
  rm -rf /var/lib/apt/lists/* && \
  echo "ja_JP.UTF-8 UTF-8" > /etc/locale.gen && \
  locale-gen ja_JP.UTF-8
ENV LC_ALL ja_JP.UTF-8

I want to use Japanese for DB, so install locale on the container OS and I am changing the locale setting to ja_JP.UTF-8.

my.conf This is a configuration file to be loaded into the container.

[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
general_log_file=/var/log/mysql/mysql.log
general_log=1

[client]
default-character-set=utf8mb4

[mysqld] This is the setting on the mysql deamon side. Set to UTF8. Also set the log output destination. Enables log output of executed SQL with general_log set to 1 or ʻON. If not specified, or if it is 0 or ʻOFF, log output will be disabled.

[client] This is the setting on the client side of the connection source. Again, the character code is UTF8.

envfile This is a separate file for the settings specified by ʻenv_file` in the yml file.

MYSQL_ROOT_PASSWORD=XXXXXXXXX
MYSQL_USER=XXXXXXXXX
MYSQL_PASSWORD=XXXXXXXXX

You can ignore it from version control by separating it into files.

I wanted to combine the password reference files into one, so This envfile is used not only for building the DB container, but also for setting the DB connection in the flask container.

I used the python-dotenv package to read from the flask side.

introdon/config_flask.py

import os
from dotenv import load_dotenv

load_dotenv('db/envfile')
MYSQL_USER = os.environ['MYSQL_USER']
MYSQL_PASSWORD = os.environ['MYSQL_PASSWORD']

(1) Import the contents of envfile into environment variables using the dotenv method load_dotenv. (2) From os.environ, specify with key and get the corresponding environment variable ③ Connect to the DB container with SQLAlchemy using the obtained user and password.

Connect to DB container

There are unexpected opportunities to connect to the started DB container, such as acquiring and confirming data. There are several ways to connect the DB container.

For containers in local environment

CLI There are two types of connection with commands.

① Attach to the container and connect to the DB

#Attach to container
docker exec -it mariadb bash

#Log in to the DB after attaching
mysql -uXXXXX -pXXXXX

② Log in to the DB directly

mysql -h 127.0.0.1 -uXXXXX -pXXXXX -P 53306

127.0.0.1 is the local loopback address. It is the so-called localhost. Use PORT: 53306 with -P 53306. 53306 is the host-side port you specified to map to the DB container in your yml file.

After ①②, you can browse and get data by operating in mysql.

Intellij It's annoying to enter it every time with CLI, I want to browse it in a table like Excel. ..

With Intellij (Jetbrains IDE), it's easy to set up, and once set up, browsing and editing is as easy as Excel. (Probably there is a similar function in Visual Studio)

It has been covered in past articles, so please refer to it. [Summary of setting method to operate AWS from IntelliJ (Jetbrains). Easy connection with #DB](https://qiita.com/akinko/items/d7001a8fe3ac87e1790c#db%E3%81%A8%E3%81%AE%E7%B0%A1%E5%8D%98%E6% 8E% A5% E7% B6% 9A) In the article AWS, not local MongoDB, not MariaDB However, you can connect to the DB container just by changing the part marked in red. The input content uses (2) local loop or port.

AWS production connection

The DB container on AWS has the same connection method as local. However, it is much easier to connect AWS with Intellij (IDE).

With CLI, after connecting to AWS with ssh, you can finally access the DB data by performing SQL operations with mysql. As you know, it is acquired by mysql in a difficult-to-read notation.

This is also [Past article](https://qiita.com/akinko/items/d7001a8fe3ac87e1790c#db%E3%81%A8%E3%81%AE%E7%B0%A1%E5%8D%98%E6%8E If you set it by referring to% A5% E7% B6% 9A), you can easily handle it by clicking like opening Excel.

mariadb_test container (DB server for TEST)

By the way, I will briefly touch on the DB container for testing. It is almost the same as the DB server.

#Excerpt
 db_test:
  build: db
  container_name: mariadb_test
  ports:
   - "63306:3306"
  volumes:
   - ./db/my.conf:/etc/mysql/conf.d/my.cnf
  networks:
   - test
  environment:
   MYSQL_DATABASE: introdon_test
  env_file: ./db/envfile #ignore because password is stated

There is no need to persist the data, just read the config file. To prevent accidents, the DATABASE name has been changed to intorodon_test, which is different from the actual one.

Docker operation with IntelliJ

Intellij (using IDE) is recommended not only for connecting the DB container but also for operating and managing Docker.

Sidebar (Docker service)

You will be able to operate it with the mouse from the sidebar of IntelliJ. You can also check the status of the container in real time.

--Log: Standard output of container --Properties: Browse container ID, imageID, etc. --Environment variables: List the set environment variables --Port Binding: List port bindings with hosts --File: List directory structure. Moreover, you can open the file by double-clicking it.

It's so convenient that I don't feel like using honest commands. .. : dancer_tone1:

Set the in-container language in the project SDK

You can develop apps with Docker driven.

--The development language can be set to the language in the Docker container. --Debugging can also be developed using Docker container --Tests can also be run using Docker containers.

In other words, it doesn't pollute the locals at all, on the contrary ** You can develop without using any locally installed programming language **.

The setting method is explained step by step below. : arrow_down:

IntelliJ settings

It is a setting for developing with python in the Docker container. Set the SDK (in this case, the programming language and version) used in the project. Project structure> Project SDK:> Edit

Normally, I think that the local default python or the virtual environment python such as pyenv is selected, but unlike usual, do as follows. Select Docker Compose and

--Server: Docker --Configuration file: docker-compose.yml file --Service: The name of the container that contains python --Environment variables: (Enter if you want to have environment variables when using) --Python interpreter path: The path to python inside the container. I think that it usually goes through the path, so only python is OK

If you save with the above settings, you can proceed with debugging and testing depending on the Docker container. : raised_hands_tone2:

test

You can already do it with the above settings, but let's make it even easier.

This Flask app is designed to switch between production, development, and test modes depending on the value of the environment variable in the flask container. Every time I test Rewrite the environment variable of docker-compose.yml, upload it, and put it back when the test is over. .. It is troublesome. I want to live as comfortably as possible.

I have entered environment variables in the run / debug settings for the following: Now if you test from the config file, it will overwrite the environment variables and then run it. You don't even have to write it back after the test.

As an aside, as you can see in Docker Compose> Command and options, if you try to manually execute this from the CLI, there will be a lot of input and it will be annoying every time. Once set with Intllij, it's just a click from the second time onwards.

in conclusion

docker-compose was hard to get at first. However, once you get used to it, web application development speeds up considerably: airplane: It's quite reusable: recycle: May you all start developing quickly and become a world where you can deploy quickly: wave: