[PYTHON] Example of pytest environment to fix database with Docker

Purpose

I want to easily execute unit tests of Python applications using databases with pytest without being affected by the environment. (The one that starts up as a daemon instead of a file such as SQLite3)

What was made

Sample code: https://github.com/hkato/pytest-docker-mysql-example

Launch MySQL from pytest on Docker, input the schema and test data, and execute the test. If you set docker-compose and support SQL related docker-entrypoint-initdb.d, it should be basically the same for other databases such as PostgreSQL. (The initialization mechanism in docker-entrypoint-initdb.d should be the same for MySQL / PostgreSQL official containers, but I don't know anything else)

Required environment and pytest plugin

• Python3
  • pytest
  • pytest-docker
  • pytest-dotenv
• Docker

Regarding pytest, I started Docker and looked at plugins that handle databases, but pytest-docker is the maintenance status and my usage / environment. Looks good for. pytest-dotenv was used by Docker and pytest to describe environment variables in .env files and make them common.

Other considerations

As a result of some consideration other than pytest-docker,

-pytest-docker-compose: A little old. Since pytest-docker handles docker-compose, the impression is that pytest-docker with solid maintenance and documentation is better. -pytest-dbfixtures Series: Compatible with mainstream databases -pytest-mysql: I haven't seen it properly, but it behaves as a basic client, so handling Docker seems to be troublesome.

After all, pytest-docker seems to be the best because it can use the container of the web application as described in the official document or can be used for general purposes regardless of the database ([Start MinIO container and replace AWS S3](https:: //qiita.com/hkato/items/89e436300c50c46624b9) I'm thinking of doing it too).

Directory structure

.
├── src                 #Application code
│   └── *.py
├── tests               #Test code
│   ├── conftest.py
│   └── test_*.py
├── initdb.d            #Database schema & test data etc.
│   ├── *.sql
│   └── *.sh
├── pytest.ini          #pytest settings
├── .env                #Environment variable settings
└── docker-compose.yml  #Docker settings

Code overview

conftest.py

Docker-compose.yml placement support

By default pytest-docker references docker-compose.yml undertests /. By setting it directly under the project (specify another directory or file name in the sample in README.md of pytest-docker), in the case of Web application etc., run the database with docker-compose up -d You can run and debug the web app server with Gunicorn, uWSGI, Uvicorn, etc. (this is not the essence of this time, so this is excluded).

Supports waiting for MySQL startup

The pytest-docker documentation describes it as a sample that launches a httpbin REST API container and waits for 200 responses from port 80. In the case of MySQL, I connected with PyMySQL as shown below and decided that Exception did not come back, that is, the Docker container started up and the initial data input was completed.

def is_database_ready(docker_ip):
    try:
        pymysql.connect(
            host=docker_ip,
            user=os.getenv('MYSQL_USER', ''),
            password=os.getenv('MYSQL_PASSWORD', ''),
            db=os.getenv('MYSQL_DATABASE')
        )
        return True
    except:
        return False

Wait for the above with a 30 second timeout in a 0.1 second cycle.

@pytest.fixture(scope="session")
def database_service(docker_ip, docker_services):
    docker_services.wait_until_responsive(
        timeout=30.0, pause=0.1, check=lambda: is_database_ready(docker_ip)
    )
    return

Execution result

Initial setting of Python environment

$ #In actual operation, it may be better to use Pipenv or Poetry for the test environment and execution environment.
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -r requirements.txt

Environment variable settings

$ cat << 'EOF' > .env
MYSQL_ROOT_PASSWORD='p@ssw0rd'
MYSQL_DATABASE='mydb'
MYSQL_USER='foo'
MYSQL_PASSWORD='pa$$Word'
EOF

When the test is executed, the Docker container is started by Fixture, and the actual unit test is executed after SQL and shell scripts are applied by the mechanism of docker-entrypoint-initdb.d in the container.

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /Users/username/tmp/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/username/tmp, configfile: pytest.ini
plugins: dotenv-0.5.2, cov-2.10.1, docker-0.10.1
collected 1 item                                                               

tests/test_users.py::test_get_name_by_id PASSED                          [100%]

---------- coverage: platform darwin, python 3.8.6-final-0 -----------
Name         Stmts   Miss  Cover
--------------------------------
src/app.py       8      0   100%


============================== 1 passed in 16.35s ==============================

Summary

--pytest-docker is convenient because you can launch services that depend on tests with Docker, not limited to databases. --It may be good to change the location of docker-compose.yml from tests / to directly under the project. Can also be used for normal debugging --Implement start wait processing according to the contents of the container in docker_services.wait_until_responsive ()