Unify the environment of the Python development team starting with Poetry

Overview

It was when I was asked to look into the environment of a certain development site. The ** environment construction procedure is given verbally **, and Anaconda is installed on Windows and additional packages are installed with pip. Of course, ** version is not specified **, so the latest version at that time will be included, so the minor version will differ depending on the entry timing. In a large team that straddles companies, it is possible that company A works but company B does not. ** Since the introduction of Lint and formatter is optional **, the commit log of git ** the diff of the added and modified part and the formatted part coexist **, and the source review takes time.

Would you like to unify the Python operating environment for all members of the development team? Would you like to create a clean source that is easy to maintain according to the code rules?

Let's start the work style reform of engineers with ** Poetry **!

goal

Understand how to code the environment using Poetry and how to distribute it to team members.
Get the project template and get a smooth start to development.

We will aim for the above. By the way, here are the main problems of the development site mentioned in the overview.

problem	What's wrong	Will Poetry solve it?
Oral environment construction procedure(Or txt)	Causes human error such as omission	Reproducibility because the environment installer is completed ◎
Pip install on Anaconda environment	In the worst case, you need to reinstall the Anaconda environment.	Don't worry because you don't use Anaconda
No version specified	It doesn't work depending on the person ...	Version can be unified with one command
Lint and formatter are optional	Hard-to-read code is hard to maintain	It can be mandatory to install the tool. Autorun needs to be combined with vscode and git.

procedure

1. Installation of poetry

We will install poetry under the python environment. It is a one-shot installation with the following command. There is no need to learn operations like Docker.

$pip install poetry

2. Create project template

After installing poetry, we will create a template for the project.

$ poetry new xxx

The part corresponding to xxx is the project name. This is the part of * pip install xxx * that corresponds to xxx. You will have a source / test directory and a Readme, pyproject.toml file. After that, push to the git repository and the initial setting is completed.

When executed, it looks like this

3. Dependent library settings

Edit the ** pyproject.toml ** file to configure the dependent libraries required by your project. pyproject.tomlはパッケージビルドに必要なデータを定義するファイルのフォーマットで、setup.py/cfgを置き換えてsetuptoolsではないビルドツールでもビルドが可能となることを目的として作られたものです。

#Add dependent library
$ poetry add xxx
#Added development library
$ poetry add -D xxx

Run!

You can run it just like pip install. If you want to specify the version, use the == operator.

$ root@16460604cb93:~/poetry_template# poetry add tensorflow==2.3.0
Creating virtualenv poetry-template-oDQfK0qA-py3.8 in /root/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... (48.4s)

Writing lock file

Package operations: 41 installs, 0 updates, 0 removals
$ root@16460604cb93:~/poetry_template# poetry add -D mypy black isort
Using version ^0.790 for mypy
Using version ^20.8b1 for black
Using version ^5.6.4 for isort

Execution result

As you execute the add command, it will be added to the pyproject.toml file. You can also edit pyproject.toml directly.

[tool.poetry]
name = "poetry_template"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
tensorflow = "2.3.0"

[tool.poetry.dev-dependencies]
pytest = "^5.2"
mypy = "^0.790"
black = "^20.8b1"
isort = "^5.6.4"
flake8 = "^3.8.4"
radon = "^4.3.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

4. Create an execution environment

We will create a virtual environment (venv) in which the environment according to the conditions described in the pyproject.toml file is installed.

Change the location where venv is created

By default, venv is created in the home directory, but it would be nice if it could be created directly under the project, such as when using it with vscode, so change the setting.

$ poetry config virtualenvs.in-project true
$ poetry config --list

Install environment

Installation is also a command. You will have a .venv directory and a poetry.lock file.

$ poetry install

When executed, it looks like this

The poetry.lock file is a file that guarantees full dependencies. Even if you didn't specify some explicit versions in the pyproject.toml file, the lock file records version information for dependencies that have been successfully installed in the past so that anyone can reproduce them at any time. In other words, if you ** distribute this file to team members ** and have them ** poetry install **, you will always have a unique environment.

5. Run Python in the environment you created

We will access the virtual environment created last. You can access the commands in the virtual environment with the poetry run command.

#Attach to python
$ poetry run python -V
Python 3.8.5
#Test run
$ poetry run pytest
======================================================= test session starts =======================================================
platform linux -- Python 3.8.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /root/poetry_template
collected 1 item

tests/test_poetry_template.py .                                                                                             [100%]

======================================================== 1 passed in 0.01s ========================================================

Actually usable project template

It's been a long time, but that's all for poetry. I hope you understand that you can code the environment with just a few commands, manage it with Git, and all project members can develop in the same environment.

We have created a project template that you can actually use, so please use it. https://github.com/TomokiHirose/poetry_template

Template overview

I am creating a project template with the following functions. Rewrite the template and use the poetry update command to rewrite the environment to suit the user.

command	Contents
poetry run pytest	./Run the tests using the file under tests
poetry run format	black(Powerful formatter)And isort(Arrangement of import order)Runs
poetry run lint	flake8(linter)And mypy(Static type checking)Runs
poetry run metrics	Performs static analysis of cyclomatic complexity index, maintenance availability index, LOC, etc.
poetry run apidoc	Generate API documentation with sphinx
poetry run testdoc	Generate Test document with sphinx[^1]
poetry build	./wheel file under dist(python installer)To generate

in conclusion

I forgot to install it! Even in that case, instead of "Please install xx", just push the fix to git and all the members hit the update command, and everyone will be in the same environment immediately! Don't you think it's very modern?

Furthermore, in cooperation with vscode, if black and flake8 under .venv are automatically executed at the time of saving, or if black is run before push by linking with the commit hook of git, the format is forced. can do. I don't think it's great to follow PEP8 [^ 2], but if you force ** anyone to write the same code **, even if advanced and newcomers are in the same project It is recommended because it will be easier to maintain because the code like this will be completed.

[^ 1]: Document that records the items to be confirmed in the test and the tester in charge. [^ 2]: PEP8 is a Python coding convention. It is not always necessary to follow it, but there is a difference in ease of maintenance depending on whether there is one standard or not.