Python package management is complicated.
Even if you are familiar with Python to some extent, many of you may not be able to grasp all of them.
I am also one of them.
Therefore, this article summarizes best practices for managing dependent packages in Python applications.
In addition, although we do not deal with dependent package management in the library, there may be some points that may be helpful.
The title of this article is "Best Practices ** Collection **". He also said, "I will ** summarize best practices **."
What does that mean.
The answer is simple. There is no single best practice for Python's dependency package management method.
Therefore, in this article, I decided to introduce various methods instead of deciding on one best practice.
Also, if you discover a new method, I will add it as needed, and I aim to make this article a collection of best practices at all times.
Traditional requirements.txt
$ pip install flake8
$ pip freeze > requirements.txt
$ cat requirements.txt
flake8==2.5.4
mccabe==0.4.0
pep8==1.7.0
pyflakes==1.0.0
$ pip install -r requieremnts.txt
After installing the required packages, it prints out all the installed packages and their versions.
It's not very convenient and may not be a best practice. However, it is listed here because it is the most common method.
requirements.txt
$ echo "python-dateutil" > requirements.txt
$ echo "flake8" > testRequirements.txt
$ pip install -r requirements.txt -r testRequirements.txt
$ pip freeze > constraints.txt
$ cat constraints.txt
flake8==2.5.4
mccabe==0.4.0
pep8==1.7.0
pyflakes==1.0.0
python-dateutil==2.5.3
six==1.10.0
$ pip install -r requirements.txt -r testRequirements.txt -c constraints.txt
This is a more practical method that supplements the simple requirements.txt method with constraints.txt.
The information written in the Constraints File is not added to the installation target and only version constraints are applied. For details, refer to Explanatory article.
Instead of writing the result of pip freeze, write requirements.txt manually. Only describe the packages that directly depend on it, and specify the version only to the minimum necessary. In most cases, instead of pip install, it is enough to put it in requirements.txt.
Instead of not writing detailed information in requirements.txt, fix the version by writing the output of pip freeze to constraints.txt.
In addition, grouping can be performed by dividing into multiple files as needed.
Note that the contents written in constraints.txt are not installed, so even if requirements.txt is grouped, constraints.txt can remain as a single file.
setup.py + constraints.txt
from setuptools import setup
requires = ['python-dateutil']
extras = {'test': ['flake8']}
setup(
name='app',
install_requires=requires,
test_requires=extras['test'],
extras_require=extras,
)
$ pip install -e .[test]
$ pip freeze > constraints.txt
$ cat constraints.txt
app==0.0.0
flake8==2.5.4
mccabe==0.4.0
pep8==1.7.0
pyflakes==1.0.0
python-dateutil==2.5.3
six==1.10.0
$ pip install -e .[test] -c constraints.txt
This is an improved pattern of setup.py + requirements.txt described later using Constraints File.
Describe the dependent packages using install_require and extra_requires in setup.py.
If you specify extra_requires as ʻextra_requires = {'test': ['flake8']} , you can install the specified package with pip install -e. [Test] `.
Constraints File is a relatively new feature, so few projects have adopted this method, but I think this pattern is better than the following pattern if it is adopted now.
setup.py + requirements.txt
from setuptools import setup
requires = ['python-dateutil']
extras = {'test': ['flake8']}
setup(
name='app',
install_requires=requires,
test_requires=extras['test'],
extras_require=extras,
)
$ pip install -e .[test]
$ pip freeze > requirements.txt
$ cat requirements.txt
app==0.0.0
flake8==2.5.4
mccabe==0.4.0
pep8==1.7.0
pyflakes==1.0.0
python-dateutil==2.5.3
six==1.10.0
$ pip install -r requirements.txt
This is a technique that was often seen when pip didn't have Constraints File yet, and is also used in well-known packages.
However, in this case, the target of test is also written in requirements.txt, so the test package will be installed even in the production environment.
requirements.txt You can do this by splitting it, but it's simpler to use the Constraints File.
pip-tools
$ pip install pip-tools
$ echo "python-dateutil" > requirements.in
$ echo "flake8" > testRequirements.in
$ pip-compile --output-file requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file requirements.txt requirements.in
#
python-dateutil==2.5.3
six==1.10.0 # via python-dateutil
$ pip-compile --output-file testRequirements.txt testRequirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file testRequirements.txt testRequirements.in
#
flake8==2.5.4
mccabe==0.4.0 # via flake8
pep8==1.7.0 # via flake8
pyflakes==1.0.0 # via flake8
$ pip-sync requirements.txt testRequirements.txt
$ pip freeze
click==6.6
first==2.0.1
flake8==2.5.4
mccabe==0.4.0
pep8==1.7.0
pip-tools==1.6.5
pyflakes==1.0.0
python-dateutil==2.5.3
six==1.10.0
A pattern that uses a third-party tool called pip-tools.
pip-compile creates requirements.txt from requirements.in.
pip-sync not only installs the required packages, but also automatically removes the unnecessary packages. Of course, pip-sync does not remove pip-tools even if you don't include pip-tools in requirements.in.
You can also group files by splitting them.
Here are some best practices for managing dependent packages in your Python app.
At the moment, we cannot say which one is the best, so we have no choice but to choose the best one according to the nature of the project.
We hope that one of these methods will become the de facto standard as soon as possible.
Recommended Posts