What is the XX file at the root of a popular Python project?

Preface

I want to understand * setup.py *, * requirements.txt *, and other files that I often see in Python projects. What kind of files are there in the first place? I was researching various things. For the time being, I was looking at the contents of a popular Python project on GitHub to imitate my predecessor, and there are configuration files for various tools, and there are such tools. Shall we? I became interested in something different from the original purpose.

That's why ** about 700 projects with the most stars in the Python project on GitHub ** are counted in the root file and the most used files are aggregated, what tool is used? I looked it up.

rule

A system often seen in Python projects

setup.py

◆ Number of projects used: 503 ◆ Tools used: setuptools

Describes all behavior when building, distributing, and installing a Python project as a library. Since the directory where * setup.py * is located is recognized as a Python project, you can pip install by specifying the local directory where * setup.py * is located, the local zip file, the Git repository, etc. Will be.

pip install path/prj/dir
pip install path/dir/file.zip
pip install git+https://github.com/{user-name}/{repo-name}

pip installs the package according to the description in setup.py.

To package the project as a library, specify python setup.py sdist or python setup.py bdist_wheel and execute.

#When creating a source code distribution
python setup.py sdist

#Wheel required to create a pre-built distribution
pip install wheel
python setup.py bdist_wheel

Reference: Building and Distributing Packages with Setuptools

MANIFEST.in

◆ Number of projects used: 402 ◆ Tools used: setuptools

If you want to include files other than source code in the distribution, write them in this file. If you write ʻinclude_package_data = True in the argument (or * setup.cfg *?) Of Setup () `, this file will be processed.

You can specify the file with more detailed conditions by specifying the package_data parameter.

Reference: https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

setup.cfg

◆ Number of projects used: 344 ◆ Tools used: setuptools

When packaging the project, I had to write various things in the arguments of the setup () function of * setup.py *. However There were problems such as being difficult to write and difficult to interpret by tools other than * setuptools *. Therefore, * setup.cfg * was created so that a series of settings can be put together in a separate file.

This eliminates the need to write arguments to setup (). (However, if you use an old build tool, you need to describe the argument)

reference

requirements.txt

◆ Number of projects used: 261 ◆ Tools used: pip

Type pip freeze on the command line to output dependency information on the command line. If you set pip freeze> requirements.txt, it will be output to the file as it is. If you are in the virtualenv (venv) environment, the packages in virtualenv will be output. The packages output by freeze do not include packages for package management such as * pip *, * setuptools *, and * wheel *.

requirements.txt example


appdirs==1.4.3
astroid==2.3.3
botocore==1.15.6

Abbreviation

When installing the dependency, it looks like the following

pip install -r requirements.txt

How to write: https://pip.pypa.io/en/latest/reference/pip_install/#example-requirements-file

The tool used is mainly pip, but Pipenv can also install packages from * requirements.txt *, and conversely generate * requirements.txt * from * Pipfile *. Doesn't Poetry seem to be possible?

pyproject.toml

◆ Number of projects used: 102 ◆ Tools used: Poetry, flit, [setuptools](https://github. com / pypa / setuptools), Pyflow, black etc.

If the package you want to install with pip install is distributed in sdist format and you need to build it yourself If the pip version is v19.0 or later and the target package has a * pyproject.toml * file, the package will be built according to PEP 517. Building packages using * pyproject.toml * is like Poetry, Pyflow Tools are supported, but it seems that * setuptools * is also supported in recent versions.

Build in the \ [tool ] table as described in the tool table section of PEP 518 Can describe tool settings for Python projects other than tools. You need to use the subtables in \ [tool ], [tool.flit.metadata], black for flit In the case of (: //github.com/psf/black), you can describe the settings of each tool by writing as [tool.black].

requirements-dev.txt

◆ Number of projects used: 51+ ◆ Tools used: pip

Requirements.txt for development / test environment. As mentioned above, the file name of * requirements.txt * is arbitrary, so Dependent package information used only in the test environment is saved under a different name from * requirements.txt *. However, there are quite a lot of name fluctuations, and there are other:

And so on.

Pipfile

◆ Number of projects used: 26 ◆ Tools used: Pipenv

A file that describes the dependent package information used by Pipenv.

Pipfile.lock

◆ Number of projects used: 22 ◆ Tools used: Pipenv

Pipenv A lock file that describes the information of the generated dependent packages. Something like * package-lock.json * in npm. Since the package described in the lock file is installed as it is with pipenv sync (pipenv sync -d if dev-package is also installed), the same environment can be reproduced among multiple users in the development environment / operation environment. ..

There was also one * pipfile.lock *. (Can it be used?)

It doesn't match the number of * Pipfile *, but everyone, let's manage the lock file properly with Git. I thought ... but according to the Pipenv documentation, "Targeting multiple versions of Python" If so, don't version control the lock file. "

  • Generally, keep both Pipfile and Pipfile.lock in version control.
  • Do not keep Pipfile.lock in version control if multiple versions of Python are being targeted.

poetry.lock

◆ Number of projects used: 15 ◆ Tools used: Poetry

A lock file that describes the information of dependent packages generated by Poetry. It is generated based on the contents described in * pyproject.toml *.

Other Python libraries

tox.ini

◆ Number of projects used: 225 ◆ Tools used: tox

A virtualenv management and command line tool for testing with multiple Python versions and interpreters. Write the settings to * tox.ini *, but it seems that it also supports * pyproject.toml *

Documentation

.coveragerc

◆ Number of projects used: 189 ◆ Tools used: coverage

Configuration file for test coverage measurement tools.

Documentation

codecov.yml

◆ Number of projects used: 102 ◆ Tools used: codecov

This is also a configuration file for another coverage measurement tool. It's less than * .coveragerc *, but it's also used in quite a few projects. Also with * .codecov.yml *.

Documentation

.pylintrc

◆ Number of projects used: 80 ◆ Tools used: pylint

The configuration file for the linter tool * pylint *. Also with * pylintrc *.

Documentation

pytest.ini

◆ Number of projects used: 69 ◆ Tools used: pytest

Documentation

.flake8

◆ Number of projects used: 55 ◆ Tools used: flake8

The configuration file of the familiar static analysis tool * flake8 *.

Documentation

conftest.py

◆ Number of projects used: 32 ◆ Tools used: pytest

A file for defining common fixtures and util functions in * pytest *. If you create it in a package, you can use it only under that package, so you can arrange it according to the scope you want to use.

mypy.ini

◆ Number of projects used: 28 ◆ Tools used: mypy

Configuration file for the static type checking tool * mypy *. .. There was also one * .mypy.ini *.

Documentation

.isort.cfg

◆ Number of projects used: 23 ◆ Tools used: isort

A configuration file for * isort *, a tool that sorts Python code imports according to rules.

Documentation

.bumpversion.cfg

◆ Number of projects used: 21 ◆ Tools used: bumpversion

bumpversion is a command line tool that appropriately increases and rewrites various versions defined in the source code.

manage.py

◆ Number of projects used: 20 ◆ Tools used: django configuration script?

django I don't really understand, but it says something like this. zulip/manage.py

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "zproject.settings")
    from django.conf import settings
    from django.core.management import execute_from_command_line
    from django.core.management.base import CommandError

    from scripts.lib.zulip_tools import log_management_command

    log_management_command(" ".join(sys.argv), settings.MANAGEMENT_LOG_PATH)

    os.environ.setdefault("PYTHONSTARTUP", os.path.join(BASE_DIR, "scripts/lib/pythonrc.py"))
    if "--no-traceback" not in sys.argv and len(sys.argv) > 1:
        sys.argv.append("--traceback")
    try:
        execute_from_command_line(sys.argv)
    except CommandError as e:
        print(e, file=sys.stderr)
        sys.exit(1)

versioneer.py

◆ Number of projects used: 15 ◆ Tools used: versioneer

This is also a tool that automatically raises the version of the project.

.pyup.yml

◆ Number of projects used: 14 ◆ Tools used: pyup

It seems that it will update the Python dependency file of the project via the pull request of GitHub / GitLab. I'm not sure.

.pep8speaks.yml

◆ Number of projects used: 11 ◆ Tools used: pep8speaks

It's a "GitHub app that automatically checks the Python code style with a pull request".

.style.yapf

◆ Number of projects used: 9 ◆ Tools used: YAPF

Google's Python code formatter. It is an abbreviation for "Yet Another Python Formatter".

tasks.py

◆ Number of projects used: 9 ◆ Tools used: Invoke?

Reference: https://equal-001.hatenablog.com/entry/2016/03/31/113000

pyinvoke is a module that can put together various commands.

Quoted from the official documentation. If you add a @ task decorator to the function as shown below,

from invoke import task

@task
def clean(c, docs=False, bytecode=False, extra=''):
    patterns = ['build']
    if docs:
        patterns.append('docs/_build')
    if bytecode:
        patterns.append('**/*.pyc')
    if extra:
        patterns.append(extra)
    for pattern in patterns:
        c.run("rm -rf {}".format(pattern))

@task
def build(c, docs=False):
    c.run("python setup.py build")
    if docs:
        c.run("sphinx-build docs docs/_build")

It seems that it can be easily called as a task from the CLI

$ invoke clean build

.landscape.yml

◆ Number of projects used: 8 ◆ Tools used: Prospector?

Prospector is a tool to analyse Python code and output information about errors, potential problems, convention violations and complexity.

It seems to be a code analysis tool.

noxfile.py

◆ Number of projects used: 7 ◆ Tools used: Nox

tox-like library

Other

Configuration files for tools that are not exclusive to Python, such as CI / CD, Docker / Vagrant, and document generation.

.travis.yml

◆ Number of projects used: 386 ◆ Tools used: Travis CI

CI。

Dockerfile

◆ Number of projects used: 111 ◆ Tools used: Docker

dockerfile.

.readthedocs.yml

◆ Number of projects used: 90 ◆ Tools used: readthedocs

Document generation tool Do you have * .readthedocs.yml * and * readthedocs.yml *?

It seems that it is used in quite a lot of projects, and many of the tools I have written so far are actually making documents with this.

.pre-commit-config.yaml ◆ Number of projects used: 88 ◆ Tools used: pre-commit

.editorconfig

◆ Number of projects used: 83 ◆ Tools used: editorconfig

A tool for formatting code in various editors and IDEs. It can be used in other than Python.

.appveyor.yml

◆ Number of projects used: 79 ◆ Tools used: appveyor

CI / CD service.

.dockerignore

◆ Number of projects used: 76 ◆ Tools used: Docker

File definitions to ignore in Docker

mkdocs.yml

◆ Number of projects used: 34 ◆ Tools used: mkdocs

It's a fast, simple and luxurious static site generator for building project documents.

docker-compose.yml

◆ Number of projects used: 33 ◆ Tools used: docker-compose

docker-compose.

package.json

◆ Number of projects used: 32 ◆ Tools used: npm

npm package file.

In projects made with Python and JavaScript such as frappe / frappe, package management on the js side is probably done with * package.json *. ..

azure-pipelines.yml

◆ Number of projects used: 28 ◆ Tools used: Azure Pipelines

.deepsource.toml

◆ Number of projects used: 14 ◆ Tools used: deepsource

Source code analysis? It seems that it can be used other than Python

environment.yml

◆ Number of projects used: 13 ◆ Tools used: conda?

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file

Vagrantfile

◆ Number of projects used: 12 ◆ Tools used: Vagrant

package-lock.json

◆ Number of projects used: 12 ◆ Tools used: npm

npm lock file. It is managed together with package.json.

_config.yml

◆ Number of projects used: 11 ◆ Tools used: jekyll

jekyll is a Ruby tool for generating static sites. For example, this becomes like this?

Jenkinsfile

◆ Number of projects used: 10 ◆ Tools used: Jenkins

.eslintrc

◆ Number of projects used: 16 ◆ Tools used: Eslint

It shouldn't be related to Python. Is it for JS?

Procfile

◆ Number of projects used: 10 ◆ Tools used: Heroku

A file that specifies the commands that your app executes when you start it on Heroku.

.codeclimate.yml

◆ Number of projects used: 10 ◆ Tools used: CodeClimate?

Code quality check, coverage measurement, etc. It can be used in other than Python.

yarn.lock

◆ Number of projects used: 9 ◆ Tools used: yarn

Yarn lock file. It is managed together with * package.json *.

runtime.txt

◆ Number of projects used: 7 ◆ Tools used: Heroku

Configuration file that specifies the runtime when deploying to Heroku

At the end

I got them in descending order of stars, so I wonder if there are few projects using modern tools? I had the impression that. There seems to be a lot to learn from well-known repositories such as Python project structure and test release scripts, so if you are interested, it may be fun to check it out. Please let us know if there are other commonly used tools / recommended tools.

Recommended Posts

What is the XX file at the root of a popular Python project?
Tasks at the start of a new python project
[python] [meta] Is the type of python a type?
Basics of Python learning ~ What is a string literal? ~
What is a recommend engine? Summary of the types
What is a python map?
[Note] Import of a file in the parent directory in Python
What is the default TLS version of the python requests module?
[Python] What is a zip function?
[Python] What is a with statement?
[Python] What is @? (About the decorator)
[python] What is the sorted key?
What is the python underscore (_) for?
On Linux, the time stamp of a file is a little past.
The idea of feeding the config file with a python file instead of yaml
Check the existence of the file with python
What kind of programming language is Python?
What is the cause of the following error?
What is "mahjong" in the Python library? ??
What is a dog? Python installation volume
The Python project template I think of.
Python Basic Course (at the end of 15)
[Python] Get the character code of the file
The story of blackjack A processing (python)
[Python3] Understand the basics of file operations
[Python] What is a formal argument? How to set the initial value
[Introduction to Python] What is the difference between a list and a tuple?
[Example of Python improvement] What is the recommended learning site for Python beginners?
Test & Debug Tips: Create a file of the specified size in Python
How to put a line number at the beginning of a CSV file
What kind of book is the best-selling "Python Crash Course" in the world?
Get a datetime instance at any time of the day in Python
I made a program to check the size of a file in Python
[Introduction to Python] What is the method of repeating with the continue statement?
What is the fastest way to create a reverse dictionary in python?
I created a script to check if English is entered in the specified position of the JSON file in Python.
Get the caller of a function in Python
The answer of "1/2" is different between python2 and 3
What is wheezy in the Docker Python image?
Make a copy of the list in Python
A note about the python version of python virtualenv
Send Gmail at the end of the process [Python]
This is the only basic review of Python ~ 1 ~
This is the only basic review of Python ~ 2 ~
What is python
[Python] A rough understanding of the logging module
Output in the form of a python array
It's a Mac. What is the Linux command Linux?
This is the only basic review of Python ~ 3 ~
At the time of python update on ubuntu
Convert the character code of the file with Python3
Remove specific strings at the end of python
A discussion of the strengths and weaknesses of Python
What to do if (base) is displayed at the beginning of the Mac terminal
Tell me what a conformal map is, Python!
Get the update date of the Python memo file.
What is Python
[Introduction to statistics] What kind of distribution is the t distribution, chi-square distribution, and F distribution? A little summary of how to use [python]
Create a python script to check if the link at the specified URL is valid 2
When a file is placed in the shared folder of Raspberry Pi, the process is executed.
Create a python script to check if the link at the specified URL is valid