One day when I was using Pipenv, I got a mysterious error.
I had a hard time figuring out the cause, so I decided to write it as a memo. Eventually it was resolved, so if you are suffering from similar symptoms and just want to know how to deal with it, please go to Solution.
The first time I noticed this error was when I wanted to set up an automated test in CI.
I was writing code to run automated tests inside docker, but I noticed the following error when running pipenv sync or pipenv install inside docker.
An error occurred while installing readme-renderer==24.0 --hash=sha256:bb16f55b259f27f75f640acf5e00cf897845a8b3e4731b5c1a436e4b8529202f --hash=sha256:c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d! Will try again.
An error occurred while installing requests==2.22.0 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31! Will try again.
An error occurred while installing requests-toolbelt==0.9.1 --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0! Will try again.
An error occurred while installing secretstorage==3.1.1 ; sys_platform == 'linux' --hash=sha256:20c797ae48a4419f66f8d28fc221623f11fc45b6828f96bdb1ad9990acb59f92 --hash=sha256:7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a! Will try again.
An error occurred while installing sphinx==2.2.1 --hash=sha256:31088dfb95359384b1005619827eaee3056243798c62724fd3fa4b84ee4d71bd --hash=sha256:52286a0b9d7caa31efee301ec4300dbdab23c3b05da1c9024b4e84896fb73d79! Will try again.
An error occurred while installing sphinx-git==11.0.0 --hash=sha256:6bf9d837de108c79fb7db585ebd590fd48f4d1f830b540420d0ca675f3b9f800! Will try again.
An error occurred while installing twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836! Will try again.
After trying various things, I found that running pipenv install or pipenv sync without a venv environment causes an error.
Since the hash is displayed, I deleted Pipfile.lock for the time being, and then executed pipenv install`, but I still get an error.
The situation confirmed at this time is as follows.
Well, what is this ...
Even after deleting both the venv environment and Pipfile.lock, an error still appears, so it is almost certain that the cause is Pipfile. The Pipfile at this time is as follows.
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
flake8 = "*"
autopep8 = "*"
isort = "*"
autoflake = "*"
pytest = "*"
pytest-cov = "*"
qiitacli = {path = "."}
twine = "*"
sphinx = "*"
sphinx-git = "*"
pypandoc = "*"
[packages]
click = ">=7.0"
qiita_v2 = ">=0.2.1"
[requires]
python_version = "3"
[scripts]
main = "python qiitacli/client.py"
I don't know what's wrong with my brain ...
I tried to look at the output with the -v option when pipenv install.
I was surprised that there was more output than I expected.
Here is the log that seems to be the cause.
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1)) readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
secretstorage; sys_platform == "linux" from https://files.pythonhosted.org/packages/82/59/cb226752e20d83598d7fdcabd7819570b0329a61db07cfbdd21b2ef546e3/SecretStorage-3.1.1-py3-none-any.whl#sha256=7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a (from keyring>=15.1->twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
Exception information:
Traceback (most recent call last):
File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
status = self.run(options, args)
File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 382, in run
resolver.resolve(requirement_set)
File "/hoge/.venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 208, in resolve
raise hash_errors
pip._internal.exceptions.HashErrors: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1)) readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
secretstorage; sys_platform == "linux" from https://files.pythonhosted.org/packages/82/59/cb226752e20d83598d7fdcabd7819570b0329a61db07cfbdd21b2ef546e3/SecretStorage-3.1.1-py3-none-any.whl#sha256=7a119fb52a88e398dbb22a4b3eb39b779bfbace7e4153b7bc6e5954d86282a8a (from keyring>=15.1->twine==3.1.0->-r /tmp/pipenv-pbslpsa_-requirements/pipenv-u629hk8s-requirement.txt (line 1))
You have added the --require-hashes option when you do pip install internally.
It seems to be hash checking mode, and I didn't know its existence until I saw this log.
ʻIn --require-hashes mode, all requirements must have their versions pinned with ==. These do notIn summary, in hash checking mode, you have to specify all module versions with==`! !! It sounds like that. (English Wakaranai)
In the first place, the version of the module with no error is not specified in Pipfile, and the mystery only deepens.
Looking at the log, it seems that the Pipenv command generates a file equivalent to requirements.txt under / tmp and executes it with the -r option added to the pip command.
I actually looked at the contents of those files.
As a method, just hit sudo cp -rf /tmp/pipenv*. repeatedly when executing Pipenv. There must be a better way, but for now.
Looking at one of the copied files, it looks like this
twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836
Hmmm, I'm generating requirements.txt with a hash value for each module and passing it to the -r option.
I checked the hash value, but it was the same as the file that was actually published.
I will try to install it in a venv environment created by hand using this requirements.txt file.
$ echo -n ' twine==3.1.0 --hash=sha256:1a87ae3f1e29a87a8ac174809bf0aa996085a0368fe500402196bda94b23aab3 --hash=sha256:ba8ba1b39987f1c22d9162f7dd3b4668388640e5f7158c15226624f88e464836' > requirements.txt
$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt --require-hashes
Collecting twine==3.1.0
Using cached https://files.pythonhosted.org/packages/74/45/1016cad7eb7cbda959a701d1dfa88b9118306677018ac3de224a6a6e7751/twine-3.1.0-py3-none-any.whl
Collecting readme-renderer>=21.0
Collecting requests-toolbelt!=0.9.0,>=0.8.0
Collecting tqdm>=4.14
Collecting requests>=2.20
Collecting keyring>=15.1
Collecting importlib-metadata; python_version < "3.8"
Requirement already satisfied: setuptools>=0.7.0 in ./venv/lib/python3.6/site-packages (from twine==3.1.0->-r requirements.txt (line 1)) (42.0.1)
Collecting pkginfo>=1.4.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
readme-renderer>=21.0 from https://files.pythonhosted.org/packages/c3/7e/d1aae793900f36b097cbfcc5e70eef82b5b56423a6c52a36dce51fedd8f0/readme_renderer-24.0-py2.py3-none-any.whl#sha256=c8532b79afc0375a85f10433eca157d6b50f7d6990f337fa498c96cd4bfc203d (from twine==3.1.0->-r requirements.txt (line 1))
requests-toolbelt!=0.9.0,>=0.8.0 from https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl#sha256=380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f (from twine==3.1.0->-r requirements.txt (line 1))
tqdm>=4.14 from https://files.pythonhosted.org/packages/bb/62/6f823501b3bf2bac242bd3c320b592ad1516b3081d82c77c1d813f076856/tqdm-4.39.0-py2.py3-none-any.whl#sha256=5a1f3d58f3eb53264387394387fe23df469d2a3fab98c9e7f99d5c146c119873 (from twine==3.1.0->-r requirements.txt (line 1))
requests>=2.20 from https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl#sha256=9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 (from twine==3.1.0->-r requirements.txt (line 1))
keyring>=15.1 from https://files.pythonhosted.org/packages/b1/08/ad1ae7262c8146bee3be360cc766d0261037a90b44872b080a53aaed4e84/keyring-19.2.0-py2.py3-none-any.whl#sha256=f5bb20ea6c57c2360daf0c591931c9ea0d7660a8d9e32ca84d63273f131ea605 (from twine==3.1.0->-r requirements.txt (line 1))
importlib-metadata; python_version < "3.8" from https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl#sha256=d5f18a79777f3aa179c145737780282e27b508fc8fd688cb17c7a813e8bd39af (from twine==3.1.0->-r requirements.txt (line 1))
pkginfo>=1.4.2 from https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl#sha256=a6d9e40ca61ad3ebd0b72fbadd4fba16e4c0e4df0428c041e01e06eb6ee71f32 (from twine==3.1.0->-r requirements.txt (line 1))
Oh! I reproduced it!
But the error is in a module different from the module described in requirements.txt ...
In this hash checking mode, it is necessary to describe all dependent packages by specifying the version.
But why don't you get an error when you normally do pipenv install?
After checking the log for a while, I found that it has an option --no-deps when installing with pip when it is working properly.
$ ['/hoge/.venv/bin/pip', 'install', '--verbose', '--upgrade', '-r', '/tmp/pipenv-9owwlzhi-requirements/pipenv-65yqxf03-requirement.txt', '-i', 'https://pypi.org/simple', '--require-hashes']
$ ['/hoge/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '/tmp/pipenv-5dvxlnyt-requirements/pipenv-pvatvj5c-requirement.txt', '-i', 'https://pypi.org/simple', '--require-hashes']
Oh, it looks like this!
--no-deps is Don't install package dependencies. That is, it is an option that does not install dependent packages.
Naruhode If this is the case, the requirements.txt mentioned earlier seems to work normally.
$ ./venv/bin/pip install -r requirements.txt --require-hashes --no-deps
Collecting twine==3.1.0
Using cached https://files.pythonhosted.org/packages/74/45/1016cad7eb7cbda959a701d1dfa88b9118306677018ac3de224a6a6e7751/twine-3.1.0-py3-none-any.whl
Installing collected packages: twine
Successfully installed twine-3.1.0
did it.
The cause has been grasped little by little.
The only remaining mystery is ** Why didn't you add the --no-deps option **?
At the end, you have to read the source of Pipenv.
https://github.com/pypa/pipenv
Let's see where the pip install ... command is actually executed when the pipenv install is executed.
The place where the pip command is actually executed was the function pip_install in pipenv / core.py.
There were various call orders such as do_install-> do_init-> do_install_dependencies-> batch_install-> pip_install, but ...
In conclusion, the following part of the batch_install function was flagged as to whether or not to add the --no-deps option.
if not PIPENV_RESOLVE_VCS and is_artifact and not dep.editable:
install_deps = True
no_deps = False
dep is a class that contains information about the modules to install.
Not PIPENV_RESOLVE_VCS, but ʻis_artifact, and the module is not ʻeditable ...
I don't understand at all
Somehow, "modules installed from source that are not flagged as editable" is related to the granting of --no-deps.
After pipenv install -h, there was something like this.
$ pipenv install -h
Abbreviation
-e, --editable TEXT An editable python package URL or path, often to a
VCS repo.
Ah,,,
Let's look again at the Pipfile that caused the error.
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
flake8 = "*"
autopep8 = "*"
isort = "*"
autoflake = "*"
pytest = "*"
pytest-cov = "*"
qiitacli = {path = "."}
twine = "*"
sphinx = "*"
sphinx-git = "*"
pypandoc = "*"
[packages]
click = ">=7.0"
qiita_v2 = ">=0.2.1"
[requires]
python_version = "3"
[scripts]
main = "python qiitacli/client.py"
Yes the cause is
qiitacli = {path = "."}
It was this guy.
I used to do pipenv install . when installing a module under development locally, but this was a problem.
The correct answer is pipenv install -e .. The resulting Pipfile looks like this.
qiitacli = {editable = true,path = "."}
It has an editable flag.
In this state, the installation is completed normally.
In the past, I used to install it with the -e option, but after I realized that I could install it without the option, I decided to install it as it was. (Perhaps)
By the way, it was written firmly in the official document. https://pipenv-ja.readthedocs.io/ja/translate-ja/basics.html#editable-dependencies-e-g-e
Let's at least understand the meaning of the options we are using. myself.
Recommended Posts