Is pyenv broken? Python 3.x references 2.x module at startup due to PYTHONPATH error

The beginning of the matter

I manage Python in the pyenv environment (to be exact, pyenv-virtualenv). I've been using JetBrains IDEs like PyCharm and ʻIntelliJ IDEA for a while, but when IntelliJ IDEA introduced its own Python virtual environment (installed in a directory called $ HOME / pyenv4intelliJ`) , It seems to pollute the python environment under pyenv and it doesn't start.

Apparently, the module (Path that specifies) that is loaded when Python starts is strange. So, this time, an article about fixing this and returning it to normal. In terms of content

--Details of error details and investigation of the cause --The story of module loading when starting Python (how pyenv works) --How to fix the error (how to write homebrew.pth)

It's not here. It's basically about python, but it seems like a little knowledge of Homebrew and pyenv will come in. I've seen issues with similar error messages in the forums of the original Python and PyCharm, so there should be quite a few people who are having trouble with this error ...? So, I will come.

Detailed elucidation of error details

Error message 1

First, let's look at the error message when calling Python3 from the Python Console in PyCharm and Terminal.

Failed to import the site module
Your PYTHONPATH points to a site-packages dir for Python 2.x bet you are running Python 3.x!
   #When executed from PyCharm ↓
     PYTHONPATH is currently: "/Applications/PyCharm.app/Contents/helpers/pydev"
   #When executed from Terminal ↓
     PYTHONPATH is currently: "$HOME/.pyenv/versions/3.5.1/bin/python"
     You should `unset PYTHONPATH` to fix this.

Error message # 2

After a while, the error message changed as follows when I played around with it.

$python
Error in sitecustomize; set PYTHONVERBOSE for traceback:
KeyError: 'PYTHONPATH'
Python 3.5.1 (default, May 19 2016, 22:59:42) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> #This will launch an interactive interface

By the way, PYTHONVERBOSE seems to be the same as starting with $ python -v with options. It will trace back at startup.

Survey results

When I read the first error message, it looks like a warning that I'm running 3.x but I'm loading 2.x site-packages (modules?). The second error message seems to be that an error has occurred in sitecustomize.py and PYTHONPATH is strange. (In conclusion, this was the same cause.)

For the time being, about site.py

Now, let's explain what the module site.py, sitecustomize.py, and site-packages that came out earlier are. (You can skip it and read it)

  1. The path that Python sees when loading a module is set in sys.path
  2. Python sets sys.path by running site.py at startup
  3. Is it like an option? : Local path setting for each project is added to the path by executing sitecustomize.py or ʻusersitecustomize.py`

This area is explained in detail on the following page.

-Official document about site.py -Blog: Find out about Python module search path

Identify the cause with brew doctor

However, I haven't set PYTHONPATH, and even if I run >>> import site; site.USERSITE from the interactive interface and check the path set in site.py, it's different. not strange. However……

$ python
>>> import sys
>>> sys.path
>>> sys.path
['', '$HOME/.pyenv/versions/3.5.1/lib/python35.zip',
 '$HOME/.pyenv/versions/3.5.1/lib/python3.5',
 '$HOME/.pyenv/versions/3.5.1/lib/python3.5/plat-darwin',
 '$HOME/.pyenv/versions/3.5.1/lib/python3.5/lib-dynload',
 '$HOME/.local/lib/python3.5/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '$HOME/.pyenv/versions/3.5.1/lib/python3.5/site-packages']

What? Oh? ?? ?? For some reason I'm referring to the Mac system default python2.x path! This is the cause! !! !! But why are you referring to the 2.x path ...?

Then, I casually tried $ brew doctor.

Warning: python is symlinked to python3
This will confuse build scripts and in general lead to subtle breakage.

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
    $HOME/.pyenv/shims/python-config
    $HOME/.pyenv/shims/python3-config
    $HOME/.pyenv/shims/python3.5-config
    $HOME/.pyenv/shims/python3.5m-config
Error in sitecustomize; set PYTHONVERBOSE for traceback:
KeyError: 'PYTHONPATH'

Warning: Your default Python does not recognize the Homebrew site-packages
directory as a special site-packages directory, which means that .pth
files will not be followed. This means you will not be able to import
some modules after installing them with Homebrew, like wxpython. To fix
this for the current user, you can run:
  mkdir -p $HOME/.local/lib/python3.5/site-packages
  echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' >> $HOME/.local/lib/python3.5/site-packages/homebrew.pth

It's a little long, but when I read the English text,

Your default Python does not recognize the Homebrew site-packages directory
as a special site-packages directory, # ... (The following is omitted)

…… The default Python (= python3.5.1 under pyenv) doesn't seem to be able to load site-packages created by homebrew (= probably created in a directory under $ HOME / .pyenv). It seems to be a thing.

I suppose. Yeah, I know.

Summary of cause investigation and how to fix

If you combine the path you checked with ʻimport sys and the python-related warning that came out with brew doctor`, you can see the following.

  1. Path set from site.py called by Python → Normal
  2. Path set from homebrew (under $ HOME / .pyenv) → Abnormal

Probably the path you set from homebrew is calling site-packages in 2.x. The path you see in this homebrew is in the file homebrew.pth in $ HOME / .local / lib / python3.5 / site-packages. (And site.py is loading this homebrew.pth. Maybe.)

When I open the inside ...

homebrew.pth


import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")

bingo! !! !! After all it was this! !! !! !! If you comment out or delete this content ...

Python is back! !! !! !! PyCharm also recognizes and starts normally! !! !! !!

As mentioned above, if you are in trouble with a similar error, please refer to it.

Recommended Posts

Is pyenv broken? Python 3.x references 2.x module at startup due to PYTHONPATH error
About the point that python 3.x got stuck due to an error due to caching_sha2_password
ModuleNotFoundError: No module named'_bz2' error in pyenv Python
Error due to conflict between python when using gurobi