[PYTHON] I want to use mkl with numpy and scipy under pyenv + poetry environment

what's this? As the title says, trying to use mkl with numpy, scipy in the pyenv + poetry environment. It would be easy if I knew the solution, but it took a long time to get there ... The environment is mac OS 10.14.6 Mojave The version of pyenv is 1.2.16-5-g7097f820 The poetry version is 1.0.3 is.

Conclusion (solution)

Perhaps many people who are looking at this article will encounter similar problems and want to solve them quickly, so I will write the conclusion (solution) first.

The project name is my-project, and the version of python used is 3.7.4. It is assumed that mkl is installed in/ opt / intel /as standard. It is assumed that the project directory is set to create .venv (poetry config virtualenvs.in-project true).

Create project directory & specify python version (normal here)

Create a project directory and specify the python version. Up to this point is normal.

$ mkdir my-project && cd $_
$ pyenv local 3.7.4

Virtual environment settings (unusual)

Normally, poetry will create .venv on its own, so you don't have to create .venv yourself, but this time

--Edit .venv / bin / activate to set environment variables for mkl --Edit .venv / pip.conf to build numpy, scipy

So make your own .venv.

$ python -m venv .venv

Set environment variables for mkl

$ echo -e "source /opt/intel/bin/compilervars.sh intel64\nsource /opt/intel/mkl/bin/mklvars.sh\nsource /opt/intel/tbb/bin/tbbvars.sh" >> .venv/bin/activate

Create .venv / pip.conf and set no-binary = numpy, scipy, no-use-pep517.

$ touch .venv/pip.conf && echo -e "[install]\nuse-pep517=false\nno-binary=numpy,scipy" >> .venv/pip.conf

Create a config file $ HOME / .numpy-site.cfg for numpy build with the following contents.

[mkl]
library_dirs = $MKLROOT/lib
include_dirs = $MKLROOT/include
mkl_libs = mkl_rt
lapack_libs =

Preparation of pyproject.toml (normal here)

Now that I'm ready, poetry init

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [my-project]:  
Version [0.1.0]:  
Description []: 
Author [hogehoge, n to skip]:  
License []:  
Compatible Python versions [^3.7]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes] no
Generated file

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["hogehoge"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"


Do you confirm generation? (yes/no) [yes]

Installation of cython

First, insert cython.

$ poetry add cython

numpy installation

Then numpy.

$ poetry add numpy

Here, if you do poetry run python -c" import numpy; numpy.show_config () ",

Traceback (most recent call last):
(Omission)
Original error was: dlopen(/path/to/my-project/.venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /path/to/my-project/.venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
  Reason: image not found

[^ 1] [^ 1]: When I enter the virtual environment with poetry shell and then start python and ʻimport numpy, this error does not appear ... but in this state, poetry add scipy` fails. So there is no choice but to add rpath. If this happens,

$ install_name_tool -add_rpath $MKLROOT/lib .venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so

To execute. Then,

$ poetry run python -c "import numpy; numpy.show_config()"
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']

Premonition of happiness ...

Install scipy

Put scipy at the end. If all goes well so far, the rest should go smoothly. (Scipy build takes time. Please wait patiently)

$ poetry add scipy

Verification

poetry run python -c "import scipy; scipy.show_config()"
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']

Thank you for your hard work.

Until the solution ...

The road to the above solution was rather long. I will send you a digest of the journey (almost my own memo). There is no problem skipping it.

I tried it normally

$ mkdir my-project
$ pyenv local 3.7.4
$ python -m venv .venv
$ poetry init

.venv / pip.conf

[install]
no-binary = numpy,scipy

Create with, and edit the other ~ / .numpy-site.cfg, .venv / bin / activate in the same way as the solution above.

poetry add numpy succeeds (although I don't want to add rpath), but when I runpoetry add scipy ...

[EnvCommandError]
Command ['/path/to/my-project/.venv/bin/pip', 'install', '--no-deps', 'scipy==1.4.1'] errored with the following return code 1, and output: 
Collecting scipy==1.4.1
  Using cached https://files.pythonhosted.org/packages/04/ab/e2eb3e3f90b9363040a3d885ccc5c79fe20c5b8a3caa8fe3bf47ff653260/scipy-1.4.1.tar.gz
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
...(Omission)...
Original error was: dlopen(/private/var/folders/9x/94rd8dwd1vg49h9bt7_0kd0w0000gn/T/pip-build-env-mb4t9me7/overlay/lib/python3.7/site-packages/numpy/core/multiarray.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
      Referenced from: /private/var/folders/9x/94rd8dwd1vg49h9bt7_0kd0w0000gn/T/pip-build-env-mb4t9me7/overlay/lib/python3.7/site-packages/numpy/core/multiarray.cpython-37m-darwin.so
      Reason: image not found

The terminal turns bright red.

Why not remove numpy from no-binary after building numpy?

.venv / pip.conf

[install]
no-binary = numpy

After installing numpy with poetry add numpy (again, you have to add rpath), then .venv / pip.conf

[install]
no-binary = scipy

If you rewrite it as poetry add scipy ... it works.

But it's a hassle ... Let's explore a little more.

Investigate the cause only with pyenv

I fold it, but pyenv alone gives the same symptom ... Isn't it useless because it is built with / private / var / ...? Try installing scipy with pip with --no-build-isolation in a pyenv-only environment and it works.

Try adding no-build-isolation in the poetry environment.

Even in poetry, I thought I would do it if I disabled build-isolation, so I changed.venv/pip.conf

[install]
build-isolation = false
no-binary = numpy,scipy

When I try it as ...

It was impossible. The terminal is bright red. I'm about to cry.

Isn't it possible to use no-use-pep517?

After researching various things,

Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour).

In the pip reference guide. ... Isn't this --no-use-pep517 possible? .venv / pip.conf

[install]
use-pep517=false
no-binary=numpy,scipy

Try as. I did well. This leads to the above solution.

Summary

It's a hassle, so I think it's better to do it unless it's too much.

reference

-"Install the MKL version of numpy / scipy on your Mac". Qiita.

Recommended Posts

I want to use mkl with numpy and scipy under pyenv + poetry environment
I want to use a virtual environment with jupyter notebook!
Use smbus with python3 under pyenv environment
I want to use MATLAB feval with python
I want to use Temporary Directory with Python2
I don't want to use -inf with np.log
I want to use Python in the environment of pyenv + pipenv on Windows 10
I want to write an element to a file with numpy and check it.
I want to handle optimization with python and cplex
Use OpenBLAS with numpy, scipy
I want to use R functions easily with ipython notebook
I want to change the Japanese flag to the Palau flag with Numpy
[Python] I want to use the -h option with argparse
I want to install a package from requirements.txt with poetry
I want to use VS Code and Spyder without anaconda! !! !!
I know? Data analysis using Python or things you want to use when you want with numpy
I want to do ○○ with Pandas
I want to debug with Python
Environment construction with pyenv and pyenv-virtualenv
I want to use a wildcard that I want to shell with Python remove
I want to solve APG4b with Python (only 4.01 and 4.04 in Chapter 4)
I want to run Rails with rails s even in vagrant environment
I want to use an external library with IBM Cloud Functions
I want to use both key and value of Python iterator
Prepare an environment to use OpenCV and Pillow with AWS Lambda
I really want to use GitHub Flavored Markdown (GFM) with Pelican!
I want to detect objects with OpenCV
After buying a new Mac, use pyenv + poetry to build a Python environment.
I wanted to use jupyter notebook with docker in pip environment (opticspy)
I want to separate the processing between test time and production environment
I want to blog with Jupyter Notebook
If you want to use NumPy, Pandas, Matplotlib, IPython, SciPy on Windows
I want to build a Python environment
I want to use Linux on mac
I want to pip install with PythonAnywhere
I want to analyze logs with Python
I want to play with aws with python
I want to use IPython Qt Console
How to use tensorflow under docker environment
I want to use only the SMTP MAIL FROM command and RCPT TO command without sending mail with Python's smtplib
Use multithreaded BLAS / LAPACK with numpy / scipy
I want to mock datetime.datetime.now () even with pytest!
I want to display multiple images with matplotlib.
I want to knock 100 data sciences with Colaboratory
I want to make a game with Python
Use multiple versions of python environment with pyenv
I want to be an OREMO with setParam!
I want to analyze songs with Spotify API 1
Using Intel MKL with NumPy / SciPy (November 2019 version)
I want to use ceres solver from python
#Unresolved I want to compile gobject-introspection with Python3
How to use tkinter with python in pyenv
How to use pip3 under proxy environment Note
I want to solve APG4b with Python (Chapter 2)
I want to do pyenv + pipenv on Windows
I want to start over with Django's Migrate
I want to write to a file with Python
I want to use the activation function Mish
[Python] I want to add a static directory with Flask [I want to use something other than static]
[Python] I want to use only index when looping a list with a for statement
I want to convert an image to WebP with lollipop