what's this? As the title says, trying to use mkl with numpy, scipy in the pyenv + poetry environment. It would be easy if I knew the solution, but it took a long time to get there ... The environment is mac OS 10.14.6 Mojave The version of pyenv is 1.2.16-5-g7097f820 The poetry version is 1.0.3 is.
Perhaps many people who are looking at this article will encounter similar problems and want to solve them quickly, so I will write the conclusion (solution) first.
The project name is my-project, and the version of python used is 3.7.4.
It is assumed that mkl is installed in/ opt / intel /as standard.
It is assumed that the project directory is set to create .venv (poetry config virtualenvs.in-project true).
Create a project directory and specify the python version. Up to this point is normal.
$ mkdir my-project && cd $_
$ pyenv local 3.7.4
Normally, poetry will create .venv on its own, so you don't have to create .venv yourself, but this time
--Edit .venv / bin / activate to set environment variables for mkl
--Edit .venv / pip.conf to build numpy, scipy
So make your own .venv.
$ python -m venv .venv
Set environment variables for mkl
$ echo -e "source /opt/intel/bin/compilervars.sh intel64\nsource /opt/intel/mkl/bin/mklvars.sh\nsource /opt/intel/tbb/bin/tbbvars.sh" >> .venv/bin/activate
Create .venv / pip.conf and set no-binary = numpy, scipy, no-use-pep517.
$ touch .venv/pip.conf && echo -e "[install]\nuse-pep517=false\nno-binary=numpy,scipy" >> .venv/pip.conf
Create a config file $ HOME / .numpy-site.cfg for numpy build with the following contents.
[mkl]
library_dirs = $MKLROOT/lib
include_dirs = $MKLROOT/include
mkl_libs = mkl_rt
lapack_libs =
Now that I'm ready, poetry init
$ poetry init
This command will guide you through creating your pyproject.toml config.
Package name [my-project]:
Version [0.1.0]:
Description []:
Author [hogehoge, n to skip]:
License []:
Compatible Python versions [^3.7]:
Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes] no
Generated file
[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["hogehoge"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
Do you confirm generation? (yes/no) [yes]
First, insert cython.
$ poetry add cython
Then numpy.
$ poetry add numpy
Here, if you do poetry run python -c" import numpy; numpy.show_config () ",
Traceback (most recent call last):
(Omission)
Original error was: dlopen(/path/to/my-project/.venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
Referenced from: /path/to/my-project/.venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
Reason: image not found
[^ 1]
[^ 1]: When I enter the virtual environment with poetry shell and then start python and ʻimport numpy, this error does not appear ... but in this state, poetry add scipy` fails. So there is no choice but to add rpath.
If this happens,
$ install_name_tool -add_rpath $MKLROOT/lib .venv/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
To execute. Then,
$ poetry run python -c "import numpy; numpy.show_config()"
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
Premonition of happiness ...
Put scipy at the end. If all goes well so far, the rest should go smoothly. (Scipy build takes time. Please wait patiently)
$ poetry add scipy
poetry run python -c "import scipy; scipy.show_config()"
lapack_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
lapack_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/include', '/opt/intel/compilers_and_libraries_2020.0.166/mac/mkl/lib']
Thank you for your hard work.
The road to the above solution was rather long. I will send you a digest of the journey (almost my own memo). There is no problem skipping it.
$ mkdir my-project
$ pyenv local 3.7.4
$ python -m venv .venv
$ poetry init
.venv / pip.conf
[install]
no-binary = numpy,scipy
Create with, and edit the other ~ / .numpy-site.cfg, .venv / bin / activate in the same way as the solution above.
poetry add numpy succeeds (although I don't want to add rpath), but when I runpoetry add scipy ...
[EnvCommandError]
Command ['/path/to/my-project/.venv/bin/pip', 'install', '--no-deps', 'scipy==1.4.1'] errored with the following return code 1, and output:
Collecting scipy==1.4.1
Using cached https://files.pythonhosted.org/packages/04/ab/e2eb3e3f90b9363040a3d885ccc5c79fe20c5b8a3caa8fe3bf47ff653260/scipy-1.4.1.tar.gz
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'error'
...(Omission)...
Original error was: dlopen(/private/var/folders/9x/94rd8dwd1vg49h9bt7_0kd0w0000gn/T/pip-build-env-mb4t9me7/overlay/lib/python3.7/site-packages/numpy/core/multiarray.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
Referenced from: /private/var/folders/9x/94rd8dwd1vg49h9bt7_0kd0w0000gn/T/pip-build-env-mb4t9me7/overlay/lib/python3.7/site-packages/numpy/core/multiarray.cpython-37m-darwin.so
Reason: image not found
The terminal turns bright red.
no-binary after building numpy?.venv / pip.conf
[install]
no-binary = numpy
After installing numpy with poetry add numpy (again, you have to add rpath), then .venv / pip.conf
[install]
no-binary = scipy
If you rewrite it as poetry add scipy ... it works.
But it's a hassle ... Let's explore a little more.
I fold it, but pyenv alone gives the same symptom ...
Isn't it useless because it is built with / private / var / ...?
Try installing scipy with pip with --no-build-isolation in a pyenv-only environment and it works.
no-build-isolation in the poetry environment.Even in poetry, I thought I would do it if I disabled build-isolation, so I changed.venv/pip.conf
[install]
build-isolation = false
no-binary = numpy,scipy
When I try it as ...
It was impossible. The terminal is bright red. I'm about to cry.
no-use-pep517?After researching various things,
Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour).
In the pip reference guide.
... Isn't this --no-use-pep517 possible?
.venv / pip.conf
[install]
use-pep517=false
no-binary=numpy,scipy
Try as. I did well. This leads to the above solution.
It's a hassle, so I think it's better to do it unless it's too much.
-"Install the MKL version of numpy / scipy on your Mac". Qiita.
Recommended Posts