Numba to speed up as Python

I've been using Cython quite a bit, but sometimes I find it annoying (for example, I need setup.py on a large scale), and I found out about Numba, which makes it easier to speed up Python. Isn't it nice to JIT with llvm like Julia? I also tried it with reference to the following article. If you are new to Numba itself, please see the link first.

-Introduction to Python Acceleration Numba -Python: Acceleration with Numba -[python: Speeding up matrix operations](http://kzky.hatenablog.com/entry/2015/04/11/python%3A_Speeding up matrix operations)

For now, it's only about how to implement it.

Installation method

For Mac 10.9 with Python 2.7

It worked below.

sudo easy_install funcsigs enum34 numba

git clone https://github.com/numba/llvmlite
cd llvmlite/
sudo python setup.py install

Please note the following.

--llvmlite can be entered with easy_install, but dylib is not enough and a runtime error (why?) --In addition to enum34, enum can be easily_installed, but it causes an error. --When I tried it on Mac OS 10.9 which is different from the machine I tried first, I got an error with import numba, but it was solved by re-installing numpy. I just built it from github with the following command without thinking too much ...

```
git clone https://github.com/numpy/numpy.git numpy_temp
cd numpy_temp
python setupegg.py bdist_egg
sudo easy_install dist/*egg
```

For Windows7 64bit Python 2.7

~~ I will try Windows7 with WinPython2.7 soon, but I think that llvm is necessary, C ++ 11 support is probably necessary, 64bit seems to be difficult, and there are many walls ... ~~ Anaconda I put it in and it passed. I used Anaconda for the first time, but if you have the possibility of gnawing numerical calculations or want to speed up your Python code, it seems good to use it positively.

Usage and impression

I will add this later, but I will dare to try it with a process other than numerical calculation. The reason is that the effect on numerical calculation has already been evaluated by the above-mentioned predecessors, and I happen to be making a complicated binary data reading / writing program instead of numerical calculation.

For use with PyInstaller

If you use PyInstaller, you can make the Python script executable and it will be easier to distribute it to others, but if you use numba, you will get the following error at runtime. This is because PyInstaller is failing to collect files that Python dynamically dlopens.

OSError: dlopen(/var/folders/qk/z4x58g2962q21q5570g2hj0c0000gn/T/_MEIRXXAjg/llvmlite/binding/libllvmlite.dylib, 6): image not found

So let's include this dylib in the build result. You can work around the problem by following the steps below. PyInstaller uses github tag v2.1.

For Mac

  1. python pyinstaller / pyinstaller.py --onefile test.py. At this time, test.spec is created and at the same time an exe is created based on this test.spec, but this exe is in the above error state.

  2. Add the following line to test.spec and instruct it to embed libllvmlite.dylib in onefiled executable.

    # -*- mode: python -*-
    a = Analysis(['test.py'],
                 pathex=['/path/to/main/script'],
                 hiddenimports=[],
                 hookspath=None,
                 runtime_hooks=None)
    
    #One line below this is added
    a.binaries += [("llvmlite/binding/libllvmlite.dylib", "/Library/Python/2.7/site-packages/llvmlite/binding/libllvmlite.dylib", 'BINARY')]
    
    pyz = PYZ(a.pure)
    exe = EXE(pyz,
              a.scripts,
              a.binaries,
              a.zipfiles,
              a.datas,
              name='test',
              debug=False,
              strip=None,
              upx=True,
              console=True )
    
  3. Use python pyinstaller / pyinstaller.py --clean test.spec and rebuild with the modified test.spec.

For Windows Anaconda

(Assuming you have installed the environment in C: \ Anaconda) First, when you try to use the Anaconda environment with PyInstaller, you will encounter an error that pywintypes cannot be found regardless of Numba. However, I was able to solve it by placing the dll in the location found in PyInstaller in the method described here.

cd C:\Anaconda\Lib\site-packages\win32\
copy pywintypes27.dll lib\
copy pythoncom27.dll lib\

After that, just write the procurement method of llvmlite.dll in .spec as in the case of Mac. If you installed it in C: \ Anaconda, the line you should add should look like this:

a.binaries += [("llvmlite/binding/llvmlite.dll", "C:\\Anaconda\\Lib\\site-packages\\llvmlite\\binding\\llvmlite.dll", 'BINARY')]

Recommended Posts

Numba to speed up as Python
Roughly speed up Python with numba
How to speed up Python calculations
[Python] Do your best to speed up SQLAlchemy
Wrap C/C ++ with SWIG to speed up Python processing. [Overview]
Project Euler 4 Attempt to speed up
[DRF] Snippet to speed up PrimaryKeyRelatedField
Don't write Python if you want to speed it up with Python
[Python] Hit Keras from TensorFlow and TensorFlow from c ++ to speed up execution
Indispensable if you use Python! How to use Numpy to speed up operations!
How to speed up instantiation of BeautifulSoup
Updated to Python 2.7.9
"Backport" to python 2
Python C / C ++ Extension Pattern-Pass data to Python as np.array
How to speed up scikit-learn like conda Numpy
Speed: Add element to end of Python array
How to output "Ketsumaimo" as standard output in Python
Call Rust from Python to speed it up! PyO3 Tutorial: Wrapping Classes Part ➀
Organize Python tools to speed up the initial movement of data analysis competitions
Call Rust from Python to speed it up! PyO3 Tutorial: Wrapping Classes Part ➁
Python program is slow! I want to speed up! In such a case ...
From setting up Raspberry Pi to installing Python environment
Trial and error to speed up heat map generation
Trial and error to speed up Android screen captures
How to install Python
How to set up a Python environment using pyenv
Python ~ Grammar speed learning ~
Changes from Python 3.0 to Python 3.5
Changes from Python 2 to Python 3.0
Python list comprehension speed
Rewrite Python2 code to Python3 (2to3)
How to install python
python decorator to retry
Introduction to Python language
All up to 775/664, 777/666, 755/644, etc.
Tips for speeding up python code correctly with numba
Introduction to OpenCV (python)-(2)
Note to daemonize python
Introducing Python 2.7 to CentOS 6.6
Specify MinGW as the compiler to use with Python
Review of atcoder ABC158, up to question E (Python)
A note on speeding up Python code with Numba
Multi-digit multiplication time up to 300 million digits in python
[Introduction to Udemy Python3 + Application] 68. Import statement and AS
Connect python to mysql
[Python MinMaxScaler] Normalize to 0 ~ 1
Call Rust from Python to speed it up! PyO3 Tutorial: Wrapping a Simple Function Part ➀
Call Rust from Python to speed it up! PyO3 Tutorial: Wrapping a Simple Function Part ➁
Posted as an attachment to Slack on AWS Lambda (Python)
Script to count and stop up to 5 seconds in Python in Blender
What I did to speed up the string search task
Review of AtCoder Beginner Contest 159, up to question E (Python)
How to import Python library set up in EFS to Lambda
Review of AtCoder Beginner Contest 163, up to question E (Python)
Review of AtCoder Beginner Contest 164, up to question E (Python)
I tried to speed up video creation by parallel processing
Review of AtCoder Beginner Contest 162, up to question E (Python)
[Python] How to set variable names dynamically and speed comparison
[Python Kivy] How to create a simple pop up window
Review of AtCoder Beginner Contest 154, up to question E (Python)
Try to bring up a subwindow with PyQt5 and Python