Create an executable file (EXE) by PyInstaller in a hybrid environment (Nimporter) of Python + Nim

Hybrid environment that I've always been interested in

In this article, I found out that there is a cool library called nimporter, and made a slight [comment](/uesseu/items/477bbabe8ea354cb9d49 # comment-4601239e890a54de0ccf), but I was always interested, so I tried to verify ..

The sample source is here.

There are many articles on PyInstaller in Qiita, so please refer to that as well.

environment

It seems that PyInstaller and pyenv and virtualenv are not compatible, so Python is running without a virtual environment.

Windows 10
Nim 1.4.2
Python 3.9
PyInstaller 4.1
Nimporter 1.0.2
Visual Studio Community 2019

Trouble during development, etc.

It seems that Nimporter is set to use VCC in Windows environment, so I installed Community Edition of Visual Studio.
Since PyInstaller does not start, I also passed the path to the Scripts folder of the Python installation folder.

First, play with Nimporter

Click here for the directory structure of this sample.

File folder	Explanation
└─ nimporter-sample	Project directory
├─nimutils	Package for nim source
│　├─`__init__.py`	Promise file
│　├─ calc.nim	Calculation sample
│　├─ thread_test.nim	Thread sample
│　└─ uuid.nim	Another library call sample
├─ nimporter_sample.py	Main python script
└─ nimporter_sample.spec.sample	Sample Spec file for PyInstaller

Import the Nim module in the package from Python

You can access Nim methods in packages (folders) just like regular Python, even if they are not in the same directory as the Python files.

`nimutils/calc.nim`


import nimpy
import strformat

proc add(a: int, b: int): int {.exportpy.} =
    echo fmt("{ a + b = }")
    return a + b

`nimporter_sample.py`


import nimporter
from nimutils import calc # nimutils/calc.import nim

# call nim method
print(calc.add(2, 4))  # 6

Try using the module installed by Nimble

You can also import a module called nuuid that was previously installed with the nimble command in the nim source and return the execution result to Python.

#Module installation
$ nimble install nuuid

`nimutils/uuid.nim`


import nimpy
import strformat
import nuuid      # import uuid library 

proc generate(): string {.exportpy.} =
    return generateUUID()

`nimporter_sample.py`


import nimporter
from nimutils import uuid

print(uuid.generate())

Try running Nim multithreading from Python

This also worked fine.

`nimutils/calc.nim`


import nimpy
import strformat
import os

proc threadFunc(param: tuple[a, b: int]) {.thread.} = 
    echo fmt("This is Thread-{param.a}")

proc threadTest(): int {.exportpy.} =
    var thr: array[0..1, Thread[tuple[a, b: int]]]
    echo "start threads"
    defer:
        echo "wait threads"
    thr[0].createThread(threadFunc, (1, 1000))
    thr[1].createThread(threadFunc, (2, 1000))
    sleep(1000)
    joinThreads(thr)

`nimporter_sample.py`


from nimutils import thread_test

#Call the method that is threading & executing
thread_test.threadTest()

However, it seems that Nim methods cannot be called from ** Python threads **. (Issue is here) Therefore, at present, when using Nimporter, it seems that it can only be called from the main thread of Python. It seems that it is not possible to call a Nim module from within a request handler in a web framework.

Creating a single Exe file with PyInstaller

Exe the Python file that called the above 3 patterns with Pyinstaller, and confirm that it works in another Windows 10 environment.

Exe is not completed in one shot, so follow the steps below to create it.

Start Pyinstaller and create a Spec file
Add Pyd file information to Spec file
Start Pyinstaller with Spec file and create Exe file
If you get an error, add the missing module to the Spec file

After that, repeat steps 3 and 4 until no error occurs when starting Exe.

Creating a Spec file

If it is a Python script with a simple configuration, you may be able to create an EXE file with Pyinstaller in one shot, but start Pyinstaller and modify the generated Spec file appropriately to prepare the environment for creating Exe. I will.

First, specify the Python script that will be the entry point and execute PyInstaller. A spec file will be generated in the same folder as the script, and an Exe file will also be created in the dist folder. However, even if you start the completed Exe file, an error will occur and it will end, so we will describe the modules etc. that are missing in the generated Spec file.

$ pyinstaller nimporter_sample.py --onefile
・ ・ ・

$ dir dist
2021/01/04  19:36    <DIR>          .
2021/01/04  19:36    <DIR>          ..
2021/01/04  19:36         7,725,805 nimporter_sample.exe
               1 File(s)      7,725,805 bytes
               2 Dir(s)  232,853,856,256 bytes free

$ dist\nimporter_sample.exe

#I get an error when I start it
Traceback (most recent call last):
  File "nimporter_sample.py", line 1, in <module>
    import nimporter
  File "c:\apps\python\python39\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "nimporter.py", line 13, in <module>
  File "c:\apps\python\python39\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "setuptools\__init__.py", line 24, in <module>
  File "c:\apps\python\python39\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "setuptools\depends.py", line 6, in <module>
ModuleNotFoundError: No module named 'setuptools.py33compat'
[431764] Failed to execute script nimporter_sample

Add Pyd files to Spec binaries

Nimporter generates a Python extension module (extension is pyd), so modify it so that Pyinstaller includes it in the Exe module. The Spec file is actually a Python script, so you can write Python code to modify the configuration file.

In this sample, the Nim file is placed in the nimutils folder, so the pyd is output to the nimutils \ __ pycache__ folder, so modify the Spec file as follows.

`nimporter_sample.spec`


# -*- mode: python ; coding: utf-8 -*-
import os

curDir = os.getcwd()
cacheDir = os.path.join(curDir, 'nimutils', '__pycache__')
pydDir = os.path.join('.', 'nimutils')
block_cipher = None

a = Analysis(['nimporter_sample.py'],
             pathex=[curDir],
             #Add the nimporter-generated pyd file as a binary
             binaries=[
               (os.path.join(cacheDir, 'calc.pyd'), pydDir),
               (os.path.join(cacheDir, 'thread_test.pyd'), pydDir),
               (os.path.join(cacheDir, 'uuid.pyd'), pydDir),
             ],

You can specify multiple binary file settings, and specify each file with a tuple (Pyd file location, where to place it when the EXE starts). In this example, it is specified that the pyd file in nimutils/__ pycache __ / should be expanded to the nimutils folder when Exe is started. Isn't it the __pycache__ folder even after extraction? I thought, but when executing Exe, even if there is pyd in the __pycache__ folder, it does not seem to be read from there.

Start Pyinstaller with Spec file and create Exe file

Specify the Spec file as an argument of PyInstaller and execute it.

#Spec file and execute
$ pyinstaller nimporter_sample.spec

#Start Exe created in dist
$ dist\nimporter_sample.exe

If you get an error, add the missing module to the Spec file

If the following error occurs when you start the Exe created by PyInstaller, add the module name to hiddenimports in the Spec file of PyInstaller to solve the error.


ModuleNotFoundError: No module named 'setuptools.py33compat'

I repeated it several times and added two modules to hidden imports and the error disappeared.

`nimporter_sample.spec`


  hiddenimports=['setuptools.py33compat','setuptools.py27compat'],

State of execution

It's a rough capture, but for your reference.

Summary

I introduced the procedure to convert a Python script using Nimporter to Exe with PyInstaller. The point was to include the Pyd file generated by Nimporter as a binary.

Although there is a limitation that modules created with Nimporter cannot be called from Python multithreading, it is possible to distribute a small utility as Exe, such as using the attractive library of Python and creating the part you want to process at high speed with Nim. Isn't it an attractive environment?

However, I think that it is not a technology that can be used in production, but is limited to hobby use (hobby) use.

Referenced site

https://qiita.com/rebellious-wimp/items/61f16389f957b2ace163