Prepare a development environment that is portable and easy to duplicate without polluting the environment with Python embeddable (Windows)

1.First of all

The license of Anaconda changed from around April of this year, and it became troublesome to build an environment on Windows for companies exceeding a certain size. There are several ways to build a Python environment, and there are various explanation pages out there. You can also use venv, but this time I will explore a development environment that emphasizes that it can be carried with a USB memory, is easy to duplicate, and does not pollute the environment. (Image of making WinPython by yourself ... I think it will be a little slimmer.)

Each method has advantages and disadvantages, so please consider whether it suits you when you refer to it.

The following articles were used as a reference How to run Python on Windows without polluting the environment as much as possible (Python embeddable version) How to install Tkinter in Python embeddable Notes that must be understood when building a PYTHON environment

2. Get Python embeddable

Get ** Windows embeddable package (64-bit) ** from here. Python Releases for Windows It will be in the following location. (For Python 3.9.1)

The zip file will be downloaded, so unzip it to a suitable location. (It may be happier if Japanese is not included in the pass)

This time, I made the following directory structure.

Any place / 　└PythonDev/ 　　└ python-x.x.x-embed-amd64

If you want to use Tensorflow, please enter 3.8. At the time of writing the article, it supports up to Python 3.8, so even if you include 3.9, you cannot use it.

3. Operation check (just in case)

Start Python.exe and check the operation.

If you want to do it from the command line, ** You can start from that directory by typing " cmd " in the address bar and pressing Enter. ** ** (Remembering this will make your work easier later)

Enter the following and start Python.


#python.Run in directory with exe
python

4. Introduce pip

Since Python is already available so far, there may be enough people, such as those who are aiming to study grammar, but it is far from a practical development environment, so various things I will introduce it.

4.1 Preparation

First, open the file _python39._pth (the number is the Python version) in the folder you unzipped earlier in Notepad, and modify the commented out import line as follows.

`_python39._pth`


#import site
↓
import site

4.2 Get pip

Download get-pip.py from the following GitHub repository and throw it into the folder you just unzipped. pypa/get-pip

I will also post a direct link (https://github.com/pypa/get-pip/raw/master/get-pip.py)

4.3 Install pip

I will execute it immediately. You can run it from the command line, but it doesn't matter if you can install it, so drag and drop it. (The message disappears in an instant, so if you don't like it, run it on the command line)

The command line is as follows.

#python.Run in directory with exe
python get-pip.py

Obviously, if you put python.exe and pip.exe in your PATH, commands can be passed from any directory, so it's easy to work. However, ** This time, the concept is that it can be carried without polluting the environment, so I will work without passing through the PATH. ** **

** After that, all pip commands appearing in the article are supposed to be "executed in the Scripts/folder" **

4.4 Conflicts of portability and countermeasures << Points to watch >>

By executing get-pip.py, a folder of Scripts/is created, and files such as pip.exe are generated in it. If you open this with a hex editor and read inside, you can see that it has the absolute path of ** python.exe near the end of the file ** If the python interpreter is not specified, it seems to go to start the interpreter with this absolute path. Therefore, even if the folder name changes, it will not work with an error such as "Fatal error in launcher: Unable to create process using ~" when executing pip.

<Workaround (emergency response)> If you specify an interpreter, the absolute path will not be referenced, so you can avoid it by specifying the command as follows.

<python.Absolute path of exe> <pip.Absolute path of exe> install <package name>

It's a little troublesome, so type the following command in the directory where python.exe is located.

#python.Run in directory with exe
python ./Scripts/pip.exe install <package name>

** Run `get-pip.py` again **. (I feel that this is easier)

Note that this seems to apply to the entire exe file created in Scripts /.

5. Introduction of Tkinter (Tcl/Tk)

This process is a prerequisite after installing pip *

Without it, you will not be able to display the plot with matplotlib etc. Also, the UI using Tcl/Tk cannot be displayed. To put it the other way around, skip it if you don't need it.

In Linux environment, you can install it with pip install python-tk, but you can't install it in Windowos.

The article that I introduced at the beginning as a reference According to How to install Tkinter in Python embeddable

Install the ** installer version of Python that is ** the same version as the embeddable version you are using
Copy Tcl/Tk related files from the installation folder
Add a line to the config file (pythonNN._pth)

This time, only 1 and 2 will be implemented, and 3 will be omitted. Instead, the directory structure should look like this: (The file in bold is the copy.)

PythonDev/ 　└ python-x.x.x-embed-amd64/ 　　　├ _tkinter.pyd 　　　├ tcl86t.dll 　　　├ tk86t.dll 　　　├ tcl/ 　　　└ Lib/ 　　　　　└ site-packages/ 　　　　　　　└ tkinter/

If you do the right thing, Python will pass the import tkinter instruction. ** If the original Python version that pulls Tcl/Tk is different or the directory is wrong, an error will occur. ** **

6. Introduce various

First, open the Scripts folder. (Location of pip.exe) Then open a command prompt and use pip to install the package you want. (Of course, you can change to that directory at the command prompt)

#For example, execute the following command in the Scripts folder
pip install seaborn pandas scikit-learn opencv-python

Since this environment is not in the PATH, it seems that the installation of packages that assume it may fail. (tensorflow, etc.) There is an explanation of countermeasures in the following article introduced at the beginning.

How to run Python on Windows without polluting the environment as much as possible (Python embeddable version) Notes that must be understood when building a PYTHON environment

According to the commentary, create a file with the extension .pth next to python.exe and It seems that it can be avoided by writing import sys; sys.path.append ('') in it.

With the above, I think that the Python environment itself is almost complete.

7. Prepare the development environment (Visual Studio Code)

From now on, we will install the Jupyter Notebook and the procedure for setting the linkage with Visual Studio Code. First of all, we will introduce the simpler Visual Studio Code.

7.1 Obtaining and deploying Visual Studio Code

Get the **. Zip version ** from the Download Page (https://code.visualstudio.com/download#).

Extract it to a suitable folder. I want to operate it as a set with the Python development environment created earlier, so is it the following folder structure? Well, I think it's okay to have a structure that is easy for individuals to do.

PythonDev/ 　├ python-x.x.x-embed-amd64/ 　└ VSCode-win32-x64-x.xx.x/

7.2 Make it boot in portable mode

VS Code has been available in portable mode for quite some time. Putting it in portable mode is easy, just create an empty folder named data.

PythonDev/ 　└ VSCode-win32-x64-x.xx.x/ ├ data/← Make this 　　 : └ Various files and folders

If you start Code.exe in this state, Visual Studio Code in the default state will start up. Extensions and user settings will be stored in this folder

7.3 Extension installation

It's the same as regular Visual Studio Code, so I'll add the necessary extensions. For the time being, I'm just touching Python, so is it something like the following? All you have to do is include the Python extension.

Japanese Language Pack for Visual Studio Code --Python (required)
Pylance
indent-rainbow
Bracket Pair Colorizer 2

If you move this folder to another environment entirely, the UI may not be translated into Japanese at the first VS Code startup after that. In that case, quit it once and open it again to fix it. (It is presumed that the extension failed to load only the first time.)

7.4 Specify the Python environment

You need to specify the environment in order to debug. There are various methods if you just set it, but this time, assuming actual development, I will create a working folder dev and put test.py in it. The image looks like the following

PythonDev/ 　├ python-x.x.x-embed-amd64/ 　├ VSCode-win32-x64-x.xx.x/ 　└ dev/ 　　└ test.py

Open the dev folder in Visual Studio Code and open test.py.

Then the Python environment is easy to understand? If it is installed, it will be loaded. (If there is no environment, a message prompting you to select it should appear) The image is the image below. In this case, my environment seems to initially specify the Python interpreter installed with Visual Studio 2019.

Since we will not use it this time, click the red circle at the bottom left → click "Enter interpreter path ..." and specify the Python embeddable interpreter (python.exe) prepared earlier.

7.5 Operation check

Check if you can debug. Insert the appropriate code and check the operation.

`test.py`


print("aaa")

Press F5 to start debugging. This time I haven't created launch.json, so I'm asked for debug settings, but you can select Python File.

Then you will get the following output:

This version of python seems to be incorrectly compiled
(internal generated filenames are not absolute).
This may make the debugger miss breakpoints.    
Related bug: http://bugs.python.org/issue1666807
aaa

The debugger is giving a warning, but it seems to be running. You're saying you might fail the break. If you can't break, debugging will be difficult, but as an editor, I'll put up with it even if I can't break. ** (If you really have a problem, let's install it quietly and pollute the environment) **

Well, in-house training, instant operation check, and simple automatic processing will be tolerable. maybe.

8. Prepare the development environment (Jupyter Notebook)

Personally, I think Visual Studio Code is sufficient, but I would like Jupyter Notebook for educational purposes.

8.1 Installation

Enter with the following command.

#For example, execute the following command in the Scripts folder
pip install notebook

8.2 Start

Since it is not in the PATH, type the command in the Scripts folder.

#Start with either of the following in the Scripts folder
jupyter notebook <Absolute or relative path of working directory>
jupyter-notebook <Absolute or relative path of working directory>

If no directory is specified, it will start in the current directory. I think it is inconvenient, so I will specify it relative to the working folder.

If the startup is successful, the standard browser will start up and the Jupyter Notebook will open.

The terminal (command prompt) is left unattended until the work is completed.

8.3 End

Type Ctrl + C in the terminal (command prompt) and it will exit after a few seconds.

8.4 Measures against the trade-off of portability

Now, Jupyter Notebook installs many executables in the Scripts / folder. As some of you may have already noticed, like pip.exe, it has the absolute path of ** python.exe inside **

Therefore, it will not start if the directory changes. As with pip, you need to specify the interpreter.

#python.Move to the exe directory and execute
python ./Scripts/jupyter-notebook.exe <Absolute or relative path of working directory>

#[Execution example]For this directory structure, python.Execute the following in the exe directory
python ./Scripts/jupyter-notebook.exe ../dev

Digression As you may have noticed by looking at the command, you haven't specified jupyter.exe. It seems that jupyter.exe just starts jupyter-notebook.exe by concatenating the string notebook passed as an argument with the string jupyter-.

As usual ** the solution is a reinstall **. Forced reinstallation is the following command.

#python.Move to the exe directory and execute
python ./Scripts/pip.exe install --force-reinstall notebook

8.5 Usability improvement

It is troublesome to start by typing commands one by one, You may want someone who doesn't understand the command to use it. If you are not spending your time in a non-essential place, you will write the jupyter-notebook startup process in a batch file.

`jupyter-notebook.bat`


cd /d %~dp0
call "./python-x.x.x-embed-amd64/python.exe" "./python-x.x.x-embed-amd64/Scripts/jupyter-notebook.exe" "./dev"

Correct the folder name of the startup command according to the folder name you are creating.

What you are doing

Move to the directory where the batch file is located
Start with the call command (give arguments in the order of python.exe specification, jupyter-notebook.exe specification, working folder specification) only. The file layout is as follows.

PythonDev/ ├ python-x.x.x-embed-amd64/← python.exe is in this. ├ dev/← jupyter-notebook working folder ├ ** jupyter-notebook.bat ← Batch file for startup ** └ (Other files and folders)

9. Using GPU

If you can't do this, the appeal will be reduced by about 40%. (Personal impression) If you set up in an environment without an NVIDIA GPU, Tensorflow will complain that there are no files even if cuda is installed. If you can't use the GPU you should be able to use, reinstall tensorflow on a GPU-equipped machine (with cuda and cudnn properly installed).

#python.Change to the exe directory
python ./Scripts/pip.exe install --force-reinstall tensorflow

Unlike before, tensorflow is easy to have the same binary for CPU version and GPU version. If you can't use the GPU, it will run on the CPU.

10. Finally

Now you have a portable development environment. (I think.)

This time, I explored how to create such an environment, but the embeddable version on which it is based is a bit special. I've confirmed that it runs as much as a simple graph drawing with matplotlib and the MNIST tutorial of Tensorflow, but I'm not sure how far it will work.

Machine learning, automatic data processing, and statistical analysis are closed in the environment, so I think it's okay, but it's unknown when linking with another system. Modules that use absolute references or environment variables also run the risk of not working. Also, please note that if you place it in a place where anyone can access it, you may have a security risk in some cases. ** Please use at your own risk while being aware of the existence of defects and conflicts. ** **

However, I think it will be a clue to solve certain problems.

--If you don't want to pollute the environment (if you can't) --If you want to carry the environment with a USB memory and run Python on a terminal without a network connection --When the proxy server is too strict in a company etc. and you cannot set up the environment many times --If you find it inconvenient to not be able to use GPU with Windows + Docker (until the official release of WSL2 for GPU) ――This is the best for safety, but some organizations have policies that can not use virtual environments, and there are problems that Docker has a high threshold for amateurs. ――When it is troublesome to set up and restore many PCs for education etc.

etc.

That's it. We hope that the content of the article will be of some help to you.