A complete guidebook to using pyenv, pip and python in an offline environment

Target person

For those who want to install various libraries on a machine that does not want to be connected to the network without any prerequisite knowledge so that python files can be executed freely.

Introduction

The reason I decided to write this article is that there seems to be no document that is aimed at people like me who have almost no prerequisite knowledge, from 0 to freely executing python files in an offline environment. (There are many partial ones) Although it was fairly easy to build an environment in an online environment, it was not as simple as an online environment in an offline environment, and it took two to three days to build the environment because it required understanding of the os and other things. is.

Also, since I built the environment while investigating and understanding the parts that I do not understand each time, there is a possibility that extra processing may be mixed in, but I would be very grateful if you could comment in that case.

Things necessary

・ Machine in online environment ・ Off-line machine

Machine OS status

On my machine

-Machine in an online environment (macOS) -Offline machine (CentOS Linux)

To readers

If an error still occurs in this flow, please leave the error content in the comment. We will reply when we can confirm.

Prerequisites

The machine in the online environment and the machine in the offline environment must be connected by SSH.

table of contents

・ Pre-environment creation in online environment (up to pyenv) ・ Creating an environment in an offline environment (up to pyenv) (where I got stuck) ・ Creating pips in an online environment -Create pip in an offline environment ・ Download the necessary libraries in the online environment (where I got stuck) -Download the necessary libraries in an offline environment (where I got stuck)

Pre-environment creation in online environment (up to pyenv)

Perform this work on a machine in an online environment.

In the example below, python 3.6.5 is inserted.

First of all, it can be anywhere, so create a folder to store the .pyenv folder. This time, I will save it in desktop/pyenv.

Terminal

$ mkdir ~/desktop/pyenv
$ cd ~/desktop/pyenv

Clone the .pyenv folder

Terminal

$ git clone git://github.com/yyuu/pyenv.git .pyenv

Check the plugin file for python 3.6.5

Let's check the target file name because it will be obtained remotely by referring to this file.

$ cat .pyenv/plugins/python-build/share/python-build/3.6.5
#require_gcc
install_package "openssl-1.0.2k" "https://www.openssl.org/source/old/1.0.2/openssl-1.0.2k.tar.gz#6b3977c61f2aedf0f96367dcfb5c6e578cf37e7b8d913b4ecb6643c3cb88d8c0" mac_openssl --if has_broken_mac_openssl
install_package "readline-8.0" "https://ftpmirror.gnu.org/readline/readline-8.0.tar.gz#e339f51971478d369f8a053a330a190781acb9864cf4c541060f12078948e461" mac_readline --if has_broken_mac_readline
if has_tar_xz_support; then
  install_package "Python-3.6.5" "https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz#f434053ba1b5c8a5cc597e966ead3c5143012af827fd3f0697d21450bb8d87a6" ldflags_dirs standard verify_py36 copy_python_gdb ensurepip
else
  install_package "Python-3.6.5" "https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz#53a3e17d77cd15c5230192b6a8c1e031c07cd9f34a2f089a731c6f6bd343d5c6" ldflags_dirs standard verify_py36 copy_python_gdb ensurepip
fi

• If you use brew, you cannot type the above command.

Related file download

Use wget to get the target file remotely. If you cannot use the wget command, refer to here to enable the wget command. Or Please make sure that the target file can be downloaded in another form. When using the wget command, download as follows.

$ wget https://www.openssl.org/source/old/1.0.2/openssl-1.0.2k.tar.gz
$ wget https://ftpmirror.gnu.org/readline/readline-8.0.tar.gz
$ wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
$ wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

Check checksum

It is done using md5sum. If you cannot enter the md5sum command, install md5sha1sum so that you can use the md5sum command. It is useful to have the brew command available.

$ md5sum *
f965fc0bf01bf882b31314b61391ae63  openssl-1.0.2k.tar.gz
????????????????????????????????  readline-8.0.tar.gz
????????????????????????????????  Python-3.6.5.tar.xz
????????????????????????????????  Python-3.6.5.tgz

You can check it like the above.

Fix plugin file

① Change https to http ② Change the URL to "http: // localhost /" ③ Change the checksum to the value of the downloaded file

$ cp .pyenv/plugins/python-build/share/python-build/3.6.5 .
$ vi 3.6.5
#require_gcc
install_package "openssl-1.0.2k" "http://localhost/openssl-1.0.2k.tar.gz#f965fc0bf01bf882b31314b61391ae63" mac_openssl --if has_broken_mac_openssl
install_package "readline-8.0" "http://localhost/readline-8.0.tar.gz#????????????????????????????????" mac_readline --if has_broken_mac_readline
if has_tar_xz_support; then
  install_package "Python-3.6.5" "http://localhost/Python-3.6.5.tar.xz#????????????????????????????????" ldflags_dirs standard verify_py36 copy_python_gdb ensurepip
else
  install_package "Python-3.6.5" "http://localhost/Python-3.6.5.tgz#????????????????????????????????" ldflags_dirs standard verify_py36 copy_python_gdb ensurepip
fi

Archiving

$ tar cvfz pyenv_files.tar.gz readline-8.0.tar.gz openssl-1.0.2k.tar.gz Python-3.6.5.tar.xz Python-3.6.5.tgz 3.6.5
$ tar cvfz pyenv_local.tar.gz .pyenv

Transfer the collected archive to the offline environment

pyenv_files.tar.gz pyenv_local.tar.gz Please transfer these two to home in the offline environment with scp etc.

This is the end of "Pre-environment creation in online environment (up to pyenv)".

Creating an environment in an offline environment (up to pyenv)

Extracting the archive file

• Please move to the transferred location.

If you transfer to home, type the following command for the time being.

$ cd ~/

Extract the file you just consolidated here

$ tar xvfz pyenv_local.tar.gz
$ tar xvfz pyenv_files.tar.gz 

Configuration file description

$ echo $SHELL

And enter

① When the execution result is/bin/bash

$ echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bash_profile
$ echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
$ . ~/.bash_profile

(2) When the execution result is/bin/zsh

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
$ echo 'eval "$(pyenv init -)"' >> ~/.zshrc
$ source ~/.zshrc

What I did was add the configuration code to a file called ".bash_profile" or ".zshrc".

If you want to know exactly what's happening, see here .

Distribution of fixed plugins

mv 3.6.5 .pyenv/plugins/python-build/share/python-build/

git build

If you can't use the git command on a machine in an offline environment, <a href="https://git-scm.com/book/ja/v2/%E4%BD%BF%E3%81%84%E5%" A7% 8B% E3% 82% 81% E3% 82% 8B-Git% E3% 81% AE% E3% 82% A4% E3% 83% B3% E3% 82% B9% E3% 83% 88% E3% Please refer to 83% BC% E3% 83% AB "> here to install. One command.

$ cd ~
$ git init 
$ git add Python-3.6.5.tgz openssl-1.0.2k.tar.gz readline-8.0.tar.gz Python-3.6.5.tar.xz
$ git commit -m'init'

http server start

You should be able to install and use bzip2-devel.

Those who cannot use

You can install from here In my case, in the offline environment, it's a Red Hat-based centOS linux, so $ sudo yum install bzip2-devel I was able to go to this command. Please note that the package management tool for linux changes depending on the type of os.

Those who can use

• Root authority is required to start on port 80.

$ sudo python -m SimpleHTTPServer 80 &

If you get an error, check here , cut the process and try again Please.

Install pyenv

$ pyenv install 3.6.5
$ pyenv versions
* system (set by /home/hoge/.pyenv/version)
  3.6.5
$ pyenv global 3.6.5
$  pyenv versions
  system
* 3.6.5 (set by /home.hoge.pyenv/version)
$ which python; python -V
~/.pyenv/shims/python
Python 3.6.5

You can now use python 3.6.5 offline. If you have any errors so far, please leave them in the comments.

Creating pips in an online environment

Download the file for the version of pip you want to install from here . I downloaded pip-20.3.3-py2.py3-none-any.whl.

Bring this file to a machine in an offline environment with scp. I transferred it to the ~/.local/lib folder of the folder I created, but it's okay wherever you like.

Creating pips in an offline environment

Since the path is set in .bash_profile, the hierarchy for entering the pip command can be anywhere.

I'm installing pip-20.3.3-py2.py3-none-any.whl from here

① Update pip ( Recommended articles for reference ) ② Uninstall pip and then reinstall it ( Recommended articles for reference ) ③ Specify the path and install normally

You can probably do it any way. This time, I will describe "③ Specify the path and install normally" that I was able to execute.

About ③

Go to the hierarchy where you downloaded pip-20.3.3-py2.py3-none-any.whl

In my case, pip-20.3.3-py2.py3-none-any.whl exists in ~/.local/lib, so execute the following command.

$ pip install --no-deps ~/.local/lib/pip-20.3.3-py2.py3-none-any.whl

as a result, In my case, I downloaded pip-20.3.3-py2.py3-none-any.whl, so

$ pip list
pip 20.3.3 from ~(python 3.6)

If it is, it is a success. In short, it's successful if the version of pip has been changed to the one you downloaded.

Download the required libraries in an online environment

Here is a description of the download method so that you can use the library you want to use such as numpy and pandas in an offline environment.

Create a folder to store the downloaded library

I created it on my desktop with the folder name pip-cache.

$ cd ~/desktop
$ mkdir pip-cache

Download any library

I'm doing pip download here, but the option description is important.

The description without general options is as follows.

$ pip download -d ./pip-cache numpy #When downloading numpy

This will download numpy to the pip-cache folder, but if I take this numpy to a machine in an offline environment and type a command like pip install numpy, I get an error.

The reason is simple: the platform is different. I'm ignorant, it took me a while to notice here.

More specifically, the above pip download was done on a machine in an online environment. And the type of os of the machine under my online environment is macOS. However, the type of os on my offline machine is centOS linux.

If you download without the --platform option, numpy for macOS will be installed, which is why it didn't work on centOS linux on a machine in an offline environment.

Therefore, I will declare the platform for centOS linux as follows.

$ pip download --platform manylinux1_x86_64 -d ./pip-cache numpy #When downloading numpy

Now it works on a machine in an offline environment.

Download in requirements.txt

This is also a slightly packed scene. What I originally wanted to do at this time was to download requirements.txt here, and the general description is

I put requirements.txt on my desktop

$ pip download -d ./pip-cache -r ~/desktop/requirements.txt  #requirements.When downloading txt

With the --platform option earlier,

$ pip download --platform manylinux1_x86_64 -d ./pip-cache -r ~/desktop/requirements.txt  #requirements.When downloading txt

However, this description throws an error. It's pretty simple once you understand the reason for this, but some libraries don't require you to write a platform. (Some are compatible and some are not)

For example, in the case of numpy

Next, in the case of astor

As you can see, numpy has a description of platform, but astor does not have a description of platform. Due to this, my requirement.txt was mixed with compatible and incompatible ones.

I checked from the error content to see if there was any good option, but the option that doesn't install the package dependencies, yes, uses --no-deps.

$ pip download --no-deps --platform manylinux1_x86_64 -d ./pip-cache -r ~/desktop/requirements.txt  #requirements.When downloading txt

You've done it safely.

Transfer downloaded libraries to machines in an offline environment

And, as before, this time, I was able to download the library required for pip-cache, so I will archive it with tar and transfer it with scp. At this time, it is better to send requirements.txt as well. As with pip, I transferred it to my own .local/lib

Download required libraries in an offline environment

Now, let's move to any layer that has been transferred and unzip the archives that were sent together. In my case, I transferred it to .local/lib, so

$ cd ~/.local/lib

$ tar xf pip-cache.tar

Finally, do a pip install. I transferred requirement.txt to home, so it will be described below.

pip install --no-index --find-links=~/.local/lib/pip-cache -r ~/requirements.txt

By the way, the option --no-index --find-links doesn't search the normal reference (PyPI), only the contents of the local directory (in my case ~/.local/lib/pip- Execute the installation with only the contents of the cache).

Please note that without --no-index --find-links, you may get an error if you are offline.

This completes the installation of the required libraries in an offline environment.

Summary

In this article, I was able to install pyenv, pip, and python in an offline environment, so I can run python as I like.

Also, if you have any opinions, such as making this part easier, I would be very happy if you could leave it in the comments.

Thank you for reading this far. I hope it will be helpful for those who have decided to run python on a machine that does not connect to the company network, or those who use a machine such as gpu that does not connect to the network for the sake of research security.

bonus

Realize a virtual environment with pyenv + virtualenv

This site is very easy to understand. However, it is for online machines. However, if you understand and read this far, you can easily realize it even in an offline environment just by checking the following site, so please try it. https://qiita.com/overflowfl/items/9bc7c9504655f6c17f4e

Reference URL

https://pip.pypa.io/en/stable/reference/pip_download/ https://webkaru.net/dev/mac-wget-command-install/ https://qiita.com/panda11/private/67d1e09ec8261e4c19f4 https://git-scm.com/book/ja/v2/%E4%BD%BF%E3%81%84%E5%A7%8B%E3%82%81%E3%82%8B-Git%E3%81%AE%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%BC%E3%83%AB http://www.s-yata.jp/docs/libbzip2/ http://edo.blog.jp/archives/1819066.html https://tech-diary.net/python-library-offline-install/ https://tnamao.hatenablog.com/entry/2017/06/29/182050