[PYTHON] Build a Pypi cache server on QNAP

Introduction

Following Building an Apt cache server on QNAP Build a Pypi cache server as a Docker container on the NAS of QNAP. We aim to speed up pip and reduce traffic in the LAN.

Pypi cache server

Devpi server is used as the cache server for Pipit. Since it is operated by Docker, search for the devpi image. For example, scrapinghub / docker-devpi was found.

As mentioned in Last time, the NAS at hand was ARM-based. Therefore, a devpi image for ARM was created separately. The Dockerfile of the created image is GitHub, and the image is DockerHub. It is located in / r / jkawamoto / armhf-devpi-server /).

start devpi server container

Container Station is used to manage containers. If not, install it from the App Center. You can refer to the Container Station manual from here.

fig6.png

Search for the container image to be used from the container creation tab. By default, the image on DockerHub is available.

fig7.png

When you find the desired image, install it.

fig8.png

If the installation is complete or already installed The container creation screen opens.

You can choose the name of the container as you like. Commands and entry points can override the Dockerfile settings here, The default is fine unless otherwise instructed. CPU and memory limits vary depending on the environment, so leave the defaults here. Next, check the network items in the detailed settings.

fig9.png

Check which port is forwarded in the network settings. devpi server uses 3141 by default. Since the port number on the host side is auto at the beginning, change it to 3141. Also, depending on the base image, unrelated ports are open, so delete them. In the case of the figure below, 6080 is unnecessary, so delete it.

fig10.png

You can also mount the directory used by the cache server in the shared folder item. After completing the above settings, create a container.

Settings on the cache server user side

To use the cache server for the pip command --index-url Optional cache server URL, Pass the IP address of the cache server as the --trusted-host option. trusted-host is needed because the cache server only listens on http, not https. It's a hassle to support https, so I decided to use it only within a trusted network. Set trusted-host.

Note that the URL of the cache server is based on the IP address xxx.xxx.xxx.xxx. It becomes http://xxx.xxx.xxx.xxx:3141/root/pypi. Note that the / root / pypi part is easy to forget. (See Documentation for URL addresses other than / root / pypi)

However, it is troublesome to add options to the pip command every time, so It can also be used as a configuration file. Describe the following contents in ~ / .pip / pip.conf.

pip.conf


[global]
index-url="http://xxx.xxx.xxx.xxx:3141/root/pypi"
trusted-host="xxx.xxx.xxx.xxx"

After that, when using pip, you will access the cache server.

Use with Docker

The most common use of pip in Docker is at build time, that is, in Dockerfile, rather than at container execution time. Use the ARG command to decide whether to use the cache server during docker build.

Dockerfile


ARG PIP_PROXY
RUN if [ -n "$PIP_PROXY" ]; then \
      echo "Set pip proxy: $PIP_PROXY"; \
      IPPORT=${PIP_PROXY#*//}; \
      mkdir -p ~/.pip/; \
      echo "[global]\nindex-url=$PIP_PROXY/root/pypi\ntrusted-host=${IPPORT%:*}" >> ~/.pip/pip.conf; \
      cat ~/.pip/pip.conf; \
    fi

Add the above items to the Dockerfile. Variables are defined in ʻARG, and values can be passed with the —build-argoption duringdocker build`.

For the Dockerfile above, enter --build-arg PIP_PROXY = http: //xxx.xxx.xxx.xxx:3141/ If you pass it to docker build, it will set ~ / .pip / pip.conf appropriately. If you do not set the PIP_PROXY variable using —build-arg, it will be skipped.

From the above, it is possible to decide the use of the cache server at build time.

Summary

Prepare a devpi server container using the Container Station of the QNAP NAS. did. People in the ARM environment take time, so it will be easier to use the image created this time. Also, the ARG command of Dockerfile is used to switch the use of the cache server at build time.

This time as well, (with Docker) the story of building a PyPI cache server and making me a little happy again was helpful.

Recommended Posts

Build a Pypi cache server on QNAP
Build a simple WebDAV server on Linux
Build a Samba server on Arch Linux
Build a web server on your Chromebook
[Part 2] Let's build a web server on EC2 Linux
Build a python environment on CentOS 7.7 for your home server
Run a Linux server on GCP
Run TensorFlow2 on a VPS server
Register as a package on PyPI
Build a python3 environment on CentOS7
Easily build a DNS server using Twisted
Build a python environment on MacOS (Catallina)
Build an NFS server on Arch Linux
Build CGI Server running on Python 3 on Docker
Build jupyter notebook on remote server (CentOS)
Build a Python + OpenCV environment on Cloud9
Run Jupyter notebook on a remote server
Build a proxy server with nginx on multiple remote servers using Ansible
Build a LAMP environment on your local Docker
Build a WardPress environment on AWS with pulumi
Simply build a Python 3 execution environment on Windows
Build a Django environment on Raspberry Pi (MySQL)
Build a python environment with ansible on centos6
Build a Python environment on Mac (Mountain Lion)
Build a Python development environment on your Mac
Build a Kubernetes environment for development on Ubuntu
Mount a directory on another server with sshfs
Building a Python environment on a Sakura VPS server
Source compile Apache2.4 (httpd 2.4.43) + PHP7.4 on Linux and build a Web server ―― 1. Apache introduction
Build a Python development environment on Raspberry Pi
Source compile Apache2.4 (httpd 2.4.43) + PHP7.4 on Linux and build a Web server --2 PHP introduction
Build a server on Linux and local network with Raspberry Pi NextCloud and desktop sharing
Source compile Apache2.4 (httpd 2.4.43) + PHP7.4 on Linux to build a Web server --3 MySQL 8.0 introduction
[CentOS 7.3] Build an FTP server on the ESXi host
Build a GVim-based Python development environment on Windows 10 (3) GVim8.0 & Python3.6
Build a Django development environment using pyenv-virtualenv on Mac
Set up a simple local server on your Mac
# 2 Build a Python environment on AWS EC2 instance (ubuntu18.04)
Build a local development environment for Laravel6.X on Mac
Set up a file server on Ubuntu 20.04 using Samba
Build a machine learning Python environment on Mac OS
Build a Python extension for E-Cell 4 on Windows 7 (64bit)
I want to find a popular package on PyPi
Build a GVim-based Python development environment on Windows 10 (1) Installation
How to build a Django (python) environment on docker
Build Apache HTTP Server and Wildfly on Oracle Linux 8
Build a Python development environment on Mac OS X
Set up a free server on AWS in 30 minutes
Build a Python environment on your Mac using pyenv
[Part 1] Let's set up a Minecraft server on Linux
CTF beginner tried to build a problem server (web) [Problem]
Build a Python development environment using pyenv on MacOS
Stray build of Python 3 on Sakura's rental server (Note)
How to run Django on IIS on a Windows server
How to build a Python environment on amazon linux 2
Build a machine learning environment natively on Windows 10 (x64)
Introducing cache server nuster
A story about building a PyPI cache server (with Docker) and making me a little happy again
Build TensorFlow on Windows
Build XGBoost on Windows
build Python on Ubuntu