Write about building a Python environment for writing Qiita Qiita

Let's write a Qiita article on machine learning for the first time in a while! !! !! I thought, but every time it was troublesome to build the environment, it was a hassle, so it was a meta-like thing that I tried to build an environment for it? I would like to write an article.

Overview

In this article, I would like to write about how to easily set up an independent python environment and connect from a browser using Docker-compose. We'll also write about how to execute code interactively from VS Code as well as access from a browser! Even if these are not for writing Qiita, I think that they can answer requests such as "I want to run it in a clean execution environment"!

Premise

This article does not cover the introduction of Docker, Docker-compose and VS Code. If you look it up, it will come out easily (Docker on Windows seems to be difficult, but ...). Also, since I specialize in data science, the programming language is Python3. I won't write in detail, but R is also partially supported. Since I am an Apple believer, I use a Mac as the OS, but I think that it will probably work in any environment as long as Docker can be used.

procedure

In this article, we will take the following steps:

1. Download Dockerfile
2. Create docker-compose.yml
3. Start the container from docker-compose and access from the browser
4. (Bonus) Comfortable coding with VS Code's Remote Desktop and jupyter's Extension

1. Download Dockerfile

The docker-stacs repository on jupyter official github has many tools for building a python environment using Jupyter. For example, if you want to do Deep Learning with tensorflow, go to tensorflow-notebook and it will look like below. You can see that there is a Dockerfile in.

In this Dockerfile

ARG BASE_CONTAINER=jupyter/scipy-notebook

There is a part called, but this is said to be an additional operation based on scipy-notebook It makes sense. And if you look at scipy-notebook, it's the same.

ARG BASE_CONTAINER=jupyter/minimal-notebook

It is written that. This is because the environment for tensorflow is the environment for Scipy (for data science with basic python) and tensorflow is added, and the environment for Scipy is the environment where scikit-learn and pandas are added to the minimal environment. It means that there is. I won't go into details, but there are other spark environments and R environments, and if you use the Dockerfile as needed, the environment will be launched in one shot! Also, since it is Docker, if you have a library you want to add personally, you only need to add a little, which is very convenient!

So, first open the directory you want to run from the terminal and clone this repository.

$ git clone https://github.com/jupyter/docker-stacks.git

And, for example, when you want to use base-notebook,

$ cp -R ./docker-stacks/base-notebook ./

If you copy the tool, the download is complete.

2. Create docker-compose.yml

It is troublesome to start jupyter every time and write a command that specifies the port, so I will start it easily using docker-compose.

docker-compose.yml

version: '3'
services:
  #Service name jupyterlab(Container name)Set to
  jupyterlab:
    #Build from Dockerfile(To be able to write additional settings)
    build:
      context: .
      dockerfile: Dockerfile

    #host's./work in the container/home/Mount on jovyan
    # Jovian means "related to Jupiter"
    volumes:
       - "./work:/home/jovyan"
      
    #Set user
    user: root
    environment:
      B_UID: 1000
      NB_GID: 100
      GRANT_SUDO: "yes"

    #Allow access to the device
    privileged: true

    #Map host 8888 to container 8888
    ports:
      - "8888:8888"

    #Reboot unless stopped
    restart: unless-stopped

    #Start jupyterlab after starting the container
    command: start.sh jupyter lab --NotebookApp.token=''

It is OK if you create the above file in the same hierarchy as the Dockerfile (I am directly under the base-notebook directory). However, if port 8888 is in use by the host, it is a good idea to set XXXX: 8888 to a different number.

3. Start the container from docker-compose and access from the browser

After doing the above, go to the same level as docker-compose.yml in the terminal,

$ docker-compose up -d

Just hit. Then, when it starts up, type localhost: 8888 in your browser The jupyter lab is opened like this. It's really easy. Since the volume is set to ./work /home/jovyan, if you create a file from the home of jupyterlab, it will be reflected in ./work on the host side. You have now built a python environment. All you have to do is install the required packages according to the article you want to write, and share the Dockerfile for reproducibility!

4. (Bonus) Comfortable coding with VS Code's Remote Desktop and jupyter's Extension

This is enough, but this time I will challenge to create an interactive execution environment with VS Code as a bonus. The advantage of doing this is that you can still partially execute the py file. I think it's very convenient to be able to easily debug and write processes that are difficult only for ipynb files, such as refactoring.

Then, I will start work immediately. If VS Code with Docker and Remote Desktop Extension installed, press Command + Shift + P to open the command operation tab as shown below. Use Open Folder in Container to open the tool you just set. (In my case, open the base-notebook)

Now, when asked for Config, choose From docker-compose. Then wait for a while, If such a screen is opened, access to the container is successful for the time being.

Speaking of jupyter, type jupyter from the Extension search and install this one displayed at the top.

When the installation is complete, the button "Reload required" will appear as shown below, so press it to reload.

After reloading, let's create a python file by deciding appropriately as jupyter_test.py in ./work. Then type "# %%" as shown below and you'll see commands like "Run Cell" above it! When you press this Run Cell, the code written in the block up to the next "# %%" will be executed interactively. If you can confirm this, you are successful!

Summary

This time, with the motivation to write an article on Qiita, I introduced how to easily set up a clean and reproducible python execution environment, and even introduced how to execute it interactively with VS Code. Even if you are not writing a blog, it is convenient to be able to prepare multiple independent python environments with a single command. After all, I realized again that Docker is amazing (vocabulary). In the future, I would like to use this environment to write some articles that are valuable in terms of data science. Thank you for watching until the end! !!