[PYTHON] Steps to attach and debug from VS Code to Jupyter Lab on a remote server

1. What to explain in this post

Follow the steps below to build an environment for debugging using Jupyter Lab and VS Code.

1. Launch Jupyter Lab on a remote server
2. Call a Python class from JupyterLab.
3. From VS Code on the local PC, remotely attach to the above Jupyter Lab notebook and debug.
4. The figure is as follows.

If you watch the video below, you can get an idea of what you can do.

2. Motivation
3. JupyterLab is convenient for writing a little code and sharing the results neatly together, but when creating a program in earnest, you probably use an editor such as VS Code.
4. On the other hand, when verifying using the program developed in this way, it is often convenient to call it from JupyterLab.
5. In this usage, we will explain how to set a breakpoint on the program called from JupyterLab and build an environment for debugging from VS Code.

3. Points of the environment to be built this time

--In VS Code, by using a mechanism called remote development, you can edit folders and files on the remote server almost the same as on your local PC by connecting with ssh. See here for details. --VS Code allows you to attach and debug Python processes on the remote server by installing a package called ptvsd on the remote server. See here for details. --In JupyterLab, it is possible to use any process (not necessarily Python) as an interpreter by using a mechanism called kernel. ――In this paper, by combining the above, the above explanation is realized.

4. Environment construction

4.1. Remote server environment construction

4.1.1. Prerequisites

--In this article, we built the environment based on the Docker image "tensorflow / tensorflow: 1.14.0-gpu-py3-jupyter". ――However, if you have already installed Jupyter Lab with anaconda etc., you can use it as it is. (I'll explain it below, but the point is that it works if ptvsd is installed on the remote server and set as the kernel of Jupyter Lab.)

4.1.2. Installation of required apps

# apt-get update
# apt-get install -y wget
# apt-get install -y language-pack-ja
# apt-get install -y ssh
# apt-get install -y git
# update-locale LANG=ja_JP.UTF-8

4.1.3. Installation of nodejs

Required to install the JupyterLab Extension. This step is not necessary if you just want to set the kernel, but it is better to support it for convenient use of Jupyter Lab.

# curl -sL https://deb.nodesource.com/setup_12.x | bash -
# apt-get install -y nodejs

4.1.4. Installing Python & JupyterLab

--In particular, you need jupyterlab and ptvsd. --This time, I will install numpy, matplotlib, scikit_learn. --As of April 2020, version 2 of jupyterlab is out, but since there are many extensions that are not yet supported, we will install 1.0.2 this time.

# pip install -r requirements.txt

requirements.txt

numpy==1.16.3
matplotlib==3.0.3
scikit_learn==0.20.3
jupyterlab==1.0.2
ptvsd==4.3.2

4.1.5. Start ssh to access from VS Code on the local PC side

Required to connect to a remote server from VS Code for remote development. This time, open the ssh port on port 19205.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.org
# sed -e s/#Port/Port/g /etc/ssh/sshd_config.org  | sed -e s/22/19205/g > /etc/ssh/sshd_config

Next, start ssh.

root@25fa703a90c4:/tf# service ssh start
 * Starting OpenBSD Secure Shell server sshd                             [ OK ]

4.2. Jupyter Lab settings

Next, set up Jupyter Lab. This time, we will start the jupyterLab server as a jupyter user.

4.2.1. Change ownership of / usr / local / share / jupyter / lab directory

First, in preparation, change ownership of the / usr / local / share / jupyter / lab directory to jupyter. If you do not do this, you will not be able to install Extensions from Jupyter Lab under startup as a jupyter user, which is inconvenient. I think there is a better way (for example, keep these files in your home directory). Here, I will simply change the ownership below.

# chown -R jupyter:jupyter /usr/local/share/jupyter/lab

4.2.2. Set ptvsd as kernel of Jupyter Lab

First, save the following file with the file name /usr/local/share/jupyter/kernels/ptvsd/kernel.json so that you can call the debug interpreter from JupyterLab. (The method of calling the interpreter set here from JupyterLab will be explained later.)

• I think that the owner of this directory is root, so save it as root.

kernel.json

{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ptvsd",
  "--host",
  "0.0.0.0",
  "--port",
  "19204",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python3 Debug",
 "language": "python"
}

It is like this.

root@25fa703a90c4:/usr/local/share/jupyter/kernels# pwd
/usr/local/share/jupyter/kernels
root@25fa703a90c4:/usr/local/share/jupyter/kernels# mkdir ptvsd
root@25fa703a90c4:/usr/local/share/jupyter/kernels# cd ptvsd
root@25fa703a90c4:/usr/local/share/jupyter/kernels/ptvsd# cat > kernel.json <<EOL
{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ptvsd",
  "--host",
  "0.0.0.0",
  "--port",
  "19204",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python3 Debug",
 "language": "python"
}
EOL
root@25fa703a90c4:/usr/local/share/jupyter/kernels/ptvsd# ls kernel.json
kernel.json

This completes the Jupyter Lab settings.

4.3. VS Code settings

Next, make settings for remote development with VS Code.

4.3.1. Preparing to connect with ssh

First of all, if you haven't done so already, you need to prepare for key exchange so that you can connect to the remote server from your local PC with ssh. I will omit the detailed procedure, but I think that here etc. will be helpful.

4.3.2. Installing the Remote Development Extension Pack

Follow these steps (https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) to install the Remote Development extension pack in Visual Studio.

4.3.3. Settings to connect to a remote server from VS Code

Follow the steps below to connect to a remote server from VS Code 1 and edit folders and files. (From the second time onward, you can also connect using the saved settings.)

1. Click the button at the bottom left of the screen

ssh user@host name-p port name

4.3.4. Installing Python extension for Visual Studio Code

Next, install the Python extension for Visual Studio Code from here to debug Python on VS Code.

This completes the connection from VS Code to the remote server. This procedure allows you to edit files and folders on the remote server with VS Code as if they were on your local PC.

5. Run the sample program

5.1. Description of sample program

It's very easy, but I will use the sample program below. First, on the remote server, save the following files in the same directory where you save the notebook.

sample.py

import numpy as np

def sample(x):
    y = np.square(x)
    return y

5.2. Access to Jupyter Lab

Next, start JupyterLab on the remote server and access JupyterLab from the browser on the local PC. (It is the same as accessing Jupyter Lab normally.)

5.2.1. Start Jupyter Lab on remote PC

jupyter lab --ip=0.0.0.0

5.2.2. Access Jupyter Lab

You can find the following message in the log when jupyterLab is started, so you can open JupyterLab by accessing the URL from your local PC. * At the time of access, the IP address will be changed to that of the remote PC.

    To access the notebook, open this file in a browser:
        file:///home/jupyter/.local/share/jupyter/runtime/nbserver-13-open.html
    Or copy and paste one of these URLs:
        http://(da9742c1f7ef or 127.0.0.1):8888/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

You can access JupyterLab by accessing the URL output when 5.2.1 JupyterLab starts from a browser on your local PC.

5.3. Specify Kernel and create notebook

When you access JupyterLab, you should see a kernel called "Python3 debug" in the Notebook column as shown below. (This is the kernel using ptvsd set in step 4.2.2.) Click this button to create a new notebook.

The upper right of the screen is "Python3 Debug", and you can see that the Kernel for Debug created earlier is used.

For the time being, create a notebook like the one below and run it.

• It seems that gist cannot be pasted on Qiita, but please refer to here for the source.

5.4. Setting up remote debugging in VS Code

5.4.1. Connect to a remote server and open the project

Follow the steps in 4.3 to connect to the remote server with VS Code. Once connected, open the sample program. It is like this.

5.4.2. Debug settings (launch.json settings)

Follow the steps below to configure debug settings.

The settings are as follows.

--port: Specify the port set in the kernel settings of ptvsd. --host: Specify the IP address of the remote server. --pathMappings: localRoot should be $ {workspaceFolder} as shown in the example below. remoteRoot specifies the folder on the remote server where the files are stored.

• If the breakpoint setting does not work, there may be a problem with the pathMappings setting, so it is a good idea to play around with it.

{
    //You can use IntelliSense to learn the available attributes.
    //Hover and display the description of existing attributes.
    //Check the following for more information: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
    
        {
            "name": "Python:attach",
            "type": "python",
            "request": "attach",
            "port": 19204,
            "host": "192.168.2.200",
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/home/jupyter/proj/docker"
                }
            ]
        }
    ]
}

This completes the debug settings. Next, let's actually debug.

5.5. Attach to JupyterLab process and debug

The debugging procedure is as follows. See also the demo video at the beginning for details.

1. After setting breakpoints on VS Code as appropriate, execute the "Python attach" set earlier.

2. Run notebook from JupyterLab

3. Debugging is performed because the processing is interrupted at the breakpoint.

4. The result of the command executed in "Debug Console" on VS Code is also displayed on JupyterLab. Especially, it is very convenient because you can display internal variables with matplotlib etc. and display them as a graph!

5. Summary So far, we have explained how to debug a Python process executed from JupyterLab by remotely attaching it from VS Code. It's very convenient, so please try it. It's a little scribbled, so there may be some deficiencies. If you have any questions, please comment and I would like to add or correct as appropriate.