[PYTHON] Building a machine learning environment with Tellus GPU server (Sakura high-power computing)

Introduction

• Described the procedure for building a machine learning environment on the Tellus GPU server (high thermal power computing).
• Check the following three items to check the operation
• GPU learning of deep learning model using PyTorch
• Browsing experiment records using MLFlow
• In-server data confirmation using QGIS

Verification environment

Apply for GPU server (via Terrass)

Tellus

• Tellus
• Japan's first hygiene data platform
• In addition to acquiring data, we also provide a development environment free of charge (JupyterLab or GPU server).

About GPU server

• Sakura High Thermal Power Computing Lineup
• About 1 million yen if you try to borrow normally
• There is a rental period, but there is no usage time limit if it is within the period
• If you do not access for one month from the last access, it will be considered as rental end
• 90 minutes / 12 hours rule exists for google colaboratory

Application flow

1. After registering as a member of Terras (free of charge), apply for a development environment.

• Free of charge

2. The period can be selected from 1 month, 3 months or more (consultation required)

• Can be renewed after the usage period ends

3. After a while after applying, the operation will contact you with your login ID.

• It depends on the availability of the server, but it takes about 1 month.

Environment construction (GPU)

Basically, follow the procedure of CUDA Toolkit / GPU card driver installation procedure

Server information

Tellus account dashboard → See development environment

Connect to server

• Describe the connection information to the server in ~ / .ssh / config

~/.ssh/config

Host tellus
     HostName [Environment host name / IP]
     User [Login ID]
     IdentityFile ~/.ssh/id_rsa

• Enter ssh tellus on the Terminal and you will be asked for the password. Enter the initial password to complete the connection.

Package update and installation

Preparation before installing GPU driver

sudo apt update
sudo apt upgrade
apt install build-essential
apt install dkms

CUDA Toolkit

• CUDA Toolkit Archive
• The latest version of CUDA as of 09/09/2020 is 11.0
• PyTorch 1.6 (latest version) supports up to 10.2, so downgrade is required
• If you use anything other than runfile, 11.0 is installed for some reason even if you specify the version, so be sure to use runfile (local) **
• If you remove sudo when running runfile, the installation failed, so add it.

wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run
sudo sh cuda_10.2.89_440.33.01_linux.run
chmod +x cuda_10.2.89_440.33.01_linux.run
sudo ./cuda_10.2.89_440.33.01_linux.run --toolkit --samples --samplespath=/usr/local/cuda-samples --no-opengl-libs

• After creating the environment variable configuration file, log out and log in again.

/etc/profile.d/cuda.sh

export CUDA_HOME="/usr/local/cuda" 
export PATH="$CUDA_HOME/bin:$PATH" 
export LD_LIBRARY_PATH="/usr/local/lib:$CUDA_HOME/lib64:$LD_LIBRARY_PATH" 
export CPATH="/usr/local/include:$CUDA_HOME/include:$CPATH" 
export INCLUDE_PATH="$CUDA_HOME/include" 

shell:/etc/profile.d/cuda.csh

export CUDA_HOME="/usr/local/cuda" 
export PATH="$CUDA_HOME/bin:$PATH" 
export LD_LIBRARY_PATH="/usr/local/lib:$CUDA_HOME/lib64:$LD_LIBRARY_PATH" 
export CPATH="/usr/local/include:$CUDA_HOME/include:$CPATH" 
export INCLUDE_PATH="$CUDA_HOME/include" 

CUDA Driver

• CUDA Driver Download
• The driver that can be installed with CUDA Toolkit is old, so install it separately.
• Add sudo at runtime like Toolkit

wget https://us.download.nvidia.com/tesla/440.95.01/NVIDIA-Linux-x86_64-440.95.01.run
chmod +x NVIDIA-Linux-x86_64-440.95.01.run
sudo ./NVIDIA-Linux-x86_64-440.95.01.run --no-opengl-files --no-libglx-indirect --dkms

cuDNN

• NVIDIA cuDNN
• Membership registration required (free)
• Download on the client side and transfer to the server with scp
• After transfer, unzip and move the contents to the specified directory

client

scp -r cudnn-10.2-linux-x64-v8.0.3.33.tgz tellus:~/

server

tar xvzf cudnn-10.2-linux-x64-v8.0.3.33.tgz
sudo mv cuda/include/cudnn.h /usr/local/cuda/include/
sudo mv cuda/lib64/* /usr/local/cuda/lib64/

Installation confirmation

• Check with nvidia-smi
• If the installation is successful, you can see the display like the image below.

Environment construction (Python)

Anaconda

• Anaconda
• After installation, you can create the environment as you like.

wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
sudo bash Anaconda3-2020.07-Linux-x86_64.sh
conda update -n base conda

• When I created the environment, it didn't work as it was, so I added the following to .bashrc (py38 is the environment name)

.bashrc

export PYTHONPATH="/home/[Login ID]/anaconda3/envs/py38/lib/python3.8:/home/[Login ID]/anaconda3/envs/py38/lib/python3.8/site-packages:$PYTHONPATH"

PyTorch

• PyTorch get-started
• When using other than conda, change the Package part and check the command.

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

MLFlow

• MLFlow
• Machine learning experiment management library

conda install -c conda-forge mlflow

• If you enter mlflow ui, the UI will be launched at localhost: 5000 and you can check the experimental results with your browser.

• Add LocalForward setting to ~ / .ssh / config so that it can be viewed on the client side browser when the UI is launched on the server side.

~/.ssh/config

Host tellus
     HostName [Environment host name / IP]
     User [Login ID]
     IdentityFile ~/.ssh/id_rsa
     LocalForward [Client side port number] localhost:5000

QGIS

• QGIS
• Viewer of data with geographic information such as GeoTiff and Shapefile
• You can also browse ordinary images without geographic information.
• It didn't work with the latest version (3.14.15), so I used 3.10.8
• Error that libprotobuf-lite.so.23 is missing

conda install -c conda-forge qgis=3.10.8

Operation check

GPU learning

• Run in two modes, CPU and GPU, and check if there is a difference in processing time
• Check GPU Memory and Volatile with nvidia-smi when executing code
• Code changed the following part referring to PyTorch CIFAR10 Tutorial
• Model is ResNet-18
• Batch size is 1024, number of workers is 8 (= number of server cores)
• CPU execution is executed by rewriting to device = torch.device ("cpu ")

cifar10.py

import os

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.models as models
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import CIFAR10
from tqdm import tqdm


batch = 1024
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


def dataloader(is_train: bool, transform: transforms.Compose) -> DataLoader:
    dataset = CIFAR10(root='./data', train=is_train, download=True, transform=transform)
    return DataLoader(dataset, batch_size=batch, shuffle=is_train, num_workers=os.cpu_count())


def model() -> nn.Module:
    model = models.resnet18(pretrained=True)
    model.fc = nn.Linear(512, 10)
    return model.to(device)


def training(net: nn.Module, trainloader: DataLoader, epochs: int) -> None:
    # loss function & optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(epochs):  # loop over the dataset multiple times
        running_loss = 0.0
        bar = tqdm(trainloader, desc="training model [epoch:{:02d}]".format(epoch), total=len(trainloader))
        for data in bar:
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data[0].to(device), data[1].to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            bar.set_postfix(device=device, batch=batch, loss=(running_loss / len(trainloader)))

    print('Finished Training')


transform = transforms.Compose(
    [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainloader = dataloader(True, transform)
net = model()
training(net, trainloader, 3)

CPU results

GPU results

• Confirmed that it was about 36 times faster by using GPU
• Confirm that the values of Memory and Volatile changed when using GPU

MLFlow

• Check if the experiment record on the server can be viewed with the browser on the client side
• Experimental code saves graph of Sin waveform with double amplitude
• Check the local forward port number at 23000

record_sin.py

from math import pi, sin

import mlflow

mlflow.set_experiment('test')
amplitude = 2.0

with mlflow.start_run() as _:
    mlflow.log_param('amplitude', amplitude)
    for i in range(360):
        sin_val = amplitude * sin(i * pi / 180.)
        mlflow.log_metric('sin wave', sin_val, step=i)

~/test_code/

python record_sin.py
mlflow ui

Result image

QGIS

• Check if the server side code (test_code) can be viewed with Data Source Manager

ssh -X tellus
qgis

• Make sure you have cifar10.py and record_sin.py

Using VS Code

• Since ssh can be used, VS Code's Remote Development can also be used.
• If you want to create jupyter notebook file with VS Code, you need to install ipykernel

conda install -c conda-forge ipykernel

in conclusion

• I chose the deb file in the CUDA Toolkit installation and wasted time, so I summarized the procedure
• If you replace the QGIS part with another viewer, you should be able to follow almost the same procedure.
• I think it is a procedure that can be used to some extent not only with high-heat computing but also with other GPU servers.

Reference page

Tellus FAQ Bamboo shoot blog-Building a PyTorch environment from the Terraus GPU server