[PYTHON] [PyTorch Tutorial ⑦] Visualizing Models, Data, And Training With Tensorboard

Introduction

This is the 7th installment of PyTorch Official Tutorial following Last time. This time, we will proceed with Visualizing Models, Data, and Training with TensorBoard.

Visualizing Models, Data, And Training With Tensorboard

In 60 Minute Blitz we have seen how to build a basic neural network and train using training data. This time, let's see how to visualize the training situation and check whether the training is in progress. Visualization uses TensorBoard. PyTorch can take advantage of the "TensorBoard" tool for training neural networks and visualizing results.

This tutorial uses Fashion-MNIST data from torchvision.datasets to illustrate some of its features. Learn how to:

1. Read the data and make the appropriate conversions. (Almost the same as the previous tutorial)
2. Set up TensorBoard.
3. Write to TensorBoard.
4. Use TensorBoard to visualize the model architecture.
5. Use TensorBoard to visualize predictions and accuracy during training.

Specifically, the following can be seen from the above five points.

• How to inspect training data
• How to track model performance during model training
• How to evaluate the performance of a model after training.

CIFAR-10 Start with the same code as the tutorial.

%matplotlib inline

# imports
import matplotlib.pyplot as plt
import numpy as np

import torch
import torchvision
import torchvision.transforms as transforms

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

#definition of transform
transform = transforms.Compose(
    [transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))])

#data set
trainset = torchvision.datasets.FashionMNIST('./data',
    download=True,
    train=True,
    transform=transform)
testset = torchvision.datasets.FashionMNIST('./data',
    download=True,
    train=False,
    transform=transform)

#Data loader
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                        shuffle=True, num_workers=2)


testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                        shuffle=False, num_workers=2)

#Classification constant
classes = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
        'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot')

#Image display helper function
# (below`plot_classes_preds`Used in function)
def matplotlib_imshow(img, one_channel=False):
    if one_channel:
        img = img.mean(dim=0)
    img = img / 2 + 0.5     #Denormalized
    npimg = img.numpy()
    if one_channel:
        plt.imshow(npimg, cmap="Greys")
    else:
        plt.imshow(np.transpose(npimg, (1, 2, 0)))

CIFAR-10 Defines a model similar to the tutorial, but with one channel instead of three images and 28x28 instead of 32x32. I'll change it a little to make it.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 4 * 4, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 4 * 4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

Define the same optimizer and loss function as before.

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

1.TensorBoard setup

Next, set up the TensorBoard. Define a SummaryWriter, an object for importing TensorBoard from torch.utils and writing to TensorBoard.

from torch.utils.tensorboard import SummaryWriter

#The default log directory is"runs"However, you can specify it here.
writer = SummaryWriter('runs/fashion_mnist_experiment_1')

Just run this line and it will create the "runs / fashion_mnist_experiment_1" directory.

2.Writing to TensorBoard

Then use make_grid to write the image to TensorBoard.

#Get a random training image
dataiter = iter(trainloader)
images, labels = dataiter.next()

#Create a grid of images
img_grid = torchvision.utils.make_grid(images)

#Display image
matplotlib_imshow(img_grid, one_channel=True)

#Write to tensor board
writer.add_image('four_fashion_mnist_images', img_grid)

(Not mentioned in this tutorial, but if you want to use Tensorboard with Google Colaboratory, load the TensorBoard notebook extension.)

%load_ext tensorboard

#When using Tensorboard with Google Colaboratory, execute Tensorboard with magic command
#tensorboard --logdir=runs
%tensorboard --logdir=runs

When run in a local environment https：//localhost:6006 You can browse the following tensorboard screens with. I was able to run TensorBoard. We'll look at the features of TensorBoard below.

3.Inspect the model using TensorBoard

One of the strengths of TensorBoard is the ability to visualize complex model structures. Let's visualize the created model.

writer.add_graph(net, images)
writer.close()

After updating the TensorBoard, you will see the "GRAPHS" tab as shown below. You can double-click Net to expand it and see the individual components that make up the model. TensorBoard has a very useful function for visualizing high-dimensional data such as image data in low-dimensional space. It will be explained below.

4.Adding a “Projector” to TensorBoard

You can visualize a low-dimensional representation of high-dimensional data via the add_embedding method.

import tensorflow as tf
import tensorboard as tb
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile

#Helper function
def select_n_random(data, labels, n=100):
    '''
Select n random data points and their corresponding labels from the dataset
    '''
    assert len(data) == len(labels)

    perm = torch.randperm(len(data))
    return data[perm][:n], labels[perm][:n]

#Select a random image and its target index
images, labels = select_n_random(trainset.data, trainset.targets)

#Get the class label for each image
class_labels = [classes[lab] for lab in labels]

#Embed log
features = images.view(-1, 28 * 28)
writer.add_embedding(features,
                    metadata=class_labels,
                    label_img=images.unsqueeze(1))
writer.close()

TensorBoard's PROJECTOR tab displays these 100 images. Each image is 784 dimensions, but is projected into 3D space. You can rotate the 3D projection by dragging. If you select "Color: Label" in the upper left and enable "Night mode", the background will be black and the image will be easier to see.

Now you know how to use TensorBoard to visualize your data. Next, let's see how to visualize training and evaluation with TensorBoard.

5.Tracking model training with TensorBoard

In the previous tutorial (https://qiita.com/sudominoru/items/61f57946799e67cedd47), we simply printed the model loss value after every 2000 iterations. In this tutorial, you will record the loss value in TensorBoard and display the predicted value with the plot_classes_preds function.

#Helper function
def images_to_probs(net, images):
    '''
It returns the predicted value and its probability with the trained model and image as arguments.
    '''
    output = net(images)
    #Convert the output probability to a prediction class
    _, preds_tensor = torch.max(output, 1)
    preds = np.squeeze(preds_tensor.numpy())
    return preds, [F.softmax(el, dim=0)[i].item() for i, el in zip(preds, output)]


def plot_classes_preds(net, images, labels):
    '''
Generates a matplotlib diagram with the trained model, image, and teacher data as arguments.
It displays the label with the highest probability that the model predicted and whether the prediction is correct.
Color it.
    「images_to_Use the "probs" function.
    '''
    preds, probs = images_to_probs(net, images)
    #Plot the image in batch with the predicted and actual labels.
    fig = plt.figure(figsize=(12, 48))
    for idx in np.arange(4):
        ax = fig.add_subplot(1, 4, idx+1, xticks=[], yticks=[])
        matplotlib_imshow(images[idx], one_channel=True)
        ax.set_title("{0}, {1:.1f}%\n(label: {2})".format(
            classes[preds[idx]],
            probs[idx] * 100.0,
            classes[labels[idx]]),
                    color=("green" if preds[idx]==labels[idx].item() else "red"))
    return fig

Learn using the same model as in the previous tutorial (https://qiita.com/sudominoru/items/61f57946799e67cedd47), but write to TensorBoard every 1000 batches instead of printing to the console. (Add_scalar function) In addition, the predicted value during training and the predicted image are output. (Add_figure function)

running_loss = 0.0
for epoch in range(1):  #Loop the dataset multiple times

    for i, data in enumerate(trainloader, 0):

        #Get the input. Data is[inputs, labels]Is a list of
        inputs, labels = data

        #Initialize the gradient
        optimizer.zero_grad()

        #Forward propagation+Backpropagation+optimisation
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 1000 == 999:    #Every 1000 batches

            # ...Record the loss value every 1000 batches
            writer.add_scalar('training loss',
                            running_loss / 1000,
                            epoch * len(trainloader) + i)

            # ...Log model predictions to Matplotlib diagrams in a random mini-batch
            writer.add_figure('predictions vs. actuals',
                            plot_classes_preds(net, inputs, labels),
                            global_step=epoch * len(trainloader) + i)
            running_loss = 0.0
print('Finished Training')

out

Finished Training

On the SCALARS tab, you can see the loss value during training. In addition, you can see the model predictions made every 1000 batches. Go to the IMAGES tab and scroll under the predictions vs. actuals visualization. If you look at it, after just 3,000 training sessions, the model can already classify shirts, sneakers, coats and more. However, the odds are not as high as in the second half of training. In Last Tutorial, we checked the accuracy rate for each label after training. Here we use TensorBoard to plot the PR curve for each class.

6.Assessing trained models with TensorBoard

# 1. test_size x num_Get probability predictions with classes Tensor
# 2. test_Get preds with size Tensor
#Takes up to 10 seconds to run
class_probs = []
class_preds = []
with torch.no_grad():
    for data in testloader:
        images, labels = data
        output = net(images)
        class_probs_batch = [F.softmax(el, dim=0) for el in output]
        _, class_preds_batch = torch.max(output, 1)

        class_probs.append(class_probs_batch)
        class_preds.append(class_preds_batch)

test_probs = torch.cat([torch.stack(batch) for batch in class_probs])
test_preds = torch.cat(class_preds)

#Helper function
def add_pr_curve_tensorboard(class_index, test_probs, test_preds, global_step=0):
    '''
"Class" from 0 to 9_Take in "index" and plot the corresponding PR curve
    '''
    tensorboard_preds = test_preds == class_index
    tensorboard_probs = test_probs[:, class_index]

    writer.add_pr_curve(classes[class_index],
                        tensorboard_preds,
                        tensorboard_probs,
                        global_step=global_step)
    writer.close()

#Plot the PR curve
for i in range(len(classes)):
    add_pr_curve_tensorboard(i, test_probs, test_preds)

The PR CURVES tab is displayed. Let's open the PR curve of each label and check it. You can see that some labels have almost 100% "area under the curve", while some labels have less of this area. This tutorial introduced you to the integration of TensorBoard with PyTorch. Of course, you can do something similar to TensorBoard with Jupyter Notebook alone, but with TensorBoard you can see it visually.

7. Finally

The above is "Visualizing Models, Data, And Training With Tensorboard". I learned how to use Tensorboard in PyTorch. Next time, I would like to proceed with the "V Torch Vision Object Detection Finetuning Tutorial".

History

2020/10/15 First edition released