[PYTHON] An amateur stumbled in Deep Learning from scratch Note: Chapter 5
Introduction
I suddenly started studying in Chapter 5 of "Deep Learning from scratch-The theory and implementation of deep learning learned with Python". This is a memo of the trip.
The execution environment is macOS Mojave + Anaconda 2019.10, and the Python version is 3.7.4. For details, refer to Chapter 1 of this memo.
(To other chapters of this memo: Chapter 1 / Chapter 2 / Chapter 3 / Chapter 4 / Chapter 5 / [Chapter 6](https: / /qiita.com/segavvy/items/ca4ac4c9ee1a126bff41) / Chapter 7 / Chapter 8 / Summary)
Chapter 5 Error back propagation method
This chapter describes the error backpropagation method for speeding up the calculation of weight parameters in neural network training.
By the way, the word "propagation" (denpa) is a translation of Propagetion, but the Japanese is chaotic because it is also translated as "propagation" (denpan). I found an article on a blog that investigated this area, so if you are interested, rest after walking> [machine learning] The translation of "Propagation" is "propagation" or "propagation"? Please.
5.1 Calculation graph
The explanation by the calculation graph is very easy to understand when learning the error back propagation method. The secret to the popularity of this book may be Chapter 5.
Learning is essential for neural networks, which requires finding the gradient of the weight parameters and calculating the derivative of each parameter. Using the calculation graph, we can see that the calculation of the derivative can be done very efficiently by back-propagating the local derivative from the back to the front.
5.2 Chain rule
This is a review of the differential formula of the composition function and the explanation that backpropagation holds. I don't think it's a problem to read it through, but if you want to understand it correctly, you need to review the differentiation of high school.
I didn't remember the differential formula at all, so I decided to review it after a long time. However, I often study after my main business is over, and when it comes to books and websites, sleepiness attacks me and it's hard to get into my head. It is quite difficult for office workers to study in the gap time.
What I found there was Tri-It, which is being done by a home teacher, Try. Although it is a video lesson, it is very easy to understand even though it is free, and it is divided into about 15 minutes, so it is ideal for studying with a smartphone while traveling or taking a break. It's a service for students, but I think it's also very suitable for reviewing working people.
For reference, here is a link to a video about the introductory part of differentiation. Please note that it is limited to private viewing.
-Video lesson Try IT> Mathematics II Limits and differential functions -Video lesson Try IT> Mathematics II differential method
5.3 Backpropagation
A description of backpropagation of add and multiply nodes. Since differentiation is simple for both addition and multiplication, backpropagation calculations are easy.
5.4 Simple layer implementation
This is the implementation of the addition layer and multiplication layer in the previous section. Since the differentiation is simple, it is easy to implement.
5.5 Implementation of activation function layer
An implementation of the activation function layer.
5.5.1 ReLU layer
Since ReLU has a simple derivative calculation, it is easy to implement and is likely to lead to faster learning. Below is the implemented code.
relu.py
# coding: utf-8
class ReLU:
def __init__(self):
"""ReLU layer
"""
self.mask = None
def forward(self, x):
"""Forward propagation
Args:
x (numpy.ndarray):input
Returns:
numpy.ndarray:output
"""
self.mask = (x <= 0)
out = x.copy()
out[self.mask] = 0
return out
def backward(self, dout):
"""Backpropagation
Args:
dout (numpy.ndarray):Derivative value transmitted from the right layer
Returns:
numpy.ndarray:Derivative value
"""
dout[self.mask] = 0
dx = dout
return dx
5.5.2 Sigmoid layer
The derivative of the sigmoid function is a little complicated, but it can be calculated little by little using a calculation graph. However, the derivative of $ y = \ frac {1} {x} $ required for the first "/" node and the derivative of $ y = exp (x) $ of the "$ exp $" node do not review the formula. It's spicy. As for the formulas in this area, Mathematics learned from concrete examples> Calculus> Organize all 59 differential formulas with importance is recommended because it is simply organized.
After the calculation by the calculation graph, it transforms into the formula (5.12) of the book, but at first I did not understand the flow of transforming the second line from the end to the last line. It will be easier to understand if you insert one line in between as follows.
\begin{align}
\frac{\partial L}{\partial y}y^2exp(-x) &= \frac{\partial L}{\partial y} \frac{1}{(1+exp(-x))^2} exp(-x) \\
&=\frac{\partial L}{\partial y} \frac{1}{1+exp(-x)} \frac{exp(-x)}{1+exp(-x)} \\
&=\frac{\partial L}{\partial y} \frac{1}{1+exp(-x)} \biggl(\frac{1+exp(-x)}{1+exp(-x)} - \frac{1}{1+exp(-x)}\biggr)← Added line\\
&=\frac{\partial L}{\partial y} y(1-y)
\end{align}
What is surprising about this result is that it is very easy to calculate with $ y $, which is the output of forward propagation. In the error back propagation method, the forward propagation is calculated first, so if you have the result, it will lead to faster learning. The characteristic that even if $ e ^ x $ is differentiated, it becomes $ e ^ x $ is also important, and the person who thinks about the sigmoid function is really amazing.
The sigmoid layer is not used in this chapter, so the code is omitted.
5.6 Implementation of Affine / Softmax layer
5.6.1 Affine layer
The calculation process is omitted in the Affine layer backpropagation formula (5.13), but @yuyasat [calculates the components of the Affine layer backpropagation steadily](https://qiita.com/yuyasat/items/ The calculation process is summarized in d9cdd4401221df5375b6) for your reference.
5.6.2 Batch version of Affine layer
Below is the implemented code. In addition, in the code of the book, there is consideration when the input data is a tensor (4D data), but it is not included because the usage is not yet known.
affine.py
# coding: utf-8
import numpy as np
class Affine:
def __init__(self, W, b):
"""Affine layer
Args:
W (numpy.ndarray):weight
b (numpy.ndarray):bias
"""
self.W = W #weight
self.b = b #bias
self.x = None #input
self.dW = None #Derivative value of weight
self.db = None #Derivative value of bias
def forward(self, x):
"""Forward propagation
Args:
x (numpy.ndarray):input
Returns:
numpy.ndarray:output
"""
self.x = x
out = np.dot(x, self.W) + self.b
return out
def backward(self, dout):
"""Backpropagation
Args:
dout (numpy.ndarray):Derivative value transmitted from the right layer
Returns:
numpy.ndarray:Derivative value
"""
dx = np.dot(dout, self.W.T)
self.dW = np.dot(self.x.T, dout)
self.db = np.sum(dout, axis=0)
return dx
5.6.3 Softmax-with-Loss layer
Backpropagation of a layer with a set of softmax function and cross entropy error. The calculation process is described in detail in Appendix A of the book, but even if $ e ^ x $ is differentiated, it becomes $ e ^ x $, and when all the one-hot vectors of the teacher label are added, it becomes 1. Etc. contribute to simplifying the formula.
The final result of $ (y_1-t_1, y_2-t_2, y_3-t_3) $ is also surprising. It does show the difference between the output of the neural network and the teacher label, and the calculation can be done at high speed. It seems that the cross entropy error is designed to be "clean" when used as a loss function of the softmax function, and it is really amazing for those who think about the cross entropy error.
Note that the backpropagation value must be divided by the number of batches to accommodate batches. In the book, there was only an explanation that "by dividing the value to be propagated by the number of batches (batch_size), the error per data is propagated to the previous layer", and I could not understand why it was necessary to divide by the number of batches. However, I understood it from the explanation of the code of @ Yoko303's Deep Learning from scratch ~ Softmax-with-Loss layer ~. In the batch version of forward propagation, the cross entropy errors are finally summed and divided by the number of batches (batch_size) to make one value. Backpropagation also requires calculation of this part, and its derivative value is $ \ frac {1} {batch_size} $. Below, I wrote the calculation graph for that part.
I think this understanding is correct, but please point out any mistakes. Below is the implemented code.
softmax_with_loss.py
# coding: utf-8
from functions import softmax, cross_entropy_error
class SoftmaxWithLoss:
def __init__(self):
"""Softmax-with-Loss layer
"""
self.loss = None #loss
self.y = None #softmax output
self.t = None #Teacher data (one-hot vector）
def forward(self, x, t):
"""Forward propagation
Args:
x (numpy.ndarray):input
t (numpy.ndarray):Teacher data
Returns:
float:Cross entropy error
"""
self.t = t
self.y = softmax(x)
self.loss = cross_entropy_error(self.y, self.t)
return self.loss
def backward(self, dout=1):
"""Backpropagation
Args:
dout (float, optional):Derivative value transmitted from the right layer. The default is 1.
Returns:
numpy.ndarray:Derivative value
"""
batch_size = self.t.shape[0] #Number of batches
dx = (self.y - self.t) * (dout / batch_size)
return dx
Note that the code in the book does not use dout in backward. Since dout is used only with 1 specified, there is no problem in operation, but I think it is probably a mistake.
5.7 Implementation of error back propagation method
5.7.1 Overall view of neural network learning
This is a review of the implementation flow. There is no particular stumbling block.
5.7.2 Implementation of neural network corresponding to error back propagation method
First is the implementation of a general-purpose function. I brought only what I needed from what I wrote in the previous chapter.
functions.py
# coding: utf-8
import numpy as np
def softmax(x):
"""Softmax function
Args:
x (numpy.ndarray):input
Returns:
numpy.ndarray:output
"""
#For batch processing x is(Number of batches, 10)It becomes a two-dimensional array of.
#In this case, it is necessary to calculate well for each image using broadcast.
#Here, np so that it can be shared in both 1D and 2D..max()And np.sum()Is axis=-Calculated by 1
#Keepdims so that you can broadcast as it is=True to maintain the dimension.
c = np.max(x, axis=-1, keepdims=True)
exp_a = np.exp(x - c) #Overflow measures
sum_exp_a = np.sum(exp_a, axis=-1, keepdims=True)
y = exp_a / sum_exp_a
return y
def numerical_gradient(f, x):
"""Gradient calculation
Args:
f (function):Loss function
x (numpy.ndarray):An array of weight parameters for which you want to check the gradient
Returns:
numpy.ndarray:Slope
"""
h = 1e-4
grad = np.zeros_like(x)
# np.Enumerate the elements of a multidimensional array with nditer
it = np.nditer(x, flags=['multi_index'])
while not it.finished:
idx = it.multi_index # it.multi_index is the element number in the list
tmp_val = x[idx] #Save original value
# f(x + h)Calculation of
x[idx] = tmp_val + h
fxh1 = f()
# f(x - h)Calculation of
x[idx] = tmp_val - h
fxh2 = f()
#Calculate the gradient
grad[idx] = (fxh1 - fxh2) / (2 * h)
x[idx] = tmp_val #Return value
it.iternext()
return grad
def cross_entropy_error(y, t):
"""Calculation of cross entropy error
Args:
y (numpy.ndarray):Neural network output
t (numpy.ndarray):Correct label
Returns:
float:Cross entropy error
"""
#If there is one data, shape it (make one data line)
if y.ndim == 1:
t = t.reshape(1, t.size)
y = y.reshape(1, y.size)
#Calculate the error and normalize by the number of batches
batch_size = y.shape[0]
return -np.sum(t * np.log(y + 1e-7)) / batch_size
And then there is the neural network class. Since it is based on the code in the previous chapter, there are many of the same parts.
two_layer_net.py
# coding: utf-8
import numpy as np
from affine import Affine
from functions import numerical_gradient
from relu import ReLU
from softmax_with_loss import SoftmaxWithLoss
class TwoLayerNet:
def __init__(self, input_size, hidden_size, output_size,
weight_init_std=0.01):
"""Two-layer neural network
Args:
input_size (int):Number of neurons in the input layer
hidden_size (int):Number of neurons in the hidden layer
output_size (int):Number of neurons in the output layer
weight_init_std (float, optional):Adjustment parameter of the initial value of the weight. The default is 0.01。
"""
#Weight initialization
self.params = {}
self.params['W1'] = weight_init_std * \
np.random.randn(input_size, hidden_size)
self.params['b1'] = np.zeros(hidden_size)
self.params['W2'] = weight_init_std * \
np.random.randn(hidden_size, output_size)
self.params['b2'] = np.zeros(output_size)
#Layer generation
self.layers = {} # Python 3.OrderedDict is unnecessary because the storage order of dictionaries is retained from 7
self.layers['Affine1'] = \
Affine(self.params['W1'], self.params['b1'])
self.layers['Relu1'] = ReLU()
self.layers['Affine2'] = \
Affine(self.params['W2'], self.params['b2'])
self.lastLayer = SoftmaxWithLoss()
def predict(self, x):
"""Inference by neural network
Args:
x (numpy.ndarray):Input to neural network
Returns:
numpy.ndarray:Neural network output
"""
#Propagate layers forward
for layer in self.layers.values():
x = layer.forward(x)
return x
def loss(self, x, t):
"""Loss function value calculation
Args:
x (numpy.ndarray):Input to neural network
t (numpy.ndarray):Correct label
Returns:
float:Loss function value
"""
#inference
y = self.predict(x)
# Softmax-with-Calculated by forward propagation of Loss layer
loss = self.lastLayer.forward(y, t)
return loss
def accuracy(self, x, t):
"""Recognition accuracy calculation
Args:
x (numpy.ndarray):Input to neural network
t (numpy.ndarray):Correct label
Returns:
float:Recognition accuracy
"""
y = self.predict(x)
y = np.argmax(y, axis=1)
t = np.argmax(t, axis=1)
accuracy = np.sum(y == t) / x.shape[0]
return accuracy
def numerical_gradient(self, x, t):
"""Calculate the gradient for the weight parameter by numerical differentiation
Args:
x (numpy.ndarray):Input to neural network
t (numpy.ndarray):Correct label
Returns:
dictionary:A dictionary containing gradients
"""
grads = {}
grads['W1'] = \
numerical_gradient(lambda: self.loss(x, t), self.params['W1'])
grads['b1'] = \
numerical_gradient(lambda: self.loss(x, t), self.params['b1'])
grads['W2'] = \
numerical_gradient(lambda: self.loss(x, t), self.params['W2'])
grads['b2'] = \
numerical_gradient(lambda: self.loss(x, t), self.params['b2'])
return grads
def gradient(self, x, t):
"""Gradient for weight parameters calculated by error backpropagation
Args:
x (numpy.ndarray):Input to neural network
t (numpy.ndarray):Correct label
Returns:
dictionary:A dictionary containing gradients
"""
#Forward propagation
self.loss(x, t) #Propagate forward to calculate loss value
#Backpropagation
dout = self.lastLayer.backward()
for layer in reversed(list(self.layers.values())):
dout = layer.backward(dout)
#Extract the differential value of each layer
grads = {}
grads['W1'] = self.layers['Affine1'].dW
grads['b1'] = self.layers['Affine1'].db
grads['W2'] = self.layers['Affine2'].dW
grads['b2'] = self.layers['Affine2'].db
return grads
The code in the book uses ʻOrderedDict, but here we use the usual dict. This is because starting with Python 3.7, the insertion order of dict` objects is saved [^ 1].
5.7.3 Gradient confirmation of error back propagation method
This code compares the gradient obtained by the error back propagation method with the gradient obtained by numerical differentiation.
gradient_check.py
# coding: utf-8
import os
import sys
import numpy as np
from two_layer_net import TwoLayerNet
sys.path.append(os.pardir) #Add parent directory to path
from dataset.mnist import load_mnist
#Read MNIST training data and test data
(x_train, t_train), (x_test, t_test) = \
load_mnist(normalize=True, one_hot_label=True)
#Two-layer neural work generation
network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)
#Data preparation for verification
x_batch = x_train[:3]
t_batch = t_train[:3]
#Gradient calculation by numerical differentiation and error back propagation method
grad_numerical = network.numerical_gradient(x_batch, t_batch)
grad_backprop = network.gradient(x_batch, t_batch)
#Check the difference between each weight
for key in grad_numerical.keys():
#Calculation of absolute value of difference
diff = np.abs(grad_backprop[key] - grad_numerical[key])
#Show average and maximum
print(f"{key}: [Average difference]{np.average(diff):.10f} [Maximum difference]{np.max(diff):.10f}")
In the book, I checked only the average of the differences in absolute values, but I also checked the maximum value of the differences in absolute values.
W1: [Average difference]0.0000000003 [Maximum difference]0.0000000080
b1: [Average difference]0.0000000021 [Maximum difference]0.0000000081
W2: [Average difference]0.0000000063 [Maximum difference]0.0000000836
b2: [Average difference]0.0000001394 [Maximum difference]0.0000002334
Since b2 is a value with about 7 digits after the decimal point, it seems that the error is a little larger than the book. There may be a bad point in the implementation. If you have any questions, please let me know: sweat:
5.7.4 Learning using the error back propagation method
Below is the learning code.
mnist.py
# coding: utf-8
import os
import sys
import matplotlib.pylab as plt
import numpy as np
from two_layer_net import TwoLayerNet
sys.path.append(os.pardir) #Add parent directory to path
from dataset.mnist import load_mnist
#Read MNIST training data and test data
(x_train, t_train), (x_test, t_test) = \
load_mnist(normalize=True, one_hot_label=True)
#Hyperparameter settings
iters_num = 10000 #Number of updates
batch_size = 100 #Batch size
learning_rate = 0.1 #Learning rate
#Record list of results
train_loss_list = [] #Changes in the value of the loss function
train_acc_list = [] #Recognition accuracy for training data
test_acc_list = [] #Recognition accuracy for test data
train_size = x_train.shape[0] #Training data size
iter_per_epoch = max(int(train_size / batch_size), 1) #Number of iterations per epoch
#Two-layer neural work generation
network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)
#Start learning
for i in range(iters_num):
#Mini batch generation
batch_mask = np.random.choice(train_size, batch_size, replace=False)
x_batch = x_train[batch_mask]
t_batch = t_train[batch_mask]
#Gradient calculation
grad = network.gradient(x_batch, t_batch)
#Weight parameter update
for key in ('W1', 'b1', 'W2', 'b2'):
network.params[key] -= learning_rate * grad[key]
#Loss function value calculation
loss = network.loss(x_batch, t_batch)
train_loss_list.append(loss)
#Recognition accuracy calculation for each epoch
if i % iter_per_epoch == 0:
train_acc = network.accuracy(x_train, t_train)
test_acc = network.accuracy(x_test, t_test)
train_acc_list.append(train_acc)
test_acc_list.append(test_acc)
#Progress display
print(f'[Number of updates]{i:>4} [Loss function value]{loss:.4f} '
f'[Training data recognition accuracy]{train_acc:.4f} [Test data recognition accuracy]{test_acc:.4f}')
#Draw the transition of the value of the loss function
x = np.arange(len(train_loss_list))
plt.plot(x, train_loss_list, label='loss')
plt.xlabel('iteration')
plt.ylabel('loss')
plt.xlim(left=0)
plt.ylim(bottom=0)
plt.show()
#Draw the transition of recognition accuracy of training data and test data
x2 = np.arange(len(train_acc_list))
plt.plot(x2, train_acc_list, label='train acc')
plt.plot(x2, test_acc_list, label='test acc', linestyle='--')
plt.xlabel('epochs')
plt.ylabel('accuracy')
plt.xlim(left=0)
plt.ylim(0, 1.0)
plt.legend(loc='lower right')
plt.show()
And it is the execution result of the same form as the previous chapter.
[Number of updates] 0 [Loss function value]2.3008 [Training data recognition accuracy]0.0926 [Test data recognition accuracy]0.0822
[Number of updates] 600 [Loss function value]0.2575 [Training data recognition accuracy]0.9011 [Test data recognition accuracy]0.9068
[Number of updates]1200 [Loss function value]0.2926 [Training data recognition accuracy]0.9219 [Test data recognition accuracy]0.9242
[Number of updates]1800 [Loss function value]0.2627 [Training data recognition accuracy]0.9324 [Test data recognition accuracy]0.9341
[Number of updates]2400 [Loss function value]0.0899 [Training data recognition accuracy]0.9393 [Test data recognition accuracy]0.9402
[Number of updates]3000 [Loss function value]0.1096 [Training data recognition accuracy]0.9500 [Test data recognition accuracy]0.9483
[Number of updates]3600 [Loss function value]0.1359 [Training data recognition accuracy]0.9559 [Test data recognition accuracy]0.9552
[Number of updates]4200 [Loss function value]0.1037 [Training data recognition accuracy]0.9592 [Test data recognition accuracy]0.9579
[Number of updates]4800 [Loss function value]0.1065 [Training data recognition accuracy]0.9639 [Test data recognition accuracy]0.9600
[Number of updates]5400 [Loss function value]0.0419 [Training data recognition accuracy]0.9665 [Test data recognition accuracy]0.9633
[Number of updates]6000 [Loss function value]0.0393 [Training data recognition accuracy]0.9698 [Test data recognition accuracy]0.9649
[Number of updates]6600 [Loss function value]0.0575 [Training data recognition accuracy]0.9718 [Test data recognition accuracy]0.9663
[Number of updates]7200 [Loss function value]0.0850 [Training data recognition accuracy]0.9728 [Test data recognition accuracy]0.9677
[Number of updates]7800 [Loss function value]0.0403 [Training data recognition accuracy]0.9749 [Test data recognition accuracy]0.9686
[Number of updates]8400 [Loss function value]0.0430 [Training data recognition accuracy]0.9761 [Test data recognition accuracy]0.9685
[Number of updates]9000 [Loss function value]0.0513 [Training data recognition accuracy]0.9782 [Test data recognition accuracy]0.9715
[Number of updates]9600 [Loss function value]0.0584 [Training data recognition accuracy]0.9777 [Test data recognition accuracy]0.9707
Compared to the results in the previous chapter, the recognition accuracy increases faster. In the end, it was about 97%. The only difference between the numerical differentiation and the error back propagation method should be the gradient calculation method, so it seems that the change from the sigmoid function to the ReLU function has led to improvement.
5.8 Summary
The calculation graph should be easy to understand. It was also well understood that the output layer and loss function were designed so that the differential value could be easily obtained.
That's all for this chapter. If you have any mistakes, I would be grateful if you could point them out. (To other chapters of this memo: Chapter 1 / Chapter 2 / Chapter 3 / Chapter 4 / Chapter 5 / [Chapter 6](https: / /qiita.com/segavvy/items/ca4ac4c9ee1a126bff41) / Chapter 7 / Chapter 8 / Summary)
[^ 1]: See "Improvement of Python's Data Model" in What's New In Python 3.7.
Recommended Posts