[PYTHON] [Deep Learning from scratch] I tried to implement sigmoid layer and Relu layer.

Introduction

This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 6 Error back propagation method **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.

Implementation of sigmoid layer

class Sigmoid: #Layer implementation of sigmoid function
    def __init__(self):
        self.out = None #The output signal of forward propagation processing is required for the back propagation processing of the sigmoid function.
        
    def forward(self, x):
        out = 1 / (1 + np.log(-x))
        self.out = out #Forward propagation output storage
        
        return out
    
    def backward(self, dout):
        dx = dout * self.out * (1 - self.out) #Processing to find the back propagation of the sigmoid function
        
        return dx

In the forward propagation process, the process is performed according to the formula of the sigmoid function, and it is returned by return. However, since the result of the forward propagation process is used in the back propagation process, it is saved in the instance variable.

In the back propagation process, it would be quite complicated if it is normal, so here we use a simplified formula to find the derivative. Find the derivative of the input value of the sigmoid layer using the previous derivative and the result of the forward propagation process saved in the instance variable.

Relu layer implementation

class Relu: #Layer implementation of Relu function
    def __init__(self):
        self.mask = None #An array of False is entered if the input signal below 0 is True and greater than 0.
        
    def forward(self, x):
        self.mask = (x <= 0)
        out = x.copy()
        out[self.mask] = 0 #Set the input signal below 0 to 0
        
        return out
    
    def backward(self, dout):
        dout[self.mask] = 0 #If the forward propagation is 0, the back propagation is also 0, so set it to 0 here.
        dx = dout #Others inherit the previous derivative
        
        return dx

In the forward propagation process, in order to implement without using if, first, all the input values that do not exceed 0 are set to True, and those that exceed 0 are set to False, and that is saved in the instance variable. This will be used later.

Next, copy the array containing the input values, and use the True and False arrays saved earlier to convert all True ones to 0. And return it with return.

In the back propagation process, all True is converted to 0 using the True and False arrays saved in the forward propagation process. If the forward propagation is 0, the reverse propagation is also 0.

After that, it inherits the previous derivative and returns it with return.