● Mean squared error 　　　 [Formula]

　　　E_n(W) = \frac{1}{2} \sum^I_{i=1}(y_n - d_n)^2

python

    def  mean_squared_error(d, y):
        return np.mean(np.square(d  - y)) / 2

☆ Confirmation test ☆ [Cross entropy (mathematical formula)]

　　　E_n(W) = -\sum^I_{i=1} d_i logy_i

Show the source code corresponding to the formulas ① to ②, Explain the process line by line.

[My answer] 　　①return -np.sum(np.log(y[np.arange(batch_size), d] + 1e-7)) / batch_size 　　②return -np.sum(np.log(y[np.arange(batch_size), d] + 1e-7)) / batch_size ⇒Because the log value may become 0 A small value 1e-7 is added so that the numerator does not become 0.

python

def  cross_entropy_error(d, y):
    if y.ndim== 1:
        d= d.reshape(1, d.size)
        y= y.reshape(1, y.size)

    #Teacher data is one-hot-In case of vector, convert to index of correct label

    if d.size== y.size:
        d= d.argmax(axis=1)
        batch_size= y.shape[0]

    return -np.sum(np.log(y[np.arange(batch_size), d] + 1e-7)) / batch_size

5-11.Section 4 "Gradient descent method"

● Gradient descent method ・ Purpose of deep learning Creating a network that minimizes errors through learning. ⇒ Discover the parameter $ w $ that minimizes the error $ E (w) $. ⇒ ** Use gradient descent method ** to optimize parameters

[Gradient descent method]

w^{(t+1)}=w^{(t)} -  ε∇E\\
∇E = \frac{\partial E}{\partial w} = \biggl \{\frac{\partial E}{\partial w_1}･ ･ ･\frac{\partial E}{\partial w_M} \biggr \}

☆ Confirmation test ☆ Let's find the corresponding source code.

[My answer] Parameters ... network [key]-= learning_rate * grad [key] $ ∇E $ ... grad = backward (x, d, z1, y)

【answer】 Parameters ... network [key]-= learning_rate * grad [key] $ ∇E $ ... grad = backward (x, d, z1, y)

• $ ∇ E $ is the error differentiated by the parameter.

[$ Ε $: Learning rate] ・ Large learning rate: It cannot converge well and diverges. ・ Small learning rate: It does not diverge, but it takes time to converge. ・ Global minimum solution ... The point that becomes the minimum as a whole

● Gradient descent algorithm About the algorithm for determining the learning rate and improving the convergence Several papers have been published and are often used. 　 ・ Momentum ・ AdaGrad ・ Adadelta ・ Adam

● Stochastic Gradient Descent (SGD) ・ Advantages of stochastic gradient descent ・ Reduction of calculation cost when data is redundant -Reducing the risk of converging on unwanted local minimal solutions ・ You can study online [Stochastic gradient descent]

w^{(t+1)}=w^{(t)} -  ε∇E_n

[Gradient descent method]

w^{(t+1)}=w^{(t)} -  ε∇E

☆ Confirmation test ☆ What is online learning? Summarize in two lines.

[My answer] Even during real-time prediction processing Weights and biases can be updated from the error.

【answer】 For example, on Facebook, etc. You can always learn using only the information of newly registered users.

● Mini batch gradient descent method -A set of randomly divided data (mini-batch) $ D_t $ Average error of samples belonging to ・ Without losing the merit of stochastic gradient descent The computing resources of the computer can be used effectively. ⇒ Thread parallelization using CPU and SIMD parallelization using GPU

[Mini batch gradient descent method]

w^{(t+1)}=w^{(t)} -  ε∇E_t\\
E_t = \frac{1}{N_t}\sum_{n \in{D_t}}E_n\\
N_t = |D_t|

[Stochastic gradient descent]

w^{(t+1)}=w^{(t)} -  ε∇E_n

☆ Confirmation test ☆

[Mini batch gradient descent method] w^{(t+1)}=w^{(t)} -  ε∇E_t

Explain the meaning of this formula in a diagram.

[My answer] Since it was a set of randomly divided data (mini-batch) $ D_t $, Group the appropriate samples and calculate for each Finally, the overall error is calculated. (I couldn't show it ...)

【answer】

● Calculation of error gradient How to calculate

∇E = \frac{\partial E}{\partial w} = \biggl \{\frac{\partial E}{\partial w_1}･ ･ ･\frac{\partial E}{\partial w_M} \biggr \}

[Numerical differentiation] A general method of generating minute numbers in a program and calculating the derivative in a pseudo manner.

\frac{\partial E}{\partial w_m} \approx \frac{E(w_m + h)- E(w_m - h)}{2h}

However, there are major disadvantages. ・ To calculate $ E (w_m + h) $ and $ E (w_m-h) $ for each parameter $ w_m $ It is necessary to repeat the calculation of forward propagation, which increases the load.

⇒ ** Use the error back propagation method **

● Deep learning development environment ・ Local: CPU, GPU ・ Cloud: AWS, GCP

• It may not be possible to use it depending on the version of the server environment. Care must be taken regarding the combination.

5-12.Section 5 "Error back propagation method"

● Calculation of error gradient How to calculate

∇E = \frac{\partial E}{\partial w} = \biggl \{\frac{\partial E}{\partial w_1}･ ･ ･\frac{\partial E}{\partial w_M} \biggr \}

[Error back propagation method] The calculated error is differentiated in order from the output layer side and propagated to the previous layer and the previous layer. A method of calculating the differential value of each parameter ** analytically ** with a minimum of calculation.

By back-calculating the derivative from the calculation result (= error), avoiding the necessary recursive calculation, The derivative can be calculated. ⇒Use the chain rule.

☆ Confirmation test ☆ In the error back propagation method, unnecessary recursive processing can be avoided. Extract the source code that holds the calculation results that have already been performed.

[My answer]

python

#Error back propagation
def backward(x, d, z1, y):
    print("\n#####Error back propagation start#####")

    grad = {}

    W1, W2 = network['W1'], network['W2']
    b1, b2 = network['b1'], network['b2']
    #Delta at the output layer
    delta2 = functions.d_sigmoid_with_loss(d, y)
    #Gradient of b2
    grad['b2'] = np.sum(delta2, axis=0)
    #Gradient of W2
    grad['W2'] = np.dot(z1.T, delta2)
    #Delta in the middle layer
    delta1 = np.dot(delta2, W2.T) * functions.d_relu(z1)
    #Gradient of b1
    grad['b1'] = np.sum(delta1, axis=0)
    #Gradient of W1
    grad['W1'] = np.dot(x.T, delta1)
        
    print_vec("Partial differential_dE/du2", delta2)
    print_vec("Partial differential_dE/du2", delta1)

    print_vec("Partial differential_Weight 1", grad["W1"])
    print_vec("Partial differential_Weight 2", grad["W2"])
    print_vec("Partial differential_Bias 1", grad["b1"])
    print_vec("Partial differential_Bias 2", grad["b2"])

    return grad

【answer】 Is it the same ...?

● Error back propagation method

\begin{align}
&E(y) = \frac{1}{2}\sum^J_{j=1}(y_j - d_j)^2 = \frac{1}{2}||y - d||^2　
⇒ Error function=Let it be a square error function.\\
&y = u^{(L)}⇒ Output layer activation function=Make it an identity map.\\
&u^{(l)} = w^{(l)}z^{(l-1)} + b^{(l)}⇒ Calculation of total input\\\\
&\frac{\partial E}{\partial w^{(2)}_{ji}}=\frac{\partial E}{\partial y}\frac{\partial y}{\partial u}\frac{\partial u}{\partial w^{(2)}_{ji}}\\\\
&\frac{\partial E(y)}{\partial y} = \frac{\partial}{\partial y}\frac{1}{2}||y - d||^2 = y - d\\\\
&\frac{\partial y(u)}{\partial u} =  \frac{\partial u}{\partial u} = 1\\
\end{align}

\frac{\partial u(w)}{\partial w_{ji}} = \frac{\partial}{\partial w_{ji}}(w^{(l)}z^{(l-1)}+b^{(l)})=\frac{\partial}{\partial w_{ji}} \left(
\begin{bmatrix} w_{11}z_1 +･ ･ ･+ w_{1i}z_i +･ ･ ･w_{1I}z_I \\
\vdots\\
w_{j1}z_1 +･ ･ ･+ w_{ji}z_i +･ ･ ･w_{jI}z_I\\
\vdots\\
w_{J1}z_1 +･ ･ ･+ w_{Ji}z_i +･ ･ ･w_{JI}z_I \end{bmatrix}
+\begin{bmatrix} b_1\\
\vdots\\
b_j\\
\vdots\\
b_J \end{bmatrix}

=\begin{bmatrix} 0\\
\vdots\\
z_i\\
\vdots\\
0 \end{bmatrix}
\right)

5-13. Explanation in Jupyter (1_3_stochastic_gradient_descent)

● Try to practice using the source code. (1) Set the activation function to the ReLU function.

(2) Set the activation function to the sigmoid function x ReLU function.

(3) Set the activation function to sigmoid function x sigmoid function.

④ Set the input value to a random value from -5 to 5.

☆ Confirmation test ☆ Find the source code that corresponds to the two spaces

\frac{\partial E}{\partial y}

⇒delta2 = functions.d_mean_squared_error(d, y)

\frac{\partial E}{\partial y}\frac{\partial y}{\partial u}

⇒？

\frac{\partial E}{\partial y}\frac{\partial y}{\partial u}\frac{\partial u}{\partial w^{(2)}_{ji}}

⇒？

[My answer] I don't understand a little. It's difficult just to watch the video normally.

【answer】 ・ Since it is an identity map, y = u. 　　　　\frac{\partial E}{\partial y}\frac{\partial y}{\partial u} = \frac{\partial E}{\partial y}\frac{\partial u}{\partial u} So delta2 = functions.d_mean_squared_error (d, y)

・ Fetch the inner product value. The inner product value of the value passed to delta2 and the transposed version of the activation function of the intermediate layer. 　　　　grad ['W2'] = np.dot(z1.T, delta2)

5-14. Thesis commentary

Obtain the knowledge necessary for design from the dissertation for implementation. Check the graphs of the experimental results as soon as possible and understand them visually.