** << Caution >> ** ** Because the heights are forcibly aligned, "$ \ frac {} {} $" is mixed in some places. Please let me know if there is any good way. Also, there are other ways to add color to the formula ... **

It will be. If the error $ E $ from the upstream is passed through this, the back propagation will be as shown in the figure. After all, there are many mathematical formulas and it looks complicated, that is, it only multiplies the partial derivative when passing through the ** compute node. ** ** Even this is a scalar, so it may be easier.

Backpropagation implementation in scalar

Let's implement it. Let's code according to the formula. The implementation destination code is [here](https://qiita.com/kuroitu/items/884c62c48c2daa3def08#layer module code preparation).

baselayer.py

    def backward(self, grad):
        """
Implementation of backpropagation
        """
        dact = grad*self.act.backward(self.x, self.y)
        self.dw = dact*self.x
        self.db = dact
        self.dx = dact*self.w

        return self.dx

The middle layer is OK as it is, but it is necessary to add some processing for the output layer. I will touch on it later.

Backpropagation in a matrix

Next, let's consider backpropagation in a matrix. This is a lot of formulas and difficult, but you can see it if you follow it properly. As before, I'll add color to the formula so that you can see it at a glance. ~~ I feel heavy ... ~~

Backpropagation theory in matrix

Consider the following calculation graph. It's the same as the one used in [Forward propagation](https://qiita.com/kuroitu/items/d22c8750e34d5d75fb6c# Forward propagation in matrix). It is a layer model with only 2 neurons, but as you can see, there are quite a lot of formulas. First, let's find the partial differential of each part as in the case of scalar.

\Rightarrow
\boldsymbol{dW} =
\left(
  \begin{array}{cc}
    \cfrac{\partial y_1}{\partial w_{1, 1}} & \cfrac{\partial y_2}{\partial w_{1, 2}} \\
    \cfrac{\partial y_1}{\partial w_{2, 1}} & \cfrac{\partial y_2}{\partial w_{2, 2}}
  \end{array}
\right)
=
\left(
  \begin{array}{c}
    x_1 \\
    x_2
  \end{array}
\right)
\left(
  \begin{array}{cc}
    \sigma'_1(m) & \sigma'_2(q)
  \end{array}
\right)
=
\boldsymbol{X}^{\top}\boldsymbol{\sigma'}

\Rightarrow
\boldsymbol{dB} =
\left(
  \begin{array}{cc}
    \cfrac{\partial y_1}{\partial b_1} & \cfrac{\partial y_2}{\partial b_2}
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
     \sigma'_1(m) & \sigma'_2(q)
  \end{array}
\right)

\Rightarrow
\boldsymbol{dX} =
\left(
  \begin{array}{cc}
    \cfrac{\partial y_1}{\partial x_1} + \cfrac{\partial y_2}{\partial x_1} & \cfrac{\partial y_1}{\partial x_2} + \cfrac{\partial y_2}{\partial x_2}
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    \sigma'_1(m) & \sigma'_2(q)
  \end{array}
\right)
\left(
  \begin{array}{cc}
    w_{1, 1} & w_{2, 1} \\
    w_{1, 2} & w_{2, 2}
  \end{array}
\right)
=
\boldsymbol{\sigma'}\boldsymbol{W}^{\top}

It looks like. Add the error from the upstream to these [element product](https://qiita.com/kuroitu/items/d22c8750e34d5d75fb6c#%E8%A1%8C%E5%88%97%E3%81%AE%E8%A6%81 By multiplying by% E7% B4% A0% E7% A9% 8D), the error as shown in the above figure is propagated. As you can see from the fact that $ x_1 $ and $ x_2 $ are branched and the values are sent to other calculation nodes, the partial differentials that have flowed need to be added and calculated. So </ font>

\left(
  \begin{array}{cc}
    \cfrac{\partial y_1}{\partial x_1} + \cfrac{\partial y_2}{\partial x_1} & \cfrac{\partial y_1}{\partial x_2} + \cfrac{\partial y_2}{\partial x_2}
  \end{array}
\right)

\begin{align}
  \underbrace{\boldsymbol{dW}}_{L \times M} &= \underbrace{\boldsymbol{X}^{\top}}_{L \times N} \underbrace{\boldsymbol{\sigma'}}_{N \times M} \\
  \underbrace{\boldsymbol{dB}}_{1 \times M} &= \underbrace{\boldsymbol{\sigma'}}_{N\times M} \\
  \underbrace{\boldsymbol{dX}}_{N \times L} &= \underbrace{\boldsymbol{\sigma'}}_{N \times M} \underbrace{\boldsymbol{W}^{\top}}_{M \times L}
\end{align}

\underbrace{\boldsymbol{dB}}_{1 \times M} = \sum_{i=1}^{N}{\underbrace{\boldsymbol{\sigma'}_i}_{1\times M}} \xrightarrow{\textrm{coding}} \textrm{sum}(\boldsymbol{\sigma'}, \textrm{axis}=0)

    def backward(self, grad):
        """
Implementation of backpropagation
        """
        dact = grad*self.act.backward(self.x, self.y)
        self.dw = self.x.T@dact
        self.db = np.sum(dact, axis=0)
        self.dx = [email protected]

        return self.dx

    def backward(self, t):
        """
Implementation of backpropagation
        """
        #When the activation function of the output layer is the softmax function and the loss function is the cross entropy error
        #Separate cases of error propagation
        if isinstance(self.act, Softmax) \
        and isinstance(self.errfunc, CrossEmtropy):
            dact = self.y - t
            self.dw = self.x.T@dact
            self.db = np.sum(dact, axis=0)
            self.dx = [email protected]
            
            return self.dx
        else:
            grad = self.errfunc.backward(self.y, t)
            return super().__init__(grad)

    def __init__(self, *, prev=1, n=1, 
                 name="", wb_width=1,
                 act="ReLU", err_func="square",
                 **kwds):
        self.prev = prev  #Number of outputs of the previous layer=Number of inputs to this layer
        self.n = n        #Number of outputs in this layer=Number of inputs to the next layer
        self.name = name  #The name of this layer

        #Set weight and bias
        self.w = wb_width*np.random.randn(prev, n)
        self.b = wb_width*np.random.randn(n)

        #Activation function(class)Get
        self.act = get_act(act)

        #Loss function(class)Get
        self.errfunc = get_errfunc(err_func)