[PYTHON] Where are matrix products and inner products used in deep learning?

About matrix product and inner product

When I was studying mathematics, the word "inner product" came up. I investigated how this "inner product" is used in neural networks. The inner product is the product of vectors, but the matrix product is the product of matrices. The inner product seems to be called "dot product".

What is inner product?

-Meaning of inner product -[Mathematics] If you understand the meaning of "inner product" graphically, you can see various things. 1 -[Dot product](https://ja.wikipedia.org/wiki/dot product)

What is matrix product?

-Definition of matrix product and its reason

Example

Consider the following two-layer neural network.

Matrix product formula

If you replace the output of the above Nural network with the matrix product formula, it looks like this.

\begin{bmatrix}
w_{11} & w_{12} & w_{13}\\
w_{21} & w_{22} & w_{23}
\end{bmatrix}

\cdot 

\begin{bmatrix}
x_{1} \\
x_{2} \\
x_{3} \\
\end{bmatrix}

Manual calculation

Apply the numbers specifically

\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{bmatrix}

\cdot 

\begin{bmatrix}
1 \\
2 \\
3 \\
\end{bmatrix}

If you calculate by hand

\begin{bmatrix}
1 \times 1 + 2 \times 2 + 3 \times 3  \\
4 \times 1 + 5 \times 2 + 6 \times 3  \\
\end{bmatrix}

=

\begin{bmatrix}
14  \\
32  \\
\end{bmatrix}
 

In Numpy

import numpy as np
w = np.array([[1,2,3],[4,5,6]])
x = np.array([[1],[2],[3]])

w.dot(x) #=> array([[14],[32]])

w.dot (x) represents the matrix product of w and x. By the way, the method name is changed from "dot product" to dot.

In Tensorflow

import tensorflow as tf
w = tf.get_variable("weight" , initializer=tf.constant(np.array([[1,2,3],[4,5,6]]).astype("int32")))
x = tf.get_variable("x" , initializer=tf.constant(np.array([[1],[2],[3]]).astype("int32")))

Session initialization

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

sess.run(tf.matmul(w, x))  #=> array([[14],[32]])

tf.matmul (w, x) represents the matrix product of w and x. By the way, the method name is matmul from" matrix multiplication ".