[PYTHON] [Explanation for beginners] TensorFlow tutorial Deep MNIST

A tutorial for TensorFlow experts is explained from an introductory perspective

I ran the tutorial for TensorFlow experts Deep MNIST for Experts. I will explain from the perspective of beginners. There are many parts that lack accuracy, but I give priority to clarity. It was very difficult compared to the tutorial for beginners: skull: ^{* Added a code so that image can be output, and posted an image of convolution / pooling processing (2017/7/27) </ sup>}

Reference link

-Installing TensorFlow on Windows was easy even for Python beginners -[Explanation for beginners] TensorFlow basic syntax and concept -[Explanation for beginners] TensorFlow tutorial MNIST (for beginners) -TensorFlow API memo -[Introduction to TensorBoard] Visualize TensorFlow processing to deepen understanding -[Introduction to TensorBoard: image] TensorFlow Visualize image processing to deepen understanding -[Introduction to TensorBoard: Projector] Make TensorFlow processing look cool -Visualize TensorFlow tutorial MNIST (for beginners) with TensorBoard -Yuki Kashiwagi's facial features to understand TensorFlow [Part 1]

Outline of processing

The other day, the photo was impressive, though the content was also impressive "I haven't eaten for nearly 80 years and amazed Indian ascetic doctors" I will explain it sensuously without using mathematical formulas, using the photo in the above as an example (I used it because the photo was very easy to understand). The TensorFlow expert tutorial Deep MNIST for Experts is primarily ** convolution **, ** pooling **, ** tightly coupled **. It consists of three processes. How to judge the image by each process is as shown in the figure below.

A little detailed processing outline

The following two Youtube videos were very easy to understand, so I will post a link (in English).

• A friendly introduction to Convolutional Neural Networks and Image Recognition
• How Convolutional Neural Networks work

In addition, as an explanation blog in Japanese, "Mathematical background of TensorFlow Tutorial-Deep MNIST for Experts (Part 1)" I understood while looking at it. Wonderful.

processing

The figure below shows the entire process using a Tensor. This time, you can see that the two-layer (multi-layer) convolution and pooling process is performed. You can see from the amount of Tensor that "find features" using a filter in convolution and "roughly organize features" in pooling.

It looks like this when the actual transition up to the second layer is represented by an image.

The explanation of convolution processing is described in detail in the article "[Explanation for beginners] Introduction to convolution processing (explained in TensorFlow)". The explanation of pooling process is described in detail in the article "[Explanation for beginners] Introduction to pooling process (explained in TensorFlow)". The explanation of image output was written in detail in the article "[Introduction to TensorBoard: image] TensorFlow Visualize image processing to deepen understanding".

Graph of TensorBoard is as shown in the figure below. Processing starts from the input below. The exact tightly coupled layer is divided into fc1_layer and fc2_layer.

code

Finally, put the code output to TensorBoard. When executed on a 2-core PC, the CPU was almost used up immediately as shown in the figure below, and other processing could not be performed on the PC during execution. Please note that it will take some time to complete the process. I'm a little unsure about the accuracy. I would be grateful if you could point out any mistakes.

import tensorflow as tf
sess = tf.InteractiveSession()
#Read MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

#TensorBoard information output directory
log_dir = '/tmp/tensorflow/mnist/logs/mnist_expert_images'

#Delete the specified directory if it exists and recreate it
if tf.gfile.Exists(log_dir):
    tf.gfile.DeleteRecursively(log_dir)
tf.gfile.MakeDirs(log_dir)

# Input placeholders &Image conversion
with tf.name_scope('input'):
    x  = tf.placeholder(tf.float32, shape=[None, 784], name='x-input')
    y_ = tf.placeholder(tf.float32, shape=[None, 10],  name='y-input')

    #Convert image to 28x28
    x_image = tf.reshape(x, [-1, 28, 28, 1])
    tf.summary.image('preprocess', x_image, 10)

#Function that processes variables and outputs TensorBoard
def variable_summaries(var):

    #Variable Summary
    with tf.name_scope('summaries'):
        mean = tf.reduce_mean(var)                      #Scalar output(average)
        tf.summary.scalar('mean', mean)
        with tf.name_scope('stddev'):
            stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))
        tf.summary.scalar('stddev', stddev)             #Scalar output(standard deviation)
        tf.summary.scalar('max', tf.reduce_max(var))    #Scalar output(Maximum value)
        tf.summary.scalar('min', tf.reduce_min(var))    #Scalar output(minimum value)
        tf.summary.histogram('histogram', var)          #Histogram output

#Weighted value
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)    #Standard deviation 0.Normal distribution random number of 1
    return tf.Variable(initial)

#Bias value
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)             #Initial value 0.1 constant
    return tf.Variable(initial)

#Convolution process
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

# Max Pooling
def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

#1st layer
with tf.name_scope('1st_layer'):

    #1st convolution layer
    with tf.name_scope('conv1_layer'):
        with tf.name_scope('weight'):
            W_conv1 = weight_variable([5, 5, 1, 32])
            variable_summaries(W_conv1)
            
            #Tensor[5,5,1,32]From[32,5,5,1]Permutation conversion and image output
            tf.summary.image('filter', tf.transpose(W_conv1,perm=[3,0,1,2]), 10)

        with tf.name_scope('bias'):            
            b_conv1 = bias_variable([32])
            variable_summaries(b_conv1)
        with tf.name_scope('activations'):            
            h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
            tf.summary.histogram('activations', h_conv1)

            #Tensor[-1,28,28,32]From[-1,32,28,28]And permutation,[-1]When[-32]Merge and output image
            tf.summary.image('convolved', tf.reshape(tf.transpose(h_conv1,perm=[0,3,1,2]),[-1,28,28,1]), 10)

#1st pooling layer
    with tf.name_scope('pool1_layer'):    
        h_pool1 = max_pool_2x2(h_conv1)

        #Tensor[-1,14,14,32]From[-1,32,14,14]And permutation,[-1]When[32]Merge and output image
        tf.summary.image('pooled', tf.reshape(tf.transpose(h_pool1,perm=[0,3,1,2]),[-1,14,14,1]), 10)

#2nd layer
with tf.name_scope('2nd_layer'):
    
    #2nd convolution layer
    with tf.name_scope('conv2_layer'):
        with tf.name_scope('weight'):
            W_conv2 = weight_variable([5, 5, 32, 64])
            variable_summaries(W_conv2)

            #Tensor[5,5,32,64]From[32*64,5,5,1]Permutation conversion and image output
            tf.summary.image('filter', tf.reshape(tf.transpose(W_conv2,perm=[2,3,0,1]),[-1,5,5,1]), 20)
        with tf.name_scope('bias'):            
            b_conv2 = bias_variable([64])
            variable_summaries(b_conv2)
        with tf.name_scope('activations'):
            h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
            tf.summary.histogram('activations', h_conv2)
            #Tensor[-1,14,14,64]From[-1,64,14,14]And permutation,[-1]When[64]Merge and output image
            tf.summary.image('convolved', tf.reshape(tf.transpose(h_conv2,perm=[0,3,1,2]),[-1,14,14,1]), 10)
    
#2nd pooling layer
    with tf.name_scope('pool2_layer'):
        h_pool2 = max_pool_2x2(h_conv2)
        
        #Tensor[-1,7,7,64]From[-1,64,7,7]And permutation,[-1]When[64]Merge and output image
        tf.summary.image('pooled', tf.reshape(tf.transpose(h_pool2,perm=[0,3,1,2]),[-1,7,7,1]), 10)

#Tightly coupled layer
with tf.name_scope('fc1_layer'):
    h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
    with tf.name_scope('weight'):
        W_fc1 = weight_variable([7 * 7 * 64, 1024])
        variable_summaries(W_fc1)
    with tf.name_scope('bias'):            
        b_fc1 = bias_variable([1024])
        variable_summaries(b_fc1)
    with tf.name_scope('activations'):
        h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
        tf.summary.histogram('activations', h_fc1)

    #Drop out
    with tf.name_scope('dropout'):
        keep_prob  = tf.placeholder(tf.float32)
        tf.summary.scalar('dropout_keep_probability', keep_prob)
        h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

#Read layer
with tf.name_scope('fc2_layer'):
    with tf.name_scope('weight'):
        W_fc2 = weight_variable([1024, 10])
        variable_summaries(W_fc2)
    with tf.name_scope('bias'):            
        b_fc2 = bias_variable([10])
        variable_summaries(b_fc2)
    with tf.name_scope('preactivations'):
        y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
        tf.summary.histogram('preactivations', y_conv)

#Cross entropy(Cross entropy)Calculation
with tf.name_scope('cross_entropy'):
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    tf.summary.scalar("cross_entropy", cross_entropy)

#Training
with tf.name_scope('train'):
    train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

#Correct answer rate calculation
with tf.name_scope('accuracy'):
    correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    tf.summary.scalar('accuracy', accuracy)

#Output all Summary
merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(log_dir + '/train', sess.graph) #Training data
test_writer  = tf.summary.FileWriter(log_dir + '/test')              #test data

#Initialization
tf.global_variables_initializer().run()

#Repeat training / test
for i in range(3000):
    batch = mnist.train.next_batch(50)
    
    #Training trace details every 100 times
    if i % 100 == 0:
        run_options  = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
        run_metadata = tf.RunMetadata()
        summary, train_accuracy, _   = sess.run([merged, accuracy , train_step],
                                                feed_dict={x: batch[0], y_:batch[1], keep_prob: 1.0},
                                                options=run_options,
                                                run_metadata=run_metadata)
        train_writer.add_run_metadata(run_metadata, 'step%03d' % i)
        train_writer.add_summary(summary, i)
        print('step %d, training accuracy %g' % (i, train_accuracy))
    #Test every 100 times
    elif i % 100 == 99:
        summary_test, train_accuracy = sess.run([merged, accuracy], feed_dict={x: mnist.test.images, y_:mnist.test.labels, keep_prob: 1.0})
        test_writer.add_summary(summary_test, i)
        
    #Training result writing
    summary, _ = sess.run([merged, train_step], feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
    train_writer.add_summary(summary, i)

#Final test result output
print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))

#Writer closed
train_writer.close()
test_writer.close()