[PYTHON] Understanding TensorFlow Arithmetic

TensorFlow is primarily a library for implementing machine learning, especially multi-layer neural networks (deep learning), but you don't need to do that hard to understand how it works.

In this article, I would like to unravel the mechanism of TensorFlow based on simple calculations such as arithmetic.

Features of TensorFlow

First, I would like to summarize the features of TensorFlow. As the name implies, TensorFlow is a tool for writing Flow (calculation processing) of Tensor (corresponding to multidimensional arrays, matrices, etc.). Its features are as follows.

Scalability
Scale according to the resources of each machine, from PCs to servers to mobile devices. In other words, even low-spec servers can work as they should, and high-spec servers with GPUs can make full use of their resources for calculations. Since this "calculation" is not limited to machine learning, it can be used in the same way in scenes that require large-scale matrix operations.
This contributes to the portability and performance that are mentioned as features.
Simple / flexible description method
As with any library, simple and flexible notation is possible.
It has a built-in automatic differentiation function, and I think it is a strength that you can learn just by defining the calculation process and the objective function.
Use from many programming languages
Currently (2015/11/12) only C ++ / Python, but in the future SWIG will provide an interface that can be used from other languages as well.
Visualization function
Comes with a visualization function of the built calculation flow called TensorBoard

Also noteworthy is that it has a track record of being used in various services (search ranking, image classification) within Google.

Please refer to the following part of the official page for details on the above contents.

How TensorFlow works

The mechanism of TensorFlow is as follows.

A TensorFlow Graph consists of Node and edge. Node represents the calculation process (green circle in the figure) or the input / output at the end (blue circle in the figure). edge represents a multidimensional array, or tensor, which is the value of the calculation result. Then Graph is assigned to Session to perform the calculation. Session calculates the computable Nodes (Nodes with all the calculation results sent from the edge) asynchronously / in parallel. When calculating, we also allocate which Device (cpu / gpu) to use.

This is the whole picture of the calculation process in TensorFlow. Please refer to the following part of the official page.

Getting Started

In addition, input / output at the end, specifically, reading files and saving calculation results, etc. are summarized in the following places.

File reading etc .: Tensorflow How to / Reading data
About variable initialization / saving / loading: [Tensorflow How to / Variables: Creation, Initialization, Saving, and Loading](http://tensorflow.org/how_tos/variables/index.html#variables-creation-initialization- saving-and-loading)

All of these are processes that are often used when actually calculating using data, so be sure to check.

Calculation by TensorFlow

Now, I would like to actually use TensorFlow to perform math ... simple operations. The code introduced this time is summarized in the following repository.

icoxfog417/tensorflow-arithmetic

You can refer to the iPython notebook from tensorflow-arithmetic.ipynb.

Installation

See below for installation instructions. It is recommended to use pip install.

Getting Started/Download and Setup

Supports Python 3 from 0.6.0 (Release 0.6.0).
Supports Windows from 0.12 (TensorFlow 0.12 supports Windows ))

Calculation

Now, let's actually calculate using TensorFlow. At first, I will write the following formula (although math is a little more than math).

y = x^2 + b

The formula written in TensorFlow is as follows.

import tensorflow as tf

def x2_plus_b(x, b):
    _x = tf.constant(x)
    _b = tf.constant(b)
    result = tf.square(_x)
    result = tf.add(result, _b)
    return result

It looks like the figure below.

Now that the expression is complete, let's actually execute it using Session.

with tf.Session() as sess:
    result = sess.run([x2_plus_b(2., 3.)])
    print(result)

The execution result (output of print) is as follows.

[7.0]

Since $ 2 ^ 2 + 3 = 7 $, you can see that the calculation is done properly. The methods for the available operations are documented.

TensorFlow API/Math

The argument can also be given in the form placeholder.

import tensorflow as tf

p_x = tf.placeholder(tf.float32)
p_b = tf.placeholder(tf.float32)
p_x2_plus_b = tf.add(tf.square(p_x), p_b)

with tf.Session() as sess:
    result = sess.run([p_x2_plus_b], feed_dict={p_x: [2.], p_b: [3.]})
    print(result)

A value is passed to placeholder using feed_dict when executing Session.

Visualization

One of the features of TensorFlow is that it has a powerful visualization tool called TensorBoard. Let's use this to visualize the calculation result of the formula.

If you installed with pip, you should be able to execute the command tensorboard. Visualization is performed using this, but first we will output the value of the object to be visualized.

import tensorflow as tf

def monitor_calculation(x, b):
    title = "b = {0}".format(b)
    c = x2_plus_b(float(x), float(b))
    s = tf.scalar_summary(title, c)
    m = tf.merge_summary([s])  # if you are using some summaries, merge them
    return m

with tf.Session() as sess:
    writer = tf.train.SummaryWriter("log", graph_def=sess.graph_def)    
    xaxis = range(-10, 12)

    for b in range(3):
        for x in xaxis:
            summary_str = sess.run(monitor_calculation(x, b))
            writer.add_summary(summary_str, x)

The point is to summarize the value calculated by tf.scalar_summary. Write the summary calculated in this way with tf.train.SummaryWriter.

When the above process is executed, the file will be output in the log folder. Then run tensorboard as follows:

tensorboard --logdir=/path/to/log-directory

(The path is definitely an absolute path). Then, I think that the graph will be displayed as shown below (* It takes a long time from the start to the display. If I thought that it would not be displayed for a long time, it might suddenly appear) ..

Now you can easily plot the calculated values and so on. At the time of machine learning, it will be possible to grasp the learning situation by plotting errors and accuracy.

Also, in the "GRAPH" tab, you can see a visualization of the constructed formula (= graph).

Please refer to the following for techniques for drawing graphs (such as creating and summarizing processes using namespaces).

TensorBoard: Graph Visualization

Also, please refer to the following for how to use TensorBoard.

TensorBoard: Visualizing Learning

For how to export the summary, refer to the mnist sample. There is a description of the process in mnist.py (using scalar_summary etc. here), and in fully_connected_feed.py, the process defined in mnist.py is combined and executed. ..

The above is the basic mechanism of TensorFlow. This content is similar to the content of Getting Started.

I think that there will be more and more articles about machine learning with TensorFlow, so I hope it helps you to understand them.