[PYTHON] Approximate sin function with TensorFlow

I will explain a program that approximates sin functions while also studying how to use TensorFlow.

Learning model

Parameter definition

About X (input value) and t (teacher data). Save training set data in Placeholder For example, [None, 5] represents a matrix of [arbitrary x 5]. t is set to [None, 1] so that the number of data can be taken arbitrarily.

About w (weight). Since it corresponds to the Variable to be optimized, it is defined as an instance of the tf.Variable class. Make a 5x1 matrix with an initial value of 0.

These create a network of input layer 1, intermediate layer 5, and output layer 1.

x=tf.placeholder(tf.float32,[None,5])
w=tf.Variable(tf.zeros([5,1]))
t=tf.placeholder(tf.float32,[None,1])

a formula

The expression of y = Xw is expressed as follows.

y=tf.matmul(x,w)

tf.matmul is a matrix multiplication function that uses the placeholder x and Variable w defined earlier. No specific value has been entered yet, and y has not been determined.

Error function

The difference between the teacher data t and the output y is the sum of squares. We will learn to minimize this squared error. Also select the optimization algorithm at this time.

loss=tf.reduce_sum(tf.square(y-t))
train_step=tf.train.AdamOptimizer().minimize(loss)

Training

Preparing a session

Variables corresponding to Variable are calculated using the "session" that is the execution environment of the training algorithm. First, prepare a new session and initialize Variable.

sess=tf.Session()
sess.run(tf.initialize_all_variables())

Training dataset assignment

Substitute the training set data into the Placeholder. train_t is the actual data.

train_t=np.array([np.sin(2.0*np.pi*i/11) for i in range(12)])
train_t=train_t.reshape([12,1])

train_x=np.zeros([12,5])
for row,m in enumerate(range(1,13)):
    for col,n in enumerate(range(0,5)):
        train_x[row][col]=m**n

Implementation of optimization

Execute the defined training algorithm train_step to correct the weights. At this time, feed_dict sets a specific value for Placeholder. You can also evaluate loss within a session and retrieve the current value, such as loss_val. Then, it outputs the value of the parameter at the time after the training is stopped (after exiting the for loop).

i=0
for _ in range(100000):
    i+=1
    sess.run(train_step,feed_dict={x:train_x,t:train_t})
    if i % 1000==0:
        loss_val=sess.run(loss,feed_dict={x:train_x,t:train_t})
        print ('Step:%d, Loss:%f'%(i,loss_val))

w_val=sess.run(w)
print w_val

test

The curve of the test result is output by the following formula. y(x)=w0+w1x+w2x^2+w3x^3+w4x^4

def predict(x):
    result=0.0
    for n in range(0,5):
        result+=w_val[n][0]* x**n
    return result

result

When the middle layer is 5 layers and the number of learning is 10,000 times Although it deviates from the plot, I think that it can be approximated.

Try changing the number of intermediate layers ... Intermediate layer 6 layers: It has shifted more.

Intermediate layer 4 layers: It is closer than 5 layers.

It is also important to specify the appropriate number of intermediate layers (hyperparameters). ..

Full code

sin.py

#-*- coding:utf-8 -*-
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

x=tf.placeholder(tf.float32,[None,5]) 
w=tf.Variable(tf.zeros([5,1]))

y=tf.matmul(x,w)

t=tf.placeholder(tf.float32,[None,1])
loss=tf.reduce_sum(tf.square(y-t))
train_step=tf.train.AdamOptimizer().minimize(loss)

sess=tf.Session()
sess.run(tf.initialize_all_variables())

train_t=np.array([np.sin(2.0*np.pi*i/11) for i in range(12)])
train_t=train_t.reshape([12,1])

train_x=np.zeros([12,5])
for row,m in enumerate(range(1,13)):
    for col,n in enumerate(range(0,5)):
        train_x[row][col]=m**n

i=0
for _ in range(100000):
    i+=1
    sess.run(train_step,feed_dict={x:train_x,t:train_t})
    if i % 1000==0:
        loss_val=sess.run(loss,feed_dict={x:train_x,t:train_t})
        print ('Step:%d, Loss:%f'%(i,loss_val))

w_val=sess.run(w)
print w_val

def predict(x):
    result=0.0
    for n in range(0,5):
        result+=w_val[n][0]* x**n
    return result

fig=plt.figure()
subplot=fig.add_subplot(1,1,1)
subplot.set_xlim(1,12)
subplot.scatter(range(1,13),train_t)
linex=np.linspace(1,12,100)
liney=predict(linex)
subplot.plot(linex,liney)
plt.show()