[PYTHON] Find polynomial approximations using TensorFlow 2.x Low-level API

Introduction

This is a study memo of Etsuji Nakai's "Learning while moving with TensorFlow and Keras-Mechanism of deep learning-Thorough explanation of convolutional neural network-".

Chapter 1 of this book deals with the content of finding a polynomial trendline by the least squares method ** using TensorFlow's ** Low-level API **. However, the code posted there was a TensorFlow 1.x compatible version, so I rewrote it to ** TensorFlow 2.x compatible ** (Note that Chapter 2 and later of the book are TF The code is 2.x compatible).

The code for TensorFlow 1.x in the book is the author's GitHub (https://github.com/enakai00/colab_tfbook/tree/tf2.0/Chapter01 You can check it from /tree/tf2.0/Chapter01)).

Google Colab. Was used as the execution environment.

What you want to do with the TF Low-level API

Given the monthly mean temperature, the parameters ($ w_0, w_1, w_2, w_3, w_4 $) of the ** 4th order polynomial approximation curve ** with the month as the variable $ x $ are as follows: The content is to find it numerically.

temp = [5.2, 5.7, 8.6, 14.9, 18.2, 20.4, 25.5, 26.4, 22.8, 17.5, 11.1, 6.6]

Try running the sample code as is in the TF 2.x environment. Failure.

Unexpectedly, I thought that it would work as it is, so I copied the above code and ran it in the TF 2.x environment of Google Colab.

As a result, the following error occurs.

x = tf.placeholder(tf.float32, [None, 5]) AttributeError: module 'tensorflow' has no attribute 'placeholder'

I made various trials and errors with the above error as a hint, but I came to the conclusion that it seems unlikely that a slight modification will be enough.

Significantly rewritten to match TF 2.x

Rewrite is TF official [here](https://www.tensorflow.org/guide/eager#variables%E3%81%A8%E3%82%AA%E3%83%97%E3%83%86 I referred to the code of% E3% 82% A3% E3% 83% 9E% E3% 82% A4% E3% 82% B6). The concept is to use ** Low-level API **, but Keras is included.

:GoogleColab.In TF2.Magic command to switch to x

%tensorflow_version 2.x

Find the parameters of the 4th order trendline

import tensorflow as tf
import numpy as np

# f(x) = w0 + w1*x + w2*x^2 + w3*x^3 + w4*x^4 
class Model(tf.keras.Model):
  def __init__(self):
    super(Model, self).__init__()
    self.w = tf.Variable(tf.zeros([5, 1]), name='w') # w0,w1,..,w4
  def call(self, inputs):
    return tf.matmul(inputs, self.w)

#Training data (input)
train_x  = np.array([[mon**n for n in range(0, 5)] for mon in range(1, 13)])
train_x =train_x.astype(np.float32)
#print(train_x.shape) #Execution result-> (12,5)

#Training data (correct answer value)
train_t = np.array([5.2, 5.7, 8.6, 14.9, 18.2, 20.4, 25.5, 26.4, 22.8, 17.5, 11.1, 6.6])
train_t =train_t.reshape([12, 1]) 
#print(train_t.shape) #Execution result-> (12, 1)

#Loss function (sum of squares of error)
loss = lambda model, x, t : tf.reduce_sum(tf.square(model(x)-t))

#Calculate slope
def grad(model, x, t):
  with tf.GradientTape() as tape:
    loss_value = loss(model, x, t)
  return tape.gradient(loss_value, [model.w])

model = Model()
optimizer = tf.keras.optimizers.Adam()

#Learning loop
print(f'Initial Loss: {loss(model, train_x, train_t):.5f}')
i = 0
for _ in range(10000):
  i += 1
  grads = grad(model, train_x, train_t)
  optimizer.apply_gradients(zip(grads, [model.w]))
  if i % 1000 == 0:
    print (f'Step: {i}, Loss: {loss(model, train_x, train_t):.5f}')

print(f'Final loss: {loss(model, train_x, train_t):.5f}')
print(f'w = {model.w.numpy()}')

Execution result

Initial Loss: 3442.96973
Step: 1000, Loss: 206.62106
Step: 2000, Loss: 107.81633
Step: 3000, Loss: 53.71612
Step: 4000, Loss: 37.42331
Step: 5000, Loss: 33.97096
Step: 6000, Loss: 33.05221
Step: 7000, Loss: 32.35994
Step: 8000, Loss: 31.79436
Step: 9000, Loss: 31.35630
Step: 10000, Loss: 31.01644
Final loss: 31.01644
w = [[ 1.3081207e+00]
 [ 1.4444064e+00]
 [ 8.0244839e-01]
 [-8.5857309e-02]
 [ 9.1365178e-04]]

Draw a trendline using the obtained parameter $ w $.

Draw a graph with a trendline

import japanize_matplotlib
import matplotlib.pyplot as plt

w = model.w.numpy().flatten()
predict = lambda x : np.dot(np.array([x**n for n in range(0, 5)]),w)

plt.figure(dpi=96)
plt.scatter(range(1, 13), train_t)
xs = np.linspace(1, 12, 100)
ys = [ predict(x) for x in xs ]
plt.plot(xs, ys)
plt.xlim(1, 12)
plt.xticks(range(1, 13))
plt.xlabel('Month')
plt.ylabel('Temperature (℃)')

problem

-Here is slower than the TF 1.x compatible code. take time.