[PYTHON] I tried to approximate the sin function using chainer (re-challenge)

Last time I tried to approximate the sin function using chainer, but it didn't work, so I tried again while visualizing various things. In conclusion, it ended in failure. I still don't understand why (I feel like I've fallen into a local solution ...). I would be very grateful if you could point out.

** Differences from last time ** -Aimed to approximate sin (0.25x) instead of sin (x) I tried to output the value of -y -Output a graph of average loss

sin_NN.py

import numpy as np
import six
import chainer
from chainer import computational_graph as c
from chainer import cuda
import chainer.functions as F
from chainer import optimizers
import matplotlib.pyplot as plt
import csv


def make_dateset():
	x_train = np.arange(0,3.14*40.0,0.5)
	y_train = np.sin(0.25 * x_train).astype(np.float32)

	f = open('sin_train.csv','ab')
	csvWriter = csv.writer(f)
	csvWriter.writerow(x_train)
	csvWriter.writerow(y_train)
	f.close()

	x_test  = np.arange(3.14*40.0,3.14 * 60.0,0.5)
	y_test = np.sin(0.25 * x_test).astype(np.float32)
	return x_train.astype(np.float32),y_train.astype(np.float32),x_test.astype(np.float32),y_test.astype(np.float32)


	
def forward(x_data,y_data,train = True,pred_flag = False):
	if pred_flag:
		x = chainer.Variable(x_data)
		train = False
	else:
		x,t = chainer.Variable(x_data),chainer.Variable(y_data)

	h1 = F.dropout(F.relu(model.l1(x)),  train=train)
	h2 = F.dropout(F.relu(model.l2(h1)),  train=train)
	h3 = F.dropout(F.relu(model.l3(h2)),  train=train)
	y = model.l4(h3)

	if pred_flag:
		return y
	else:
		return F.mean_squared_error(y,t)


if __name__ == "__main__":

	
	x_train,y_train,x_test,y_test = make_dateset()
	x_train = x_train.reshape(len(x_train),1)
	y_train = y_train.reshape(len(y_train),1)
	x_test = x_test.reshape(len(x_test),1)
	y_test = y_test.reshape(len(y_test),1)


	xp = np
	
	batchsize = 20
	N = len(x_train)
	N_test = len(x_test)
	n_epoch = 500
	n_units = 10

	model = chainer.FunctionSet(l1=F.Linear(1, n_units),
								l2=F.Linear(n_units, n_units),
								l3=F.Linear(n_units, n_units),
								l4=F.Linear(n_units, 1))

	optimizer = optimizers.Adam()
	optimizer.setup(model.collect_parameters())


	loss_means = []

	for epoch in six.moves.range(1, n_epoch + 1):
		print('epoch', epoch)
		
		#train
		perm = np.random.permutation(N)
		sum_loss = 0
		sum_accuracy = 0
		for i in six.moves.range(0, N, batchsize):
			x_batch = xp.asarray(x_train[perm[i:i + batchsize]])
			y_batch = xp.asarray(y_train[perm[i:i + batchsize]])
			optimizer.zero_grads()
			loss = forward(x_batch, y_batch)
			loss.backward()
			optimizer.update()
			sum_loss += float(cuda.to_cpu(loss.data)) * len(y_batch)
		print "train mean loss = ",sum_loss/N

		#evaluation
		sum_loss = 0
		sum_accuracy = 0
		for i in six.moves.range(0, N_test, batchsize):
			x_batch = xp.asarray(x_test[i:i+batchsize])
			y_batch = xp.asarray(y_test[i:i+batchsize])
			loss = forward(x_batch, y_batch, train=False)
			sum_loss += float(cuda.to_cpu(loss.data)) * len(y_batch)

			###################################################
		if epoch == 200:
			#Create predict data
			x_pre = np.arange(3.14*80.0,3.14*120.0,0.5)
			x_pre = x_pre.astype(np.float32)
			y_pre = np.sin(0.25 * x_pre).astype(np.float32)
			y_pre = y_pre.reshape(1,len(y_pre))
			answer = []

			#predict
			for g in range(0,len(x_pre)-1):
				xx =  np.asarray([[x_pre[g]]])
				y1 = forward(x_data = xx,y_data = None,train = False,pred_flag=True)
				answer.append(y1.data[0][0])

			f = open('sin_pre.csv','ab')
			csvWriter = csv.writer(f)
			csvWriter.writerow(x_pre)
			csvWriter.writerow(y_pre[0])
			csvWriter.writerow(answer)
			f.close()
			####################
		print "test mean loss = ",sum_loss/N_test
		loss_means.append(sum_loss/N_test)

	f = open('loss_means.csv','ab')
	csvWriter = csv.writer(f)
	csvWriter.writerow(loss_means)
	f.close()

-Deep Learning parameters Mini batch size 20 Number of epochs (number of learning) 500 Number of units 1-20-20-1 Activation function ReLu (rectified linear function) How to update Adam Loss error function Mean square error function

Training data

First, I plotted the data to be trained this time (train data) y=sin(0.25x)

Created as training data by 0.5 in the range of 0 <x <3.14 * 40

Test data was created at 0.5 intervals in the range of 3.14 * 40.0 <x <60.0

Loss average graph

I couldn't approximate the sin function too much, so I plotted it for confirmation. The initial average loss value is too large and seems to be 0 from around epoch13, but it is actually wandering around 0.5. It decreases to 0.5, but does not decrease at all. Even if the number of batches and the number of units were changed, the decrease stopped at around 0.5.

Forecast graph

I tried to output what kind of function it actually becomes. epoch is 200 o'clock

Blue is the correct answer (original 0.25 * sin (x) function), and orange is the output function of Deep Learning. Well, I feel that the average loss will be 0.5. I think I should have tampered with the learning rate as a solution when I fell into a local solution, but I'm still studying, so what should I do?

Please help someone (laughs)