This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 7 Learning Techniques **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.
Hyperparameters are parameters that must be set by human power, which is required when creating a neural network. For example, the number of layers and the number of neurons.
This hyperparameter has a great influence on the performance of the neural network, so I definitely want to optimize it, but it is very difficult when it comes to human power. So, let's leave this to the machine, so we will try to implement it to automatically find the optimum value of hyperparameters.
To put it simply, we use random values as hyperparameters to measure the learning results and narrow down the optimum value range from the results.
#Hyperparameter tuning
from sklearn.model_selection import train_test_split
def hayper_tyning(lr_m, lr_d, wd_m, wd_d, x_train, t_train, sanpule = 2):
lr_list = []
wd_list = []
x_train = x_train[:500]
t_train = t_train[:500]#Because it takes a lot of time
(train_x, hayper_x, train_t, hayper_t) = train_test_split(x_train, t_train, test_size=0.2, random_state=0)
for i in range(sanpule):
train_acc_list = []
test_acc_list = []
lr = 10 ** np.random.uniform(lr_d, lr_m)
weight_decay_lambda = 10 ** np.random.uniform(wd_d, wd_m)
lr_list.append(lr)
wd_list.append(weight_decay_lambda)
network = MutiLayerNet(input_size = 784, hiden_size_list = [50], output_size = 10, weight_decay_lambda = weight_decay_lambda)
for i in range(1, 101):
grads = network.gradient(train_x, train_t)
for p in ['W1','b1','W2','b2']:
network.params[p] = network.params[p] - (lr * grads[p])
if i % 100 == 0:
train_acc = network.accuracy(train_x, train_t)
test_acc = network.accuracy(hayper_x, hayper_t)
train_acc_list.append(train_acc)
test_acc_list.append(test_acc)
#The rest is a graph or narrowing down
Recommended Posts