[PYTHON] Tuning Keras parameters with Keras Tuner

Hyperparameter tuning modules such as Optuna are coming out, but I would like to try the recently introduced hypertuning module for Keras Keras Tuner. ..

Basically, it is a Japanese translation of here.

conditions

The following modules are required to operate Keras Tuner.

--Python 3.6 and above --TensorFlow 2.0 or higher

The Keras used is also tf.keras.

Installation

You can install it with pip.

pip install -U keras-tuner

By the way, if you have AutoKeras installed, you already have Keras Tuner installed. (Auto Keras uses Keras Tuner for parameter tuning)

You can also install from the source.

Simple sample

First, let's take a simple sample as an example to see how to use it.

from tensorflow import keras
from tensorflow.keras import layers

from kerastuner.tuners import RandomSearch

(x, y), (val_x, val_y) = keras.datasets.mnist.load_data()
x = x.astype('float32') / 255.
val_x = val_x.astype('float32') / 255.

x = x[:10000]
y = y[:10000]

def build_model(hp):
    model = keras.Sequential()
    model.add(layers.Flatten(input_shape=(28, 28)))
    for i in range(hp.Int('num_layers', 2, 20)):
        model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
                               activation='relu'))
    model.add(layers.Dense(10, activation='softmax'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'])
    return model

tuner = RandomSearch(
    build_model,
    objective='val_accuracy',
    max_trials=5,
    executions_per_trial=3,
    directory='test_dir',
    project_name='helloworld')

tuner.search_space_summary()

tuner.search(x=x,
             y=y,
             epochs=3,
             validation_data=(val_x, val_y))

tuner.results_summary()

The dataset used is the familiar MNIST.

import

To use "Keras Tuner", import "Random Search".

Model building

First, prepare a function for building a model. Prepare the hyperparameter variable "hp" as an argument.

There are two ways to adjust the value in this example.

--Value range --hp.Int (name, start, end, tick) --The default value for ticks is 1 --Specified value --hp.Choice (name, [list of values])

Search algorithm

An instance that performs random search (RandomSearch) is generated as a search algorithm. You can also specify Hyperband (https://arxiv.org/pdf/1603.06560.pdf) (Hyperband).

Example

from kerastuner.tuners import Hyperband

tuner = Hyperband(
    hypermodel,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

Here, in addition to setting the function for building the model, set the number of trials (max_trials) and the number of models for each trial (executions_per_trial). The results are saved in the "test_dir / helloworld" directory.

Display the contents of the search space with search_space_summary ().

[Search space summary]
 |-Default search space size: 4
 > num_layers (Int)
 |-default: None
 |-max_value: 20
 |-min_value: 2
 |-sampling: None
 |-step: 1
 > units_0 (Int)
 |-default: None
 |-max_value: 512
 |-min_value: 32
 |-sampling: None
 |-step: 32
 > units_1 (Int)
 |-default: None
 |-max_value: 512
 |-min_value: 32
 |-sampling: None
 |-step: 32
 > learning_rate (Choice)
 |-default: 0.01
 |-ordered: True
 |-values: [0.01, 0.001, 0.0001]

In this example, the values for each parameter are displayed.

Search

Search for the best parameter with search (). The argument is the same as the so-called "fit ()".

The results are displayed with results_summary ().

[Results summary]
 |-Results in test_dir\helloworld
 |-Showing 10 best trials
 |-Objective(name='val_accuracy', direction='max')
[Trial summary]
 |-Trial ID: 5dae177f590b1ff7f9a549cda6ae9567
 |-Score: 0.9282999634742737
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.0001
 |-num_layers: 10
 |-units_0: 256
 |-units_1: 352
 |-units_10: 416
 |-units_11: 448
 |-units_12: 480
 |-units_13: 128
 |-units_14: 64
 |-units_15: 32
 |-units_16: 512
 |-units_17: 256
 |-units_18: 96
 |-units_19: 64
 |-units_2: 480
 |-units_3: 320
 |-units_4: 64
 |-units_5: 512
 |-units_6: 320
 |-units_7: 512
 |-units_8: 320
 |-units_9: 64
[Trial summary]
 |-Trial ID: 496aa846dabfafb3c67270e3ce810234
 |-Score: 0.9157333374023438
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 3
 |-units_0: 64
 |-units_1: 416
 |-units_2: 32
[Trial summary]
 |-Trial ID: c516cbd03faf4aa32cf8182ab34eb114
 |-Score: 0.8071333765983582
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.0001
 |-num_layers: 18
 |-units_0: 160
 |-units_1: 384
 |-units_10: 32
 |-units_11: 32
 |-units_12: 32
 |-units_13: 32
 |-units_14: 32
 |-units_15: 32
 |-units_16: 32
 |-units_17: 32
 |-units_2: 320
 |-units_3: 512
 |-units_4: 416
 |-units_5: 416
 |-units_6: 96
 |-units_7: 128
 |-units_8: 160
 |-units_9: 32
[Trial summary]
 |-Trial ID: 81260e9782e1bc81da957360c6322371
 |-Score: 0.7860667109489441
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 9
 |-units_0: 64
 |-units_1: 512
 |-units_2: 224
 |-units_3: 32
 |-units_4: 32
 |-units_5: 32
 |-units_6: 32
 |-units_7: 32
 |-units_8: 32
[Trial summary]
 |-Trial ID: eb9da62f11d1bb75b11b9d05c79ae7ec
 |-Score: 0.11349999904632568
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 20
 |-units_0: 224
 |-units_1: 288
 |-units_10: 32
 |-units_11: 64
 |-units_12: 448
 |-units_13: 64
 |-units_14: 512
 |-units_15: 96
 |-units_16: 256
 |-units_17: 64
 |-units_18: 32
 |-units_19: 32
 |-units_2: 352
 |-units_3: 480
 |-units_4: 128
 |-units_5: 160
 |-units_6: 224
 |-units_7: 480
 |-units_8: 224
 |-units_9: 352

In this example, 5 trials are displayed. (In order of best grade)

If you want to get it as a model

Example

models = tuner.get_best_models(num_models=2)

Use tuner.get_best_models () like this.

Derivation of class

You can also define the model construction in classes instead of functions. In that case, it will be derived from the HyperModel class. The only function that needs to be implemented is build ().

from tensorflow import keras
from tensorflow.keras import layers

from kerastuner.tuners import RandomSearch
from kerastuner import HyperModel

class MyHyperModel(HyperModel):

    def __init__(self, num_classes):
        self.num_classes = num_classes

    def build(self, hp):
        model = keras.Sequential()
        for i in range(hp.Int('num_layers', 2, 20)):
            model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
                               activation='relu'))
        model.add(layers.Dense(self.num_classes, activation='softmax'))
        model.compile(
            optimizer=keras.optimizers.Adam(
                hp.Choice('learning_rate',
                          values=[1e-2, 1e-3, 1e-4])),
            loss='sparse_categorical_crossentropy',
            metrics=['accuracy'])
        return model

hypermodel = MyHyperModel(num_classes=10)

tuner = RandomSearch(
    hypermodel,
    objective='val_accuracy',
    max_trials=10,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             epochs=5,
             validation_data=(val_x, val_y))

At this time, you can specify the default value for the value to be adjusted.

Example

hp.Int('units',
       min_value=32,
       max_value=512,
       step=32,
       default=128)

Embedded model

Models for ResNet and Xception are already available.

Example

from kerastuner.applications import HyperResNet

hypermodel = HyperResNet(input_shape=(128, 128, 3), num_classes=10)

Example

from kerastuner.applications import HyperXception

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

If you want to change the parameters to be adjusted in the built-in model, prepare hyperparameters with names as shown below and set them when creating an instance of the search algorithm. At that time, set tune_new_entries to False.

from kerastuner.applications import HyperXception
from kerastuner import HyperParameters
from kerastuner.tuners import Hyperband

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

hp = HyperParameters()
hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])

tuner = Hyperband(
    hypermodel,
    hyperparameters=hp,
    tune_new_entries=False,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

On the contrary, if you do not want to adjust it, you can set a fixed value. Specify the value with fixed () and set tune_new_entries = True.

Example

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

hp = HyperParameters()
hp.Fixed('learning_rate', value=1e-4)

tuner = Hyperband(
    hypermodel,
    hyperparameters=hp,
    tune_new_entries=True,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

In addition, the variables "optimizer", "loss" and "metrics" can be changed directly.

Example

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

tuner = Hyperband(
    hypermodel,
    optimizer=keras.optimizers.Adam(1e-3),
    loss='mse',
    metrics=[keras.metrics.Precision(name='precision'),
             keras.metrics.Recall(name='recall')],
    objective='val_precision',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

Other functions

In addition, the following functions are available.

-Distributed Tuning --Perform parameter adjustment in parallel -Customize Tuner --Correspondence to GAN and reinforcement learning --Used outside the model building function (preprocessing, data extension, test time extension)

• Details are omitted here.

Summary

I think it can be used when you want to tune parameters a little with Keras.

Reference URL

https://tksmml.hatenablog.com/entry/2020/02/01/093000