[PYTHON] Evaluate CNN performance with a custom merit function

I was doing data analysis and had to create and evaluate the CNN evaluation function myself, so make a note of how to create it.

Evaluation function to create

This time, I created the coefficient of determination (R2) as an evaluation function. R2 is expressed by the following formula.

program

The program of the evaluation function (R2) itself is as follows.

def r2(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true - y_pred)) 
    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) 
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

Embed this function in the model's metric as follows: This time we are using the CNN model.

def build_model():

    model = Sequential()
    model.add(Dense(500, activation='relu', input_shape=(X_train.shape[1],), kernel_initializer='he_normal'))
    model.add(Dropout(0.1))
    
    model.add(Dense(300, activation='relu', kernel_initializer='he_normal'))
    model.add(Dropout(0.1))

    model.add(Dense(100, activation='relu', kernel_initializer='he_normal'))
    model.add(Dropout(0.1))

    model.add(Dense(1, activation='sigmoid'))
    
    model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['mae', r2])
    
    return model

By executing this model as follows, R2 will also be displayed and output.

reg = KerasRegressor(build_fn=build_model, validation_data=(X_test, y_test), 
                     batch_size=100, 
                     epochs=5, 
                     verbose=1)
history = reg.fit(X_train, y_train)

Train on 11671 samples, validate on 2060 samples
Epoch 1/5
11671/11671 [==============================] - 4s 366us/step 
- loss: 1.1265 - mean_absolute_error: 0.2145 - r2_keras: -1.7520 - val_loss: 0.6966 - val_mean_absolute_error: 0.1376 - val_r2_keras: 0.0823
Epoch 2/5
11671/11671 [==============================] - 4s 311us/step 
- loss: 0.7213 - mean_absolute_error: 0.1264 - r2_keras: 0.1204 - val_loss: 0.6822 - val_mean_absolute_error: 0.1165 - val_r2_keras: 0.3338
Epoch 3/5
11671/11671 [==============================] - 4s 311us/step 
- loss: 0.6593 - mean_absolute_error: 0.1153 - r2_keras: 0.3085 - val_loss: 0.6790 - val_mean_absolute_error: 0.1015 - val_r2_keras: 0.4019
Epoch 4/5
11671/11671 [==============================] - 4s 308us/step 
- loss: 0.6433 - mean_absolute_error: 0.0993 - r2_keras: 0.4104 - val_loss: 0.6678 - val_mean_absolute_error: 0.0991 - val_r2_keras: 0.4225
Epoch 5/5
11671/11671 [==============================] - 4s 315us/step 
- loss: 0.6362 - mean_absolute_error: 0.0953 - r2_keras: 0.4335 - val_loss: 0.6646 - val_mean_absolute_error: 0.0982 - val_r2_keras: 0.4332

By changing the contents of the evaluation function, it is possible to evaluate with various custom functions.