# [PYTHON] I tried to organize the evaluation indexes used in machine learning (regression model)

• Articles sent by data scientists from the manufacturing industry
• This time, the evaluation indexes used in the regression model are organized.

## Introduction

This time, I tried to organize the evaluation index (regression model) of machine learning. We also summarized the points when using each evaluation index.

## 1. Coefficient of determination

The coefficient of determination $R ^ 2$ evaluates the goodness of the estimated regression model. The formula is as follows.

\begin{eqnarray}
R^2 = 1 - \sum \frac{(y_i - \hat{y_i})^2}{(y_i - \bar{y})^2}
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $\ bar {y}$: Average of measured values

item Overview
Evaluation criteria ・ The closer it is to 1, the better the value
Interpretation ・ Evaluate the goodness of the regression model
Characteristic ・ As the number of explanatory variables increases, it approaches 1.
・ Non-linear model cannot be evaluated

The python code is below.

#Library for score calculation
from sklearn.metrics import r2_score

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of the coefficient of determination R2
r2_score(true, pred)

1. MAE MAE (Mean Absolute Error) evaluates the magnitude of the error in the predicted value. The formula is as follows.
\begin{eqnarray}
MAE = \frac{1}{N} \sum|y_i - \hat{y_i}|
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $N$: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・ The size of the average deviation (error) between the predicted value and the measured value
Characteristic ・ We do not attach much importance to cases that are largely off-prediction.
・ If a good model is selected based on MAE, the maximum error tends to increase.

The python code is below.

#Library for score calculation
from sklearn.metrics import mean_absolute_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of MAE
mean_absolute_error(true, pred)

1. MSE In MSE (Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.
\begin{eqnarray}
MSE = \frac{1}{N} \sum(y_i - \hat{y_i})^2
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $N$: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・予測値と実測値のズレの大きさとInterpretationできる
-Similar to MAE, but different from the actual mean error like MAE
Characteristic ・ We attach great importance to cases that are largely out of the forecast.
・ MSE tends to increase significantly if the forecast is largely missed.

The python code is below.

#Library for score calculation
from sklearn.metrics import mean_squared_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of MSE
mean_squared_error(true, pred)

1. RMSE In RMSE (Root Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.
\begin{eqnarray}
RMSE = \sqrt{\frac{1}{N} \sum(y_i - \hat{y_i})^2}
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $N$: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・予測値と実測値のズレの大きさとInterpretationできる
-Similar to MAE, but different from the actual mean error like MAE
Characteristic ・ We attach great importance to cases that are largely out of the forecast.
・ If the forecast is largely missed, the RMSE tends to increase significantly.

The python code is below.

#Library for score calculation
import numpy as np
from sklearn.metrics import mean_squared_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate RMSE value
np.sqrt(mean_squared_error(true, pred))

1. MAPE MAPE (Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.
\begin{eqnarray}
MAPE = \frac{1}{N} \sum|\frac{y_i - \hat{y_i}}{y_i}|
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $N$: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation -Ratio of the average deviation (error) of the predicted value to the size of the measured value
Characteristic ・ Emphasis is placed on the size of the ratio of prediction error to the measured value.
・ Cannot be used in cases where the measured value is 0

The python code is below.

#Library for score calculation
import numpy as np

#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

#Calculate the value of MAPE
np.mean(np.abs((true - pred) / true)) * 100

1. SMAPE SMAPE (Symmetric Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.
\begin{eqnarray}
SMAPE = \frac{100}{N} \sum|\frac{2(y_i - \hat{y_i})}{(|y_i|+|\hat{y_i}|)}|
\end{eqnarray}


$y_i$: Measured value of $i$ th sample $\ hat {y_i}$: Predicted value of $i$ th sample $N$: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation -Ratio of the average deviation (error) of the predicted value to the size of the measured value
Characteristic ・ Emphasis is placed on the size of the ratio of prediction error to the measured value.
・ Unlike MAPE, it can be used even when the measured value is 0.

The python code is below.

#Library for score calculation
import numpy as np

#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

#Calculate SMAPE value
100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))


## Script summary

Organize the scripts so far.

#Library for score calculation
import numpy as np
import pandas as pd
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error

def calculate_scores(true, pred):
"""Calculate all metrics

Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value

Returns
-------
scores : pd.DataFrame
Results of summarizing each evaluation index
"""
scores = pd.DataFrame({'R2': r2_score(true, pred),
'MAE': mean_absolute_error(true, pred),
'MSE': mean_squared_error(true, pred),
'RMSE': np.sqrt(mean_squared_error(true, pred)),
'MAPE': mape(true, pred),
'SMAPE': smape(true, pred)},
index = ['scores'])
return scores

def mape(true, pred):
"""Calculate MAPE

Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value

Returns
-------
np.array :mape calculation result
"""
return np.mean(np.abs((true - pred) / true)) * 100

#SMAPE calculation
def smape(true, pred):
"""Calculate SMAPE

Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value

Returns
-------
np.array :Calculation of smape
"""
return 100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))

def main():
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

scores = calculate_scores(true, pred)

print(scores)

if __name__ == "__main__":
main()

R2 MAE MSE RMSE MAPE SMAPE
scores 0.808824 0.14 0.026 0.161245 9.888889 10.254971

## finally

Thank you for reading to the end. If you have a request for correction, we would appreciate it if you could contact us.