- Articles sent by data scientists from the manufacturing industry
- This time, the evaluation indexes used in the regression model are organized.

This time, I tried to organize the evaluation index (regression model) of machine learning. We also summarized the points when using each evaluation index.

The coefficient of determination $ R ^ 2 $ evaluates the goodness of the estimated regression model. The formula is as follows.

```
\begin{eqnarray}
R^2 = 1 - \sum \frac{(y_i - \hat{y_i})^2}{(y_i - \bar{y})^2}
\end{eqnarray}
```

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ \ bar {y} $: Average of measured values

item | Overview |
---|---|

Evaluation criteria | ・ The closer it is to 1, the better the value |

Interpretation | ・ Evaluate the goodness of the regression model |

Characteristic | ・ As the number of explanatory variables increases, it approaches 1. ・ Non-linear model cannot be evaluated |

The python code is below.

```
#Library for score calculation
from sklearn.metrics import r2_score
#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]
#Calculate the value of the coefficient of determination R2
r2_score(true, pred)
```

- MAE MAE (Mean Absolute Error) evaluates the magnitude of the error in the predicted value. The formula is as follows.

```
\begin{eqnarray}
MAE = \frac{1}{N} \sum|y_i - \hat{y_i}|
\end{eqnarray}
```

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item | Overview |
---|---|

Evaluation criteria | ・ The closer to 0, the better the value |

Interpretation | ・ The size of the average deviation (error) between the predicted value and the measured value |

Characteristic | ・ We do not attach much importance to cases that are largely off-prediction. ・ If a good model is selected based on MAE, the maximum error tends to increase. |

The python code is below.

```
#Library for score calculation
from sklearn.metrics import mean_absolute_error
#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]
#Calculate the value of MAE
mean_absolute_error(true, pred)
```

- MSE In MSE (Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.

```
\begin{eqnarray}
MSE = \frac{1}{N} \sum(y_i - \hat{y_i})^2
\end{eqnarray}
```

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item | Overview |
---|---|

Evaluation criteria | ・ The closer to 0, the better the value |

Interpretation | ・予測値と実測値のズレの大きさとInterpretationできる -Similar to MAE, but different from the actual mean error like MAE |

Characteristic | ・ We attach great importance to cases that are largely out of the forecast. ・ MSE tends to increase significantly if the forecast is largely missed. |

The python code is below.

```
#Library for score calculation
from sklearn.metrics import mean_squared_error
#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]
#Calculate the value of MSE
mean_squared_error(true, pred)
```

- RMSE In RMSE (Root Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.

```
\begin{eqnarray}
RMSE = \sqrt{\frac{1}{N} \sum(y_i - \hat{y_i})^2}
\end{eqnarray}
```

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item | Overview |
---|---|

Evaluation criteria | ・ The closer to 0, the better the value |

Interpretation | ・予測値と実測値のズレの大きさとInterpretationできる -Similar to MAE, but different from the actual mean error like MAE |

Characteristic | ・ We attach great importance to cases that are largely out of the forecast. ・ If the forecast is largely missed, the RMSE tends to increase significantly. |

The python code is below.

```
#Library for score calculation
import numpy as np
from sklearn.metrics import mean_squared_error
#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]
#Calculate RMSE value
np.sqrt(mean_squared_error(true, pred))
```

- MAPE MAPE (Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.

```
\begin{eqnarray}
MAPE = \frac{1}{N} \sum|\frac{y_i - \hat{y_i}}{y_i}|
\end{eqnarray}
```

item | Overview |
---|---|

Evaluation criteria | ・ The closer to 0, the better the value |

Interpretation | -Ratio of the average deviation (error) of the predicted value to the size of the measured value |

Characteristic | ・ Emphasis is placed on the size of the ratio of prediction error to the measured value. ・ Cannot be used in cases where the measured value is 0 |

The python code is below.

```
#Library for score calculation
import numpy as np
#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])
#Calculate the value of MAPE
np.mean(np.abs((true - pred) / true)) * 100
```

- SMAPE SMAPE (Symmetric Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.

```
\begin{eqnarray}
SMAPE = \frac{100}{N} \sum|\frac{2(y_i - \hat{y_i})}{(|y_i|+|\hat{y_i}|)}|
\end{eqnarray}
```

item | Overview |
---|---|

Evaluation criteria | ・ The closer to 0, the better the value |

Interpretation | -Ratio of the average deviation (error) of the predicted value to the size of the measured value |

Characteristic | ・ Emphasis is placed on the size of the ratio of prediction error to the measured value. ・ Unlike MAPE, it can be used even when the measured value is 0. |

The python code is below.

```
#Library for score calculation
import numpy as np
#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])
#Calculate SMAPE value
100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))
```

Organize the scripts so far.

```
#Library for score calculation
import numpy as np
import pandas as pd
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
def calculate_scores(true, pred):
"""Calculate all metrics
Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value
Returns
-------
scores : pd.DataFrame
Results of summarizing each evaluation index
"""
scores = pd.DataFrame({'R2': r2_score(true, pred),
'MAE': mean_absolute_error(true, pred),
'MSE': mean_squared_error(true, pred),
'RMSE': np.sqrt(mean_squared_error(true, pred)),
'MAPE': mape(true, pred),
'SMAPE': smape(true, pred)},
index = ['scores'])
return scores
def mape(true, pred):
"""Calculate MAPE
Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value
Returns
-------
np.array :mape calculation result
"""
return np.mean(np.abs((true - pred) / true)) * 100
#SMAPE calculation
def smape(true, pred):
"""Calculate SMAPE
Parameters
----------
true : np.array
Measured value
pred : np.array
Predicted value
Returns
-------
np.array :Calculation of smape
"""
return 100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))
def main():
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])
scores = calculate_scores(true, pred)
print(scores)
if __name__ == "__main__":
main()
```

R2 | MAE | MSE | RMSE | MAPE | SMAPE | |
---|---|---|---|---|---|---|

scores | 0.808824 | 0.14 | 0.026 | 0.161245 | 9.888889 | 10.254971 |

Thank you for reading to the end. If you have a request for correction, we would appreciate it if you could contact us.

Recommended Posts