[PYTHON] Typical indicators of classification

Hello. This time I'm going to be a little enthusiastic about ** classification accuracy **: relaxed:

What is an index?

An indicator is a measure of how accurately a class of data is assigned. This model is correct! !! !! So please adopt! !! No one is convinced even if it is said.

A concrete example of life around us is ** deviation value **. The deviation value has a role as a ** standard ** for studying. I don't think the deviation values are all, but some criteria are needed. Because without standards, judgments can lead to unfair decisions. In the academic world, there are various standards such as report cards and resumes. Also, each standard has a different axis.

This is the same in the world of classification. There are many indicators for evaluating the quality of a model built using machine learning. I would like to introduce four of them this time. --Compliance rate (* precision ) --Recall rate ( recall ) --F value ( F-Value ) --Correct answer rate ( accuracy *)

Before introducing these, I would like to explain the existence of ** confusion matrix **.

Confusion matrix

In conclusion, a confusion matrix is a ** matrix that summarizes the results of classification **. Huh? I'm sure some people thought that, but please be assured that I will explain it from now on.

Positive and negative examples

In the world of classification, there is the idea of "interested" and "not interested". Also "I'm interested" is called ** regular example ** "I'm not interested" is a ** negative example **.

For example, if you expect the email sent to be spam I'm interested in which emails are ** spam **, so The positive example is spam and the negative example is regular email. Did you get an idea of the positive and negative examples?

With this in mind, take a look at the image below.

Description of the confusion matrix

The confusion matrix is divided into four types. True positive (upper left): When the actual data is a positive example and machine learning itself is also classified as a positive example. (Correct answer) False negative (upper right): Actual data is a positive example, but machine learning is classified as a negative example. (Incorrect answer) False positive (lower left): Actual data is a negative example, but machine learning is also classified as a positive example. (Incorrect answer) True negative (lower right): When the actual data is a negative example and machine learning itself is also classified as a negative example. (Correct answer)

Mathematically, there can only be these four types. Because, depending on the number of cases, 2 (actual data is 2 types (positive example OR negative example)) * 2 (prediction is also 2 types) = ** 4 **

The explanation of the confusion matrix is around here. Based on this figure, I would like to look at the four indicators.

Compliance rate

Conformance rate: ** Indicates what percentage of the machine learning model predicted to be a correct example ** (whether it was a correct example). If you write it in a formula using the symbols in the above figure,

Compliance rate= \frac{tp}{tp + fp}

Can be written. In other words, a high precision rate means that the percentage of data that was predicted to be a positive example and was actually a positive example is high.

Recall

Recall rate: ** Indicates the percentage of actual positive data that is predicted to be correct **. This has a relationship that is very similar to the precision rate. I'll explain that later. If you write the recall rate in a mathematical formula first,

Recall= \frac{tp}{tp + fn}

In other words, high recall means that the ratio of data that can be predicted to be positive is high compared to data that is positive.

Relationship between precision and recall

These two metrics are in a ** trade-off ** relationship. In other words, if you try to increase the recall rate, the precision rate will decrease, and vice versa.

If you can't imagine it, read the example below. It was very easy to understand. Relationship between reproduction and precision

F value

F value: Harmonic mean of precision and recall That is,

F value= \frac{2}{\frac{1}{Compliance rate} + \frac{1}{Recall}} = \frac{2 \times Compliance rate \times Recall}{Compliance rate + Recall}

The F value is an index that is emphasized when aiming for a good ** balance ** value for both indicators by taking the harmonic mean of the precision rate and recall rate.

Correct answer rate

Correct answer rate: Shows the percentage of data that matches the forecast and the actual result, regardless of whether it is a positive example or a negative example. That is,

Correct answer rate= \frac{tp + tn}{tp + fp + fn + tn}

An index that determines how well the answer matches the overall prediction result.

Summary

How was that? There are many perspectives on determining whether a machine learning model is good or bad. It seems that you need to find out what kind of model you want throughout the task. This time I summarized --Compliance rate

• Recall --F value --Correct answer rate was. There are many indicators in this world. I don't think it's necessary to remember the indicators, but it's important to know. Thank you very much.

References

Evaluation function of machine learning that I can't hear anymore Understanding the Confusion Matrix