[PYTHON] I tried to visualize the model with the low-code machine learning library "PyCaret"

Overview

• PyCaret, a low-code machine learning library, is finally v1.0.
• Since ** visualization ** of machine learning model is convenient, let's focus on ** visualization ** of the model and summarize it.
• If you check Source, [Yellowbrick @ HP](https://www. It seems that you are using scikit-yb.org/en/latest/) (Yellowbrick@qiita).
• In addition, Qiita is also featured in the following PyCaret tags.
• I tried using PyCaret at the fastest speed
• Introduction to the library "PyCaret" that automates machine learning

things to do

The list is as follows, but it can be executed in a few lines by automating pycaret.

• ① Load data (credit card default)
• ② Pretreatment
• ③ Model comparison (performance comparison between algorithms)
• ④ Parameter tuning
• ⑤ Visualization of the model (★ This is the main, so I will explain here at the beginning ★)

Try it (⑤ Visualization of model)

• The procedure is ** 1st and last **, but since the purpose of this article is ** visualization, I would like to treat it as the first **.

⑤ Model visualization

• Visualize the characteristics of the model for the completed model (tuned_model).
• By passing the completed model to the evaluate_model function, the following menu will be displayed.
• How was the tuned_model completed? Will be described later. First, explain visualization.

evaluate_model(tuned_model)

• With just a click, you can plot various plots such as ROC curve, learning curve, confusion matrix, etc.
• I would like to pick up what I was interested in and take a look at each one.

AUC (ROC curve)

• Although it says AUC, you can draw a ROC curve.
• Since it was classified into 2 classes, 2 types are drawn for the Positive / Negative class.

Confusion Matrix

• This is also a confusion matrix that you often see. It is output as a heat map.

• Binary classification looks lonely, but multi-value classification makes various mistakes.

Error

• Which of Positive / Negative did you make the prediction for each actual class? Is displayed.

• If this is also a multi-value classification, you can feel more useful.

Dicision Boundary

• Decision boundary. The ** Credit dataset ** is "default or not?" Unbalanced data.

• Therefore, since the Positive class is ** very few **, it is difficult to see the boundaries.

• Although it is a multi-value classification, the decision boundary can be confirmed in a data set that is balanced to some extent.

• The following is the lightGBM decision boundary, so you can see the jagged boundaries of the ** Tree-based algorithm **.

• This is a bonus, but you can understand the characteristics of the algorithm by comparing the decision boundaries.

#	Logistic Regression	K Nearest Neighbour	Gaussian Process
boundary
Feature	Because of the linear algorithm The decision boundary is also straight	Grouping nearby points Boundary	Be aware of the bell curve Smooth curved surface

Threshold

• Precision / recall / f-measure is output for each threshold value.
• It can be used to consider how much the threshold should be set for the required prediction characteristics.

Precision Recall

• The Precision-Recall graph can also be used to discuss predictive characteristics.
• What thresholds do both Precision and Recall meet the prediction characteristics? Examination of Can be used for.

Learning Curve

• The learning curve (number of learnings vs. accuracy) is displayed.
• Scores are displayed for Train and CV sets, which can be used to determine Underfittng / Overfitting.
• For the learning curve and judgment of under / overfitting, refer to Materials of Dr. Andrew Ng of Stanford.

Validation Curve

• Train set / CV set scores are displayed for the regularization parameters for each model.

• For LightGBM, take the max_depth (controlling the depth of the tree) parameter on the horizontal axis.

• For this model

• When max_depth = 4, the generalization performance (CV score) is high.

• Above that, the generalization performance does not increase, but the train set is (slightly) overfitted.

• Therefore, it seems better to control max_depth. It can be used for judgment such as *.

• The horizontal axis is different for each algorithm because the parameters that control regularization are different for each model.

• For example, in logistic regression, the regularization parameter is ** C **, so the horizontal axis is C.

• The parameters on the horizontal axis for each algorithm are summarized below.

• See source code for more information (https://github.com/pycaret/pycaret/blob/master/classification.py#L2871-L2941). LDA is not supported.

Feature Importance

• Which features does this model emphasize? Can be displayed.

Manifold Learning

• Displays the result of manifold learning (dimensional compression) using t-SNE.
• Does the feature quantity used and the data itself have resolution rather than a model?
• In the case of binary classification, you can check whether it can be separated into Positive / Negative.
• In this data, the feature amount is increased from 24-> 90 columns in the pre-processing (described later), so 90 columns are made 2D by dimensional compression and the result is visualized.

Dimensions

• This is also a visualization of the data itself, [RadViz Visualizer] Display the results of (https://www.scikit-yb.org/en/latest/api/features/radviz.html).

Try it ((1) Data load- (4) Tuning)

① Data load

• Pycaret contains various data. See Getting Data --PyCaret for more information.
• This time I would like to predict the credit card default.

from pycaret.datasets import get_data
#Load the credit dataset.
#If you specify the profile option as True, pandas-EDA by profiling runs.
data = get_data('credit',profile=False)

② Pretreatment

• The credit dataset is a binary classification (predicted in ** default ** columns).
• Therefore, import the classification library and specify default for target.

from pycaret.classification import *
exp1 = setup(data, target = 'default')

• Various pre-processing runs automatically.
• For this dataset, features are expanded from 24 columns to 90 columns.

③ Model comparison

• Below you can model for multiple classification algorithms.
• It depends on the characteristics of the forecast you are looking for, but first, list them in AUC.

compare_models(sort="AUC")

• GBM, XGB, CatBoost, LightGBM, etc. are lined up

④ Parameter tuning

• Random Forest, which has the best AUC, is fine, but I will try it with lightGBM, which is relatively quick to calculate.

tuned_model = tune_model(estimator='lightgbm')

*The algorithms that can be specified are as follows.docstringBut you can check it.

#Summary *I have written the method of visualizing the model separately, so I would like to finish organizing it by application at the end. *Assuming input data-> modeling-> results, I would like to group them for the following 5 purposes. * A)Understand the input data and features themselves. * B)Understand the features that the model is looking at. * C)Determine the model's learning status (insufficient learning, overfitting). * D)Consider the predictive characteristics of the model and the thresholds at which the objectives can be achieved. * E)Understand the prediction performance and prediction results of the model.

Use	Perspective	Visualization means
A)Understand the input data and features themselves.	Is positive / negative data separable?	Manifold Learning
	Same as above	Dimensions
B)Understand the features that the model is looking at.	Which features are important	Feature Importance
C)Determine the model's learning status (insufficient learning, overfitting).	Can prediction performance be improved by increasing the number of learnings?	Learning Curve
	Is overfitting suppressed by regularization?	Validation Curve
D)Consider the predictive characteristics of the model and the thresholds at which the objectives can be achieved.	Which threshold value corresponds to the desired prediction characteristic?	Threshold
	What is the relationship between Precision and Recall?	Precision Recall
E)Understand the prediction performance and prediction results of the model.	What is the AUC (Predictive Performance)?	AUC
	Understand the boundaries of the results	Decision Boundary
	Understand how to make mistakes	Confusion Matrix
	Same as above	Error

#Finally *Thank you for staying with us. *If you don't mindLike, shareI would be happy if you could. *If there is a response to some extent, I will write a masterpiece (parameter explanation, etc.).