Last time University of Tsukuba Machine Learning Course: Study sklearn while creating the Python script part of the assignment (10) https://github.com/legacyworld/sklearn-basic

Exercise 5.4 Creating ROC curves and tuning regularization parameters

Commentary on Youtube: 6th (1) 58 minutes 30 seconds The iris (iris) data of sklearn is used, but the following conditions are used for simplification.

--Use only sepal width and sepal length --Do not use petal width / length --Only consider whether virginica or not --Actually, there are 3 types: setosa, versicolor, and virginica.

5.4_irisdata.png

Click here for source code

`python:Homework_5.4.py`


import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm,metrics
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris  
iris = load_iris()
# sepal width(Sepal width),sepal length(One-sided length)Use only
features = [0,1]
#Just consider whether virginica or not
target = 2
X = iris.data[:,features]
y = iris.target
# y =Not 2(Not virginica)If-1
y[np.where(np.not_equal(y,target))] = -1
y[np.where(np.equal(y,target))] = 1
#Draw data only
plt.figure(0,figsize=(5,5))
plt.scatter(X[:, 0][np.where(y==1)], X[:, 1][np.where(y==1)], color='b',label=iris.target_names[target])
plt.scatter(X[:, 0][np.where(y==-1)], X[:, 1][np.where(y==-1)], color='r',label=f"not {iris.target_names[target]}")
plt.xlabel(iris.feature_names[features[0]])
plt.ylabel(iris.feature_names[features[1]])
plt.legend()
plt.savefig("5.4_irisdata.png ")
# SVM (C=0.01)
clf = svm.SVC(kernel='linear', C=0.01)
clf.fit(X, y)

#ROC curve
fpr,tpr,th = metrics.roc_curve(y,clf.decision_function(X))
plt.clf()
plt.plot(fpr,tpr)
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC")
plt.savefig("5.4_ROC_curve.png ")
#False Positive Rate is 0.True Positive Rate near 2
index = np.where(fpr>=0.2)[0].tolist()[0]
print(f"FPR = {fpr[index-1]} TPR = {tpr[index-1]}")
print(f"FPR = {fpr[index]} TPR = {tpr[index]}")

#Draw while changing the intercept
for i in range(0,110,10):
    dec = clf.decision_function(X)
    #Move the intercept from maximum to minimum
    intercept = np.max(dec) - i/100*(np.max(dec)-np.min(dec))
    #TPR and FPR
    TP = np.where((np.signbit(dec-intercept)==False) & (np.signbit(y)==False))
    FP = np.where((np.signbit(dec-intercept)==False) & (np.signbit(y)==True))
    FPR = len(FP[0])/len(y)
    TPR = len(TP[0])/len(y)
    #Index of wrong prediction
    index = np.where(np.signbit(dec-intercept)!=np.signbit(y))
    x_miss = X[index]
    y_miss = y[index]
    #drawing
    plt.clf()
    plt.figure(0,figsize=(5,5))
    plt.scatter(X[:, 0][np.where(y==1)], X[:, 1][np.where(y==1)], color='b',label=iris.target_names[target])
    plt.scatter(X[:, 0][np.where(y==-1)], X[:, 1][np.where(y==-1)], color='r',label=f"not {iris.target_names[target]}")
    plt.xlabel(iris.feature_names[features[0]])
    plt.ylabel(iris.feature_names[features[1]])
    plt.title(f"Decision boundary Moved {i}% {-intercept:.3f}\nFPR = {FPR:.3f} TPR = {TPR:.3f}")
    #Drawing the decision boundary
    xlim = plt.xlim()
    ylim = plt.ylim()
    #Make a 30x30 grid
    xx = np.linspace(xlim[0], xlim[1], 30)
    yy = np.linspace(ylim[0], ylim[1], 30)
    YY, XX = np.meshgrid(yy, xx)
    xy = np.vstack([XX.ravel(), YY.ravel()]).T
    #Classification at each grid
    Z = clf.decision_function(xy).reshape(XX.shape)-intercept
    #Draw decision boundaries using contour lines level=0 corresponds to it
    plt.contour(XX,YY,Z,levels=[0])
    plt.scatter(x_miss[:,0][np.where(y_miss==1)],x_miss[:,1][np.where(y_miss==1)],color='b',s=100,linewidth=1, facecolors='none', edgecolors='k')
    plt.scatter(x_miss[:,0][np.where(y_miss==-1)],x_miss[:,1][np.where(y_miss==-1)],color='b',s=100,linewidth=1, facecolors='none', edgecolors='k')
    plt.legend()
    plt.savefig(f"5.4_decision_boundary_{i}.png ")

#C to 0.01 to 1,AUC score when changing to 000
c_values = [0.01,0.02,0.05,0.1,0.2,0.5,1,2,5,10,100,1000]
aucs = []
# TRAIN/VALICATION/Divided into 3 for TEST
X_tr_val,X_test,y_tr_val,y_test = train_test_split(X,y,test_size=0.2,random_state=1)
X_tr,X_val,y_tr,y_val = train_test_split(X_tr_val,y_tr_val,test_size=0.2,random_state=1)
print(len(X_tr_val),len(X_test),len(X_tr),len(X_val))
for c_value in c_values:
    clf = svm.SVC(kernel='linear',C=c_value).fit(X_tr,y_tr)
    dec = clf.decision_function(X_val)
    auc = metrics.roc_auc_score(y_val,dec)
    aucs.append(auc)

#Recalculate with the best C
best_c_index = np.argmax(aucs)
best_clf = svm.SVC(kernel='linear', C=c_values[best_c_index]).fit(X_tr_val,y_tr_val)
test_predict = best_clf.predict(X_test)
print(f"Best value of parameter C c_values[best_c_index]")

plt.clf()
plt.plot(c_values,aucs,label="AUC Scores")
plt.plot(c_values[best_c_index],aucs[best_c_index],marker ='o',markersize=10,label="Best Score")
plt.legend()
plt.xscale("log")
plt.savefig("5.4_AUC.png ")

ROC curve

In the lecture, the intercept (threshold value) was moved, but in this program, the intercept is drawn every 10%. The explanation of why you are doing this is very detailed here. https://qiita.com/TsutomuNakamura/items/ef963381e5d2768791d4

The predictions when moving the intercept and the mistakes (circled) are shown below. 5 (2).png 5 (3).png

The higher the threshold, the higher the accuracy. In other words, something that is not virginica is not made virginica, but there are more cases where virginica is not virginica. On the contrary, if the threshold value is lowered, virginica can be correctly judged as virginica, but non-virginica will also be virginica.

The former is a criminal judgment (some criminals may be overlooked, but false charges are not allowed) The latter is the judgment of cancer screening (a benign tumor that is not cancer is also judged to be cancer and makes the patient extra scared, but there is no problem because the damage is only scary)

The ROC curve looks like this 5.4_ROC_curve.png

Now this problem in the challenge "When the probability of predicting virginica even though it is not virginica is suppressed to 0.2 or less, what is the maximum probability of correctly predicting virginica data as virginica?" In short, what is the maximum value of True Positive Rate when False Positve Rate <= 0.2? In other words. This is the output inside the program

FPR = 0.18 TPR = 0.8
FPR = 0.2 TPR = 0.8

When the FPR is around 0.2, the TPR is about 0.8, so the probability of making a correct prediction is 80%.

AUC The final problem is to move the regularization parameters and plot the AUC. First, as a preparation, the data is divided into three parts.

--X_tr_val, y_tr_val: 120 pieces --Data for SVC again after the best C is decided --X_test, y_test: 30 pieces --Data for the final accuracy_score calculation --X_tr, y_tr: 96 pieces --Training data to find the optimum C --X_val, y_val: 24 pieces --Data for issuing AUC Score

AUC graph 5.4_AUC.png

predict_proba and decision_function

In the lecture, predict_proba is used in the AUC part, but I used decision_function because I saw some articles that predict_proba was sometimes mistaken. https://qiita.com/rawHam/items/3bcb6a68a533f2b82a85

Past posts

University of Tsukuba Machine Learning Course: Study sklearn while creating the Python script part of the task (11)

Exercise 5.4 Creating ROC curves and tuning regularization parameters

python:Homework_5.4.py

ROC curve

predict_proba and decision_function

Past posts

`python:Homework_5.4.py`