Last time University of Tsukuba Machine Learning Course: Study sklearn while creating the Python script part of the assignment (14) https://github.com/legacyworld/sklearn-basic

Challenge 6.6 Logistic Regression and Log Likelihood

Commentary on Youtube is the 8th (1) per 20 minutes The task itself was so easy that I'm programming a little more myself to understand logistic regression. The problem of finding $ E (w) $ for each training sample ($ x_ {1i}, x_ {2i}, i = (1,2, \ cdots, 10) $). $ E (w) $ is expressed as follows.

E(w) = -\sum_{n=1}^{N}{t_n\,ln\hat t_n + (1-t_n)\,ln(1-\hat t_n)}

In this example N = 10 t_n = (1,0,0,1,1,1,0,1,0,0)

After finding $ E (w) $, we also use logistic regression to find $ w $. Click here for source code

`python:Homework_6.6.py`


import numpy as np
from sklearn.linear_model import LogisticRegression

#Sigmoid function
def sigmoid(w,x):
    return 1/(1+np.exp(-np.dot(w,x)))

#Cross entropy loss
def cross_entropy_loss(w,x,y):
    y_sig = sigmoid(w,x)
    return -np.sum(y*np.log(y_sig)+(1-y)*np.log(1-y_sig),axis=1)

X = np.array([[1.5,-0.5],[-0.5,-1.0],[1.0,-2.5],[1.5,-1.0],[0.5,0.0],[1.5,-2.0],[-0.5,-0.5],[1.0,-1.0],[0.0,-1.0],[0.0,0.5]])
X = np.concatenate([X,np.ones(10).reshape(-1,1)],1)
y = np.array([1,0,0,1,1,1,0,1,0,0])
w = np.array([[6,3,-2],[4.6,1,-2.2],[1,-1,-2]])
print(f"E(w1) = {cross_entropy_loss(w,X.T,y)[0]:.3f} E(w2) = {cross_entropy_loss(w,X.T,y)[1]:.3f} E(w3) = {cross_entropy_loss(w,X.T,y)[2]:.3f}")

for c_value in [10**(a-2) for a in range(5)]:
    clf = LogisticRegression(C=c_value).fit(X,y)
    w = np.array([[clf.coef_[0][0],clf.coef_[0][1],clf.intercept_[0]]])
    print(f"C = {c_value} w = {w} E(w) = {cross_entropy_loss(w,X.T,y)}")

Execution result

E(w1) = 1.474 E(w2) = 1.832 E(w3) = 6.185
C = 0.01 w = [[ 0.02956523  0.00018875 -0.01756914]] E(w) = [6.84341713]
C = 0.1 w = [[ 0.26242317  0.01451582 -0.1445077 ]] E(w) = [6.19257501]
C = 1 w = [[ 1.38391039  0.32530732 -0.55198479]] E(w) = [3.91381807]
C = 10 w = [[ 3.9100986   1.36910424 -1.28870173]] E(w) = [1.77721899]
C = 100 w = [[ 9.40098848  3.40849535 -3.23672119]] E(w) = [0.57516562]

In logistic regression, the L2 regularization parameter is moved from 0.01 to 100, and the cross entropy loss is also displayed for each calculated $ w $. It can be seen that the absolute value of $ w $ naturally increases where regularization is not very effective (C is large).

By the way, if regularization is not done, it will be as follows

Homework_6.6.py:13: RuntimeWarning: divide by zero encountered in log
  return -np.sum(y*np.log(y_sig)+(1-y)*np.log(1-y_sig),axis=1)
Homework_6.6.py:13: RuntimeWarning: invalid value encountered in multiply
  return -np.sum(y*np.log(y_sig)+(1-y)*np.log(1-y_sig),axis=1)
No regularization w = [[57.89037518 20.53048228 -9.91476711]] E(w) = [nan]

The absolute value of $ w $ becomes too large and 1 is returned in the calculation of the sigmoid function, making it impossible to calculate the log and an error occurs.

Past posts

University of Tsukuba Machine Learning Course: Study sklearn while creating the Python script part of the task (15)

Challenge 6.6 Logistic Regression and Log Likelihood

python:Homework_6.6.py

Past posts

`python:Homework_6.6.py`