[PYTHON] Challenge image classification by TensorFlow2 + Keras 5 ~ Observe images that fail to classify ~

Introduction

This is a study memo (5th) about image classification (Google Colaboratory environment) using TensorFlow2 + Keras. The subject is the classification of handwritten digit images (MNIST), which is a standard item.

--Challenge image classification by TensorFlow2 + Keras series -1. Move for the time being -2. Take a closer look at the input data -3. Visualize MNIST data -4. Let's make a prediction with the trained model -5. Observe images that fail to classify -6. Try preprocessing and classifying images prepared by yourself -7. Understanding layer types and activation functions -8. Select optimization algorithm and loss function -9. Try learning, saving and loading the model

Last time made a prediction (classification) using a trained model. This time, in the model created by TF Official HP Tutorial, ** What kind of image does the prediction (classification) fail? ** Also ** What kind of misclassification will occur? I will observe **.

Specifically, matplotlib will output the following:

■ Cases where the correct answer value "6" could not be predicted (classified) correctly

We will also create a diagram showing the following misclassification relationships.

Extraction of handwritten digit data that failed to be classified

From the 10,000 test input data (images of handwritten numbers), ** extract those that could not be classified correctly ** using the model of the TF tutorial.

By the way, when the trained model was evaluated by model.evaluate (x_test, y_test, verbose = 2), the accuracy rate was 0.9759. Therefore, $ (1-0.9759) \ times 10,000 = 241 $ handwritten digits have failed to be classified (the trained model generated for each training run will be slightly different, resulting in classification. Please note that the image that fails will also be slightly different).

For this extraction process, we use the library "pandas", which is convenient for data analysis. Please refer to other articles for that, as we do not include explanations about the basics of pandas.

The following code stores the correct and predicted values in columns in the data frame. Here, the row index matches the x_test index.

python

import numpy as np
import pandas as pd
p_test = model.predict_classes(x_test) #Forecast
df = pd.DataFrame({'Correct answer value':y_test, 'Predicted value':p_test})
display(df.head(5))  #Display the first 5 lines
display(df.tail(5))  #Display 5 lines from the end

The execution result is as follows. For the first 5 cases and the last 5 cases, the correct answer value and the predicted value all match.

Next, ** extract the rows where the correct and predicted values do not match ** and store them in a new data frame df2. It also sorts.

python

#Extract rows where "correct answer" and "prediction" do not match
df2 = df[df['Correct answer value']!=df['Predicted value']]
display(df2.head(5))

#Sort in ascending order 1st key'Correct answer value', 2nd key'Predicted value'
df2 = df2.sort_values(['Correct answer value', 'Predicted value'])
display(df2.head(5))

The execution result is as follows.

From this, we can see that x_test [9634] is "** the correct answer is" 0 ", but the prediction is" 1 "**".

List display of handwritten digit images that failed to be classified

The x_test [9634] that failed to classify ** what kind of handwritten number **, and ** mistakenly predicts "1". How convinced was that? ** I'm curious.

** What kind of handwritten numbers are ** can be seen by using matplotlib's ʻimshow (...) `as explained in 2nd. You can output and check it.

Also, ** How confident was it that I mistakenly predicted "1" ** was explained in 4th. As you can see, it can be retrieved from the information obtained by model.predict (...). Specifically, it can be obtained by doing the following.

python

idn = 9634
s = model.predict( np.array([x_test[idn]]) ) #Output layer value
s = s[0]
print( f'Predictive classification is "{s.argmax()}, The output value of the corresponding neuron in the output layer{s[s.argmax()]:.2f}' ) 
print( f'The correct answer classification is "{y_test[idn]}, The output value of the corresponding neuron in the output layer{s[y_test[idn]]:.2f}' )

The predictive classification is "1", and the output value of the corresponding neuron in the output layer is 0.62. The correct answer classification is "0", and the output value of the corresponding neuron in the output layer is 0.00.

Alternatively, you can use the code (report output of predictive classification) shown at the end of 4th to check as follows.

The correct answer, "0", isn't too faint. It may be unavoidable because it is an image with dust on the upper left, but if you can observe these things, you may be able to get some hints about preprocessing **.

However, it is troublesome to output these individually, so I would like to ** output them all together **.

python

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patheffects as pe

for t in range(0,10):

  print(f'■ Correct answer value "{t}Could not be predicted (classified) correctly for')

  #Extract the row with the correct answer value t
  index_list = list(df2[df2['Correct answer value']==t].index.values)

  #matplotlib output
  n_cols = 7
  n_rows = ((len(index_list)-1)//n_cols)+1
  fig, ax = plt.subplots(nrows=n_rows, ncols=n_cols, figsize=(6.5, 0.9*n_rows), dpi=120)
  for i,ax in enumerate( np.ravel(ax) ):
    if i < len(index_list):
      
      p = index_list[i]
      ax.imshow(x_test[p],interpolation='nearest',vmin=0.,vmax=1.,cmap='Greys')

      #Prediction (classification) is displayed in the upper left
      t = ax.text(1, 1, f'{p_test[p]}', verticalalignment='top', fontsize=8, color='tab:red')
      t.set_path_effects([pe.Stroke(linewidth=2, foreground='white'), pe.Normal()]) 

      #The values of neurons in the output layer corresponding to the prediction (separation) are displayed in parentheses.
      t = ax.text(5, 2, f'({s_test[p].max():.1f})', verticalalignment='top', fontsize=6, color='tab:red')
      t.set_path_effects([pe.Stroke(linewidth=2, foreground='white'), pe.Normal()]) 
      
      #Hide scale etc.
      ax.tick_params(axis='both', which='both', left=False, labelleft=False, 
                     bottom=False, labelbottom=False)
      
      #Show index in blue
      ax.set_title(index_list[i],fontsize=7,pad=1.5,color='tab:blue')

    else :
      ax.axis('off') #Margin processing

  plt.show()

This is the execution result (only the part output as text is slightly formatted).

--** Index number ** with ** light blue number ** </ font> at the top of each image frame is x_test --The ** red number </ font> ** in the upper left of the image is the ** predicted value ** (that is, the value that was incorrectly predicted (classified)). --The ** red number </ font> ** in the upper left of the image corresponds to the above in the output layer ** Neuron output value ** --For example, in the first example, " 1 (0.6) </ font>", but this is (wrongly) classified as "1", and the output layer " It means that the output value of the neuron corresponding to "1" was "0.6".

import pandas as pd

p_test = model.predict_classes(x_test) #Forecast
df = pd.DataFrame({'Correct answer value':y_test, 'Predicted value':p_test})

#Cross tabulation
dfc = pd.crosstab(index=df['Correct answer value'], columns=df['Predicted value']) 
display(dfc)

!pip install japanize-matplotlib
import japanize_matplotlib

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patheffects as pe
import matplotlib.ticker as ticker
import matplotlib.colors

p_test = model.predict_classes(x_test) #Forecast
df = pd.DataFrame({'Correct answer value':y_test, 'Predicted value':p_test})

#Cross tabulation
dfc = pd.crosstab(index=df['Correct answer value'], columns=df['Predicted value']) 
#display(dfc)

for i in dfc.index.values :
  dfc.at[i,i] = 0.0

#Output as a heat map
plt.figure(dpi=160)

plt.imshow(dfc,interpolation='nearest',cmap='Oranges')
plt.plot([0,0],[9,9])

n = len(dfc.columns) #Number of items
plt.gca().set_xticks(range(n))
plt.gca().set_xticklabels(dfc.columns)
plt.gca().set_yticks(range(n))
plt.gca().set_yticklabels(dfc.columns)

plt.tick_params(axis='x', which='both', direction=None, 
                top=True, bottom=False, labeltop=True, labelbottom=False)
plt.tick_params(axis='both', which='both', top=False, left=False )

#Grid settings
plt.gca().set_xticks(np.arange(-0.5, n-1), minor=True);
plt.gca().set_yticks(np.arange(-0.5, n-1), minor=True);
plt.grid( which='minor', color='white', linewidth=1)

plt.gca().xaxis.set_label_position('top') 
plt.xlabel('Predicted value')
plt.ylabel('Correct answer value')

plt.plot([-0.5,n-0.5],[-0.5,n-0.5],color='black',linewidth=0.75)

#Display correlation coefficient (characters have borders)
tp = dict(horizontalalignment='center',verticalalignment='center')
ep = [pe.Stroke(linewidth=3, foreground='white'),pe.Normal()]
for y,i in enumerate(dfc.index.values) :
  for x,c in enumerate(dfc.columns.values) :
    if x != y :
      if dfc.at[i,c] !=  0:
        t = plt.text(x, y, f'{dfc.at[i,c]}',**tp)
        t.set_path_effects(ep)