Color page judgment of scanned image with python

Overview

• The program is available at https://github.com/rennone/is_color_page. I am doing self-catering of books, and I have digitized more than 1000 books (mainly manga). Due to the size of the PDF and the ease of reading on tablets, black and white pages are grayscaled and contrasted.

In order to automate that, we implemented a process to determine whether the scanned image is a color page or a black and white page. (Well, the scanner has automatic discrimination, but I want to keep the original data in color.)

In conclusion, python + SVM gave simple and relatively accurate results. I referred to https://qiita.com/kazuki_hayakawa/items/18b7017da9a6f73eba77 when implementing it. Thank you very much.

import sklearn
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
import cv2
import numpy as np

class IsColorPageSVM:
    def __init__(self, kernel='linear', random_state=None):
        self.svm = SVC(kernel=kernel, random_state=random_state)
        self.scaler =  StandardScaler() 

    def learn(self, vec_array, label_array, test_size=0.3, random_state=None):
        #Separate training data and test data
        vec_train, vec_test, label_train, label_test = sklearn.model_selection.train_test_split(vec_array, label_array, test_size=test_size, random_state=random_state )

        #Data standardization process
        self.scaler.fit(vec_train)
        #Model learning
        self.svm.fit(self.scaler.transform(vec_train), label_train)

        #Returns the correct answer rate for training data and test data
        return self.test(vec_train, label_train), self.test(vec_test, label_test)

    #If you pass test data and answer, it will return the correct answer rate
    def test(self, vec_array, label_array):
        vec_array_std   = self.scaler.transform(vec_array)
        pred_train        = self.svm.predict(vec_array_std)
        return sklearn.metrics.accuracy_score(label_array, pred_train) 

Why SVM

This is because it specializes in two-class classification and seems to have high accuracy. The other reason is that it seemed like it could be done with python soon. If you use the trendy deep learning, you may get different results.

data form

If you insert the image data as it is, the number of dimensions will vary depending on the image, so conversion processing will be applied. It may be possible to use Reshape for 128 * 128, but I found out that it would be better if it was a color judgment. Tell me goo posted the following.

I calculated the standard deviation of each HSV element using cvAvgSdv. After that, as far as I tried with the image I have, The saturation (S) is now roughly as follows. Color: Approximately 40 to 50 with many single colors. It usually looks like that, over 80, over 100. Black and white: around 20. Yellowing: Around 20 to 30. * I didn't have many samples, so it's reasonable. From now on, it seems that it can be judged by setting the threshold value around 30-40.

Then, I think it's okay to make a judgment based on this, without having to do machine learning. I thought so and tried it with the image I had, but it was mostly correct, but the page that was too burnt was judged as color, but if I raised the threshold, the subtle color page was judged as black and white. There was a problem that it was done. In that case, it is necessary to judge the hue as well, and it is impossible to judge the threshold value by the scale! That's why I decided to do machine learning.

So, convert the image to HSV format, get the standard deviation of all HSVs, and then add the mean/maximum/minimum values ​​and try.

Finally, I tested the following 12-dimensional vector as a data format.

(Standard deviation of H, Standard deviation of S, Standard deviation of V, Mean of H, Mean of S, Mean of V, Maximum value of H, Maximum value of S, Maximum value of V, Minimum value of H Value, minimum value of S, minimum value of V)

Test method

The data of the color page/black and white page was read as follows, and the average value trained 100 times was calculated. At that time, we also conducted a test after extracting some dimensions with the meaning of measuring how much of the 12-dimensional vectors contributed to learning.

def main():
    #Read 2D array csv file
    color_vec_array = np.loadtxt('color_page.csv', delimiter=',')
    gray_vec_array = np.loadtxt('gray_page.csv', delimiter=',')
    #Since the training data for black and white pages is large for color images, make them the same size.
    gray_vec_array = gray_vec_array[: len(color_vec_array),:]

    #Merge into one 2D array
    vec_array = np.append(color_vec_array, gray_vec_array, axis=0)

    #Color image label is 0,Set the label of the black and white image to 1.
    label_array = [0] * len(color_vec_array) +[1] * len(gray_vec_array)
    
    # i,j is the dimension of the vector to extract
    for i in range(0,vec_array.shape[1]):
        for j in range(i+1,vec_array.shape[1] + 1):
            train_score, test_score = 0.0, 0.0
            #Take the average of the results of training 100 times
            test_num = 100
            print('{0:2}:{1:2}Element data'.format(i, j), end='  ')
            for n in range(test_num):
                data_list = vec_array[:,i:j]
                model = IsColorPageSVM()                    
                train, test = model.learn(data_list, label_array)
                train_score += train
                test_score += test
            print(u'Correct answer rate(Training/test):{0:0.3} - {1:0.3}'.format(train_score/test_num, test_score/test_num))

if __name__ == '__main__':
    main()

result

The following is the correct answer rate for each dimension extracted. The item of the number of dimensions indicates from where to where of the 12 dimensions was used (3: 6 means that the 3rd dimension from the 3rd element to the element immediately before the 6th is used). .. Looking at this, the standard deviation of S alone is a considerable accuracy rate. (The area matches what was said in the data format of ↑)

The highest accuracy rate of the test data was 99% of 0: 4 data (standard deviation of HSV element + mean value of H) and 3: 9 (mean value of HSV + maximum value of HSV). What is surprising is that the 3: 9 score, which does not use the standard deviation, also gives the result of the 1st place tie (this is the best result because the score of the training data is also the 1st place). Regarding this, I wrote a little in the chapter on data formats, but it may be because I labeled the image so that monochromatic color pages (such as pages with a single blue color) are also color-judged.

Naturally, information on only the maximum and minimum values ​​and only the brightness is almost unrecognizable (since there are two categories, around 50% is virtually random). The minimum value would be 0 if there were black pixels, and the brightness doesn't matter for color or black and white pages.

Future plans

Regarding color judgment, the only way to do it in the future is to reshape the image data and use it as it is, or try other methods such as deep learning. After that, I also want to judge whether it is an illustration of a novel or a text page, so I may do that too (I may write it because I got a decent result with the same feeling) That's it. Thank you for reading until the end.