[PYTHON] [Anomaly detection] Detect image distortion by deep distance learning

There is a lot of research on image anomaly detection using deep learning.

As an approach to detect abnormal products using only normal data for industrial products, So far, @ daisukelab's Self-supervised learning is the most influential. In this study As a problem, the accuracy of some images was sluggish, and I made a trial and error on how to create abnormal class data **. There was a comment.

Therefore, in this article, I would like to explore ** various ** automatic generation methods for abnormal images.

• This is the presentation material of Python Data Analysis Study Group # 16.
• The entire code is placed in here.

From the conclusion

• In the previous study Learning with self-supervised learning, the two classes of "normal" and "abnormality drawing lines" were compared with "distortion" in this paper. "Additional anomalies" were added and trained by ** 3 class deep distance learning ** (see the figure below). With the +3 class, the latest method of deep distance learning "AdaCos" can also be used.
• This method has improved accuracy with some data, but with some data, it has become a ** double-edged sword **.

Image consideration

Prior research has succeeded in catching many defects in a very simple method. In addition, @ daisukelab's fasi.ai code is very clean and readable. If you are interested in image anomaly detection, we recommend that you read it.

In previous studies, the AUC was below 0.9 for the four data of Screw, Transistor, Grid, and Pill of MVtech AD. As a result, it was a content of trial and error.

This time, I would like to target data other than Pill. The three data other than Pill are as follows.

Looking at the abnormal image, there are many things that ** "distortion" rather than scratches and dirt, and this detection is Since it is difficult, it is presumed that AUC is difficult to grow.

Therefore, we will search for a method to automatically generate ** "distortion data" from normal data ** and boost the anomaly detection performance.

Automatic generation of "distortion data"

There seem to be various methods for creating distortion data, but in this article, we will explain how to rotate a part of the image. I adopted it.

This is easy to do with Keras' Data Generator.

from keras.preprocessing.image import ImageDataGenerator
    small_fig = fig[begin1:end1,begin2:end2]

    datagen = ImageDataGenerator(
               rotation_range=30,
               width_shift_range=0.2,
              height_shift_range=0.2,
              horizontal_flip=False,
              vertical_flip=False)

    for d in datagen.flow(np.expand_dims(small_fig, axis=0), batch_size=1):
        # datagen.Since flow loops infinitely, it exits the loop
        break

    fig[begin1:end1,begin2:end2] = d[0]

Applies to meaningful places

However, there is no point in rotating part of the ** background image **.

Therefore, we implemented an algorithm that automatically detects the object.

With Screw etc., the direction and position of the test piece are slightly different in the training data. On the other hand, if the training data is averaged, an image in which the test piece disappears can be obtained. （2.Training data average）。

After that, the difference between the averaged image and the target image is taken and binarized (Binary map). By doing so, the range of the test piece can be detected automatically. While performing automatic detection If you rotate a part of the image, you will get the following image.

I got a distorted image.

Procedure of this method

• Prepare a normal image.
• Draw a thin line on the normal image and generate an image of the "abnormal line drawing" class. (In the experiment, OpenCV is used to generate the image shown below)
• Apply the proposed method to a normal image to generate a "distorted anomaly" class image.
• Learn the classification problems of the prepared 3 classes (see the figure below) by deep distance learning.
• After learning, calculate the degree of abnormality by LOF and cosine similarity.

Experiment

Experiments were conducted using MVtech AD. I put the whole code here [https://github.com/shinmura0/AutoEncoder_vs_MetricLearning/blob/master/AnomalyDetection_self_supervised.ipynb). It can be executed in Colab.

conditions

• For deep distance learning, "L2-Softmax Loss", "ArcFace", "AdaCos" ([3 or more classes](https://qiita. Since it became com / shinmura0 / items / d2a3a46c3fb287a1ea0e #% E5% 88% B6% E7% B4% 84), ** Applicable! **) is used.
• "L2-Softmax Loss" calculates the degree of abnormality by LOF
• "ArcFace" and "AdaCos" calculate the cosine similarity for all learned (normal) data and adopt the largest value (for "L2-Softmax Loss" and "ArcFace", here //qiita.com/shinmura0/items/06d81c72601c7578c6d3#2019%E5%B9%B4metric-learning%E8%B7%9D%E9%9B%A2%E5%AD%A6%E7%BF%92%E3%81 % AB% E3% 82% 88% E3% 82% 8B% E7% 95% B0% E5% B8% B8% E6% A4% 9C% E7% 9F% A5) Please refer to the article)
• epoch is 10, optimization method is Adam
• Batch size is 32
• Base model is MobileNet V2 (using trained model, that is, transfer learning)
• Abnormal classes (lines, distortions) are drawn randomly for each batch, so the function used for learning uses Keras train_on_batch.
• Try 10 times to calculate AUC
• Data uses Screw, Transistor, Grid

AUC results

In the experimental results, class 2 (two classes, "normal" and "abnormal with a line") is the baseline, class3 (class2 class + "distorted anomaly") is the proposed method.

• Screw

In Screw, when the proposed method (class3) is applied to class2, the AUC decreases. It's gone. The reason for the decline will be discussed later.

• Grid

In Grid, as expected, the proposed method (class3) is better than the baseline (class2). AUC has risen. I think I was able to detect the distortion as I aimed. Abnormal part detected We will verify that it is done by visualization.

• Transistor

In Transistor, AUC increased when the proposed method (class3) was applied as in Grid. An improvement of 0.1 or more can be confirmed for ArcFace.

Visualization results

As shown in Previous research, I also tried to visualize the abnormal part by Grad-CAM. However, it is based on "AdaCos". Therefore, there are two abnormal classes, I'm running Grad-CAM in each class.

First is the Grid data.

The visualization of "Distortion" fits nicely on all data. However, if there are multiple abnormalities, it seems that only one can be detected.

Next is Transistor.

In Transistor, the visualization of "Slight line" fits well for most data. The result is the opposite of Grid. Which class of anomalies should be seen around here, depending on the type of data It may change.

Experiment (extra edition)

Finally, let's change the base model (MobileNetV2 → EfficentNetB0).

conditions

The conditions are almost the same as in the previous experiment. The differences are as follows.

• The base model is EfficentNetB0 (using the trained model, that is, transfer learning)
• Try 5 times to calculate AUC

AUC results

• The median is difficult to see, but it is listed later in the table.

• Screw

With Screw, the AUC is less than 0.5 even in class 2.

• Grid

With Grid, the result is similar to MobileNet V2. After all, ** class3 gives a better score. ** **

• Transistor

With Transistor, the result is similar to MobileNet V2. After all, you will get a better score if you use class3.

Summary of experimental results

Let's compare the results so far by the median.

Excluding Screw, the AUC increases ** when the proposed method (class3) is applied to the baseline (class2). You can check. Also, as for the types of distance learning, it seems that there is such a difference. I can't see it, but ArcFace shows a relatively good score.

The difference in AUC by the CNN model seems to be good for EfficientNet B0, which has achieved AUC over 0.9. Applying EfficientNetB7 or something may increase the AUC more significantly (although in the image) The minimum size seems to be 600 x 600. heavy. By the way, B0 is 224 x 224. )

Consideration (Why did Screw's AUC decrease?)

Consider the cause of the decrease in AUC of Screw.

Simply put, by adding the "Distorted Anomaly" class ** "Large" distortion is considered anomalous Now detectable **, but ** "small" distortions are now recognized as near normal ** It seems that it has become.

In the two-class classification of the previous research, by drawing a thin line, not only scratches and discoloration, but also the tip of the screw By drawing a line, I was able to simulate "small" distortion. However, in the third class With the introduction, CNN tasks become more difficult.

Until now, in the two-class classification, "small" distortion was a rather thin line class. On the other hand, due to the three-class classification, ** CNN's special amount extraction specializes in line detection and rotation detection, ** The "small" distortion seems to be close to normal class. As a result, the decrease in AUC It is presumed that he was invited.

The figure above is a diagram in which Grad-CAM is applied to AdaCos (class3) trained by Screw. All images It is an abnormal image. As you can see, the small distortion at the tip of the screw has not been detected at all.

Furthermore, if you change from MobileNetV2 to EfficientNetB0, the AUC will be 0.5 even in class2. The result was lower. This is because EfficientNetB0 can extract features more clearly. The result is that the feature extraction of "thin lines" goes too well, and the "small" distortion approaches normal. It seems that it has been done.

Then, isn't it possible to simulate "small" distortion by reducing the range of partial rotation? However, if the range is made too small, it cannot be detected by CNN and learning will not proceed at all. Become.

To improve this, increase the resolution of the image or align the orientation of the test product. It is necessary to devise a special image. As mentioned in Prior research, If anomalous data is available, you can also look closely at the image and analyze it. It seems to be effective.

As a recommended procedure, first of all, two classes are classified by self-supervised learning. afterwards, If you try 3 classifications and the AUC worsens, analyze the image well. It will be a process.

Summary

• "Distorted anomaly" class was added, and learning was performed by 3 classes of deep distance learning + self-supervised learning.
• In Transistor and Grid, ** AUC increase (maximum AUC: 0.75 → 0.87) was confirmed. ** On the other hand, in Screw, ** on the contrary, it caused a decrease in AUC. ** **
• Therefore, whether or not to apply this method must be carefully judged by looking at the ** data. ** **
• Pill has many images of discoloration abnormality rather than distortion, so this method is not suitable, so I have not tried it.
• Almost the same accuracy for the types of deep distance learning. However, "AdaCos" is highly recommended because it does not require parameter tuning.