[PYTHON] [AI] Deep Learning for Image Denoising

Introduction

It's been a long time since Deep Learning has firmly established its position in the field of opportunity learning. There are many fields of application of deep learning, but this time I would like to introduce a brief summary and some demos about the application to "Image Denoising", which is extremely useful and practical. .. We also do normal denoising and denoising when only noise images are available as a setting close to the actual problem.

What is Image Denoising?

Image Denoising is literally "noise reduction from an image". Noise is added to photos and video data taken with a camera due to various factors such as the shooting environment, which causes a decrease in visibility compared to the original image. Restoring the original image from an image with such noise is called denoising, and it is applied not only to the images we usually see, but also to the field of medical images such as noise removal from CT and MRI images. It has been.

Deep Learning for Image Denoising

Smoothing processes such as median filters, Gaussian filters, and moving averages have long been used as basic methods for removing noise from images. In the field of machine learning, methods using neural networks have been proposed for a long time, but they are not the mainstream methods. However, starting with the invention of convolutional neural networks, deep learning has shown overwhelming performance in the image domain, and its application to the field of denoising has rapidly progressed. It is one of the fields that are still being researched day and night, and is one of the main topics at top computer vision conferences.

Below, we will introduce some of the main methods and demos of denoising using deep learning.

Demo 1. Denoise from handwritten characters

I will use MNIST (handwritten character image) for the data this time as well. Learn a model that removes noise by using an image with noise as input data and the original image as correct data, and denoises a new noise image.

Input data: 50,000 noise images with Gaussian noise added to the correct answer data Correct answer data: 50,000 original images

As a model to learn, in addition to the usual CNN (Convolutional Neural Network), we will compare with the following three.

1. CAE（Convolutional Autoencoder）

2. DnCNN（Denoising Convolutional Neural Network）

3. Win5-RB（Wide Inference Network 5layer + Resnet and BatchNormalization）

4. CAE（Convolutional Autoencoder） Autocoder is a type of neural network architecture that trains the output data to approach the input data, and is mainly used in the field of dimensionality reduction. Encoder and Decoder are paired, and the structure is such that the input data is compressed into latent variables by Encoder and restored by Decoder from the compressed latent variables.

By making the dimension of the latent variable smaller than the input data, it is possible to capture low-dimensional features that can reproduce the input data, and since a neural network is used for the Encoder and Decoder, non-linear dimension reduction is possible. It is used not only for dimension reduction, but also for noise reduction due to the properties of "compressing to latent variables to capture essential features" and "recovering images with Decoder", and turns the input image into a noise image. Denoise is possible by doing this (also known as Denoising Autoencoder).

CAE replaces the neural network part of Autoencoder's Encoder and Decoder with a convolutional neural network, and by using the convolutional neural network, the accuracy of denoising from image data is greatly improved.

2. DnCNN（Denoising Convolutional Neural Network） DnCNN, as the name "Denosing" suggests, is a CNN aimed at removing noise. DnCNN is a multi-layered CNN consisting of 17 layers, in which Convolution and Batch Normalization are repeatedly applied to the input image.

Batch Normalization is a method of normalizing the output of each layer of the neural network, which can suppress the problem of internal covariate shift (the distribution of each layer changes during training) and contributes to the improvement of learning efficiency and accuracy. I will. Also, one of the major features of DnCNN is that it uses the framework of ResNet (residual network), so it estimates the noise component from the noise image instead of directly estimating the image with the noise removed, and the noise. The original image is estimated by removing the components from the noise image. In the framework of denoise, the input and output values are often close to each other, including when the noise level is low, and it is inefficient to perform learning close to the identity map with a nonlinear map using a finite neural network, so the difference is learned. This will improve learning efficiency. For example, if there is no difference, it is not necessary to train the conformal map with a neural network, and the weight of each layer can be simply set to 0. As an implementation method, by bypassing the input value as it is and connecting it to the final layer, the neural network learns the difference.

3. Win5-RB（Wide Inference Network 5layer + Resnet and BatchNormalization） Win5-RB is a model in which Wide Inference Network (Win) has 5 layers and is connected by Resnet (R) and Batch Normalization (B). In the task of denoising, not only the layer depth but also the number of filters It is claimed that it is effective to make the kernel size larger than the normal CNN, and it is named Wide Inference. From denoising experiments using various data, 128 filters and kernel size 7x7 have been adopted, and the expressive power of each layer is high due to the number of filters and the size of the kernel size, and the network configuration is shallower than DnCNN. I will.

Model building

Model construction is done using Keras on Google Colaboratory. The outline of each model is as follows.

** Various learning parameters **

--Number of epochs: 10 --Batch size: 128 --Number of filters for each layer: 64 (128 for Win5-RB only) --Kernel size: 3x3 (Win5-RB only 7x7)

result

As a test, the denoising result from a noise image with Gaussian noise added is as follows.

In CNN, something like blurring appears in some parts, but in CAE, DnCNN, Win5-RB, noise can be removed well at a level where the difference from the original image can hardly be seen.

Win5-RB has the highest accuracy in MSE (Mean Squared Error) with 10,000 test images.

Demo on Flask

Using the trained model, I created a demo that can be processed in real time in Flask. This is a demo at CAE, but you can see that it removes noise well.

Demo 2. Denoise in a noise data only situation

In Demo 1, we built a model that removes noise components using a noise image and a noise-free original image. As you may have already noticed, is it possible to obtain the original image without noise in the first place? In the actual problem, it is often the case that the original image without noise is not available, and all that is available is the image with noise. Considering such a real problem, we will do denoising in a situation where only noise images can be obtained.

Noise2Noise Noise2Noise (N2N) is a noise removal opportunity learning method that uses only noise images during learning, and has performance comparable to learning methods that use noise-free images (correct answer data) even though only noise images are used. It is a technique that can be put out. In a word, by learning the conversion from a noise image to a noise image (Noise2Noise), you will learn the conversion from a noise image to a noise-free image (correct answer data).

Below is a brief explanation. The task of removing noise from the noise image $ \ hat {x} _i $ and the noise-free original image $ y_i $ pair $ (\ hat {x} _i, y_i) $ is to minimize the following experience loss. It is formulated.

\tag{1}
\underset{\theta}{argmin} \displaystyle \sum_i L(f_\theta(\hat{x}_i),y_i)

Where $ f_ {\ theta} $ is the conversion parameter (eg CNN weight) under the condition of the loss function $ L $. On the other hand, in Noise2Noise, the correct image is also a noise image, so the following experience loss where $ y_i $ becomes $ \ hat {y} _i $ is minimized.

\tag{2}
\underset{\theta}{argmin} \displaystyle \sum_i L(f_\theta(\hat{x}_i),\hat{y}_i)

In Noise2Noise, target distribution with input conditionsp(y|x)Parameters to optimize even if is replaced by any distribution with the same expected value\thetaClaims that it will not change. In other words, even if the target is a noise image, the expected value is$E \{ \hat{y} _i | \hat{x} _i \} = y _i Because it is(1)When(2)Obtained when optimizingf _\theta$は等価であるWhenいう事になります(However, the variance of the estimation result is the average of the variance of the noise at the target divided by the number of training samples.)。

This is easy to understand in the context of regression, and if you want to infer the underlying model from the resulting samples, there are innumerable combinations of samples in which the same model can be inferred. Therefore, even if the observed value varies due to noise, if the expected value of the variation is the same as the original sample ($ E \ {\ hat {y} _i | \ hat {x} _i \ } = y _i $), the model obtained by minimizing the loss from the sample is the same as the model obtained by minimizing the loss in the noise-free image. This is the reason why it is possible to learn the conversion from a noise image to a noise-free image (correct answer data) by learning the conversion from a noise image to a noise image (Noise2Noise).

Model building

Training uses noise images for input and target respectively.

The model uses Win5-RB and uses the same learning parameters as in Demo 1.

result

As a test, the denoising result from a noise image with Gaussian noise added is as follows.

Even though I only use noise images for learning, the noise is surprisingly removed. When I first read the dissertation, I was skeptical, but I was quite surprised to see this result. In this way, N2N can remove noise even in situations where it is difficult to obtain correct answer data, so it can be seen that it is an epoch-making technology that greatly expands the range of usable scenes.

In the paper, it is reported that the performance was comparable to the learning method using noise-free images (correct answer data), but looking at the MSE value for 10,000 test images, demo 1 using the correct answer data. It is a high value compared to.

For consideration, let's look at the histograms of the pixel values of the images during training and testing (0 corresponds to black and 1 corresponds to white pixel values).

Since the target image during training in Demo 1 is the original image without noise, there are many 0s in the pixel value, while the target image in Demo 2 is a noise image, so the number of 0s decreases due to noise, and the pixels. You can see that the values are evenly distributed. Looking at the estimated image, Win5-RB in Demo 1 shows a distribution close to the test image (a distribution where 0 and 1 are clearly separated), but the estimated image in Demo 2 is centered from 0 and 1. The distribution is based on. In other words, in learning from a noise image to a noise image, it is understood that this result is obtained because it contributes to the minimization of the loss function by giving a value that is close to each other rather than giving 0 or 1 clearly. I will. The handwritten character image used this time is close to a binary image, and the bias is extremely large, so it is inferred that such a result was obtained. Looking at the image of the actual estimation result of Demo 2, it can be seen that the image is a little thin as a whole, although the shape is firmly grasped, and the distribution of pixel values is like a natural image. It seems that higher performance will come out when it is gentle on the left and right.

Summary

We introduced the application of Deep Learning to the field of Image Denoising, and performed noise reduction in cases where learning is based on correct images without noise and cases where only noise images are available, which are close to actual problems. This time, we focused on Gaussian noise, but the paper also shows that it is an effective method for salt and pepper noise and other noises.

This time, we removed noise, but I would like to introduce not the level of noise, but also the restoration by deep learning when there is a large defect in the image.