[PYTHON] What ICCV2019 Best paper SinGAN can't do [Practice]
Introduction
This article is the 18th day entry of CyberAgent20 new graduate engineer Advent Calender 2019. I am a graduate student doing computer vision research using deep learning. twitter → https://twitter.com/revi_matsu This time, it is the ** SinGAN ** practice edition that won the Best paper at the ICCV 2019 held the other day.
How to read this article
[Those who want to get a rough overview] Please read this article. [For those who want to know a little specifications] This article → I read "SinGAN" lightly and summarized it briefly [Those who want to know the details] This article → “SinGAN” was read briefly and summarized → Paper )
What is SinGAN
Please refer to the following article for the explanation about SinGAN. Here is a brief summary of the points. ICCV2019 Best Paper "SinGAN" I read it briefly and summarized it briefly
SinGAN is a GAN-based one-shot learning method. What is amazing is that ** one learning image can be applied to various tasks **. Usually, learning of deep learning, not limited to GAN, requires a large number of learning images. However, of course, due to the difficulty of collecting data, implementation with a small number of learning images has become a theme of high demand, but of course it is not an easy problem.
With SinGAN, it is possible to learn features that could not be extracted so far. The figure below is an example of Random sampling (learning with one learning image and generating a similar sample image from noise). In the conventional PSGAN and Deep Texture Synthesis, only the ** texture (atmosphere) ** of the training image is retained and the image is generated, but it can be seen that the shape information of the building or building cannot be retained. On the other hand, it can be confirmed that SinGAN also reproduces something like a building.
SinGAN can be applied to other tasks at a high level, and a large number of experiments are described in the paper.
This time, in addition to actually conducting an experiment from the author github, the practical edition based on the case of applying it to a task not mentioned in the paper I wish I could introduce.
Practical edition
The code uses here. The code uses python3.6 and the framework uses pytorch. This time, I applied it to the following two tasks. ・ Animation (Introduced in the paper) ・ Semantic segmentation (not introduced in the paper)
Animation Here, a gif is generated from one image. In the paper, the example of lightning was used, but this time I was writing an article about Christmas at Advent Calender the other day, so I want to ride it. ・ @ Dai7Igarashi, Make something like Christmas with FramerMotion! -Android animation implementation using @ airagu950, Lottie
So, I will create a Christmas-like animation from one learning image. </ Font>
Learning image
Generated animation
You can see how snowy things are randomly generated while maintaining the style. Let's use a sample that can confirm whether the style is maintained a little more.
Learning image
Generated animation
** It can be confirmed that the snowman retains its shape **. In this way, information with large features in the image is highly feature-preserved, and conversely, with a small feature amount like snow Due to the random noise of the input when Random sampling You can see that small changes occur. (I don't care about how it's snowing.) Don't you think this is really interesting? ??
Semantic Segmentation Semantic Segmentation is a task not proposed in the paper, and it is considered very difficult to realize this task with One-shot. The main reason for this is that Segmentation generally uses a method of assigning a label determined for each pixel, and that it does not learn the conversion of an image to a segmentation image by an in-vehicle camera. ** SinGAN is good at extracting features from teacher images, and there is a high probability that this part of the image that will be the input during the test is 〇〇! I have not learned that. ** **
However, this time I wanted to check how much feature extraction (I feel that the expression of feature retention is more appropriate), so I applied original SinGAN to Semantic Segmentation. ..
Simply put, ** image to image ** is applied directly. This time, I used the in-vehicle camera data set and the cell image data set.
The actual segmentation result is as shown in the figure below. Most of the predictions were made, but it seems difficult to apply segmentation directly to tasks. However, in the cell image, some parts where the cell nucleus can be identified can be confirmed. Data such as cell images, in which the composition of the entire image does not change easily from image to image, may indicate the possibility of segmentation to some extent.
In any case, this article simply forcibly drops the Segmentation into the existing SinGAN model.
I look forward to future development.
in conclusion
This article was a practical version of SinGAN. There were many tasks that could be done with the original SinGAN, and all of them were of a high level, and some of them did not seem to be one-shot learning. I am looking forward to applying and developing it to other tasks as it seems to be one of the interesting research fields.
Also, this time I posted an article to CyberAgent 20 new graduate engineer Advent Calender 2019 as a candidate for CyberAgent graduated in 2020, but in fact, CyberAgent has started hiring 21 graduate engineers! !! </ font> I decided to join CyberAgent through a long-term internship, but CyberAgent found ** various services and businesses and their connections ** very interesting. I feel that such an environment is a very good place for self-growth. Isn't it exciting to hone your skills and contribute to the company? I decided to join the company hoping that it would be rewarding and my growth. If you are interested in it, please come! A chance to greatly expand your horizons! https://www.cyberagent.co.jp/careers/special/engineer2021/?utm_source=twitter&utm_medium=sns&utm_campaign=social
Recommended Posts