Introduction († Horror conversion †)

Before conversion	After conversion

Did you understand?

This utilizes GAN (Generative Adversarial Network) to give photos ** "horror" ** features and ** convert **.

Let's set aside the principle, and first look at the possibilities of GAN!

Seasonal conversion, aurora conversion, fireworks conversion

Before conversion	After conversion

These are paired with "Summer and Spring", "Clouds and Aurora", "Sky and Fireworks", etc. using CycleGAN [1] ** Features exchanged * We are doing *. It does not just change the color, but the conversion is done while keeping the prototype like "translation (conversion) of the image".

If you want to know more about the principle of CycleGAN, please refer to this article. Thank you very much for my help.

Landscape † Different World Transformation †

This time, I turned this technology into a photo ** "landscape conversion" ** ... No! !!

I used it for ** "† Different World Conversion †" **! !!

Take a look at the four months of struggle and the results.

† Fantasy conversion †

First is † fantasy conversion †. I created and learned a dataset with a pair of "landscape (Kagoshima)" and "fantasy / fantastic". After all, abstract expression was difficult, and it was very difficult to select images.

Before conversion	After conversion

came out! The already beautiful scenery of Kagoshima becomes even more ** fantastic and mysterious **! ?? Is the grain of light "just noise" ...? No, no, it looks like ** "a fairy is flying" ** to my eyes. Overall ** the blue color is very beautiful **.

† Horror conversion †

Next is the † horror conversion † introduced at the beginning.

** * * * Browsing attention * * * * *.

Before conversion	After conversion

Please see this cityscape ...! !! ** Do you have the courage to step in? ** ** I want to use it as a texture for horror games as it is.

Here, I created a data set with a pair of "cityscape" and "horror video" and learned. In other words, I kept showing the play screens of "SIREN" and "Silent Hill" to the learning AI. This is the reason why the cityscape is ** red-black ** overall (I tried several, but the river did not turn red. Unfortunately).

† Undersea city conversion †

Next is † Undersea City-Atlantis-Transformation †. ** "If you mix the characteristics of" underwater "with" landscape ", it will become an" undersea city "? 』** I created a data set and learned from this simple idea.

Before conversion	After conversion

It's wonderful ...! ** The entrance to Ryugu Castle ** …… Can you say that? The tree behind it looks like a coral reef. ** God of water ... Thank you. ** ** † Undersea City-Atlantis-Transformation † collects not only "underwater" images but also "diving" images. I was able to create an excellent dataset.

† Iceberg conversion †

Just the other day, there was a special project of Frozen 2 to "ice-pick Mt. Fuji and Sakurajima". ** What if I do the same with GAN? ** ** I created a dataset with a pair of "Sakurajima" and "Iceberg" and learned!

Before conversion	After conversion

** No, it's overkill. ** ** When I went to see the conversion results with excitement, I was so scared that I cried because it was an ice age.

Is this the rebellion of AI ...? If we created the dataset more carefully and learned the shape of the mountain, the result might not have been so terrifying. If you reopen it with ** † Ice Age Conversion † **, it will be a perfect perfection.

About learning method

Image collection for training datasets

CycleGAN is ** unsupervised learning **. No labeling is required. The great thing about CycleGAN is that even if the background and composition are different, you can learn well if you have the same number. In this conversion, images are collected around ** 1000 each ** to create a data set. The usual 1000 cityscapes are in the form of "train A", and the 1000 streets in another world are in the form of "train B".

That said, I'm not a ** resident of another world **, so I don't have 1000 photos of another world ** in my camera ...

Therefore, I mainly created the training data set by the following two methods.

① Collect images by Google image search

The python library "icrawler" is useful. For details, see This article. Thank you for all the help you have given me.

I searched for images with keywords such as "Aurora" and created a dataset.

② Take a video and cut out the image in frame units

With method (1), noise data is mixed too much depending on the search word, making it very difficult to select learning images. In such a case, it is easier to image the video that is the image of style conversion with OpenCV etc. in frame units. For details, see This article. Thank you for all the help you have given me.

For example, in "horror conversion", I recorded the most scary scenes in horror games for a total of about 30 minutes, and cut out images with a target of 1000 images.

Learn with CycleGAN

It's finally time to study with CycleGAN. It's good to spin it on your own PC, but ** Google Colabratory ** is amazing. It is a service that allows you to use Jupyter Notebook in the cloud. For details, see This article. Thank you for all the help you have given me.

Roughly speaking, it is a cloud IDE that ** lends a GPU for free **. It's amazing. If you're lucky, you can also borrow a Tesla P100 (a GPU for about 900,000 yen) for free. It's amazing. Since it is a cloud IDE, it does not depend on the location or your own machine specifications. All you can study in the background. It's amazing. I was too comfortable and charged Google Drive. I love google.

Source code

https://colab.research.google.com/drive/1KmwC-eOU3Z02ZiCmgFv-rDG4HxzK-8zG ↑ Here are the notes I actually used to learn ** † Different World Conversion † **. Pytorch version of CycleGAN [2] is implemented on Google Colabratory. I am still uncertain about my knowledge, but I hope it will be helpful as I have a memo for myself and a commentary. First of all, please come from the standard "horse → zebra" style conversion of the sample.

Web application release

I will write it soon ngrok is amazing!

References

[1]Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, arXiv preprint arxiv:1703.10593, 2017.

[2]@inproceedings{CycleGAN2017, title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss}, author={Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A}, booktitle={Computer Vision (ICCV), 2017 IEEE International Conference on}, year={2017} } @inproceedings{isola2017image, title={Image-to-Image Translation with Conditional Adversarial Networks}, author={Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A}, booktitle={Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on}, year={2017} }

Landscape † Different World Conversion by AI †