[PYTHON] Try to edit a new image using the trained StyleGAN2 model

1.First of all

In the previous ** "Does Yui Aragaki live in the latent space of StyleGAN2?" **, ** Learned StyleGAN2 model ** It was found that ** new images ** that are not used for learning also have ** high image generation ability **.

This time, I would like to use the ** trained StyleGAN2 model ** to see how much ** new images that are not used for training can be edited **.

The code was created using ** Google Colab ** and posted on ** Github **. Please try to move it.

2. What is Style Mixing?

StyleGAN does not generate an image from one latent variable like GAN so far, but uses a Mapping network to generate an image from 18 latent variables w (this is called style). Taking advantage of this feature, editing called ** Style Mixing ** becomes possible.

スクリーンショット 2020-07-08 13.40.30.png

The 18 latent variables ** w0 to w17 ** are layers with 9 resolutions (4 x 4, 8 x 8, 16 x 16, 32 x 32, 64 x 64, 128 x 128, 256 x 256, 512. Two are connected to x512, 1024 x 1024).

Latent variables affect image generation differently depending on the resolution. At low resolution, it affects the big picture such as face orientation, face shape, and hairstyle, and as the resolution increases, it affects details such as eyes and mouth. I will.

Here, you can mix the features of the two ** images ** by swapping only the part of the image A and B with the ** latent variable w **, which is called ** Style Miximg **. is.

3. Face orientation

スクリーンショット 2020-07-08 13.32.06.png This image shows the result of replacing the latent variables ** w0, w1 ** in the blue frame ** Row_pic ** with those in the red frame ** Col_pic **. ** w0 and w1 mainly affect the orientation of the face and the presence or absence of glasses **, so only the orientation of the face can be changed independently.

4. Laughter

スクリーンショット 2020-07-08 13.52.42.png This image is the Row pic ** w4, w5 ** replaced with Col_pic. It is mainly w4, w5 ** that affect the shape of the mouth, which is the point of laughter. It feels like the way you open your mouth is moving as it is, so you can edit the nuances of how you laugh.

5. Glasses

スクリーンショット 2020-07-08 14.15.56.png This image is Row_pic with ** w0, w1, w2 ** replaced with Col_pic. ** It is mainly w0, w1, w2 ** that affect the glasses. Since it covers the direction of the face, the direction of the face also changes at the same time.

What's interesting is that the shape of the glasses doesn't move as it is, but it seems that Row_pic has ** individual glasses attributes **. Therefore, it is difficult to wear glasses of the intended shape.

6. Youth

スクリーンショット 2020-07-08 14.26.43.png This image is Row_pic with ** w4, w5, w6, w7 ** replaced with Col_pic. ** Leave w2, w3 ** affected by face shape and hairstyle, and replace only ** w4, w5 ** affecting mouth shape and ** w6, w7 ** affected eye shape I have. If you change the shape of your eyes and mouth, you will be quite young.

7. Get older

スクリーンショット 2020-07-08 14.38.48.png As before, w2 and w3, which affect the shape of the face and hairstyle, are left as they are, and only w4, w5, w6, and w7, which affect the shape of the mouth and eyes, are replaced **. Just changing the shape of the eyes and mouth makes it look a little older.

8. Virilization

スクリーンショット 2020-07-08 14.43.20.png This is a bonus. Similarly, only w4, w5, w6, w7 are replaced, but the result is not good (laughs).

9. Summary

スクリーンショット 2020-07-08 16.34.55.png

This is a rough summary of the relationships between the main latent variables w related to the elements of the face image. ** w8 and above only affect ** such as contrast and color, and do not seem to directly affect the shape of the face.

I think that the trained StyleGAN2 model has ** high image editing ability ** even for ** new images **.

(reference) Play StyleGAN !! ~ Image editing without additional learning ~

Recommended Posts

Try to edit a new image using the trained StyleGAN2 model
Try to model a multimodal distribution using the EM algorithm
Try a similar search for Image Search using the Python SDK [Search]
Try to infer using a linear regression model on android [PyTorch Mobile]
[NNabla] How to add a quantization layer to the middle layer of a trained model
Try adding fisheye lens distortion to the image
How to code a drone using image recognition
Reinforcement learning 10 Try using a trained neural network.
Try to create a new command on linux
Until you try to let DNN learn the truth of the image using Colab
Created a fooling image for the caption generative model
How to write a GUI using the maya command
(Python) Try to develop a web application using Django
Try loading the image in a separate thread (OpenCV-Python)
[Python] Mask the image into a circle using Pillow
I tried to compress the image using machine learning
Create a web page that runs a model that increases the resolution of the image using gradio, which makes it easy to create a web screen
I tried to predict the infection of new pneumonia using the SIR model: ☓ Wuhan edition ○ Hubei edition
I tried using PI Fu to generate a 3D model of a person from one image
Implementation of VGG16 using Keras created without using a trained model
Try to evaluate the performance of machine learning / regression model
[python] Change the image file name to a serial number
Try to evaluate the performance of machine learning / classification model
I made a function to check the model of DCGAN
How to generate a query using the IN operator in Django
I made a VGG16 model using TensorFlow (on the way)
Convert the image data (png) at hand to a .pbm image
[Introduction to Tensorflow] Understand Tensorflow properly and try to make a model
Try using Jupyter's Docker image
Dot according to the image
Try using the Twitter API
Try using the Twitter API
Try using the PeeringDB 2.0 API
Try to select a language
The image is a slug
Try to generate a death metal jacket image with DCGAN + scrape the metal database site for that
I tried to predict the number of domestically infected people of the new corona with a mathematical model
Introduction to Deep Learning for the first time (Chainer) Japanese character recognition Chapter 3 [Character recognition using a model]
[Python] Try to graph from the image of Ring Fit [OCR]
I tried to implement anomaly detection using a hidden Markov model
Try to extract a character string from an image with Python3
[Go] Create a CLI command to change the extension of the image
The simplest way to build a Spleeter usage environment using Windows
Try to model the cumulative return of rollovers in futures trading
Try to write a program that abuses the program and sends 100 emails
Try to operate the database using Python's ORM Peewee (August 2019 version)
Evaluate the performance of a simple regression model using LeaveOneOut cross-validation
I tried to cut out a still image from the video
Try using Elasticsearch as the foundation of a question answering system
Try to solve the function minimization problem using particle swarm optimization
Generate a Docker image using Fabric
Try to draw a Bezier curve
Try using pynag to configure Nagios
Try to introduce the theme to Pelican
Try to get statistics using e-Stat
Convert a string to an image
Try using the Python Cmd module
Cython to try in the shortest
Try blurring the image with opencv2
Creating a learning model using MNIST
The fastest way to try EfficientNet