[PYTHON] [DeepFake] Finally, my dog ​​started talking

① Introduction

If there is any output, it's fun ** to feel like doing AI. This time, I would like to try the topic ** DeepFake **.

② What is Deep Fake?

** DeepFake ** is a coined word that combines ** deep learning ** and ** fake **, and is a ** human image composition technology ** based on artificial intelligence. If you look at Sample, you can see the fun, but this time I would like to use " first-order-model "to make the dog of the still image speak. I will.

③TRY Let's do it now! As for the environment, the GPU can be used to run the program, so ** Google Colab ** and data storage are done with ** Google Drive **.

■ Advance preparation

Please use ** Google Drive **. As a preliminary preparation, various data (checkpoints, videos, still images) are stored in ** Google Drive **.

➊ Create data storage folder

Create a " first-order-motion-model "folder directly under My Drive in ** Google Drive **.

➋ Checkpoint DL

Store the shared "trained checkpoints" in your ** Google Drive **. Just open the checkpoint link and drag and drop it onto the " first-order-motion-model "you created earlier. This method actually creates a shortcut, but that's okay.

Click here to download ▶ Google Drive or Yandex Disk

If you use the shortcut, an error may occur when reading the checkpoint, but this is because the download from the shared folder is restricted. In that case, wait about a day and try again. If you don't like this, download it to your local PC and upload the actual file to ** Google Drive **. The actual file is about 700MB in size.

➌ Storage of video data and still image data

Shooting and trimming can be done easily with a mobile phone.

-Take a still image of the dog you want to talk to. —— Shoot a video of a person talking. (It will be more interesting if you snap your eyes or shake your head vertically and horizontally)

Place the video and still images cropped into squares in the " first-order-motion-model "folder. The file name should be as follows.

--Video: movie001.mp4 --Still image: pic001.png

■ DeepFake processing

Please do it with ** Google Colab **. ** DeepFake ** processing is performed using the data stored in ** Google Drive **.

➊git clone First, git clone " first-order-model ".

.s

!git clone https://github.com/AliaksandrSiarohin/first-order-model

➋ Move current directory

Move to " first-order-model ".

.s

%cd first-order-model

➌ Resize

To resize still images and videos to 256x256, do the following:

import imageio
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from skimage.transform import resize
from IPython.display import HTML
import warnings
warnings.filterwarnings("ignore")

source_image = imageio.imread('/content/drive/My Drive/first-order-motion-model/pic001.png')
reader = imageio.get_reader('/content/drive/My Drive/first-order-motion-model/movie001.mp4')


#Resize image and video to 256x256

source_image = resize(source_image, (256, 256))[..., :3]

fps = reader.get_meta_data()['fps']
driving_video = []
try:
    for im in reader:
        driving_video.append(im)
except RuntimeError:
    pass
reader.close()

driving_video = [resize(frame, (256, 256))[..., :3] for frame in driving_video]

def display(source, driving, generated=None):
    fig = plt.figure(figsize=(8 + 4 * (generated is not None), 6))

    ims = []
    for i in range(len(driving)):
        cols = [source]
        cols.append(driving[i])
        if generated is not None:
            cols.append(generated[i])
        im = plt.imshow(np.concatenate(cols, axis=1), animated=True)
        plt.axis('off')
        ims.append([im])

    ani = animation.ArtistAnimation(fig, ims, interval=50, repeat_delay=1000)
    plt.close()
    return ani
    

HTML(display(source_image, driving_video).to_html5_video())

➍ Checkpoint reading

Load pre-trained checkpoints.

from demo import load_checkpoints
generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml', 
                            checkpoint_path='/content/drive/My Drive/first-order-motion-model/vox-cpk.pth.tar')

➎ Deep Fake creation

Create ** DeepFake **. The created video will be "../generated.mp4".

from demo import make_animation
from skimage import img_as_ubyte

predictions = make_animation(source_image, driving_video, generator, kp_detector, relative=True)

#save resulting video
imageio.mimsave('../generated.mp4', [img_as_ubyte(frame) for frame in predictions], fps=fps)
#video can be downloaded from /content folder

HTML(display(source_image, driving_video, predictions).to_html5_video())

④ Above

Thank you for your hard work. It's super fun because it feels like it's done when there is some output. I'm surprised that ** DeepFake ** can be done so easily. Everything is ** power ** of " first-order-model ".