[PYTHON] A beginner tried coloring line art with chainer. I was able to do it.

Deplaning is tai2an, which is mainly in charge of work at P-FN, which is said to be commoditized and soldering has higher added value. At the Amazon Picking Challenge, which was broadcast nationwide on NHK, I made a sticky hand and a slide full of Robocon feeling. However, I wanted to do some deep learning, so I started studying chainer a month or two ago. I wanted to color the line art because it was a big deal, so I tried various things.

Line art coloring is supervised learning, so you need a dataset of line art and colored images (preferably in large quantities). This time, OpenCV is used to properly extract line art from color images.

Extraction example →　

Collect color images to create a line art and you have a dataset. (I'm using about 600,000 this time)

Regarding the form of the network, I used a network called U-net, which is used by merging the output of the layer when convolving at the beginning and deconvolving at the end. It seems to be suitable when referring to a line art and coloring it. (U-net is also used in px2px) For the time being, I tried to learn to minimize the square of the error from the correct answer. I understand that the network definition itself is generally OK if the output of the layer and the input of the next layer are matched, but it is a bit difficult to understand because there was not much way to create a dataset that you define yourself. I thought it was.

Here is the result of studying overnight and feeding test data ↓

Well, I can feel the feeling of the neural network, "I can understand the skin color somehow, but I don't know anything else, I can't guess the color of the hair or clothes of the illustration character." .. ..

That's where the Adversary Net comes in. Known as Ad Auntie. Auntie is a guy who learns the difference between a real image and a colored image of a neural network and stingy. So, if you put out only skin-colored sepia-like images, you will learn in one shot and you will end up eating bad things.

However, if the ad woman is too strong, the colored side with the bent navel will be blurred, so be careful.

With this kind of feeling, the colors are surely attached, but it becomes an art system as if it was a line art. (Well, this may be an art path like this,

If you turn the learning while paying attention to the balance with the difference component from the original image. .. ..

It's getting colored! Huh

It ’s even better! I also learned a network that reduces the 512x512px line art, paints 128x128px in the first stage, and colors 512x512px in the second stage (the network shape is almost the same in the second stage, but the input is 4ch and it is new. Retrain. This is no ad woman)

The result is here ↓

(I borrowed this line art published by @ lio8644.)

It's okay.

It's not bad.

As long as I use the extracted line art for testing, it looks pretty good, but what about the actual line art?

I borrowed a line art from a tag of pixiv's painting system and tested it. (Basically all are CNN, so even if the aspect ratio changes a little, it can be handled.

Denden

great.

It became a colorful monster, but there are some like this.

Feeling safely summarized

By the way, after all there is something like wanting you to paint this color here, right? That's why I changed the input of the first stage to 4ch so that I can give additional hints.

Tsumari

You can set requirements such as brown hair, sweater light blue, and skirt navy blue.

If you give a rough impression that you want it to look like this, it will be pretty good.

(If the line art is good, I feel like it's beautiful if I paint it, but w

It is also possible to give a lot of hints in quite detail. (It's a little difficult to understand, but I'm hitting a lot of hints

Merry Christmas! !!

With this, I'm also promoted from a craftsman to a drawing craftsman!

So, I think that the automatic coloring of line art and the coloring with hints have been done considerably.

I don't think I can beat the proper painting of the painters, but I think it's convenient to paint roughly. It would be convenient if it was faster to color the manga roughly than to paste the tone. (This neural network is strong for skin color ... Do you know what you want to say?)

As a side note, there are still some weaknesses.

For example, since you are learning Adversary Net and hints at the same time, the ripple effect when giving hints may become unstable.

↑ I wanted only the swimsuit to be painted in a different color, but the colors of the other parts have also changed significantly. If you use it only as an easy coloring tool, it may be more stable to learn with hints only.

Also, since the color is painted after reducing it once, even if the line art is too thick / thin, there are cases where the line is crushed or skipped and it does not work, or even if you instruct fine painting with hints In some cases, it will not be reflected.

It would be cool if one NN could handle all the details, but it seems necessary to make adjustments for each application when using it as a tool.

Line art borrower http://www.pixiv.net/member_illust.php?mode=medium&illust_id=31274285 http://www.pixiv.net/member_illust.php?mode=manga&illust_id=43369404 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=56689287 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=40487409 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=10552795 https://twitter.com/lio8644

• As for the extracted line art, I could not find the original reference immediately, so I will omit it. I'm sorry

By the way, the network structure this time is the same for both the first and second stages, and it looks like this

unet.py

    class UNET(chainer.Chain):
        def __init__(self):
            super(UNET, self).__init__(
                c0 = L.Convolution2D(4, 32, 3, 1, 1),
                c1 = L.Convolution2D(32, 64, 4, 2, 1),
                c2 = L.Convolution2D(64, 64, 3, 1, 1),
                c3 = L.Convolution2D(64, 128, 4, 2, 1),
                c4 = L.Convolution2D(128, 128, 3, 1, 1),
                c5 = L.Convolution2D(128, 256, 4, 2, 1),
                c6 = L.Convolution2D(256, 256, 3, 1, 1),
                c7 = L.Convolution2D(256, 512, 4, 2, 1),
                c8 = L.Convolution2D(512, 512, 3, 1, 1),

                dc8 = L.Deconvolution2D(1024, 512, 4, 2, 1),
                dc7 = L.Convolution2D(512, 256, 3, 1, 1),
                dc6 = L.Deconvolution2D(512, 256, 4, 2, 1),
                dc5 = L.Convolution2D(256, 128, 3, 1, 1),
                dc4 = L.Deconvolution2D(256, 128, 4, 2, 1),
                dc3 = L.Convolution2D(128, 64, 3, 1, 1),
                dc2 = L.Deconvolution2D(128, 64, 4, 2, 1),
                dc1 = L.Convolution2D(64, 32, 3, 1, 1),
                dc0 = L.Convolution2D(64, 3, 3, 1, 1),

                bnc0 = L.BatchNormalization(32),
                bnc1 = L.BatchNormalization(64),
                bnc2 = L.BatchNormalization(64),
                bnc3 = L.BatchNormalization(128),
                bnc4 = L.BatchNormalization(128),
                bnc5 = L.BatchNormalization(256),
                bnc6 = L.BatchNormalization(256),
                bnc7 = L.BatchNormalization(512),
                bnc8 = L.BatchNormalization(512),

                bnd8 = L.BatchNormalization(512),
                bnd7 = L.BatchNormalization(256),
                bnd6 = L.BatchNormalization(256),
                bnd5 = L.BatchNormalization(128),
                bnd4 = L.BatchNormalization(128),
                bnd3 = L.BatchNormalization(64),
                bnd2 = L.BatchNormalization(64),
                bnd1 = L.BatchNormalization(32)
        )

    def calc(self,x, test = False):
        e0 = F.relu(self.bnc0(self.c0(x), test=test))
        e1 = F.relu(self.bnc1(self.c1(e0), test=test))
        e2 = F.relu(self.bnc2(self.c2(e1), test=test))
        e3 = F.relu(self.bnc3(self.c3(e2), test=test))
        e4 = F.relu(self.bnc4(self.c4(e3), test=test))
        e5 = F.relu(self.bnc5(self.c5(e4), test=test))
        e6 = F.relu(self.bnc6(self.c6(e5), test=test))
        e7 = F.relu(self.bnc7(self.c7(e6), test=test))
        e8 = F.relu(self.bnc8(self.c8(e7), test=test))

        d8 = F.relu(self.bnd8(self.dc8(F.concat([e7, e8])), test=test))
        d7 = F.relu(self.bnd7(self.dc7(d8), test=test))
        d6 = F.relu(self.bnd6(self.dc6(F.concat([e6, d7])), test=test))
        d5 = F.relu(self.bnd5(self.dc5(d6), test=test))
        d4 = F.relu(self.bnd4(self.dc4(F.concat([e4, d5])), test=test))
        d3 = F.relu(self.bnd3(self.dc3(d4), test=test))
        d2 = F.relu(self.bnd2(self.dc2(F.concat([e2, d3])), test=test))
        d1 = F.relu(self.bnd1(self.dc1(d2), test=test))
        d0 = self.dc0(F.concat([e0, d1]))

        return d0

Ad Auntie

adv.py

class DIS(chainer.Chain):
    def __init__(self):
        super(DIS, self).__init__(
                c1 = L.Convolution2D(3, 32, 4, 2, 1),
                c2 = L.Convolution2D(32, 32, 3, 1, 1),
                c3 = L.Convolution2D(32, 64, 4, 2, 1),
                c4 = L.Convolution2D(64, 64, 3, 1, 1),
                c5 = L.Convolution2D(64, 128, 4, 2, 1),
                c6 = L.Convolution2D(128, 128, 3, 1, 1),
                c7 = L.Convolution2D(128, 256, 4, 2, 1),
                l8l = L.Linear(None, 2, wscale=0.02*math.sqrt(8*8*256)),

                bnc1 = L.BatchNormalization(32),
                bnc2 = L.BatchNormalization(32),
                bnc3 = L.BatchNormalization(64),
                bnc4 = L.BatchNormalization(64),
                bnc5 = L.BatchNormalization(128),
                bnc6 = L.BatchNormalization(128),
                bnc7 = L.BatchNormalization(256),
        )

    def calc(self,x, test = False):
        h = F.relu(self.bnc1(self.c1(x), test=test))
        h = F.relu(self.bnc2(self.c2(h), test=test))
        h = F.relu(self.bnc3(self.c3(h), test=test))
        h = F.relu(self.bnc4(self.c4(h), test=test))
        h = F.relu(self.bnc5(self.c5(h), test=test))
        h = F.relu(self.bnc6(self.c6(h), test=test))
        h = F.relu(self.bnc7(self.c7(h), test=test))
        return  self.l8l(h)