[PYTHON] Shakedrop's Keras implementation

Introduction

Algo Age Forest: grinning: Here is a slide summarizing the papers around Shakedrop and an implementation of Keras.

About shakedrop

Slides (announced at DL Hacks)

https://www.slideshare.net/DeepLearningJP2016/dl-hacks-shakedrop-by-keras

Implementation

reference

・ Https://github.com/jonnedtc/Shake-Shake-Keras ・ Https://github.com/owruby/shake-drop_pytorch

shakedrop implementation

shakedrop_model.py

import keras
from keras import Input
from keras import backend as K
from tensorflow import distributions as tfd

class Shakedrop(layers.Layer): #Define a custom layer

    def __init__(self, num_of_unit, num_of_layers, **kwargs):
        super(Shakedrop, self).__init__(**kwargs)
        self.num_of_unit = num_of_unit #What number resblock
        self.num_of_layers = num_of_layers #Number of layers in the entire model

    def build(self, input_shape):
        super(Shakedrop, self).build(input_shape)

    def call(self, x):
        batch_size = K.shape(x)[0]
        alpha = K.random_uniform((batch_size, 1, 1, 1),  minval=-1.0)
        beta = K.random_uniform((batch_size, 1, 1, 1))
        p = 1 - (self.num_of_unit / (2 * self.num_of_layers)) #The closer to the output, the easier it is to shake
        bernoulli = tfd.Bernoulli(probs=p).prob(1)
        
        def x_shake():
            # stop_Switch between forward and backward using gradient
            return (1 - bernoulli) * (beta * x + K.stop_gradient((alpha - beta) * x))

        def x_even():
             #p becomes the expected value of b as it is
            return p * x

        #X when learning_x when shake and test_even
        return K.in_train_phase(x_shake, x_even)

    def compute_output_shape(self, input_shape):
        return input_shape[0]

Built-in

It can be used with any model that has a resnet structure as follows.

resblock.py

return layers.Add(
    [inputs, Shakedrop(num_of_unit=num_of_unit, num_of_layers=num_of_layers)(x)])

At the end

We are currently looking for human resources! There are various internships with training that you can learn machine learning from scratch, so please feel free to contact us if you are interested: relaxed:

Wantedly: https://www.wantedly.com/companies/company_5667111/projects Homepage: https://www.algoage.net/ Twitter: https://twitter.com/algoage