[PYTHON] Write your own activation function with Pytorch (hard sigmoid)

Porting from Chainer to Pytorch

I'm doing deep learning in my research, but the other day I learned that the Chainer update was coming to an end, so I decided to change the framework to Pytorch as well as the developer. As a starting point, I decided to port from the existing Chainer program to Pytorch.

Basically, all I had to do was change the name of the function, but on the way I noticed that Pytorch doesn't have HardSigmoid. So let's make it ourselves.

I actually wrote it

... but it's written in the official reference, so I just did it almost exactly. --> https://pytorch.org/docs/master/autograd.html

python

class MyHardSigmoid(torch.autograd.Function):
    @staticmethod
     def forward(ctx, i):
         ctx.save_for_backward(i)
         result = (0.2 * i + 0.5).clamp(min=0.0, max=1.0)
         return result

     @staticmethod
     def backward(ctx, grad_output):
         grad_input = grad_output.clone()
         result, = ctx.saved_tensors
         grad_input *= 0.2
         grad_input[result < -2.5] = 0
         grad_input[result > -2.5] = 0
         return grad_input

If you do not write @staticmethod, a warning will appear. The official one is an exponential function, but we will change it to a hard sigmoid.

First, forward () is forward-propagated. hard sigmoid () has the following formula, so I wrote it so.

h(x) = \left\{
\begin{array}{ll}
0 & (x \lt -2.5) \\
0.2x + 0.5 & (-2.5 \leq x \leq 2.5)  \\
1 & (2.5 \lt x)
\end{array}
\right.

Then with backward (), this writes backpropagation. The differential coefficient is as follows.

\frac{\partial h(x)}{\partial x} = \left\{
\begin{array}{ll}
0 & (x \lt -2.5) \\
0.2 & (-2.5 \leq x \leq 2.5)  \\
0 & (2.5 \lt x)
\end{array}
\right.

And finally apply this to the model. (The contents of the model are appropriate.)

model.py

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        hard_sigmoid = MyHardSigmoid.apply
        return hard_sigmoid(self.conv2(x))

This is perfect! !! ... should be