[PYTHON] What you can and cannot do with Tensorflow 2.x

Tensorflow 2.x I don't know anything

As a result of implementing the method with TF 2.x, he is a person who is on fire. It's been about a year since TF 2.x came out, but I will summarize what TF 2.x has done, what it hasn't done, and the methods that have recently come to light.

Can you do this? Isn't this way wrong? If there is something like this, please point it out and correct it. In the future, those who suffer from this ~~ shit ~~ framework will be saved.

What you can do with Tensorflow 2.x

PyTorch, Keras-like writing

With the availability of Eager Execution, tf.session has been deprecated and the interpreter can now execute tf functions. This made debugging easier and also easier to run.

Also, due to the flow of integration with keras, implementation using tf.keras.layer (nn.Module? In PyTorch) has been promoted. On the other hand, existing functional implementations are now aggregated under tf.nn, for example BatchNormalization is tf.nn.batch_normalization.

Use of TPU

It seems that the use of TPU is becoming more convenient. Articles around here is very easy to read, so please read it.

What you can't do with Tensorflow 2.x

~~BatchNormalization~~ You may think that Batch Normalization is not possible, but unexpectedly this layer has many holes and it is difficult to find a proper implementation (I wonder if there is no proper implementation yet ...)

What's particularly problematic is that it can't handle operations on multiple GPUs / TPUs **, so this one is PyTorch. #syncbatchnorm) and MxNet have been implemented, but TF 2.x has not been implemented yet. The time when I relied on horovod for MultiGPU / TPU (TF 1.x series) was realized from the support of that person, but strangely I tried to implement it inside TF It is postponed //github.com/tensorflow/community/blob/master/rfcs/20181016-replicator.md#global-batch-normalization).

This problem seems to be annoying throughout the TF 2.x family, as many researchers escape to PyTorch. ) It's a tough problem.

The latest issue currently open is probably here so please join us (I'm also digging).

--2020/2/3 The layer itself has been added. However, it is doubtful that it has been tested, so don't expect too much.

Multi GPU / TPU Arithmetic (WIP)

For the time being, TF 2.x can calculate with multiple GPUs / TPUs! That said, the reality is that it's full of Experimental support, as you can see in Official. Therefore, there is a high possibility that the rolling function will change, and it is difficult to expect backward compatibility.

Initialization of variables with initial batch data

Initialization with initial batch data, which is rarely used in deep learning, cannot be done as long as this is also observed. The way to do this is to rewrite the layer weights directly from the outside according to the initial batch, which requires designing an external function to initialize and a mechanism to call that external function during training. This makes it difficult to use the usual tf.keras.Model and tf.keras.Sequential (at least fit () cannot be used and must be trained in a custom training loop). (Example)

class IdentityWithInit:
   def build(self, input_shape: tf.TensorShape):
        self.initialized = self.add_weight(
        self.built = True
   def initialize_parameter(self, x: tf.Tensor):
        tf.print("initialized {}".format(self.name)) 

   def __init__(self):

   def call(self, x:tf.Tensor):
       if not self.initialized:
       return x

Gradient calculation (2.0.0)

In 2.0.0, the gradient calculation seems to accident in a special case. (I thought it was sane, but it seems to be apt)

But 2.1.0 Has been fixed. (I also tested it at hand, but it was fixed.)

Use of slashes

It's rarely used when writing very long sentences in Python, but Tensorflow doesn't seem to use Python's parsing tree (?), So I can't use it. https://github.com/tensorflow/tensorflow/issues/35765

Bad example

variable * decay * \ 

Good example

(variable * decay * 

Using Python 3.8

Support for Python 3.8 seems to start in mid-2020/02. (It seems that you can try it by doing a source build)


How to use Tensorflow 2.x

Creating a Task class

I've noticed this recently so I don't know everything, but apparently the official [some implementations](https://github.com/tensorflow/models/blob/master/official/transformer/v2/ As far as transformer_main.py) is seen, it seems that it is recommended to create a class for training and inference separately from the model class.

Basically, the functions to be implemented are training, test train,test (eval)and xxx_step, and for xxx_step, it is a function that processes one batch and tf.function decorator. The pattern surrounded by is often seen.

class MyTask:
  def __init__(self, args):
     self.loss = tf.metrics.Mean(name='loss', dtype=tf.float32)

  def train(self):
    def train_step(x: tf.Tensor, y: tf.Tensor):
      _y = self.model(x)
      loss = loss_fn(_y, y)

    for epoch in range(self.epochs):
      for x, y in tqdm(self.train_dataset):
        train_step(x, y)
      for x, y in self.val_dataset:
        val_step(x, y)
      print('EPOCH {} train: loss {} / val: loss {}'.format(epoch + 1, 
         self.loss.result(), self.val_loss.result()))
  def test(self):
    def test_step(x: tf.Tensor, y: tf.Tensor):

custom Layer ** kwargs

I'm writing a Tensorflow class What is this argument? I thought, but this is a more important argument than I expected, and it is very useful when switching inference / training in the BatchNormalization layer or Dropout layer, for example.

For example

class CustomLayer(Layer):
  def __init__(self):
    self.conv = Conv2D(...)
    self.bn = BatchNormalization(...)

  def build(self, input_shape):

  def call(self, x: tf.Tensor, **kwargs):
    y = self.conv(x, **kwargs)
    y = self.bn(y, **kwargs)

cl = CustomLayer()

BatchNormalization can be executed in inference mode by setting cl (x, training = False).

Recommended Posts

What you can and cannot do with Tensorflow 2.x
Python | What you can do with Python
What you can do with API vol.1
What you can do with programming skills
What you can do with the Python standard library statistics
Consider what you can do with Python from the Qiita article
You will be an engineer in 100 days --Day 35 --Python --What you can do with Python
Links to do what you want with Sublime Text
What is pip and how do you use it?
What you can't do with hstack or vstack with dstack
Until you can do simple image recognition with Jupyter
What to do if you run python in IntelliJ and end with an error
[AWS] What to do when you want to pip with Lambda
What to do if you can't sort files with subscripts
[Python] What do you do with visualization of 4 or more variables?
Real-time object detection Android app with TensorFlow and Camera X
What to do with Magics install
What to do if you can't install pyaudio with pip #Python
What to do if you get a UnicodeDecodeError with pip install
What to do with PYTHON release?
What to do if you can't build your project with Maven
What to do if you get the message "" ~ .pkg "is corrupted and cannot be opened" when installing wxPython on Mac OS X
What to do if you can't find well with grep's -f option
You can also check the communication of DB and cache with curl
What to do if you couldn't send an email to Yahoo with Python.
What to do if you get angry in TensorFlow v2 without attribute'app'
What to do if you get a TypeError with numpy min, max
Until you can borrow VPS with Conoha and authenticate public key with SSH
From pandas1.0.0, you can specify S3 / GCS URL with read_pickle () and to_pickle ()!
You can do it with Python! Structural analysis of two-dimensional colloidal crystals
What to do if you get Could not fetch URL 443 with pip
Learn data distributed with TensorFlow Y = 2X
What do you roughly say "sudo su"?
Compare raw TensorFlow with tf.contrib.learn and Keras
Data analysis, what do you do after all?
Let's summarize what you want to do.
You can do it in 5 minutes !? Create a face detection API with FastAPI and OpenCV and publish it on Heroku
What to do if you get angry with swapon failed: Operation not permitted
If you guys in the scope kitchen can do it with a margin ~ ♪
[For beginners] You can do it from scratch! Creating APIs with AWS SAM and outputting OpenAPI documentation in Python
Until you CI what you made with Django with Jenkins
Multivariate LSTM and data preprocessing in TensorFlow 2.x
Can you challenge Kaggle with just your iPad?
You can manage overlapping characters with multiple plt.text.
What to do if mod_fcgid cannot resolve UnicodeEncodeError
You can easily create a GUI with Python
What to do if pip cannot be installed
What should I do with DICOM in MPEG2?
You can now display slides with nbviewer! [Jupyter]
Self-made window manager (Do you have an X window manager? 2)
If you get lost with HTTP redirects 301 and 302
What are you using when testing with Python?
What to do if you get an OpenSSL error when installing Python 2 with pyenv
What to do if pvcreate produces a lot of WARNING and cannot be created
Until you can install blender and run it with python for the time being
How to install and use pyenv, what to do if you can't switch python versions
What to do if you get an Import Error when importing matplotlib with Jupyter