This article is the 20th day article of TensorFlow 2.0 Advent Calendar 2019. I think the major change in TensorFlow 2.0 is that EagerExecution is now the default, allowing you to write in imperative languages and writing more freely in Pythonic style. However, on the other hand, there is a problem that performance and portability are sacrificed, but in order to solve it so that you can benefit from both Graph mode in 1.x and Eager mode in 2.x. Introduced in tf.function. In this article, I'll introduce how to use tf.function and what you should know when using it. Basically, it is a summary of Official Site, so if you want to know more, please refer to that as well.
It's easy to use, either add a collator with @ tf.function to the function that describes the heavy processing you want to optimize, or define a function and feed it to the tf.function method to add a function for Graph mode separately. It is a method of creating.
function_test.py
import tensorflow as tf
@tf.function
def f(x,y):
return x + y
#Or
def g(x,y):
return x * y
h = tf.function(g)
Also, if you call another function in @ tf.function, the scope extends to that function, so you don't have to bother checking all the functions and adding @ tf.function (or rather, we recommend it). Function). Therefore, it seems that you can easily benefit from Graph mode just by adding it to the heavy processing part for the time being. However, if it is a simple writing style like the one in the tutorial, this is not a big deal, but if you try to do something a little complicated, you will not think that you do not know the specification of tf.function. You need to be careful to do this. At the beginning of Official Site, there is the following description.
--Don't rely on Python's unique behavior like Object Mutation or Python's list
--tf.function performs best with TensorFlow Ops rather than with Numpy or Python primitive types.
--If in doubt, write for x in y
There are some items that are difficult to interpret, but I think it's easier to understand if you look at a concrete example, so let's take a look.
First of all, for the sake of simplicity, prepare the following simple function.
tracing_test.py
import tensorflow as tf
@tf.function
def double(a):
print("Tracing with, {}".format(a) )
return a + a
It is a simple function that doubles the input argument and returns it, and also includes the process of printing the input argument. The argument works with integers, real numbers, and strings. Let's do this with some patterns.
tracing_test.py
print(double(3))
print()
print(double(2))
print()
print(double(tf.constant(1.1)))
print()
print(double(tf.constant('a')))
print()
print(double(tf.constant('b')))
The result is as follows.
Tracing with, 3
tf.Tensor(6, shape=(), dtype=int32)
Tracing with, 2
tf.Tensor(4, shape=(), dtype=int32)
Tracing with, Tensor("a:0", shape=(), dtype=float32)
tf.Tensor(2.2, shape=(), dtype=float32)
Tracing with, Tensor("a:0", shape=(), dtype=string)
tf.Tensor(b'aa', shape=(), dtype=string)
tf.Tensor(b'bb', shape=(), dtype=string)
The result is a little strange. The print statement is not executed only as a result of executing the last tf.constant ('b') as an argument. If you try running the above program again, you will get even more strange results.
tracing_test.py
print(double(3))
print()
print(double(2))
print()
print(double(tf.constant(1.1)))
print()
print(double(tf.constant('a')))
print()
print(double(tf.constant('b')))
The result is as follows.
tf.Tensor(6, shape=(), dtype=int32)
tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor(2.2, shape=(), dtype=float32)
tf.Tensor(b'aa', shape=(), dtype=string)
tf.Tensor(b'bb', shape=(), dtype=string)
The correct value will be returned, but the print statement written in the middle will not be executed at all. What does this mean?
Tracing
In fact, this strange behavior involves a process called Tracing when tf.function builds and optimizes a function on a computational graph. tf.function converts a function that describes not only TensorFlow-derived but also Python-specific processing into a computational graph. And it omits the Python-specific processing (print statement in this case) that is not actually related to the execution of the calculation graph. But why was it done at first? That's because when tf.function converts a function into a computational graph, a process called Tracing runs. Functions written in Python have no explicit type in their arguments. Therefore, while it is convenient to be able to enter various values, it is a problem from the tf.function side trying to create an optimal calculation graph. Therefore, when a value or type that has not been entered in the argument is input and the function is called for the first time, a process called Tracing is performed to execute all Python-specific processing in the function. I said "values and types that have never been included in the arguments", but strictly speaking, the criteria are as follows.
--In the case of Python primitive types, Tracing when different values come in --In the case of Python objects, if you get an object with a different id, Tracing --In the case of Tensor derived from TensorFlow, if a different type or shape comes in, Tracing
Therefore, the strange behavior of Karakuri is as follows.
tracing_test.py
print(double(3)) #Tracing because it is the value I see for the first time
print()
print(double(2)) #Tracing because it is the value I see for the first time
print()
print(double(tf.constant(1.1))) #Tracing because it is the value I see for the first time
print()
print(double(tf.constant('a'))) #Tracing because it is the first shape to see
print()
print(double(tf.constant('b'))) #Optimized graph execution because it is the type shape we saw before
Once you run Tracing, TensorFlow saves the resulting computational graph internally. Then, the next time a previously traced value or type / shape argument is entered, the optimized calculation graph will be executed. Therefore, in the above program, the Python print statement was not executed in the last call, and all the print statements were not executed when it was executed again.
So I will return to the beginning,
--Don't rely on Python's unique behavior like Object Mutation or Python's list
It means that. It is Print earlier, but if you write it in tf.print instead, it will be executed every time. You can also prevent strange behavior by making full use of TensorFlow-derived functions, such as using tf.summary or using tf.Variable if you want to update various values in the function. It also improves performance. However, please note that I am not saying that you should not include any Python-specific processing. The advantage of being able to program more flexibly by being able to use it together with Pythonic writing is great. Just be aware that if you define a function without thinking about anything and add tf.function, it will behave strangely.
It's nice to be able to benefit from both Graph and Eager modes with TensorFlow 2.0, but it creates functions that rely too much on Python-specific features other than the pitfalls mentioned above. You may step on an unexpected bug. If you use TensorFlow, use methods derived from TensorFlow as much as possible, and when using Python-specific functions, design with consideration for the behavior and tracing of AutoGraph. The Official Site has a lot of other things to watch out for and how to control this behavior. If you need to implement your own model from now on and need to use tf.function, please read it.
Recommended Posts