When using a self-made function for the loss function during learning or the conversion part of the data set in TensorFlow (2.x), I tried to confirm "Is the value as expected?" Print ( Calling) may not output the value.
You can use a debugger such as tfdbg, but here's a simpler way to do so-called "print debugging".
For tfdbg → Use tfdbg to crush Keras nan and inf --Qiita
In the code below, it is assumed that ↓ is written.
import tensorflow as tf
Eager Execution is now the default in TensorFlow 2.x, so as long as you are assembling Tensor with the interpreter, you can just do print () to see the value of Tensor. You can also get the value as ndarray with .numpy ().
x = tf.constant(10)
y = tf.constant(20)
z = x + y
print(z) # tf.Tensor(30, shape=(), dtype=int32)
print(z.numpy()) # 30
Since it is slow to evaluate the value of Tensor sequentially at the time of learning / evaluation, it is possible to define the function to be processed in the graph mode by adding the @ tf.function decorator. ** Operations defined in graph mode are processed together as a calculation graph (outside Python), so you cannot see the values in the calculation process. ** **
@tf.function
def dot(x, y):
tmp = x * y
print(tmp)
return tf.reduce_sum(tmp)
x = tf.constant([1, 2])
y = tf.constant([3, 1])
w = tf.constant([0, 2])
print(dot(x, y))
print(dot(x, w))
# Tensor("mul:0", shape=(2,), dtype=int32)
# tf.Tensor(5, shape=(), dtype=int32)
# tf.Tensor(4, shape=(), dtype=int32)
You can print the calculation result outside of dot (), but you cannot see the value inside.
Moreover, even if you call dot () multiple times, the print () inside is basically executed only once at the timing of analyzing the graph.
Of course, in this case you can remove the @ tf.function () and you will see the value.
def dot(x, y):
tmp = x * y
print(tmp)
return tf.reduce_sum(tmp)
x = tf.constant([1, 2])
y = tf.constant([3, 1])
w = tf.constant([0, 2])
print(dot(x, y))
print(dot(x, w))
# tf.Tensor([3 2], shape=(2,), dtype=int32)
# tf.Tensor(5, shape=(), dtype=int32)
# tf.Tensor([0 4], shape=(2,), dtype=int32)
# tf.Tensor(4, shape=(), dtype=int32)
It may be implicitly executed in graph mode, such as when map () processing for tf.data.Dataset or when using your own loss function in Keras. In this case, you cannot see the value in the middle with print () even if you do not add ** @ tf.function. ** **
def fourth_power(x):
z = x * x
print(z)
z = z * z
return z
ds = tf.data.Dataset.range(10).map(fourth_power)
for i in ds:
print(i)
# Tensor("mul:0", shape=(), dtype=int64)
# tf.Tensor(0, shape=(), dtype=int64)
# tf.Tensor(1, shape=(), dtype=int64)
# tf.Tensor(16, shape=(), dtype=int64)
# :
tf.print()
Use tf.print () to see the value of Tensor in the process running in graph mode.
tf.print | TensorFlow Core v2.1.0
You can display the values as follows, or you can use tf.shape () to display the dimensions and size of the Tensor. Please also use it for debugging when you get angry when the dimensions and sizes do not match for some reason in your own function.
@tf.function
def dot(x, y):
tmp = x * y
tf.print(tmp)
tf.print(tf.shape(tmp))
return tf.reduce_sum(tmp)
x = tf.constant([1, 2])
y = tf.constant([3, 1])
w = tf.constant([0, 2])
print(dot(x, y))
print(dot(x, w))
# [3 2]
# [2]
# tf.Tensor(5, shape=(), dtype=int32)
# [0 4]
# [2]
# tf.Tensor(4, shape=(), dtype=int32)
If you want to debug Dataset.map (), you can use tf.print ().
def fourth_power(x):
z = x * x
tf.print(z)
z = z * z
return z
ds = tf.data.Dataset.range(10).map(fourth_power)
for i in ds:
print(i)
# 0
# tf.Tensor(0, shape=(), dtype=int64)
# 1
# tf.Tensor(1, shape=(), dtype=int64)
# 4
# tf.Tensor(16, shape=(), dtype=int64)
# :
Then, if you don't think about anything and you should output the contents of Tensor with tf.print (), that is not the case.
tf.print () is executed when the process is actually executed as a calculated graph.
In other words, if it is found that the types and dimensions do not match at the time of graph analysis without performing ** calculation and an error occurs, the processing of tf.print () will not be executed. ** It's complicated ...
@tf.function
def add(x, y):
z = x + y
tf.print(z) #This print will not be executed
return z
x = tf.constant([1, 2])
y = tf.constant([3, 4, 5])
ret = add(x, y)
# ValueError: Dimensions must be equal, but are 2 and 3 for 'add' (op: 'AddV2') with input shapes: [2], [3].
In such a case, it is better to use ordinary print () to check if the data of the desired shape is passed.
@tf.function
def add(x, y):
print(x) #This print is executed during calculation graph analysis
print(y)
z = x + y
return z
x = tf.constant([1, 2])
y = tf.constant([3, 4, 5])
ret = add(x, y)
# Tensor("x:0", shape=(2,), dtype=int32)
# Tensor("y:0", shape=(3,), dtype=int32)
# ValueError: Dimensions must be equal, but are 2 and 3 for 'add' (op: 'AddV2') with input shapes: [2], [3].
For example, consider counting the number of times your function has been called in graph mode and displaying how many times it has been called.
count = 0
@tf.function
def dot(x, y):
global count
tmp = x * y
count += 1
tf.print(count, tmp)
return tf.reduce_sum(tmp)
x = tf.constant([1, 2])
y = tf.constant([3, 1])
w = tf.constant([0, 2])
print(dot(x, y))
print(dot(x, w))
# 1 [3 2]
# tf.Tensor(5, shape=(), dtype=int32)
# 1 [0 4]
# tf.Tensor(4, shape=(), dtype=int32)
"1" will be displayed the second time as well.
This is because count + = 1 as Python code is only executed once during graph analysis.
The correct answer is to use tf.Variable () and ```assign_add () `` as shown below.
tf.Variable | TensorFlow Core v2.1.0
count = tf.Variable(0)
@tf.function
def dot(x, y):
tmp = x * y
count.assign_add(1)
tf.print(count, tmp)
return tf.reduce_sum(tmp)
x = tf.constant([1, 2])
y = tf.constant([3, 1])
w = tf.constant([0, 2])
print(dot(x, y))
print(dot(x, w))
# 1 [3 2]
# tf.Tensor(5, shape=(), dtype=int32)
# 2 [0 4]
# tf.Tensor(4, shape=(), dtype=int32)
Performance up with tf.function | TensorFlow Core (Official document)