tensorflow.set_random_seed function or the seed argument of the random number operator.When implementing machine learning code in TensorFlow, we often use pseudo-random numbers (hereinafter simply "random numbers") for weight initialization and so on. As a result, the result changes each time the code is re-executed, and it is not possible to distinguish between the change in the result due to changes in data and parameters and the change in the result due to random numbers. Random numbers are generated based on a value called a "seed", which, unless otherwise specified, uses a somewhat variable default value (often a time or process ID). TensorFlow also provides a way to fix the random seed, but I was a little worried about its behavior, so I tried to organize it.
In addition to TensorFlow, we will also discuss fixing random number seeds in the random module and NumPy package.
The content of this article is based on the following versions:
In the Python standard random module, you can set the random seed with the random.seed function.
The experimental code is as follows.
seed_random.py
import random
random.seed(0)
print("1:", random.random())
print("2:", random.random())
print("3:", random.random())
random.seed(0)
print("4:", random.random())
print("5:", random.random())
print("6:", random.random())
The result when executed 3 times is as follows. The same value is output each time a random number seed is set or executed.
$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845
$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845
$ python3 seed_random.py
1: 0.8444218515250481
2: 0.7579544029403025
3: 0.420571580830845
4: 0.8444218515250481
5: 0.7579544029403025
6: 0.420571580830845
In the NumPy package, you can set the random seed with the numpy.random.seed function.
The experimental code is as follows.
seed_numpy.py
import numpy as np
np.random.seed(0)
print("1:", np.random.rand(3))
print("2:", np.random.rand(3))
print("3:", np.random.rand(3))
np.random.seed(0)
print("4:", np.random.rand(3))
print("5:", np.random.rand(3))
print("6:", np.random.rand(3))
The result when executed 3 times is as follows. The same value is output each time a random number seed is set or executed.
$ python3 seed_numpy.py
1: [ 0.5488135 0.71518937 0.60276338]
2: [ 0.54488318 0.4236548 0.64589411]
3: [ 0.43758721 0.891773 0.96366276]
4: [ 0.5488135 0.71518937 0.60276338]
5: [ 0.54488318 0.4236548 0.64589411]
6: [ 0.43758721 0.891773 0.96366276]
$ python3 seed_numpy.py
1: [ 0.5488135 0.71518937 0.60276338]
2: [ 0.54488318 0.4236548 0.64589411]
3: [ 0.43758721 0.891773 0.96366276]
4: [ 0.5488135 0.71518937 0.60276338]
5: [ 0.54488318 0.4236548 0.64589411]
6: [ 0.43758721 0.891773 0.96366276]
$ python3 seed_numpy.py
1: [ 0.5488135 0.71518937 0.60276338]
2: [ 0.54488318 0.4236548 0.64589411]
3: [ 0.43758721 0.891773 0.96366276]
4: [ 0.5488135 0.71518937 0.60276338]
5: [ 0.54488318 0.4236548 0.64589411]
6: [ 0.43758721 0.891773 0.96366276]
In the TensorFlow package, you can set the random seed with the tensorflow.set_random_seed function or the seed argument of the random number operator.
There are two types of random number seeds in TensorFlow called "graph level" and "operation level".
You also need to be careful about the order of the settings, and you need to set the random seeds "after creating the graph" and "before generating the operator".
Reference: Constants, Sequences, and Random Values --tf.set_random_seed (seed)
The experimental code is as follows.
seed_tensorflow1.py
import tensorflow as tf
with tf.Graph().as_default():
tf.set_random_seed(0)
rand_op1 = tf.random_normal([3])
rand_op2 = tf.random_normal([3], seed=1)
print("session1")
with tf.Session() as sess1:
print("op1-1:", sess1.run(rand_op1))
print("op1-2:", sess1.run(rand_op1))
print("op2-1:", sess1.run(rand_op2))
print("op2-2:", sess1.run(rand_op2))
print("session2")
with tf.Session() as sess2:
print("op1-1:", sess2.run(rand_op1))
print("op1-2:", sess2.run(rand_op1))
print("op2-1:", sess2.run(rand_op2))
print("op2-2:", sess2.run(rand_op2))
The result when executed 3 times is as follows.
$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
$ python3 seed_tensorflow1.py
session1
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
session2
op1-1: [-1.40955448 -0.53668278 -0.56523788]
op1-2: [-1.07107699 0.4139019 2.29180121]
op2-1: [ 0.68034536 0.8777824 -0.64773595]
op2-2: [-1.21607268 0.95542693 -0.16866584]
Here is an example of setting a random seed before creating a graph. The experimental code is as follows.
seed_tensorflow2.py
import tensorflow as tf
tf.set_random_seed(0) #Set random seeds before creating the graph
with tf.Graph().as_default():
rand_op1 = tf.random_normal([3])
rand_op2 = tf.random_normal([3], seed=1)
with tf.Session() as sess1:
print("op1-1:", sess1.run(rand_op1))
print("op1-2:", sess1.run(rand_op1))
print("op2-1:", sess1.run(rand_op2))
print("op2-2:", sess1.run(rand_op2))
The result when executed 3 times is as follows.
Rand_op2, which sets the random number seed at the operation level, outputs the same value three times, but rand_op1 outputs a different value three times.
$ python3 seed_tensorflow2.py
op1-1: [ 0.22292495 0.67142487 0.80927771]
op1-2: [-1.11188841 0.57819426 0.19088539]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow2.py
op1-1: [-0.96930581 1.48854125 0.52752507]
op1-2: [ 0.91646689 0.85830265 -0.18069462]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow2.py
op1-1: [ 0.79890805 1.35702407 -0.12329593]
op1-2: [ 1.69212222 0.22590902 -0.73435217]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
The following example shows how to set the random seed after generating the random operator. The experimental code is as follows.
seed_tensorflow3.py
import tensorflow as tf
with tf.Graph().as_default():
rand_op1 = tf.random_normal([3])
rand_op2 = tf.random_normal([3], seed=1)
tf.set_random_seed(0) #Set random number seed after generating random number operator
with tf.Session() as sess:
print("op1-1:", sess.run(rand_op1))
print("op1-2:", sess.run(rand_op1))
print("op2-1:", sess.run(rand_op2))
print("op2-2:", sess.run(rand_op2))
The result when executed 3 times is as follows.
Similar to the above example, rand_op2, which sets the random number seed at the operation level, outputs the same value three times, but rand_op1 outputs a different value three times.
$ python3 seed_tensorflow3.py
op1-1: [ 0.03802272 1.69988739 -0.59952497]
op1-2: [ 0.62614048 0.07537607 -1.19501412]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow3.py
op1-1: [ 1.02020776 1.70896292 0.45571503]
op1-2: [-0.46230376 -1.22950804 0.51038951]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
$ python3 seed_tensorflow3.py
op1-1: [ 1.31134415 -1.12688231 0.1805287 ]
op1-2: [-0.57391566 0.94440365 -1.07441545]
op2-1: [-0.81131822 1.48459876 0.06532937]
op2-2: [ 0.55171245 -0.13107552 -0.04481386]
It is unconfirmed information, but it seems that it will not be the same value even if you set a random number seed when running on GPU.
Reference: Mention that GPU reductions are nondeterministic in docs · Issue # 2732 · tensorflow / tensorflow
We received very useful information from Mr. Hirai, a fellow study group on TensorFlow. Thank you here.
Recommended Posts