[TensorFlow] Python ⇔ Protocol Buffers ⇔ GPU / Distributed Computing
Introduction
Deep learning framework TensorFlow reduces switching between Python and GPU / distributed computers and streamlines calculations by externalizing various processes via Protocol Buffers.
Protocol Buffers is a technology that supports Google's distributed computing, and is a mechanism for serializing data structures in a language-independent and platform-independent manner. Currently, C ++, C #, GO, Java and Python are supported.
TensorFlow is a mechanism that builds a graph with processing as a node and calculates at once. Example of graph
Below, I would like to see how the TensorFlow graph is serialized into Protocol Buffers format.
Sample program
Let's add with TensorFlow. Addition 1 is executed 3 times with the initial value 0.
add.py
import tensorflow as tf
state = tf.Variable(0, name="counter")
one = tf.constant(1)
new_value = tf.add(state, one)
update = tf.assign(state, new_value)
init_op = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init_op)
print(sess.run(state))
for _ in range(3):
sess.run(update)
print(sess.run(state))
TensorBoard graph
** Display graph with TensorBoard **
Protocol Buffers Below, we'll look at what happens when a Python program using TensorFlow is converted to a Protocol Buffers node.
Variable definition
- Python
state = tf.Variable(0, name="counter")
- Protocol Buffers: Variable named counter
name: "counter"
op: "Variable"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "shape"
value {
shape {
}
}
}
attr {
key: "shared_name"
value {
s: ""
}
}
- Protocol Buffers: Definition of initial value 0 (constant)
name: "counter/initial_value"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 0
}
}
}
- Protocol Buffers: Assign initial value 0 to variable counter
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}
- Protocol Buffers: Read the variable counter
name: "counter/read"
op: "Identity"
input: "counter"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
Definition of constant (addition value)
- Python
one = tf.constant(1)
- Protocol Buffers: Definition of constant 1
name: "Const"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 1
}
}
}
Definition of addition operation
- Python
one = tf.constant(1)
- Protocol Buffers: Defines addition operations
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
key: "T"
value {
type: DT_INT32
}
}
Substitute the value of the addition result into a variable
- Python
update = tf.assign(state, new_value)
- Protocol Buffers: Substitute values for addition results into variables
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}
in conclusion
If you understand that TensorFlow uses Protocol Buffers behind a Python program, you'll understand why TensorFlow processing is plugged into that location.
TensorFlow provides various basic processes (API).
- Graph
- Variable
- Tensor
- Math
- String
- Control syntax
- Assertion
- Image
- Video
- Input / output
- Neural network
- Operation summary
- Unit test
It's also interesting to try using TensorFlow as a GPU computing / distributed computing framework instead of as a deep learning framework.
(reference)
Graph display program
graph = tf.get_default_graph()
summary_writer = tf.train.SummaryWriter('log_valiable', graph)
operations = graph.get_operations()
for operation in operations:
print("======================")
print("=== name ===")
print(operation.name)
print("=== type ===")
print(operation.type)
print("=== inputs ===")
for input in operation.inputs:
print(input)
print("=== control_inputs ===")
for control_input in operation.control_inputs:
print(control_input)
print("=== outputs ===")
for output in operation.outputs:
print(output)
print("=== node_def ===")
print(operation.node_def)
print("=== op_def ===")
print(operation.op_def)
print("=== traceback ===")
print(operation.traceback)
print("")
Console output
0
1
2
3
======================
=== name ===
counter/initial_value
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/initial_value"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 0
}
}
}
=== op_def ===
None
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 293, '_init_from_args', 'initial_value, name="initial_value", dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 657, 'convert_to_tensor', 'ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 180, '_constant_tensor_conversion_function', 'return constant(v, dtype=dtype, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
counter
=== type ===
Variable
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter"
op: "Variable"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "shape"
value {
shape {
}
}
}
attr {
key: "shared_name"
value {
s: ""
}
}
=== op_def ===
name: "Variable"
output_arg {
name: "ref"
type_attr: "dtype"
is_ref: true
}
attr {
name: "shape"
type: "shape"
}
attr {
name: "dtype"
type: "type"
}
attr {
name: "container"
type: "string"
default_value {
s: ""
}
}
attr {
name: "shared_name"
type: "string"
default_value {
s: ""
}
}
is_stateful: true
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 300, '_init_from_args', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py', 146, 'variable_op', 'container=container, shared_name=shared_name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 490, '_variable', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
counter/Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("counter/Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}
=== op_def ===
name: "Assign"
input_arg {
name: "ref"
type_attr: "T"
is_ref: true
}
input_arg {
name: "value"
type_attr: "T"
}
output_arg {
name: "output_ref"
type_attr: "T"
is_ref: true
}
attr {
name: "T"
type: "type"
}
attr {
name: "validate_shape"
type: "bool"
default_value {
b: true
}
}
attr {
name: "use_locking"
type: "bool"
default_value {
b: true
}
}
allows_uninitialized_input: true
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 317, '_init_from_args', 'validate_shape=validate_shape).op'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
counter/read
=== type ===
Identity
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== control_inputs ===
=== outputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/read"
op: "Identity"
input: "counter"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
=== op_def ===
name: "Identity"
input_arg {
name: "input"
type_attr: "T"
}
output_arg {
name: "output"
type_attr: "T"
}
attr {
name: "T"
type: "type"
}
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 327, '_init_from_args', 'self._snapshot = array_ops.identity(self._variable, name="read")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py', 1128, 'identity', 'result = _op_def_lib.apply_op("Identity", input=input, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
Const
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("Const:0", shape=(), dtype=int32)
=== node_def ===
name: "Const"
op: "Const"
attr {
key: "dtype"
value {
type: DT_INT32
}
}
attr {
key: "value"
value {
tensor {
dtype: DT_INT32
tensor_shape {
}
int_val: 1
}
}
}
=== op_def ===
None
=== traceback ===
[('./valiable.py', 7, '<module>', 'one = tf.constant(1)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
Add
=== type ===
Add
=== inputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Add:0", shape=(), dtype=int32)
=== node_def ===
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
key: "T"
value {
type: DT_INT32
}
}
=== op_def ===
name: "Add"
input_arg {
name: "x"
type_attr: "T"
}
input_arg {
name: "y"
type_attr: "T"
}
output_arg {
name: "z"
type_attr: "T"
}
attr {
name: "T"
type: "type"
allowed_values {
list {
type: DT_HALF
type: DT_FLOAT
type: DT_DOUBLE
type: DT_UINT8
type: DT_INT8
type: DT_INT16
type: DT_INT32
type: DT_INT64
type: DT_COMPLEX64
type: DT_COMPLEX128
type: DT_STRING
}
}
}
=== traceback ===
[('./valiable.py', 8, '<module>', 'new_value = tf.add(state, one)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py', 71, 'add', 'result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("Add:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}
=== op_def ===
name: "Assign"
input_arg {
name: "ref"
type_attr: "T"
is_ref: true
}
input_arg {
name: "value"
type_attr: "T"
}
output_arg {
name: "output_ref"
type_attr: "T"
is_ref: true
}
attr {
name: "T"
type: "type"
}
attr {
name: "validate_shape"
type: "bool"
default_value {
b: true
}
}
attr {
name: "use_locking"
type: "bool"
default_value {
b: true
}
}
allows_uninitialized_input: true
=== traceback ===
[('./valiable.py', 9, '<module>', 'update = tf.assign(state, new_value)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
======================
=== name ===
init
=== type ===
NoOp
=== inputs ===
=== control_inputs ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
key: "T"
value {
type: DT_INT32
}
}
attr {
key: "_class"
value {
list {
s: "loc:@counter"
}
}
}
attr {
key: "use_locking"
value {
b: true
}
}
attr {
key: "validate_shape"
value {
b: true
}
}
=== outputs ===
=== node_def ===
name: "init"
op: "NoOp"
input: "^counter/Assign"
=== op_def ===
name: "NoOp"
=== traceback ===
[('./valiable.py', 11, '<module>', 'init_op = tf.initialize_all_variables()'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1063, 'initialize_all_variables', 'return initialize_variables(all_variables())'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1051, 'initialize_variables', 'return control_flow_ops.group(*[v.initializer for v in var_list], name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2645, 'group', 'return _GroupControlDeps(dev, deps, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2603, '_GroupControlDeps', 'return no_op(name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py', 184, 'no_op', 'result = _op_def_lib.apply_op("NoOp", name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 756, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
Protocol Buffers Benchmark
I actually checked if Protocol Buffers are really slow @ 2016/8/24
Recommended Posts