** Supported from chainer v.1.16.0 **
The long-awaited protobuf 3.0.0 has been released. In particular, up to 2 series did not work with Python 3, and the official release of 3 series that supports Python 3 was long-awaited.
As you may not know, protobuf's Python interface internally has two implementations, a Python implementation and a C ++ implementation, the latter being overwhelmingly faster. Reading Caffe's modelzoo file in Chainer is unusually slow because it uses the former Python implementation. For example, loading the model file of alexnet took about 120 seconds in the Python implementation, but it took about 10 seconds, which is 10 times faster in the C ++ implementation. For some time, due to the read file size limit (there was a buffer size limit), it was not possible to read huge files such as those published by modelzoo, and it seems that a method to avoid this has recently been introduced.
To use the C ++ implementation of protobuf, you need to get it from source. This may change in the future. First, grab the C ++ source for protobuf, unzip it, and install it.
$ wget https://github.com/google/protobuf/releases/download/v3.0.0/protobuf-cpp-3.0.0.tar.gz
$ tar -xzf protobuf-cpp-3.0.0.tar.gz
$ cd protobuf-3.0.0 && ./configure && make && make install && ldconfig
Next is the download and build of the python package. Work in the same directory as before. If you specify --cpp_implementation here, the C ++ implementation will be used.
$ wget https://github.com/google/protobuf/releases/download/v3.0.0/protobuf-python-3.0.0.tar.gz
$ tar -xzf protobuf-python-3.0.0.tar.gz
$ cd protobuf-3.0.0/python && python setup.py install --cpp_implementation
That's all.
Be sure to unzip both C ++ and Python sources in the same directory. Also, please install C ++ first. It seems that the C ++ library is specified as a relative directory in setup.py. In the future it may be possible to install from pip as well.
It has been pointed out in various places that pickle makes loading faster. This may be fine for the time being. Please note that pickle is not compatible between versions (eg python2 and python3).