[PYTHON] DEEP PROBABILISTIC PROGRAMMING --- "Deep Learning + Bayes" Library --- Introduction of Edward

WHY

Edward

We will introduce it from the following viewpoints with expectations for mounting and future growth.

--Co-authored by Google Research --BackEnd is Tensorflow --There is something like Caffee's Model Zoo and there is an ecosystem for published models ――It converges faster than deep learning and is likely to become the next trend

I have read Deep Probabilistic Programming, so I will introduce an excerpt from this content.

What thing?

Introducing a library that is more flexible than conventional deep learning and enables probabilistic programming with high computational efficiency. The same model can be used with different inference machines. Since the model representation can be reused as part of inference, a wide variety of network configurations are possible. The backend is made of Tensorflow and there is something like Caffee's Model Zoo Probability Zoo

What is amazing compared to previous research?

・ Computational efficiency is better than deep learning ・ Creating flexible inference parts ・ 35 times faster with logistic regression than Stan and PyMC3

Where is the key to technology and method?

・ Consists of random value generation and inference ・ Created with open technology (backend is Tensorflow) -The part that includes the construction of the model in the inference part is different from the conventional one. ・ Bayesian Recurrent Neural Network with Variable Length

Concrete example

Bayesian Recurrent Neural Network with Variable Length

Normal: normal distribution

Learn the mean and variance of the normal distribution of Weight and regularization

GAN

A network for generation and a network for identification are configured, and data parameters are generated from the distribution. Predicted values are also generated from the distribution

Speculator composite

EM Algorithm

How did you verify it was valid?

Comparison of each method

Comparison with conventional library

Speed verification with PyMC and Stan

About 20 times that of Stan About 40 times that of PyMC

Spec
12-core Intel i7-5930K CPU 3.5GHz NVIDIA Titan X(Maxwell) GPU

Task
generate posterior samples with Hamiltonian Monte Carlo

Data
Covertype dataset (N = 581012, D = 54; responses were binarized)

Other
100 HMC iterations, with 10 leapfrog updates per iteration and a step size of 0.5/N

35x speedup from stan 

Is there a discussion?

The latest technology wants to incorporate more traditional Bayesian techniques I want to be able to apply it even to huge data

What paper should I read next?

The following paper is recommended because you can see how to use it in earnest. Tran, Dustin, et al. "Edward: A library for probabilistic modeling, inference, and criticism." arXiv preprint arXiv:1610.09787 (2016).

Reference

Deep Probabilistic Programming