f(E[x]) \ge E[f(x)]

Is called the Jensen's inequality. This proof has already been explained in various places (for example, here), so it is omitted here. I will.

To intuitively understand this inequality $ f (E [x]) \ ge E [f (x)] $, let's graphically represent an example using random numbers.

First, let's assume that x is a random variable that follows a normal distribution, and create a random number generated from it. Also, convert that x with an upwardly convex function of $ f (x) =-x ^ 2 + 10 $. The histogram at the top of the graph below is the distribution of x that follows a normal distribution, and the histogram on the right is the distribution that $ x ^ 2 $ follows. In other words, Jensen's inequality is ** greener than the ** red circle ** below (after taking the expected value, that is, taking the average of the histogram above and then converting with $ f (x) $). It shows that it is larger than the circle ** (converted with $ f (x) $ and then take the expected value, that is, the average value of the histogram on the right).

The following is an animation of the average of the normal distribution, which is the distribution of x, shifted. In each case you can see that the green circle is below the red circle.

What is this inequality good for? ?? ??

Jensen's inequality

f(E[x]) \ge E[f(x)]

Wants to maximize $ f (E [x]) $, but when it is unclear what this function is, $ E [f (x)] $ if it can be calculated. Since E [f (x)] $ can be treated as the lower limit of $ f (E [x]) $, by maximizing the computable $ E [f (x)] $, the original target $ It is possible to maximize f (E [x]) $.

Often used is because $ \ log (\ cdot) $ is an upwardly convex function

log \int p(x)f(x)dx \ge \int p(x) log f(x)dx

Like, $ \ log (\ cdot) $ is put in the integral so that it can be calculated.

It's a little confusing, but here's the animation for $ \ log (\ cdot) $. If you look at the histogram on the right side, it is distorted downwards, so you can feel that the average value shifts downwards. It can be seen that the green circle is below the red circle by that amount.

reference

1.8 Random variable inequalities 1 http://mcm-www.jwu.ac.jp/~konno/pdf/statg-1-8r.pdf

Python code that describes the graph used in this article https://github.com/matsuken92/Qiita_Contents/blob/master/General/Jensens-inequality.ipynb