[PYTHON] Notes on KL divergence between Poisson distributions

The calculation of KL divergence of two probability distributions according to Poisson distribution is summarized as a memo.

Premise

The two probability distributions $ p and q $ follow a Poisson distribution with parameters $ \ lambda_p and \ lambda_q $, respectively. The probability density function of $ p $ is defined as follows.

p(x) = e^{-\lambda_p} \frac{\lambda_p^x}{x!}

The KL divergence of the discrete probability distribution is as follows.

D_{KL}(p, q) = \sum_{x=0}^{\infty} p(x) \log \frac{p(x)}{q(x)}

a formula

\begin{equation*}\begin{split}D_{KL}(p,q) &= \sum_{x=0}^{\infty} p(x) \log \frac{p(x)}{q(x)} \\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot \log \frac{e^{-\lambda_p} \frac{\lambda_p^x}{x!}}{e^{-\lambda_q} \frac{\lambda_q^x}{x!}} \\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot ( \log e^{-\lambda_p} \frac{\lambda_p^x}{x!} - \log e^{-\lambda_q} \frac{\lambda_q^x}{x!} )\\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot (-\lambda_p +\log \frac{\lambda_p^x}{x!} +\lambda_q - \log \frac{\lambda_q^x}{x!} )\\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot (\lambda_q -\lambda_p +\log \frac{\lambda_p^x}{x!} \frac{x!}{\lambda_q^x} )\\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot (\lambda_q -\lambda_p +\log \frac{\lambda_p^x}{\lambda_q^x} )\\&=\sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot (\lambda_q -\lambda_p) + \sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot \log \frac{\lambda_p^x}{\lambda_q^x} \\&=\lambda_q -\lambda_p + \sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^x}{x!} \cdot x \log \frac{\lambda_p}{\lambda_q} \\&=\lambda_q -\lambda_p + \sum_{x=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^{x-1}}{(x-1)!} \lambda_p \cdot \log \frac{\lambda_p}{\lambda_q}\\&=\lambda_q -\lambda_p + \sum_{t=0}^{\infty} e^{-\lambda_p} \frac{\lambda_p^{t}}{t!} \lambda_p \cdot \log \frac{\lambda_p}{\lambda_q} &\mbox{$t=x-1$To}\\&=\lambda_q -\lambda_p + e^{-\lambda_p} e^{\lambda_p} \lambda_p \cdot \log \frac{\lambda_p}{\lambda_q} \\&=\lambda_q -\lambda_p + \lambda_p \cdot \log \frac{\lambda_p}{\lambda_q}\\\end{split}\end{equation*}

Verification

I tried to see if the calculation of KL divergence using scipy's pmph and entropy and the result of the above formula are equal.

import numpy as np
from scipy.stats import poisson, entropy


x = np.linspace(0, 100, 101)

#Define parameter λ as alpha
alpha_p = 1
alpha_q = 5
p = poisson.pmf(x, alpha_p)
q = poisson.pmf(x, alpha_q)
kl = entropy(p, q)

#Calculation based on the above formula
kl_alpha = alpha_q - alpha_p + alpha_p * np.log(alpha_p / alpha_q)

print("KL divergence using the entropy function:", kl)
print("KL divergence using alpha:", kl_alpha)

Output result

KL divergence using the entropy function: 2.3905620875658995
KL divergence using alpha: 2.3905620875658995