This is a question similar to 1 and 2. I wan to find out what the relationship between Kullback-Leibler divergence and L2 norm. Finally, I get the answer in one book named "Smoothing of Multivariate Data: Density Estimation and Visualization". But I can not prove the Lemma in this book. Does anyone can help me to prove the Lemma together. And my probability density function is a GMM. Thanks in advance.
For the sake of completeness I quote the Lemma 11.6:
Let $D_k$ be the Kullback-Leibler distance defined in (11.8). We have $$D_k^2(f,f_0)\le\int_{\{x\in \mathbb{R}^d:f_0(x)\gt0\}}\frac{(f-f_0)^2}{f_0}.$$ The upper bound is called the $\chi^2$-divergence between $f$ and $f_0$. In particular if $\inf_{x\in \mathbb{R}^d}$$f_0(x)\gt0$, then $$D_k^2(f,f_0)\le\frac{\lVert f-f_0\rVert_2^2}{\inf_{x\in \mathbb{R}^d}f_0(x)}.$$ Also, if $f$ and $f_0$ are both bounded and bounded away from zero, then $$D_k^2(f,f_0)\ge\int_{\{x\in \mathbb{R}^d:f_0(x)\gt0\}}(f-f_0)+C\lVert f-f_0\rVert_2^2,$$ for a positive constant C.
Some other formulas maybe used are given as follows:$$D_k^2(f,g)=\int_{\mathbb{R}^d\cap\{x:g(x)\gt0\}}f\log_e(\frac{f}{g})\quad\quad(11.8)$$ $$\lVert f-\hat f\rVert_P^P=\int_{R^d}\lvert f-\hat f\rvert^P$$
$\endgroup$ 6 Reset to default
