3 Problem Set: Inequalities...
1. Let $(p_1, \dots, p_n)$ and $(q_1, \dots, q_n)$ be two probabilistic distributions, e.g
$p_i, q_i > 0$ for all $i$; $\sum_{1 \le i \le n} p_i = \sum_{1 \le i \le n} q_i = 1$. Define
$KLD(p||q) = \sum_{1 \le i \le n} p_i(\log(p_i) - \log(q_i)).$
Finally define the following function on $[0, 1]$:
$K(t) = \sum_{1 \le i \le n} p_i(\log(p_i) - \log(p_i + t(q_i - p_i))).$
(Note that $K(0) = 0$ and $K(1) = KLD(p||q))$.
Prove that its derivative $K(t)' = t \sum_{1 \le i \le n} \frac{(p_i - q_i)^2}{p_i + t(q_i - p_i)}.$
2. Prove that $KLD(p||q) = \int_0^1 t \sum_{1 \le i \le n} \frac{(p_i - q_i)^2}{p_i + t(q_i - p_i)} dt.$
3. Prove that $\sum_{1 \le i \le n} \frac{(p_i - q_i)^2}{p_i + t(q_i - p_i)} \ge \sum_{1 \le i \le n} |p_i - q_i|^2.$
4. Prove that $KLD(p||q) \ge \frac{1}{2} (\sum_{1 \le i \le n} |p_i - q_i|^2).$
5. (Big Bonus Question!) The last inequality is pretty famous
What is its name?
