Problem 2 - Probability

1.

Let $\ln$ by the natural logarithm. Show that $\ln x \leq x - 1$ for $x>0$.

2.

For probability density functions $p$ and $q$ of random variable $x$ on a domains $I$, $D(p\vert\vert q)$ is defined by

$$D(p\vert\vert q) \equiv \int_I p \ln \frac{p}{q} dx.$$

Show that $D(p\vert\vert q)\geq 0$ holds. When does this inequality hold by equality?

3.

For probability density functions $p_1$, $p_2$, $q_1$, $q_2$ of random variable $x$, show

$$D(p_1\vert\vert q_1) + D(p_2\vert\vert q_2) \geq 2D\left(\frac{p_1+p_2}{2}\vert\vert\frac{q_1+q_2}{2}\right)$$

4.

Let $p$,$q$ be normal distributions with mean $\mu_p$, $\mu_q$ and standard deviation $\sigma_p$, $\sigma_q$, respectively (i.e. $p=p(x)$ is given by

$$p(x)=\frac{1}{\sqrt{2\pi}\sigma_p}\exp\left(-\frac{(x-\mu_p)^2}{2\sigma_p^2}\right)$$

and similar for $q$). Compute $D(p\vert\vert q)$.

1.

Let us consider the following function $f(x)=\ln x - x + 1$. Its first derivative is $f'(x)=\frac{1}{x}-1$. This function is positive if $0<x<1$, negative is $x>1$ and is 0 if $x=1$. Thus $f$ is increasing from $x>0$ to $x<1$, it reaches 0 in 1 and then decreases for $x>1$. Thus, $\forall x>0$, $f(x)\leq 0$. That is:

$$\forall x > 0, \quad \ln x \leq x - 1$$

2.

$D(p\vert\vert q)=\int_I p\ln \frac{p}{q}dx$
$=\int_I p\left( -\ln\frac{q}{p}\right)dx$
$\geq \int_I p\left( \frac{q}{p} - 1 \right)dx$
$= \int_I (q - p)dx$
$= \int_I q dx - \int_I p dx$
$= 0$ since both $\int_I q dx$ and $\int_I p dx$ are equal to 1.

3.

$D(p_1\vert\vert q_1) + D(p_2\vert\vert q_2) - 2D\left(\frac{p_1+p_2}{2}\vert\v... ...+ \int_I p_2\ln\frac{p_2}{q_2}dx - \int_I (p_1+p_2)\ln\frac{p_1+p_2}{q_1+q_2}dx$
$= \int_I p_1\ln\frac{p_1(q_1+q_2)}{q_1(p_1+p_2)}dx + \int_I p_2\ln \frac{p_2(q_1+q_2)}{q_2(p_1+p_2)}dx$
$\geq \int_I p_1\left(\frac{q_1(p_1+p_2)}{p_1(q_1+q_2)} - 1\right)dx + \int_I p_2\left( \frac{q_2(p_1+p_2)}{p_2(q_1+q_2)} - 1\right)dx$
$= \int_I \left( q_1\frac{p_1+p_2}{q_1+q_2} - p_1 + q_2\frac{p_1+p_2}{q_1+q_2} - p_2 \right) dx$
$= 0$

4.

$D(p\vert\vert q) = \int_I \frac{1}{\sqrt{2\pi}\sigma_p} \exp\left(-\frac{(x-\m... ...p}\exp\left(\frac{(x-\mu_p)^2\sigma_q^2}{(x-\mu_q)^2\sigma_p^2}\right)\right)dx$

TODO: What am I supposed to find?