Problem 1 - Probability

1.

For $t>0$, prove $\log t \geq 1 - \frac{1}{t}.$

2.

For functions $p(x)$, $q(x)$ defined over $-\infty < x < \infty$ such that

$$p(x)>0,\quad q(x)>0,$$

$$\int_{-\infty}^\infty p(x)dx = \int_{-\infty}^\infty q(x)dx = 1,$$

prove the following equality holds:

$$I(p,q)=\int_{-\infty}^\infty p(x)\log \frac{p(x)}{q(x)}dx \geq 0.$$

3.

For functions $p(x)$ and $q(x)$ defined by

$$p(x)=\frac{1}{\sigma_1\sqrt{2\pi}}\exp\left( -\frac{(x-x_1)^2}{2\sigma_1^2} \right),$$

$$q(x)=\frac{1}{\sigma_2\sqrt{2\pi}}\exp\left( -\frac{(x-x_2)^2}{2\sigma_2^2} \right),$$

compute $I(p,q)$. Here, $x_1, x_2, \sigma_1, \sigma_2$ are constants.

1.

For $t>0$, we study the function defined by $f(t)=\log t + \frac{1}{t} - 1$ whose first derivative is $f'(t)= \frac{t-1}{t^2}$. This function is $<0$ for $0<t<1$ and $>0$ for $t>1$. This means, $f$ is decreasing on the way to 1 and increasing afterwards. It reaches its minimum in 1 and this minimum has 0 as a value. From this, we conclude $\forall t > 0, f(t)\geq 0$ which is equivalent to the statement we're asked about.

2.

$$\forall x\in \mathbb{R}, p(x)\log \frac{p(x)}{q(x)} \geq p(x)\left( 1 - \frac{q(x)}{p(x)}\right) = p(x) - q(x)$$

We integrate the inequality and get:

$$\int_{-\infty}^\infty p(x)\log \frac{p(x)}{q(x)}dx \geq \int_{-\infty}^\infty p(x) dx - \int_{-\infty}^\infty q(x)dx = 1 - 1 = 0$$

since $p$ and $q$ are probability density functions.

3.

$$\int_{-\infty}^\infty \frac{1}{\sqrt{2\pi}\sigma_1}\exp\left(-\fr... ...gma_2}{\sigma_1}\exp \frac{\sigma_2^2}{\sigma_1^2}\frac{(x-x_1)^2}{(x-x_2)^2}dx$$

TODO: What am I suppose to find?