Question

3. Consider random variables x and y, distributed according to a joint probability p(x, y). The mutual information between the two variables is defined by \begin{equation} M(x,y) = \sum_{x,y} p(x, y) \ln \left(\frac{p(x, y)}{p_x(x)p_y(y)}\right), \end{equation} where $p_x$ and $p_y$ denote the unconditional probabilities for x and y. (a) Relate M(x, y) to the entropies S(x, y), S(x) and S(y) obtained from the corresponding probabilities. (b) Calculate the mutual information for the joint Gaussian form \begin{equation} p(x, y) \propto \exp\left(-\frac{1}{2}ax^2 - \frac{1}{2}by^2 - cxy\right). \end{equation} (0.1)

          3. Consider random variables x and y, distributed according to a joint probability p(x, y). The
mutual information between the two variables is defined by
\begin{equation}
M(x,y) = \sum_{x,y} p(x, y) \ln \left(\frac{p(x, y)}{p_x(x)p_y(y)}\right),
\end{equation}
where $p_x$ and $p_y$ denote the unconditional probabilities for x and y.
(a) Relate M(x, y) to the entropies S(x, y), S(x) and S(y) obtained from the corresponding
probabilities.
(b) Calculate the mutual information for the joint Gaussian form
\begin{equation}
p(x, y) \propto \exp\left(-\frac{1}{2}ax^2 - \frac{1}{2}by^2 - cxy\right).
\end{equation} (0.1)

3. Consider random variables x and y, distributed according to a joint probability p(x, y). The
mutual information between the two variables is defined by

M(x,y) = ∑x,y p(x, y) ln((p(x, y))/(px(x)py(y))),

where px and py denote the unconditional probabilities for x and y.
(a) Relate M(x, y) to the entropies S(x, y), S(x) and S(y) obtained from the corresponding
probabilities.
(b) Calculate the mutual information for the joint Gaussian form

p(x, y) ∝(-(1)/(2)ax^2 - (1)/(2)by^2 - cxy).
(0.1)

Added by Robert P.

Question

Please give Ace some feedback