Question

The entropy of a discrete random variable X is defined as: H(X) = -p(x) log p(x), where p(x) = P(X = x), and H(X) is interpreted as the average information revealed by a random outcome. (a) Compute the entropy of a discrete uniform random variable X taking values in {1,2, ..., m}. (b) Compute the entropy of a Bernoulli random variable X with parameter . Recall that if X ~ Bernoulli(0), then p(x) = 0^x(1 - 0)^(1-x), x = 0,1. (c) For what value of *, H(X) is maximized as a function of o? (d) The cross entropy between two discrete random variables X and Y is defined as: H(X,Y) = -p(x,y) log p(x, y), where p(x, y) = P(X = x, Y = y). Assume that X and Y are independent, i.e., p(x, y) = p(x)p(y). Show that H(X,Y) = H(X) + H(Y).

          The entropy of a discrete random variable X is defined as: 
H(X) = -p(x) log p(x), where p(x) = P(X = x), and H(X) is interpreted as the average information revealed by a random outcome.
(a) Compute the entropy of a discrete uniform random variable X taking values in {1,2, ..., m}.
(b) Compute the entropy of a Bernoulli random variable X with parameter . Recall that if X ~ Bernoulli(0), then p(x) = 0^x(1 - 0)^(1-x), x = 0,1.
(c) For what value of *, H(X) is maximized as a function of o?
(d) The cross entropy between two discrete random variables X and Y is defined as: H(X,Y) = -p(x,y) log p(x, y), where p(x, y) = P(X = x, Y = y).
Assume that X and Y are independent, i.e., p(x, y) = p(x)p(y).
Show that H(X,Y) = H(X) + H(Y).

Show more…

Added by Brandon C.

Elementary Statistics a Step by Step Approach

Allan G. Bluman 9th Edition

Instant Answer

Solved by Expert Umar Sohail Qureshi

01/23/2022

Step 1

.., m}, we need to find the probability distribution p(x) first. Since X is a discrete uniform random variable, each value in {1,2, ..., m} has an equal probability of occurring. Show more…

Show all steps

lock

Thanks for your feedback!

The entropy of a discrete random variable X is defined as: H(X) = -p(x) log p(x), where p(x) = P(X = x), and H(X) is interpreted as the average information revealed by a random outcome. (a) Compute the entropy of a discrete uniform random variable X taking values in {1,2, ..., m}. (b) Compute the entropy of a Bernoulli random variable X with parameter . Recall that if X ~ Bernoulli(0), then p(x) = 0^x(1 - 0)^(1-x), x = 0,1. (c) For what value of *, H(X) is maximized as a function of o? (d) The cross entropy between two discrete random variables X and Y is defined as: H(X,Y) = -p(x,y) log p(x, y), where p(x, y) = P(X = x, Y = y). Assume that X and Y are independent, i.e., p(x, y) = p(x)p(y). Show that H(X,Y) = H(X) + H(Y).

Play audio

Feedback

Powered by NumerAI

Umar Sohail Qureshi and 70 other subject Intro Stats / AP Statistics educators are ready to help you.

Ask a new question

Labs

Want to see this concept in action?

NEW

Explore this concept interactively to see how it behaves as you change inputs.

View Labs

Key Concepts

Recommended Videos

Recommended Textbooks

Transcript

Ace is your personal tutor. It breaks down any question with clear steps so you can learn.

Start Using Ace

Continue solving this problem

Create a free account to:

View full step-by-step solution
Ask follow-up questions with Ace AI
Save progress and study later