Entropy. Consider a distribution over n possible outcomes,...

Entropy. Consider a distribution over $n$ possible outcomes, with probabilities $p_{1}, p_{2}, \ldots, p_{n}$. (a) Just for this part of the problem, assume that each $p_{i}$ is a power of 2 (that is, of the form $\left.1 / 2^{k}\right) .$ Suppose a long sequence of $m$ samples is drawn from the distribution and that for all $1 \leq i \leq n,$ the $i^{\text {th }}$ outcome occurs exactly $m p_{i}$ times in the sequence. Show that if Huffman encoding is applied to this sequence, the resulting encoding will have length \[ \sum_{i=1}^{n} m p_{i} \log \frac{1}{p_{i}}. \] (b) Now consider arbitrary distributions- -that is, the probabilities $p_{i}$ are not restricted to powers of $2 .$ The most commonly used measure of the amount of randomness in the distribution is the entropy \[ \sum_{i=1}^{n} p_{i} \log \frac{1}{p_{i}}. \] For what distribution (over $n$ outcomes) is the entropy the largest possible? The smallest possible?

Entropy. Consider a distribution over $n$ possible outcomes, with probabilities $p_{1}, p_{2}, \ldots, p_{n}$.
(a) Just for this part of the problem, assume that each $p_{i}$ is a power of 2 (that is, of the form $\left.1 / 2^{k}\right) .$ Suppose a long sequence of $m$ samples is drawn from the distribution and that for all $1 \leq i \leq n,$ the $i^{\text {th }}$ outcome occurs exactly $m p_{i}$ times in the sequence. Show that if Huffman encoding is applied to this sequence, the resulting encoding will have length
\[
\sum_{i=1}^{n} m p_{i} \log \frac{1}{p_{i}}.
\]
(b) Now consider arbitrary distributions- -that is, the probabilities $p_{i}$ are not restricted to powers of $2 .$ The most commonly used measure of the amount of randomness in the distribution is the entropy
\[
\sum_{i=1}^{n} p_{i} \log \frac{1}{p_{i}}.
\]
For what distribution (over $n$ outcomes) is the entropy the largest possible? The smallest possible?

Key Concepts

Degenerate Distribution

A degenerate distribution is one in which one outcome has a probability of one and all other outcomes have a probability of zero. This distribution has the smallest possible entropy because there is no uncertainty in the occurrence of outcomes.

Probability Distribution

A probability distribution assigns probabilities to each of a set of outcomes in such a way that the total probability sums to one. This concept forms the basis for defining entropy and influences the design of optimal coding schemes like Huffman coding.

Uniform Distribution

The uniform distribution is a specific type of probability distribution in which all outcomes are equally likely. In terms of entropy, the uniform distribution maximizes uncertainty, thereby yielding the highest possible entropy among all distributions over a given number of outcomes.

Huffman Coding

Huffman coding is an algorithm used in data compression that assigns variable-length codes to input characters based on their frequencies. The algorithm ensures that more frequent symbols have shorter codes, resulting in an optimal prefix-free code that minimizes the average code length in a fixed sample sequence.

Entropy

Entropy, in the context of information theory, is a measure of the average amount of information produced by a stochastic source of data. It is defined as the sum over all outcomes of the probability of each outcome multiplied by the logarithm of the inverse of that probability, quantifying the uncertainty inherent in the distribution.

Transcript

Please give Ace some feedback