The following table gives the frequencies of the letters of...

The following table gives the frequencies of the letters of the English language (including the blank for separating words) in a particular corpus. $$\begin{array}{cc|cc|cc} \text { blank } & 18.3 \% & \text { r } & 4.8 \% & \text { y } & 1.6 \% \\ \text { e } & 10.2 \% & \text { d } & 3.5 \% & \text { p } & 1.6 \% \\ \text { t } & 7.7 \% & \text { l } & 3.4 \% & \text { b } & 1.3 \% \\ \text { a } & 6.8 \% & \text { c } & 2.6 \% & \text { v } & 0.9 \% \\ \text { 0 } & 5.9 \% & \text { u } & 2.4 \% & \text { k } & 0.6 \% \\ \text { i } & 5.8 \% & \text { m } & 2.1 \% & \text { j } & 0.2 \% \\ \text { n } & 5.5 \% & \text { w } & 1.9 \% & \text { x } & 0.2 \% \\ \text { s } & 5.1 \% & \text { f } & 1.8 \% & \text { q } & 0.1 \% \\ \text { h } & 4.9 \% & \text { g } & 1.7 \% & \text { z } & 0.1 \% \end{array}$$ (a) What is the optimum Huffman encoding of this alphabet? (b) What is the expected number of bits per letter? (c) Suppose now that we calculate the entropy of these frequencies \[ H=\sum_{i=0}^{26} p_{i} \log \frac{1}{p_{i}} \] (see the box in page 156 ). Would you expect it to be larger or smaller than your answer above? Explain. (d) Do you think that this is the limit of how much English text can be compressed? What features of the English language, besides letters and their frequencies, should a better compression scheme take into account?

The following table gives the frequencies of the letters of the English language (including the blank for separating words) in a particular corpus.
$$\begin{array}{cc|cc|cc}
\text { blank } & 18.3 \% & \text { r } & 4.8 \% & \text { y } & 1.6 \% \\
\text { e } & 10.2 \% & \text { d } & 3.5 \% & \text { p } & 1.6 \% \\
\text { t } & 7.7 \% & \text { l } & 3.4 \% & \text { b } & 1.3 \% \\
\text { a } & 6.8 \% & \text { c } & 2.6 \% & \text { v } & 0.9 \% \\
\text { 0 } & 5.9 \% & \text { u } & 2.4 \% & \text { k } & 0.6 \% \\
\text { i } & 5.8 \% & \text { m } & 2.1 \% & \text { j } & 0.2 \% \\
\text { n } & 5.5 \% & \text { w } & 1.9 \% & \text { x } & 0.2 \% \\
\text { s } & 5.1 \% & \text { f } & 1.8 \% & \text { q } & 0.1 \% \\
\text { h } & 4.9 \% & \text { g } & 1.7 \% & \text { z } & 0.1 \%
\end{array}$$
(a) What is the optimum Huffman encoding of this alphabet?
(b) What is the expected number of bits per letter?
(c) Suppose now that we calculate the entropy of these frequencies
\[
H=\sum_{i=0}^{26} p_{i} \log \frac{1}{p_{i}}
\]
(see the box in page 156 ). Would you expect it to be larger or smaller than your answer above? Explain.
(d) Do you think that this is the limit of how much English text can be compressed? What features of the English language, besides letters and their frequencies, should a better compression scheme take into account?

Key Concepts

Entropy

Entropy in information theory is a measure of the average uncertainty or information content inherent in a set of symbols, computed using the probabilities of those symbols. It represents the theoretical lower bound on the average length of any lossless encoding of a source, serving as a benchmark for the performance of practical compression schemes.

Advanced Compression Techniques

Beyond simple encoding of individual letters, more effective compression of natural language text can benefit from exploiting additional structure such as common letter pairs (bigrams), words, grammatical patterns, and contextual redundancies. These higher-order dependencies, which capture the syntax and semantics of the language, enable compression algorithms to achieve rates closer to the true entropy of the text by modeling the statistical dependencies beyond single-letter frequencies.

Huffman Coding

Huffman coding is a method for constructing an optimal prefix code for a set of symbols based on their probabilities. It assigns shorter binary codes to more frequent symbols and longer codes to less frequent symbols, ensuring that the overall expected code length is minimized, making it a key tool in lossless data compression.

Expected Codeword Length

The expected number of bits per letter in an encoding scheme is calculated by summing, over all symbols, the product of each symbol’s probability and the length of its encoded binary string. This measure provides the average cost of encoding each letter and is crucial for assessing the efficiency of a compression algorithm.

Transcript

Please give Ace some feedback