• Home
  • Textbooks
  • Foundations of coding: theory and applications of error-correcting codes
  • Data Compression and Entropy

Foundations of coding: theory and applications of error-correcting codes

Jiri Adamek

Chapter 3

Data Compression and Entropy - all with Video Answers

Educators


Chapter Questions

Problem 1

A message is written in source symbols $A, B, C, D$, and $A$ appears seven times more often than any of the other symbols. Find a binary coding which does not require more than 1.4 bits per symbol in the average. (Use extensions.)

Check back soon!

Problem 2

Find the entropy of the following information source:
$$
\begin{array}{l|cccccc}
\hline \text { Symbol } & 1 & 2 & 3 & 4 & 5 & 6 \\
\text { Probability } & 0.1 & 0.1 & 0.45 & 0.05 & 0.2 & 0.1 \\
\hline
\end{array}
$$

Check back soon!

Problem 3

A shooter who hits the target with probability $\frac{1}{2}$ shoots twice. Another shooter has the probability $\frac{1}{3}$ of hitting the target, and he shoots three times. Whose target carries "more information" (i.e., has a larger entropy)?

Check back soon!

Problem 4

Given information sources $S_1$ and $S_2$, denote by $S_1 \times S_2$ the information source of all pairs of symbols $\left(s_1, s_2\right)$ with $s_i$ a symbol of $S_i(i=1,2)$. Prove that
$$
H\left(S_1 \times S_2\right) \leq \boldsymbol{H}\left(S_1\right)+H\left(S_2\right)
$$
and the equality takes place iff $S_1$ and $S_2$ are independent [i.e., the probability of $\left(s_1, s_2\right)$ is the product of the probability of $s_1$ (in $\left.S_1\right)$ and $s_2$ (in $S_2$ )].

Check back soon!

Problem 5

A channel transmits equiprobable symbols 0 and 1 . What is the probability of receiving the message 01101 ? What is the entropy of fivesymbol messages?

Check back soon!

Problem 6

An information source consists of 128 equiprobable symbols. What is the length of a message of entropy 42 bits?

Check back soon!

Problem 7

What is the entropy of a message consisting of three words of length 8 each, if the source alphabet has symhols $A, B, C$ which
(1) are equiprobable?
(2) have probabilities $0.58,0.33$, and 0.09 ?

Check back soon!

Problem 8

The effectivity of an information source is defined as the ratio between the entropy (in bits) and the average length of a binary Huffman code. Prove that the effectivity lies between 0 and 1 , and discuss the two extremal values. Find the effectivity of the three sources of Figure 5 in Chapter 2.

Check back soon!

Problem 9

Find the effectivity of
(1) the source $A, B, C, D$ in which $A$ has double the frequency of the other three symbols,
(2) the second extension of that source.

Check back soon!

Problem 10

What is the effectivity of a binary source $S$ in which 0 has probability 0.89 ? Find an extension of $S$ with effectivity at least $90 \%$.

Check back soon!

Problem 11

Find binary instantaneous codes for the sources in Figure 4 and compute their effectivity.
$$
\begin{array}{l|cccccccc}
\hline \text { Symbol } & \text { A } & \text { B } & \text { C } & \text { D } & \text { E } & \text { F } & \text { G } & \text { H } \\
\hline \text { Prob. (1 } \text { at } \text { source) } & 6 \% & 15 \% & 15 \% & 7 \% & 5 \% & 30 \% & 18 \% & 4 \% \\
\text { Prob. (2 }{ }^{\text {nd }} \text { source) } & \frac{1}{4} & \frac{1}{4} & \frac{1}{8} & \frac{1}{8} & \frac{1}{16} & \frac{1}{16} & \frac{1}{16} & \frac{1}{16} \\
\text { Prob. (3 }{ }^{\text {rd }} \text { source) } & 0.2 & 0.18 & 0.16 & 0.14 & 0.1 & 0.1 & 0.6 & 0.4 \\
\hline
\end{array}
$$

Check back soon!

Problem 12

Suppose a huge binary message contains twice more 0's than 1's. Find a coding which uses
(1) at most 0.94 bits per symbol
(2) at most 0.9 bits per symbol.

Check back soon!