Huffiman codes. Consider a random variable X that takes six values {A, B, C, D, E, F} with proba

Huffiman codes. Consider a random variable X that takes six...

Key Concepts

Bounds on Average Code Length

In coding theory, especially with Huffman codes, establishing bounds on average code length is critical. These bounds, such as proving that the average length of the binary Huffman code is less than or equal to that of a binary conversion of a quaternary code plus a constant, help quantify the performance gap between different coding approaches. Such bounds are useful for evaluating how close a practical coding strategy is to the theoretical optimum.

Average Code Length and Efficiency

The average code length of a coding scheme is calculated by summing the products of each symbol's probability and its codeword length, serving as a measure of the efficiency of the code. Achieving a lower average length means a more efficient code, which is a central objective in data compression and coding theory.

Code Transformation and Conversion

The process of code transformation involves converting a code from one alphabet to another while preserving optimality properties, as far as possible. In the context of this problem, a quaternary code is transformed to a binary code by mapping each symbol of the higher base to a fixed binary string, typically of fixed length. This conversion can affect the overall average code length and performance compared to directly designed binary codes.

Tightness of Bounds

The concept of tightness in the context of bounds refers to whether a given theoretical limit can be achieved by some instance or example. In the study of code conversion between quaternary and binary systems, it is important to note examples where the lower bound is achieved (making it tight) and understand that while upper bounds provide limits on performance, they may not be improvable in all cases. Demonstrating tightness involves constructing or rigorous analysis of specific distributions where the bounds are met exactly.

Binary Huffman Code

A binary Huffman code is a specific application of Huffman coding where the code tree is binary, meaning that each node has at most two children. The binary code assigns a sequence of 0s and 1s to each symbol such that the total average length — defined as the sum of the product of the probability of each symbol and its corresponding codeword length — is minimized while ensuring that no codeword is a prefix of another.

Huffman Coding

Huffman coding is a greedy algorithm used to construct an optimal prefix code that minimizes the expected average length of the codewords based on symbol probabilities. The algorithm builds a binary tree (or a tree of any desired branching factor) by iteratively combining the two (or more) least probable symbols until a complete tree is formed, resulting in a variable-length code that is optimal for the given distribution.

Quaternary Huffman Code

A quaternary Huffman code extends the Huffman coding idea to an alphabet of four symbols, where the tree is constructed such that each internal node has up to four children. This approach uses a higher branching factor, which can lead to different average code lengths compared to the binary code. The process of constructing a quaternary tree involves combining the lowest probability symbols in groups that reflect the target alphabet size.

Transcript

00:01 They give us these frequencies here, and they want us to use hoffman coding to encode the symbols.

00:07 And we're going to use two different ways to tiebreak when we're looking for the minimum frequency.

00:13 So the first way is we're going to take the tree with the largest number of vertices and then combine those.

00:19 And then the other way is going to be using the lowest amount of vertices and combining those trees.

00:27 Then they want us to figure out the average number it's required to encode each symbol as well as the variation.

00:36 And then they want us to determine what is the smaller variance.

00:42 So let's go ahead and do this first way.

00:47 So first.

00:49 So this was most vertices.

00:56 So let's go ahead and make our little tree to start.

01:00 So it'd be a, b, c, d, and e.

01:07 So we're going to have 0 .4, 0 .2, 0 .2, 0 .1, 0 .1 .1.

01:16 So 0 .1 to 0 .1 are our smallest.

01:20 So let's go ahead and scoot these down a little bit.

01:22 So we're going to turn these into a single tree.

01:26 So it doesn't matter if we put 0 .1 to left or d to left or e to left.

01:31 Left since both of them have the same frequency.

01:33 So let's just do it this way.

01:34 So we have 0 and 1.

01:37 And then we combine those into 0 .2.

01:41 Let's get rid of those two frequencies and scoot our letters up.

01:48 All right.

01:50 Next, well, we would have 0 .2, 0 .2 and 0 .2.

01:55 So since we're combining the most vertices, we for sure have to have this one over here on the right.

02:03 And then we just need to decide between these other two.

02:09 And again, it won't matter which one we choose, so i'm just going to pick c.

02:14 So i'll combine these two together.

02:19 And it won't matter which one we put to the left, which one we put to the right.

02:25 So i'll just put c to the left, and then our tree over here to the right.

02:30 So that's going to be 0 .1.

02:32 And again, let's get rid of these old frequencies, and then we can add those together to give us 0 .4.

02:43 All right.

02:44 Let me scoot these up a little bit.

02:50 So now we have 0 .4, 0 .2, and 0 .4.

02:55 Well, 0 .2 for sure has to go.

02:59 And then it's going to be between 0 .4 or a or our tree here.

03:05 But remember, we're going to choose the one with the most vertices.

03:08 So that means we want to choose the tree over here on the right.

03:18 And now remember we want the larger frequency to be to the left, the smaller to be to the right.

03:25 So we have the 0 .4 to left, 0 .2 to the right.

03:29 So this would be our tree like this, and then we would add those up to give us 0 .6.

03:38 And then let's go ahead and get rid of those old frequencies.

03:43 And now all we need to do is compare 0 .6 and 0 .4, well, 0 .6.

03:51 Is larger, so that should go to left.

03:57 So then we can come over here.

04:00 And let's just get rid of these frequencies, since this is going to be the last one.

04:06 And again, to the left is zero, and to the right is one.

04:09 So let's go ahead and write out this.

04:13 So that would be a is 1.

04:17 B is 0 .1.

04:21 C is going to be 0 ,000.

04:27 D is going to be 0 .0.

04:30 010, so 0 ,010, and then e, well, the only difference between d and e is that last position.

04:40 So that would be 0 -0 -1 -1.

04:44 1, 2, 3, 4.

04:46 All right, i didn't miss any letters.

04:47 All right, now they want us to figure out average number of bits.

04:51 So let's do that.

04:52 So let's figure out how many bits each of these has.

04:55 And we can put the frequency here, multiply, add everything up.

04:58 And then we'll also need this for our variance.

05:03 So, a only has one bit, b has two, c has three, d has four, and e has four.

05:15 And then a's frequency was 0 .4, b was 0 .2, c is 0 .2, d is 0 .1, and e is 0 .1.

05:25 So now we want to multiply these horizontally.

05:31 So that should give us 0 .4.

05:34 0 .4, 0 .6, 0 .4, and 0 .4.

05:40 And then we want to add those up.

05:45 And it looks like we get 2 .2.

05:48 So this is the average bits for the most vertices.

05:57 And so actually, one thing i should say is that your coding here might be slightly different, because remember we just arbitrarily chose some of them since they had the same amount.

06:06 So we still had to make some arbitrary choice.

06:10 We didn't have an exact way to do it.

06:12 So yours might be slightly different for this, but the average bits should still be the same.

06:17 All right, let's go ahead and find the variance for this one, though.

06:22 And so remember to get variance.

06:25 So i'll just write this down here.

06:28 This is going to be equal to the sum of, so our weights that we got.

06:38 So w -i -minus the average, and then we square this and then multiply this by the frequency.

06:56 Oh, and by wake here, we just mean the number of bits...