5. Data Compression

5.6 Huffman Codes

Huffman-Codes are equivalent to the game “20 questions”:

• the code = sequences of answers to the questions
  • 0101 = no, yes, no, yes for example
• we want to find the questions that allow us to minimise the number of questions we have to ask
  • i.e. recursively split the set of codewords in half (yes or no)

Note: main property of a Huffman-Code

• by construction, any huffman-code assigns the two least likely symbols the same code, with only the last digit differing!

We have to do it from the back greedily like this because forward (or any other order) doesn’t work:

• Counterexample for the forward procedure. Take $p=(0.3,0.3,0.2,0.2)$, symbols $a,b,c,d$.
• Forward insertion. Start with $a=0$, $b=1$. Add $c$ by splitting the least likely leaf, $b$: $b=10$, $c=11$. Add $d$ by splitting the least likely leaf, now $c$: $c=110$, $d=111$. The lengths are $(1,2,3,3)$, so

$$L=0.3+0.6+0.6+0.6=\mathrm{2.1.}$$

• Huffman. Merge $c,d$ into $0.4$, giving $(0.3,0.3,0.4)$. Merge $a,b$ into $0.6$, giving $(0.6,0.4)$. Expanding back gives lengths $(2,2,2,2)$, so

$$L=2.$$

Huffman-Codes are optimal → via proof

• reduction
• induction on the construction of the code
  • start with 2 least likely → obvious code
  • add next $x_i$
    • it’s more likely
    • so the two from before are the ones we extend (from $p^{\prime }$ to $p$ via the extension by one symbol)
  • thus we get next step of Huffman-construction
  • this is still optimal
  • repeat