Website: https://isi.ee.ethz.ch/teaching/courses/it1.html
Buch Information Theory von Cover & Thomas.
Oxford Video Lectures: https://www.youtube.com/watch?v=ScX2aBFyrVU&list=PL4d5ZtfQonW3iAhXvTYCnoGEeRhxhKHMc
Notes
Exercises
Published on wednesday, solutions published the week after.
Lecture Contents to Book
Mapping of Elements of Information Theory (Cover & Thomas, 2nd ed.) onto the IT I course (Lapidoth, HS 2026). Based on the official topic list, the lecture program (cites §2.1–2.10 and §5.1–5.8 so far), and the 2018 exam.
Core: read fully
| Chapter | Sections | Content / why |
|---|---|---|
| 2 Entropy, Relative Entropy, MI | 2.1–2.8, 2.10 | Entropy, conditional entropy, MI, chain rules, Jensen, log-sum, convexity of , concavity of and (Thm 2.7.2–2.7.4), data processing, Fano |
| 3 AEP | 3.1–3.2 | Weak typicality , compression via the typical set (Exam 2018 P2a-ii) |
| 5 Data Compression | 5.1–5.6, 5.8 | Code classes, Kraft, McMillan (5.5), bounds on optimal length, block coding to approach , Huffman and its optimality. Thm 5.4.3 (wrong code costs ) is exactly Exam P4 |
| 7 Channel Capacity | 7.1–7.7, 7.9, 7.12, 7.13 | Capacity examples, (weakly) symmetric channels, properties of , definitions, joint typicality, coding theorem, converse via Fano, feedback, source–channel separation (Exam P5–P7) |
| 8 Differential Entropy | 8.1–8.6 | Whole chapter, including “Gaussian maximizes ” (Thm 8.6.5) |
| 9 Gaussian Channel | 9.1–9.2 | , achievability and converse |
| 10 Rate Distortion | 10.1–10.2, 10.3.1–10.3.2, 10.4–10.6 | Definitions, binary source , Gaussian source, converse, achievability, strongly typical sequences (Exam P7, P8) |
Borderline: skim, decide based on lectures/exercises
| Section | Comment |
|---|---|
| 2.9 Sufficient statistics | Not on topic list; nice application of DPI |
| 3.3 High-probability sets | Sometimes used to argue the typical set is “the smallest” high-probability set |
| 5.7 Comments on Huffman | Short, cheap to read |
| 7.8 Zero-error codes, 7.10 Equality in converse | Short; 7.10 helps understanding the converse |
| 9.3 Bandlimited, 9.4 Parallel Gaussian | 9.4 (water-filling) is the book’s best worked Kuhn–Tucker example |
| 10.3.3 Reverse water-filling, 10.7 Characterization of | Only if lectures cover them |
| 11.1 Method of types | Background for strong typicality |
Not relevant
- Ch. 4 (entropy rates, Markov sources), apart from the Markov chain definition used in §2.8
- Ch. 6 (gambling)
- §5.9–5.11 (Shannon–Fano–Elias, competitive optimality, generating distributions from coins)
- §7.11 (Hamming codes), §9.5–9.6 (colored noise, Gaussian feedback), §10.8 (Blahut–Arimoto)
- Ch. 11–17 (statistics/Sanov, max-entropy, universal coding/Lempel–Ziv, Kolmogorov complexity, network IT, portfolio theory, inequalities)
Practice material
End-of-chapter problems of C&T Ch. 2, 5, 7, 10 are the best practice in the book; many course exercises are taken from or adapted from them.
Exam 2018 to book map
| Problem | Topic | Sections |
|---|---|---|
| P1 | Optimal codes, block coding, entropy of BEC output | 5.4–5.6, 5.8, 2.1, 7.1 |
| P2 | Weak and strong typicality, typicality under functions | 3.1, 10.6 (+ lecture def.) |
| P3 | Converse for compression with side information | 2.10, 7.9 (structure) |
| P4 | Shannon code for mismatched law | 5.4 (Thm 5.4.3) |
| P5 | Capacity of specific channels | 7.1–7.3 + Kuhn–Tucker (lecture) |
| P6 | Channel with random but fixed state | 7.x + lecture/exercises |
| P7 | Separation, binary rate distortion | 7.13, 10.3.1 |
| P8 | Rate distortion for a function of the source | 10.2–10.3.1 + exercises |