Information and Coding Theory

1. Source Coding.- 1.1 Definitions and Examples.- 1.2 Uniquely Decodable Codes.- 1.3 Instantaneous Codes.- 1.4 Constructing Instantaneous Codes.- 1.5 Kraft’s Inequality.- 1.6 McMillan’s Inequality.- 1.7 Comments on Kraft’s and McMillan’s Inequalities.- 1.8 Supplementary Exercises.- 2. Optimal Codes.- 2.1 Optimality.- 2.2 Binary Huffman Codes.- 2.3 Average Word-length of Huffman Codes.- 2.4 Optimality of Binary Huffman Codes.- 2.5 r-ary Huffman Codes.- 2.6 Extensions of Sources.- 2.7 Supplementary Exercises.- 3. Entropy.- 3.1 Information and Entropy.- 3.2 Properties of the Entropy Function.- 3.3 Entropy and Average Word-length.- 3.4 Shannon-Fano Coding.- 3.5 Entropy of Extensions and Products.- 3.6 Shannon’s First Theorem.- 3.7 An Example of Shannon’s First Theorem.- 3.8 Supplementary Exercises.- 4. Information Channels.- 4.1 Notation and Definitions.- 4.2 The Binary Symmetric Channel.- 4.3 System Entropies.- 4.4 System Entropies for the Binary Symmetric Channel.- 4.5 Extension of Shannon’s First Theorem to Information Channels.- 4.6 Mutual Information.- 4.7 Mutual Information for the Binary Symmetric Channel.- 4.8 Channel Capacity.- 4.9 Supplementary Exercises.- 5. Using an Unreliable Channel.- 5.1 Decision Rules.- 5.2 An Example of Improved Reliability.- 5.3 Hamming Distance.- 5.4 Statement and Outline Proof of Shannon’s Theorem.- 5.5 The Converse of Shannon’s Theorem.- 5.6 Comments on Shannon’s Theorem.- 5.7 Supplementary Exercises.- 6. Error-correcting Codes.- 6.1 Introductory Concepts.- 6.2 Examples of Codes.- 6.3 Minimum Distance.- 6.4 Hamming’s Sphere-packing Bound.- 6.5 The Gilbert-Varshamov Bound.- 6.6 Hadamard Matrices and Codes.- 6.7 Supplementary Exercises.- 7. Linear Codes.- 7.1 Matrix Description of Linear Codes.- 7.2 Equivalence of Linear Codes.- 7.3 Minimum Distance of Linear Codes.- 7.4 The Hamming Codes.- 7.5 The Golay Codes.- 7.6 The Standard Array.- 7.7 Syndrome Decoding.- 7.8 Supplementary Exercises.- Suggestions for Further Reading.- Appendix A. Proof of the Sardinas-Patterson Theorem.- Appendix B. The Law of Large Numbers.- Appendix C. Proof of Shannon’s Fundamental Theorem.- Solutions to Exercises.- Index of Symbols and Abbreviations.