ECE423 Information and Coding Theory

Topics covered include the source-channel communication model, the axiomatic model to define uncertainty and entropy, important information quantities such as joint entropy, conditional entropy, relative entropy and mutual information, information equalities and inequalities, the asymptotic equipartition principle, entropy rate, entropy of Markov chains, instantaneously decodable and prefix codes, Kraft/McMillan inequalities, Shannon’s source coding theorems, Huffman entropy coding, Shannon, Shannon-Fano, Shannon-Fano-Elias and arithmetic coding, mathematical definition of a communication channel, mutual information and its importance in communications, channel capacity, channel coding, Shannon’s channel coding theorems, channel capacity of symmetric channels, feedback channels, continuous sources and differential entropy, Gaussian channel, continuous channel capacity, water-filling principle, rate-distortion theory, Shannon’s rate-distortion theorems, Gaussian source with squared error distance and Bernoulli source with Hamming distance rate-distortion curves, multiple source quantization, inverse water-filling principle, Arimoto-Blahut algorithms for channel capacity and rate-distortion calculation, network coding.
All topics are covered in the classroom with a presentation of corresponding theory and examples, followed by homework assignments that help students understand and apply subjects taught in classroom. A midterm exam covers the first half of the course (source-coding) while a final exam covers the second half (channel coding/rate-distortion). Optional projects help students advance their knowledge in selected topics, requiring both theoretical analysis and software implementations.

• Source-channel communication model
• Axiomatic model of uncertainty and entropy
• Other information quantities – joint entropy, conditional entropy, relative entropy and mutual information
• Information equalities and inequalities
• Asymptotic equipartition principle
• Entropy rate, entropy of Markov chains
• Instantaneously decodable and prefix codes, Kraft/McMillan inequalities
• Shannon’s source coding theorems
• Huffman entropy coding, Shannon, Shannon-Fano, Shannon-Fano-Elias and arithmetic coding
• Mathematical definition of a communication channel, mutual information and its importance in communications
• Channel capacity, channel coding
• Shannon’s channel coding theorems
• Channel capacity of symmetric channels, feedback channels
• Continuous sources and differential entropy
• Gaussian channel, continuous channel capacity
• Water-filling principle
• Rate-distortion theory
• Shannon’s rate-distortion theorems
• Gaussian source with squared error distance and Bernoulli source with Hamming distance rate-distortion curves
• Multiple source quantization, inverse water-filling principle
• Arimoto-Blahut algorithms for channel capacity and rate-distortion calculation
• Network coding

Homework assignments, midterm and final exams, as well as optional projects are also significant components of the course.

This course introduces students to Shannon’s information theory, the mathematical basis of telecommunications.
This module is an advanced, deeply mathematical class that provides solid and rigorous foundation for telecommunication and signal compression subjects. It is essential for students who wish to continue in post-graduate studies in these fields. It can lead to multiple topics suitable for diploma theses.
By the end of the course, students must be able to solve source-coding and channel-coding problems, explain fundamental issues in telecommunications and solve related optimization problems. Typical students will have acquired the following skills:

• Understanding the nature of entropy and coding principles, as well as channel capacity and rate-distortion for discrete and continuous sources.
• Ability to calculate related information quantities for selected sources and channels.
• Optimize system parameters for multiple source/multiple channel systems.
• Present and explain their results in engineering terms and a professional manner
• The skills to continue further and study advanced related subjects.