Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 × 2hr lectures)
Class limit: max. 10 students
Prerequisites: Undergraduate-level probability and statistics (distributions, Bayes theorem), linear algebra (matrix operations, eigenvalues), and basic multivariate calculus. No prior physics or thermodynamics assumed.
Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for reasoning about the foundations of intelligent systems.
Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The course covers:
- the Boltzmann distribution, free energy, and the partition function as a generating function,
- Shannon entropy and its formal equivalence to thermodynamic entropy; the exponential family as the MaxEnt family,
- Maxwell's demon and Landauer's principle: the thermodynamic cost of decision-making,
- information geometry: the Fisher metric, dually flat geometry, and natural gradient descent,
- multi-information, an entropy game, quantum information,
- further advanced topics such as probability transport, Schrödinger bridges, and information-theoretic limits on intelligent agency.
Each lecture is followed by a formative self-study exercise exploring the session's central concept from thermodynamic, information-theoretic, and Bayesian perspectives.
Equip students to reason rigorously about entropy across thermodynamics, information theory, and Bayesian inference, and to evaluate claims about intelligent systems using information-theoretic constraints.
Four take-home mini-project worksheets (15% each, 60% total), each consisting of a short Python notebook and a written reflection. Four short in-class Moodle quizzes (10% each, 40% total).
Shannon (1948), Bell System Technical Journal; Jaynes (1957), Physical Review; Landauer (1961), IBM Journal; Amari & Nagaoka (2000), Methods of Information Geometry (Chapters 1–3). Cover & Thomas (2006) and MacKay (2003) as background references.