Introduction to MDP and RL 2026-2027 (IMRL)
Table of Contents
Syllabus
We will focus on the formalization of decision making for a robotic agent. What strategy should it implement to solve a complex problem with unknown and/or random events? We will use as our main formalism reinforcement learning.
This lecture consists of an introduction and 6 classes. Each class presents a new notion and is followed by a lab session where students are required to implement the new notion to check their understanding. Every lab session starts by a 10-minute quiz, on material presented the weeks before.
IMRL Introduction
Lecture
Introduction and Markov Decision Processes (part 1)
- Introduction to decision problem for autonomous agents
- Markov Decision Processes (MDP)
- The agent-environment model
- Introduction to MDP
- Policy and Decision making
- Concepts: Gains, State Values, State-Action Values
References
- Introduction Course on AI @ Berkeley MDP1 and MDP2
- Introduction to RL chapters 3 and 4 (many examples, beware notations differ slightly)
- Mathematical foundations of RL, 3 first chapters, the best book for this course
- Pieter Abbeel's Lecture slides and video (with convergence sketch of proof)
- David Silver's Lecture : slides on MDP
IMRL Session 1
Lecture
Markov Decision Processes (part 2) + introduction Monte Carlo methods and temporal difference methods
- Markov Decision processes (MDP)
- Optimal (decision) policy
- gain, state values, state-action values
- optimal policy by policy iteration
- optimal policy by value iteration
Lab Session
Introduction python/pytorch/colab: link to lab page if you don't know python
How to get a Frisbee on a frozen lake? link to lab page (1st part)
References
- Introduction to AI MDP1 and MDP2
- Introduction to RL chapters 3 and 4
- Mathematical foundations of RL, 3 first chapters, the best book for this course
- Pieter Abbeel's lecture slides and video (with sketch of convergence proofs)
- David Silver's lecture slides on MDP and dynamic programming
- On iterative algorithms and convergence proofs: slides by Tom Mitchell
IMRL Session 2
Lecture
Monte Carlo methods and temporal difference methods
- Markov Decision Process (MDP) recap
- Monte-Carlo method
- Temporal Difference methods
- Learning \(Q\) table with Q-learning and SARSA
Lab Session
How to get a Frisbee on a frozen lake? link to lab page (2nd part)
References
- Introduction to RL chapters 5 and 6 (examples, and sketches of proof, and many variants)
- Mathematical foundations of RL
- Pieter Abbeel's course slides (pages 1 to 17) and the part corresponding to the video
- David Silver's course : slides on dynamic programming
IMRL Session 3
Lecture
Lab Session
Mountain Car and tabular SARSA/Q-Learning link to lab page
References
- Introduction to RL chapters 9 and 10
- Pieter Abbeel slides (pages 1 to 11 for examples of functions)
IMRL Session 4
Lecture
Function approximation and input encoding
- input encoding and features extraction
- introduction to neural networks
Lab Session
Mountain Car with function and one-hot encoding link to lab page
References
- Introduction to RL chapter 11
- Pieter Abbeel's lecture slides (pages 1 to 11 for examples of function)
- article on mathematical properties of the tile coding representation
IMRL Session 5
Lecture
Direct policy learning : policy gradient algorithm
Lab Session
Mountain Car by function with tile coding link to lab page
IMRL Session 6
Lecture
Follow-up Direct policy learning : policy gradient algorithm and actor/critic policy learning
Lab Session
Lunar Lander link to page
IMRL Exam 10:00-12:00
Only Annotated handouts authorized
General References
- Mainly Mathematical foundations of RL and Introduction to RL 2nd edition
- Pieter Abbeel's lectures are also recommended, for instance advanced robotics and the other lectures referred on this page, but note that they cover more advanced notions than us.
- Moodle INFO2 and Moodle INFO2A