IMRL LAB, Policy Gradient/REINFORCE
Table des matières
In this final TP, we want to apply the method of direct policy optimization by gradient ascent to the problem of controlling a spacecraft, which may perhaps allow you to get an internship at SpaceX.
LunarLander Environment
This environment allows you to control a spacecraft that must land, gently, on the platform indicated by two flags.
Modeling as a Markov Decision Process
Reminder: An MDP is a tuple \((S,A,H,T,R,\gamma)\) with
- \(S\) the set of states
- \(A\) the set of actions
- \(H\) the horizon
- \(T\) the transition function (also called dynamics)
- \(R\) the reward function
- \(\gamma\) the discount factor
In our case, we will have:
- \(S\) an environment state is a vector of dimension 8: abscissa and ordinate of the module, its velocity (in \(x\) and \(y\)), its angle, its angular velocity, and two booleans, each indicating if the corresponding leg is in contact with the ground. We will use tile coding to encode these states (jointly with the actions)
- 4 actions: do nothing, ignite the left engine, ignite the central engine, ignite the right engine
- \(H\) is considered infinite (in practice the number of actions is limited to 200)
- We assume the dynamics are unknown (our agent doesn't know the physics)
- The reward is described here.
- the discount factor is arbitrarily set to 0.99.
Using Gymnasium
The goal of the TP is to create an agent for this environment, and have it learn an optimal policy \(\pi\) by REINFORCE.
TP Steps:
- Download the notebook here
- Open a browser window on colab
- Load the notebook (File → Open Notebook …)
- Read/evaluate/complete the notebook
- Save the notebook, and download it (File → Download .ipynb/)
- Log in to the LMS, and go to the robotics course page
- Submit your notebook on the submission interface for the relevant week
Remarks
You can install jupyter notebook on your personal computer and work locally. You will need to install jupyter and the necessary libraries in this case.