https://en.wikipedia.org/wiki/Mountain_car_problem

A cart is found on rails that run in a straight line between two mountains. The goal is to bring the cart to the summit of the right mountain (at the flag); however, the cart's motor is not powerful enough to climb the mountain in one go. The only solution to succeed is therefore to move forward then backward alternately to gain speed.

Through the environment, you can know the position of the cart on the x-axis as well as its speed (position and speed are bounded). Three actions are possible: accelerate to the left, do nothing, accelerate to the right.

The cart begins each attempt to reach the flag at the summit of the right mountain from a random position and zero velocity. Each attempt lasts a maximum of 200 time steps. If at time step 200, the cart has not reached the summit, it's a failure.

The reward is zero if the cart's position is greater than or equal to 0.5 (the flag's position) and -1 otherwise.

Reminder: An MDP is a tuple \((S,A,H,T,R,\gamma)\) with

• \(S\) the set of states
• \(A\) the set of actions
• \(H\) the horizon
• \(T\) the transition function (also called dynamics)
• \(R\) the reward function
• \(\gamma\) the discount factor

In our case, we will have:

• \(S\) we place ourselves in the framework of a finite number of states. We decide to store states in a table indexed by two intervals (one for position, and one for velocity). We divide the set of positions into intervals of size 0.1 and the set of velocities into intervals of 0.01.
• 3 actions (left/nothing/right) numbered 0 to 2
• \(H=200\) (or infinite if simpler to model)
• We assume the dynamics are unknown (our agent doesn't know the physics)
• the reward for each state is 0 if we are to the right of the flag, -1 otherwise.
• the discount factor is arbitrarily set to 0.9.

The website Gymnasium environment offers models for many robotics tasks and more generally reinforcement learning. It is mainly used to test and compare new algorithms. The task we're interested in today is Mountain Car.

Through the Python module gym you can load these environments and create an agent that interacts with them.

The goal of the TP is to create an agent for this environment, and have it learn a policy \(\pi\) through the SARSA, Q-Learning and Expected SARSA algorithms seen last time.

1. Download the notebook here
2. Open a browser window on colab
3. Load the notebook (File → Open Notebook …)
4. Read/evaluate/complete the notebook
5. Save the notebook, and download it (File → Download .ipynb/)
6. Log in to the LMS, and go to the robotics course page
7. Submit your notebook on the submission interface for the relevant week

You can install jupyter notebook on your personal computer and work locally. You will need to install jupyter and the necessary libraries in this case.