IMRL, LAB 3

This week, we return to the frozen lake problem, but to solve it we will use the Monte-Carlo method, assuming that we do not know the dynamics \(T\). Good news: no more tinkering with env.unwrapped.P!

The first part of the TP repeats the previous assignment. Only the notebook changes.

• The Frozen Lake!

Original English Version here

It is winter. You are playing frisbee with your friends in the park when an errant throw causes the frisbee to land in the middle of the lake. The water is frozen over almost the entire surface, but there are a few holes where the ice has melted. If you find yourself above one of these holes, you will fall into the icy water. Currently, there is a global shortage of frisbees, so it is imperative that you go get the frisbee from the lake and bring it back. Be careful, the ice is slippery, and you will not necessarily move in the direction you chose.

The lake will be modeled by a graph, represented informally by an illustration like the one below:

SFFFFFFF
FFFFFFFF
FFFHFFFF
FFFFFHFF
FFFHFFFF
FHHFFFHF
FHFFHFHF
FFFHFFFG

Each letter corresponds to a vertex, and there is an arc between a vertex and each of its neighbors (up/down/left/right).

The letters indicate what each state contains:

S

your starting position on the frozen lake

G

the position of the frisbee

F

the ground is frozen on this vertex

H

there is a hole

The goal is to provide a sequence of actions to reach vertex G from your initial position S while avoiding holes H. There are 4 actions (left/down/right/up) that indicate which neighboring vertex you want to reach. Be careful: just because you decide to go in one direction doesn't mean you'll get there—you may slip and end up randomly on another neighboring vertex!

• Modeling as a Markov Decision Process

Reminder: An MDP is a tuple \((S,A,H,T,R,\gamma)\) with

• \(S\) the set of states
• \(A\) the set of actions
• \(H\) the horizon
• \(T\) the transition function (also called dynamics)
• \(R\) the reward function
• \(\gamma\) the discount factor

In our case, there will be:

• 64 states numbered 0 to 63 from left to right and top to bottom
• 4 actions (left/down/right/up) numbered 0 to 3
• \(H\) is infinite. The shortest action sequences from S to G are of length 14, but longer (and infinite) sequences exist.
• For transitions, we assume this time that the dynamics are unknown (you can refer to the previous TP otherwise)
• the reward is 1 if we are in the state labeled G, and 0 otherwise
• the discount factor is arbitrarily set to 0.9
• Using Gymnasium (OpenAI Gym)

The Gymnasium website offers environment models for many robotics and reinforcement learning tasks. It is mainly used to test and compare new algorithms. The task that interests us today is Frozen Lake.

Through the Python module gym you can load these environments and create an agent that interacts with them.

The goal of the TP is to create an agent for this environment and have it learn a policy \(\pi\) via the Monte-Carlo method seen in class that allows the agent to reliably retrieve the frisbee.