{"cells":[{"cell_type":"markdown","id":"39eddb07-69e1-414f-ba3b-7f5e77ef8b4a","metadata":{},"source":"Q-learning for Mountain Car\n===========================\n\n**Author:** Joseph Le Roux\n\n**Date:** 2026-07-19\n\n"},{"cell_type":"markdown","id":"12b3517c-02a1-479f-a21c-1d5c273075bc","metadata":{},"source":["- AUTHOR: Joseph Le Roux\n","- DATE: 2026-07-19\n","- DESCRIPTION: Q-learning for Mountain Car\n"]},{"cell_type":"markdown","id":"872b8c43-9588-4562-b7c6-f6903db8d363","metadata":{},"source":["## Environment\n\n"]},{"cell_type":"markdown","id":"2e904dd4-629e-4032-847f-c5e746f992cf","metadata":{},"source":["To visualize this environment (render function), you need a graphical interface.\n\nIf you are using a notebook on your personal computer, nothing to do. If you are using Colab, it is not possible to directly visualize environments, but you can record an agent's evolution in its environment as a video and then play it back.\n\nFor this, the following code is necessary. (warning, it is quite lengthy, especially the first time)\n\n"]},{"cell_type":"code","execution_count":1,"id":"485ac285-ee78-48b6-9070-c41efd04b95b","metadata":{},"outputs":[],"source":["# save your best videos\n\n!pip install pyvirtualdisplay\n!sudo apt-get install -y xvfb ffmpeg"]},{"cell_type":"code","execution_count":1,"id":"b6b5f653-b801-4641-88d3-ec1de17d0863","metadata":{},"outputs":[],"source":["import gymnasium as gym\nfrom gymnasium.wrappers import RecordEpisodeStatistics, RecordVideo\n\nimport random\nimport matplotlib\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport math\nimport glob\nimport io\nimport base64\nfrom IPython.display import HTML\n\nfrom IPython import display as ipythondisplay\n\nfrom pyvirtualdisplay import Display\ndisplay = Display(visible=0, size=(1400, 900))\ndisplay.start()\n\n\"\"\"\nUtility functions to enable video recording of gym environment and displaying it\nTo enable video, just do \"env = wrap_env(env)\"\"\n\"\"\"\n\ndef show_video(dir='./video'):\n  mp4list = glob.glob(dir+'/*.mp4')\n  for mp4 in mp4list:\n    video = io.open(mp4, 'r+b').read()\n    encoded = base64.b64encode(video)\n    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay\n                loop controls style=\"height: 400px;\">\n                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n             </video>'''.format(encoded.decode('ascii'))))\n  else:\n    print(\"Could not find video\")\n\n\ndef wrap_env(env, dir='./video', freq=1000):\n  env = RecordVideo(\n      env,\n      video_folder=dir,\n      name_prefix=\"training\",\n      episode_trigger=lambda x: x % freq == 0  # Only record every freq episode\n      )\n  return env"]},{"cell_type":"markdown","id":"b59bf9df-4b37-4efc-92fb-6050250c43b3","metadata":{},"source":["If you are using a Jupyter notebook on your own computer (not Colab), evaluate the following code instead:\n\n"]},{"cell_type":"code","execution_count":1,"id":"7a50a071-9ce0-4dd5-b305-5b5459d731fd","metadata":{},"outputs":[],"source":["# gym contains the RL problem interface through environment\nimport gymnasium\n#for math + autodiff (for neural networks)\nimport torch"]},{"cell_type":"code","execution_count":1,"id":"454c45d5-d358-4c09-86cb-aff254b0afb0","metadata":{},"outputs":[],"source":["env = gym.make('MountainCar-v0', render_mode=\"rgb_array\")\nprint(env)\nenv = wrap_env(env,1) #to be able to record the video, comment otherwise\n\n#usage example\nenv.reset() #to initialize the env\nenv.render() # display (to buffer if wrapped, otherwise to screen)\nenv.step(env.action_space.sample()) #move randomly\nenv.render() # display\nenv.close()  # close env (necessary when wrapped)\nshow_video() # when wrapped"]},{"cell_type":"markdown","id":"e8608168-aa27-42e8-96aa-ffd77cb3849d","metadata":{},"source":["By default, the set of states is not finite. It is a subset of $\\mathbb{R}^2$ (position, velocity), but the coordinates are bounded. You can verify this through the following variables:\n\n"]},{"cell_type":"code","execution_count":1,"id":"69466d95-9850-4d26-a5ae-93cf6d5c5c21","metadata":{},"outputs":[],"source":["print(env.observation_space)\nprint(env.observation_space.low)\nprint(env.observation_space.high)"]},{"cell_type":"markdown","id":"cb1235f7-299b-4c43-ba8f-6484dbaa6efb","metadata":{},"source":["1.  What interval does the cart position belong to?\n2.  What interval does the cart velocity belong to?\n\n**your answer here**\n\n"]},{"cell_type":"code","execution_count":1,"id":"6074a087-593f-4797-a5b9-99bd832b06db","metadata":{},"outputs":[],"source":["def evaluate(env, policy = None, trials=1000, render_last_trials=0):\n\n  success = 0\n  lengths = []\n  fails = []\n\n  for trial in range(trials):\n    obs,_ = env.reset()\n\n    terminated, truncated = False, False\n    length = 0\n\n    while not (terminated or truncated):\n      action = env.action_space.sample () if policy is None else policy.get_policy(env, obs)\n      obs,reward,terminated,truncated,_ = env.step(action)\n      length +=1\n\n      if terminated and (obs[0] >= 0.5):\n        success +=1\n        lengths.append(length)\n      if truncated or (terminated and obs[0] < 0.5):\n          fails.append(length)\n\n  lengths = torch.tensor(lengths if len(lengths) >0 else [0], dtype=torch.float)\n  fails = torch.tensor(fails if len(fails) >0 else [0], dtype=torch.float)\n\n  print('----------------------------------------------')\n  print('You succeeded {:.2f} % of the time'.format((success/trials) * 100))\n\n  print('On average you need {:.0f} moves to succeed'.format(torch.mean(lengths)))\n  print('The shortest solution found has {:.0f} moves'.format(torch.min(lengths)))\n  print('The longest solution found has {:.0f} moves'.format(torch.max(lengths)))\n\n  print('The shortest failure has {:.0f} moves'.format(torch.min(fails)))\n  print('The longest failure has {:.0f} moves'.format(torch.max(fails)))\n  print('----------------------------------------------')"]},{"cell_type":"code","execution_count":1,"id":"20fe6215-0659-4ac7-91cd-79cdb772b250","metadata":{},"outputs":[],"source":["def test(agent= None,maxit=1000,with_video=True):\n  # load the mountain car problem\n  env = gym.make('MountainCar-v0', render_mode=\"rgb_array\")\n  if agent is not None:\n    #agent.reset(env)\n    with torch.no_grad():\n      agent.compute_policy(env, max_iterations=maxit, debug=True)\n\n  evaluate(env,agent,trials=100, render_last_trials=10)\n\n  if with_video:\n    env = wrap_env(env,freq=1)\n    evaluate(env,agent,trials=1)\n\n  env.close()\n  if with_video:\n    show_video()"]},{"cell_type":"code","execution_count":1,"id":"900ec7e0-fd67-42ab-999f-e238f0a31856","metadata":{},"outputs":[],"source":["# uncomment to verify everything works correctly\n# test(None,5000)"]},{"cell_type":"markdown","id":"a02fadb5-eed8-4ab6-9da5-c173d0f6e1c8","metadata":{},"source":["## Implementation\n\n"]},{"cell_type":"code","execution_count":1,"id":"07f751f6-19ad-43b2-814f-da6d142d99fd","metadata":{},"outputs":[],"source":["class QLearning:\n  def __init__(self, env, step_x = 10, step_v = 100):\n    self.buckets = torch.tensor([step_x,step_v])\n\n    self.reset(env)\n\n  #initialize tables Q[x,v,a] and pi[x,v]\n  def reset(self, env):\n    pass\n    ### your code begins here\n    #self.Q =  ...\n    #self.pi = ...\n\n\n  def get_epsilon_greedy_policy(self, env, state, epsilon):\n    ### your code begins here\n\n\n  # do max_iterations trials. Each time:\n  # generate a trajectory using a epsilon-greedy policy based on Q\n  # update Q/pi according to Q-learning method\n  def compute_policy(self, env, gamma = 0.9, max_iterations=100000, base_epsilon = 0.8, alpha =0.2, debug = False):\n\n    epsilon = base_epsilon\n    sum_rewards = 0\n    for trial in range(max_iterations):\n      #your code begins here\n\n      obs = env.reset()\n      terminated, truncated = False, False\n      done\n\n      rewards = 0\n\n      while not (terminated or truncated):\n        #your code begins here\n        #\n        #\n        rewards += reward\n\n      #print average trajectory total reward every 1000 trajectory\n      sum_rewards += rewards\n      if trial % 1000 == 0:\n        print(sum_rewards/1000)\n        sum_rewards = 0\n\n      epsilon -= base_epsilon/max_iterations\n\n    #update policy\n    # your code here\n\n\n  # translate a (x,v) point to a robot state\n  def obs_to_state(self, env, obs):\n      ### your code here\n\n\n  def get_policy(self, env, obs):\n      ### your code here"]},{"cell_type":"code","execution_count":1,"id":"99129111-d703-488f-8177-68e16bd3bd07","metadata":{},"outputs":[],"source":["agent = QLearning(env)\ntest(agent,5000)"]},{"cell_type":"code","execution_count":1,"id":"ba55024a-74f6-406d-8722-9e70aa686ae7","metadata":{},"outputs":[],"source":["test(agent, 10000)"]},{"cell_type":"markdown","id":"94d2f11d-b297-46e1-b714-ccedf5dfb8b8","metadata":{},"source":["## Bonus\n\n"]},{"cell_type":"markdown","id":"4d4cad27-dda4-4ba2-9ed9-247230cf2ffc","metadata":{},"source":["Implement the SARSA algorithm by using your ****QLearning**** class implementation as a base.\n\nThe main difference is the update formula: you should not use the action with the best Q value but instead the action that was sampled to continue the trajectory.\n\n"]},{"cell_type":"code","execution_count":1,"id":"d3ac58be-d7ba-41fd-96f8-74fe159e0ac3","metadata":{},"outputs":[],"source":["class SARSA:\n  pass"]},{"cell_type":"code","execution_count":1,"id":"e4377959-5b8b-42c3-9a93-2c1ceb52ebb4","metadata":{},"outputs":[],"source":["agent = SARSA()\ntest(agent,5000)"]},{"cell_type":"code","execution_count":1,"id":"31f3fb42-f4a7-4c39-b667-180c8d1f4092","metadata":{},"outputs":[],"source":["test(agent, 10000)"]},{"cell_type":"markdown","id":"7ad232a1-f24c-4928-800b-f29969078433","metadata":{},"source":["## Super Bonus\n\n"]},{"cell_type":"markdown","id":"9b6f9a3e-ebe3-41f8-8cc0-78342435e0e5","metadata":{},"source":["Implement the **Expected SARSA** algorithm by using your ****QLearning**** class implementation as a base.\n\nThe main difference is the update formula: you should not use the action with the best Q value but instead use the average of action scores weighted by their probability according to the policy. You can use softmax on the action scores to provide this probability.\n\n"]},{"cell_type":"code","execution_count":1,"id":"2bf92c60-137f-4da7-8336-3094cd1a39bb","metadata":{},"outputs":[],"source":["class ExpectedSARSA:\n  pass"]},{"cell_type":"code","execution_count":1,"id":"ae862c75-80e0-43fa-819c-89c3de9ba5ef","metadata":{},"outputs":[],"source":["agent = ExpectedSARSA()\ntest(agent,5000)"]},{"cell_type":"code","execution_count":1,"id":"654a7795-a41a-48c9-9e6b-13394a8e2f61","metadata":{},"outputs":[],"source":["test(agent, 10000)"]}],"metadata":{"org":{"AUTHOR":"Joseph Le Roux","DATE":"2026-07-19","DESCRIPTION":"Q-learning for Mountain Car"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":5}