{"cells":[{"cell_type":"markdown","id":"e87f1b4b-8300-4bc3-83fe-c35344bdf59f","metadata":{},"source":"Lunar Lander and Policy Gradient\n================================\n\n**Author:** Joseph Le Roux\n\n**Date:** 2026-07-19\n\n"},{"cell_type":"markdown","id":"c877f911-485e-4420-b1df-37275e8cbbbc","metadata":{},"source":["- AUTHOR: Joseph Le Roux\n","- DATE: 2026-07-19\n","- DESCRIPTION: Lunar Lander and Policy Gradient\n"]},{"cell_type":"markdown","id":"54bc19ed-b8c8-484b-b428-9acc13394e65","metadata":{},"source":["## Environment\n\n"]},{"cell_type":"markdown","id":"e8c4f0c4-4143-4122-aa03-c5f3c92f86eb","metadata":{},"source":["To visualize this environment (render function), you need a graphical interface.\n\nFor this, the following code is necessary. (warning, it is quite lengthy, especially the first time)\n\n"]},{"cell_type":"code","execution_count":1,"id":"c56a7eb7-1c0d-4397-a97d-028e70a824c4","metadata":{},"outputs":[],"source":["# save your best videos\n\n!pip install pyvirtualdisplay\n!sudo apt-get install -y xvfb ffmpeg>&1"]},{"cell_type":"markdown","id":"c5b1642b-bcb2-4b7f-ad39-ae179cd79094","metadata":{},"source":["Import necessary packages and define functions **show<sub>video</sub>** and **wrap<sub>video</sub>**.\n\n"]},{"cell_type":"code","execution_count":1,"id":"1e305afd-016c-4990-910c-d4192bbf52e7","metadata":{},"outputs":[],"source":["!pip install swig\n!pip install \"gymnasium[box2d]\"\n\nimport torch\nimport gymnasium as gym\nfrom gymnasium.wrappers import RecordEpisodeStatistics, RecordVideo\n\nimport random\nimport matplotlib\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport math\nimport glob\nimport io\nimport base64\nfrom IPython.display import HTML\n\nfrom IPython import display as ipythondisplay\n\nfrom pyvirtualdisplay import Display\ndisplay = Display(visible=0, size=(1400, 900))\ndisplay.start()\n\n\"\"\"\nUtility functions to enable video recording of gym environment and displaying it\nTo enable video, just do \"env = wrap_env(env, freq)\"\"\n\"\"\"\n\ndef show_video(dir='./video'):\n  mp4list = glob.glob(dir+'/*.mp4')\n  for mp4 in mp4list:\n    video = io.open(mp4, 'r+b').read()\n    encoded = base64.b64encode(video)\n    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay\n                loop controls style=\"height: 400px;\">\n                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n             </video>'''.format(encoded.decode('ascii'))))\n  else:\n    print(\"Could not find video\")\n\n\ndef wrap_env(env, dir='./video', freq=1000):\n  env = RecordVideo(\n      env,\n      video_folder=dir,\n      name_prefix=\"training\",\n      episode_trigger=lambda x: x % freq == 0  # Only record every freq episode\n      )\n  return env"]},{"cell_type":"markdown","id":"a33cdd19-f61c-4e85-af6a-0c8bff3e8ea0","metadata":{},"source":["## Environnement **Lunar Lander**\n\n"]},{"cell_type":"markdown","id":"19799991-aba4-4640-90ba-4dd93f2747f7","metadata":{},"source":["Check that it is working:\n\n"]},{"cell_type":"code","execution_count":1,"id":"125d33e4-dce4-479b-819f-578774d4d783","metadata":{},"outputs":[],"source":["env = gym.make(\"LunarLander-v3\", render_mode=\"human\")\nprint(env.unwrapped.enable_wind, env.unwrapped.gravity)"]},{"cell_type":"markdown","id":"b3c97f47-73fb-4be2-bc6b-5a90e12c5b07","metadata":{},"source":["And generate the first (short!) video.\n\n"]},{"cell_type":"code","execution_count":1,"id":"4c8a1986-4b57-4a3d-ba20-1d5100f48128","metadata":{},"outputs":[],"source":["env = gym.make(\"LunarLander-v3\", render_mode=\"rgb_array\")\nprint(env)\nenv = wrap_env(env) #to be able to record the video, comment otherwise\n\n#exemple d'utilisation\nenv.reset() #to initialize the env\nenv.render() # display (to buffer if wrapped, otherwise to screen)\nenv.step(env.action_space.sample()) #move randomly\nenv.render() # display\nenv.close()  # close env (necessary when wrapped)\nshow_video() # when wrapped"]},{"cell_type":"markdown","id":"c0e073f8-da9e-4daf-afc5-fa0fe7800555","metadata":{},"source":["As before\n\n"]},{"cell_type":"code","execution_count":1,"id":"78f0feb8-4dce-41a2-8cd2-d532ce05b368","metadata":{},"outputs":[],"source":["def evaluate(env, policy = None, trials=1000, render_last_trials=0):\n\n  success = 0\n  lengths = []\n  fails = []\n\n  for trial in range(trials):\n    obs = env.reset()\n\n    terminated, truncated = False, False\n    length = 0\n    rewards = 0\n    success = False\n\n    while not (terminated or truncated):\n      action = env.action_space.sample () if policy is None else policy.get_policy(env, obs)\n      obs,reward,done,_ = env.step(action)\n      length +=1\n      rewards += reward\n\n      if (terminated or truncated):\n        if rewards >=200:\n          success +=1\n        lengths.append(rewards)\n\n\n\n  print('----------------------------------------------')\n  print('You succeeded {:.2f} % of the time'.format((success/trials) * 100))\n\n  print('On average, you retrieve {:.0f} rewards (vs. +200 for success landing)'.format(torch.mean(torch.tensor(rewards))))\n  print('----------------------------------------------')"]},{"cell_type":"code","execution_count":1,"id":"bdaea327-dd43-4ca5-8b3a-c991cbd00bd5","metadata":{},"outputs":[],"source":["def test(agent= None,maxit=1000,with_video=True):\n  # load the Lunar Lander problem\n  env = gym.make(\"LunarLander-v3\", render_mode=\"rgb_array\")\n  if agent is not None:\n    agent.compute_policy(env, max_iterations=maxit, debug=True)\n\n  evaluate(env,agent,trials=100, render_last_trials=10)\n\n  if with_video:\n    env = wrap_env(env)\n    evaluate(env,agent,trials=1)\n\n  env.close()\n  if with_video:\n    show_video()"]},{"cell_type":"code","execution_count":1,"id":"3b75f987-9b56-42a7-b2c5-9bd7d0a2ae0b","metadata":{},"outputs":[],"source":["# uncomment to check implementation\ntest(None,5000)"]},{"cell_type":"markdown","id":"a84791f3-a7f8-4e3d-b520-c1df32c5e30c","metadata":{},"source":["## Implementation of Tile Coding (by Mr Richard Sutton, himself)\n\n"]},{"cell_type":"markdown","id":"de8c0e71-b13b-48cd-974b-d7099d5f3b9a","metadata":{},"source":["Below, I have copied the implementation of tile coding ditributed by Richard Sutton as support for his book **Introduction to Reinforcement Learning**.\n\nSee Usage [here](http://incompleteideas.net/tiles/tiles3.html).\n\n"]},{"cell_type":"code","execution_count":1,"id":"f3dc79e0-5e02-46ff-88f4-41e5405aece6","metadata":{},"outputs":[],"source":["basehash = hash\n\n#Integer Hash Table\nclass IHT:\n    \"Structure to handle collisions\"\n    def __init__(self, sizeval):\n        self.size = sizeval\n        self.overfullCount = 0\n        self.dictionary = {}\n\n    def __str__(self):\n        \"Prepares a string for printing whenever this object is printed\"\n        return \"Collision table:\" + \\\n               \" size:\" + str(self.size) + \\\n               \" overfullCount:\" + str(self.overfullCount) + \\\n               \" dictionary:\" + str(len(self.dictionary)) + \" items\"\n\n    def count (self):\n        return len(self.dictionary)\n\n    def fullp (self):\n        return len(self.dictionary) >= self.size\n\n    def getindex (self, obj, readonly=False):\n        d = self.dictionary\n        if obj in d: return d[obj]\n        elif readonly: return None\n        size = self.size\n        count = self.count()\n        if count >= size:\n            if self.overfullCount==0: print('IHT full, starting to allow collisions')\n            self.overfullCount += 1\n            return basehash(obj) % self.size\n        else:\n            d[obj] = count\n            return count\n\ndef hashcoords(coordinates, m, readonly=False):\n    if type(m)==IHT: return m.getindex(tuple(coordinates), readonly)\n    if type(m)==int: return basehash(tuple(coordinates)) % m\n    if m==None: return coordinates\n\nfrom math import floor, log\nfrom itertools import zip_longest\n\ndef tiles (ihtORsize, numtilings, floats, ints=[], readonly=False):\n    \"\"\"returns num-tilings tile indices corresponding to the floats and ints\"\"\"\n    qfloats = [floor(f*numtilings) for f in floats]\n    Tiles = []\n    for tiling in range(numtilings):\n        tilingX2 = tiling*2\n        coords = [tiling]\n        b = tiling\n        for q in qfloats:\n            coords.append( (q + b) // numtilings )\n            b += tilingX2\n        coords.extend(ints)\n        Tiles.append(hashcoords(coords, ihtORsize, readonly))\n    return Tiles\n\ndef tileswrap (ihtORsize, numtilings, floats, wrapwidths, ints=[], readonly=False):\n    \"\"\"returns num-tilings tile indices corresponding to the floats and ints, wrapping some floats\"\"\"\n    qfloats = [floor(f*numtilings) for f in floats]\n    Tiles = []\n    for tiling in range(numtilings):\n        tilingX2 = tiling*2\n        coords = [tiling]\n        b = tiling\n        for q, width in zip_longest(qfloats, wrapwidths):\n            c = (q + b%numtilings) // numtilings\n            coords.append(c%width if width else c)\n            b += tilingX2\n        coords.extend(ints)\n        Tiles.append(hashcoords(coords, ihtORsize, readonly))\n    return Tiles"]},{"cell_type":"markdown","id":"4a9ae3e1-1056-402c-90b5-f9c83b214327","metadata":{},"source":["Test and Understand what is going on:\n\n"]},{"cell_type":"code","execution_count":1,"id":"60e4a9ab-6682-45dc-9278-fa4fb3c49c5e","metadata":{},"outputs":[],"source":["iht = IHT(4096)\nprint(tiles (iht, 8, [0.1, 0.2], ints=[0]))\nprint(tiles (iht, 8, [0.1, 0.2], ints=[1]))\n\nprint(tiles (iht, 8, [0.1, 0.3]))\nprint(tiles (iht, 8, [0.1, 0.4]))\nprint(tiles (iht, 8, [0.1, 0.5]))"]},{"cell_type":"markdown","id":"2caf8186-726d-4570-91f5-51bc4b98352f","metadata":{},"source":["## Policy Gradient and Monte-Carlo &#x2013; Reinforce\n\n"]},{"cell_type":"markdown","id":"bd984095-2af4-4268-9629-95d5e2e3b839","metadata":{},"source":["Complete class below to implement parameter learning through policy function:\n\n"]},{"cell_type":"code","execution_count":1,"id":"1b100a3a-3716-44e1-9442-e08aada0d329","metadata":{},"outputs":[],"source":["torch.serialization.add_safe_globals([IHT])\n\nclass PolicyGradient:\n  def __init__(self, file=None):\n\n    #find the optimal tiling size: this one seems to be ok...\n    self.max_size =  98364 #65596 #8192 #16384 #32768\n    self.num_tilings = 4\n    self.tiling_dim = 8\n\n\n    ## Initialize index hash table (IHT) for tile coding.\n    self.iht = IHT(self.max_size)\n\n    # we will use a module to implement an efficient version of the gradient descent/ascent called Adam (see https://arxiv.org/abs/1412.6980)\n    # and use the torch API, this means:\n    # 1. that we need to embed our parameters in an object of class 'Parameter'\n    self.parameters = torch.nn.parameter.Parameter(data=torch.zeros(self.max_size))\n    # 2. that we will have to compute and store gradients before calling the method 'step()' on the optimizer object, defined as:\n    self.optimizer = torch.optim.Adam([self.parameters], lr=1e-3)\n\n    # we will store the model so we can save/load it easily.\n    # Here we load the model if a filename is provided\n    if file is not None:\n      d = torch.load(file,map_location='cpu')\n      self.parameters = d['parameters']\n      self.optimizer.load_state_dict(d['opt'])\n      self.iht = d['iht']\n\n    # Tilecoding software partitions at integer boundaries, so must rescale\n    # dimension space to span tiling_dim x tiling_dim region.\n    self.scale = self.tiling_dim / (env.observation_space.high - env.observation_space.low)\n\n  ## returns a list l of lists l(a)\n  ## where l(a) is the  tile coding for state s and action a\n  ## hint: use function \"tiles\"...\n  def featurize(self, state):\n    actions = [i for i in range(4)]\n    state= self.scale * s\n    #print(scaled_s)\n    ### begin your code here\n    pass\n    ### end your code here\n\n\n  # save the model to file\n  def save(self, file='/content/drive/My Drive/save.pt'):\n    torch.save({'parameters': self.parameters,\n                'opt': self.optimizer.state_dict(),\n                'iht': self.iht},\n                file)\n\n  def compute_policy(self, env, gamma=0.99, max_iterations=100000, alpha = 1e-3, debug=True, avg_baseline=True):\n    alpha = alpha / self.num_tilings\n\n    #number of parameter updates\n    counter = 0\n    # do we need to generate another trajectory?\n    loop = True\n\n    # sum of rewards since the beginning\n    tot_rewards = 0\n\n    #discount\n    gamma = torch.tensor(gamma)\n\n    #number of trajectories (also called episodes)\n    episodes = 0\n\n    while loop:\n\n      terminated, truncated = False, False\n      #initial state\n      state = env.reset()\n      tot_reward = 0.0\n\n      rewards = []\n      actions = []\n      scores = []\n      stateaction_features = []\n\n      episodes +=1\n\n      #generate a trajectory/episode\n      while not (terminated or truncated):\n\n        if (counter % 10000) == 0: self.save()\n\n        counter += 1\n        if counter > max_iterations: loop = False\n\n        # call featurize and transform the list of list of features into a matrix  4xself.max_size\n\n\n        #compute score\n\n\n        #choose an action: do not do it manually, use method sample in torch.distributions.categorical.Categorical\n\n        # perform action\n\n        state = next\n        tot_reward += reward\n\n        #append useful information for the parameters' update\n        actions.append(action)\n        rewards.append(reward)\n        scores.append(score)\n        stateaction_features.append(stateaction_feature)\n\n\n      # trajectory is finished\n      # compute gains for each state in the trajectory\n      gains = torch.empty(len(rewards))\n\n\n\n      #using simple baseline (average gain)\n      if avg_baseline:\n        pass\n\n      #create  gradient if absent as the null vector\n      if self.parameters.grad is None:\n        self.parameters.grad = torch.zeros_like(self.parameters)\n\n      #then update gradient\n      correction = None #replace with correct computation\n      for i in range(len(stateaction_features)):\n       #print(state_features[i].size())\n       for a in range(4):\n          self.parameters.grad[stateaction_features[i][a]] +=  correction[i][a]\n\n      #call step() on on optimizer to perform gradient descent/ascent\n      self.optimizer.step()\n      # reset gradients to None\n      self.optimizer.zero_grad()\n\n      tot_rewards += tot_reward\n\n      if debug and ((episodes)%100 == 0):\n        print(counter, counter/episodes, tot_rewards / episodes)\n\n\n  def get_policy(self, env, state):\n    # returns policy: compute score -> softmax -> sample\n    # cf. method sample() in torch.distributions.categorical.Categorical\n    # your code here\n    pass"]},{"cell_type":"markdown","id":"00697b41-9212-4ab9-bdbd-bdaf105e65e9","metadata":{},"source":["Create the directory for models, in order to reload them in case of disconnection\n\n"]},{"cell_type":"code","execution_count":1,"id":"c85aa7ae-838e-4a8b-90fe-92c56d15406d","metadata":{},"outputs":[],"source":["from google.colab import drive\ndrive.mount('/content/drive')"]},{"cell_type":"code","execution_count":1,"id":"9d179a95-226f-4992-b293-0563ec88170e","metadata":{},"outputs":[],"source":["agent = PolicyGradient()\ntest(agent, 5e4)"]},{"cell_type":"code","execution_count":1,"id":"de74e820-c151-4902-9ad7-109c31f8456d","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e4)"]},{"cell_type":"code","execution_count":1,"id":"f2cb2cc9-2764-4e4c-a8e0-0c30015b8e01","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e4)"]},{"cell_type":"code","execution_count":1,"id":"59a3b0c9-3aef-4c6d-a1e5-2ad7373a9a54","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"4ff56161-0a9e-4d6d-94fc-d876b7ed2efe","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"0261810d-51ec-44d9-8284-567a7691e599","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"5f908bc7-0e4c-4633-9252-29cc27a93976","metadata":{},"outputs":[],"source":["agent = PolicyGradient('/content/drive/My Drive/save.pt')\ntest(agent, 5e5)"]},{"cell_type":"markdown","id":"eaab546c-280b-49d0-8d41-56ce1f20bf85","metadata":{},"source":["## Bonus: Baseline Implementation Using Temporal Difference\n\n"]},{"cell_type":"markdown","id":"f2c704b9-53af-4bd5-9743-3c25a2e2d832","metadata":{},"source":["Instead of using the average return as a baseline, we can use a state-dependent baseline. The idea is to subtract from the return of a state along the current trajectory the expected return for that state, that is, the state value $V(s)$ approximated by a linear function $\\hat{v}(s;w)$.\n\nThe intuition is that if $G_t - \\hat{v}(s_t,w)$ is positive (resp. negative), it means the action following $s_t$ is better (resp. worse) than average, and we should therefore increase (resp. decrease) its probability.\n\nWe can thus use the temporal difference method (as in Q-Learning) to learn the parameters $w$ of $\\hat{v}$.\n\n$\\nabla_w MSE_v(s,w) = \\nabla_w \\frac{1}{2}(v(s) - \\hat{v}(s,w))^2$\n\n$\\nabla_w MSE_v(s,w) \\approx \\nabla_w \\frac{1}{2}(U(s) - \\hat{v}(s,w))^2$ with the Bellman estimator $U(s) = r + \\gamma \\hat{v}(s',w)$\n\n$\\nabla_w MSE_v(s,w) \\approx (\\hat{v}(s,w) - U(s))\\nabla_w \\hat{v}(s,w)$\n\nSince $\\hat{v}(s,w)$ is a linear function, implemented as $\\sum_i w_i \\times x(s)_i$, we finally obtain the approximate gradient:\n\n$\\nabla_w MSE_v(s,w) \\approx (\\hat{v}(s,w) - U(s)) \\times x(s)$\n\n"]},{"cell_type":"code","execution_count":1,"id":"a6ac46a5-2815-41d8-bc0d-49a5293a93bd","metadata":{},"outputs":[],"source":["from numpy import e\n\nclass PolicyGradientBL:\n  def __init__(self, file=None):\n\n    self.max_size =  65596 #8192 #16384 #32768\n    self.num_tilings = 4\n    self.tiling_dim = 8\n\n\n    ## Initialize index hash table (IHT) for tile coding.\n    self.iht = IHT(self.max_size)\n\n    self.parameters = torch.nn.parameter.Parameter(data=torch.zeros(self.max_size))\n    self.optimizer = torch.optim.Adam([self.parameters], lr=1e-3)\n\n\n    self.max_size_bl = 16384\n    self.num_tilings_bl = 4\n    self.tiling_dim_bl = 8\n\n    self.iht_bl = IHT(self.max_size_bl)\n\n    self.parameters_bl = torch.nn.parameter.Parameter(data=torch.zeros(self.max_size_bl))\n    self.optimizer_bl = torch.optim.Adam([self.parameters_bl], lr=1e-3)\n\n    if file is not None:\n      d = torch.load(file,map_location='cpu')\n      self.parameters = d['parameters']\n      self.optimizer = d['opt']\n      self.iht = d['iht']\n\n      self.parameters_bl = d['parameters_bl']\n      self.optimizer_bl = d['opt_bl']\n      self.iht_bl = d['iht_bl']\n\n\n\n\n    # Tilecoding software partitions at integer boundaries, so must rescale\n    # dimension space to span tiling_dim x tiling_dim region.\n    self.scale = self.tiling_dim / (env.observation_space.high - env.observation_space.low)\n\n  ## returns a list l of lists l(a)\n  ## where l(a) is the  tile coding for state s and action a\n  ## hint: use function \"tiles\"...\n  def featurize(self, s):\n    pass\n\n\n  def save(self, file='/content/drive/My Drive/save_with_bl.pt'):\n    torch.save({'parameters': self.parameters,\n                'opt': self.optimizer,\n                'iht': self.iht,\n                'parameters_bl': self.parameters_bl,\n                'opt_bl': self.optimizer_bl, 'iht_bl': self.iht_bl,},\n               file)\n\n\n  def compute_policy(self, env, gamma=0.99, max_iterations=100000, alpha = 1e-3, debug=True):\n    alpha = alpha / self.num_tilings\n\n    counter = 0\n    loop = True\n\n    tot_rewards = 0\n\n    gamma = torch.tensor(gamma)\n\n    episodes = 0\n\n    while loop:\n\n      done = False\n      state = env.reset()\n      tot_reward = 0.0\n\n      rewards = []\n      actions = []\n      scores = []\n      state_action_features = []\n      state_features = []\n      state_scores = []\n\n\n      episodes +=1\n\n      while not done:\n\n        if counter % 10000 == 0: self.save()\n\n        counter += 1\n        if counter > max_iterations: loop = False\n\n        # call featurize and transform the list of list of features into a matrix  4xself.max_size\n        # do it for the policy function and for the gain estimator  function\n\n\n        #compute scores (for the policy and for the gain estimator)\n\n\n        #choose an action: do not do it manually, use method sample in torch.distributions.categorical.Categorical\n\n        # perform action\n\n\n        state = next\n        tot_reward += reward\n\n        #store useful information for update\n        actions.append(action)\n        rewards.append(reward)\n        scores.append(score)\n        state_action_features.append(state_action_feature)\n        state_features.append(state_feature)\n        state_scores.append(state_score)\n\n\n      # trajectory is finished\n      #compute gains\n      gains = torch.empty(len(rewards))\n\n      # use baseline\n      baselines = None #replace with state values\n      gainsbl = gains - baseline\n\n      #gradient of errors\n      correction = None\n\n      if self.parameters.grad is None:\n        self.parameters.grad = torch.zeros_like(self.parameters)\n\n      for i in range(len(state_action_features)):\n       for a in range(4):\n          self.parameters.grad[state_action_features[i][a]] +=  correction[i][a]\n\n\n      #gradient of errors MSE for v\n      corrections_bl = None\n\n      #print(corrections_bl.size())\n\n      if self.parameters_bl.grad is None:\n        self.parameters_bl.grad = torch.zeros_like(self.parameters_bl)\n      for i in range(len(state_features)):\n          self.parameters_bl.grad[state_features[i]] +=  corrections_bl[i]\n\n      self.optimizer.step()\n      self.optimizer.zero_grad()\n\n      self.optimizer_bl.step()\n      self.optimizer_bl.zero_grad()\n\n      tot_rewards += tot_reward\n\n      if debug and ((episodes)%100 == 0):\n        print(counter, counter/episodes, tot_rewards / episodes)\n\n\n  def get_policy(self, env, state):\n    # returns policy (sample from probability)\n    # your code here\n    pass"]},{"cell_type":"code","execution_count":1,"id":"4010ce01-70fa-4e2f-aa12-7f1b9ee2d9fe","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL()\ntest(agent, 5e4)"]},{"cell_type":"code","execution_count":1,"id":"8e92b86e-4804-4380-9607-cb3d85f64394","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e4)"]},{"cell_type":"code","execution_count":1,"id":"91f9a300-3c50-44b3-aaef-a972a7d1534c","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"d0cd7607-7025-4b5d-8d96-d0297558ba7d","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"5dde0774-caae-45c2-b0e2-0ecfedd18b38","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"1b61f028-6618-4df8-9173-0b288b81fafb","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e5)"]},{"cell_type":"code","execution_count":1,"id":"e1f3ac18-d72e-4f9c-acd7-a70030b962fb","metadata":{},"outputs":[],"source":["agent = PolicyGradientBL('/content/drive/My Drive/save_with_bl.pt')\ntest(agent, 5e5)"]}],"metadata":{"org":{"AUTHOR":"Joseph Le Roux","DATE":"2026-07-19","DESCRIPTION":"Lunar Lander and Policy Gradient"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":5}