%%capture
!apt install python-opengl
!apt install ffmpeg
!apt install xvfb
!pip install pyvirtualdisplay
!pip install pyglet==1.5.1

# Virtual display
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()

<pyvirtualdisplay.display.Display at 0x7ea7ee05c680>

# !pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt

# 1. Install the game engines first
!pip install git+https://github.com/ntasfi/PyGame-Learning-Environment.git
# !pip install git+https://github.com/simoninithomas/gym-games.git

# 2. Install the supporting libraries with modern versions
!pip install pyyaml>=6.0.1
!pip install huggingface_hub imageio-ffmpeg imageio
!pip install gymnasium
!pip install pygame

Collecting git+https://github.com/ntasfi/PyGame-Learning-Environment.git
  Cloning https://github.com/ntasfi/PyGame-Learning-Environment.git to /tmp/pip-req-build-xhmmddx1
  Running command git clone --filter=blob:none --quiet https://github.com/ntasfi/PyGame-Learning-Environment.git /tmp/pip-req-build-xhmmddx1
  Resolved https://github.com/ntasfi/PyGame-Learning-Environment.git to commit 3dbe79dc0c35559bb441b9359948aabf9bb3d331
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (from ple==0.0.1) (2.0.2)
Requirement already satisfied: Pillow in /usr/local/lib/python3.12/dist-packages (from ple==0.0.1) (11.3.0)
Building wheels for collected packages: ple
  Building wheel for ple (setup.py) ... done
  Created wheel for ple: filename=ple-0.0.1-py3-none-any.whl size=50769 sha256=af724bf21da9dbe284d48946126e5c06b23471495f8a902a25da1c01fcf90dae
  Stored in directory: /tmp/pip-ephem-wheel-cache-4ooy8r89/wheels/6d/3c/74/aa0f046a54330af388e34b880213857c59e03b701cdcd9c38f
Successfully built ple
Installing collected packages: ple
Successfully installed ple-0.0.1
Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.12/dist-packages (1.4.1)
Requirement already satisfied: imageio-ffmpeg in /usr/local/lib/python3.12/dist-packages (0.6.0)
Requirement already satisfied: imageio in /usr/local/lib/python3.12/dist-packages (2.37.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (3.24.2)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (2025.3.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (1.2.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (0.28.1)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (26.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (6.0.3)
Requirement already satisfied: shellingham in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (1.5.4)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (4.67.3)
Requirement already satisfied: typer-slim in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (0.24.0)
Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (4.15.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (from imageio) (2.0.2)
Requirement already satisfied: pillow>=8.3.2 in /usr/local/lib/python3.12/dist-packages (from imageio) (11.3.0)
Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface_hub) (4.12.1)
Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface_hub) (2026.1.4)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface_hub) (1.0.9)
Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface_hub) (3.11)
Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub) (0.16.0)
Requirement already satisfied: typer>=0.24.0 in /usr/local/lib/python3.12/dist-packages (from typer-slim->huggingface_hub) (0.24.0)
Requirement already satisfied: click>=8.2.1 in /usr/local/lib/python3.12/dist-packages (from typer>=0.24.0->typer-slim->huggingface_hub) (8.3.1)
Requirement already satisfied: rich>=12.3.0 in /usr/local/lib/python3.12/dist-packages (from typer>=0.24.0->typer-slim->huggingface_hub) (13.9.4)
Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer>=0.24.0->typer-slim->huggingface_hub) (0.0.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=12.3.0->typer>=0.24.0->typer-slim->huggingface_hub) (4.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=12.3.0->typer>=0.24.0->typer-slim->huggingface_hub) (2.19.2)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=12.3.0->typer>=0.24.0->typer-slim->huggingface_hub) (0.1.2)
Requirement already satisfied: gymnasium in /usr/local/lib/python3.12/dist-packages (1.2.3)
Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (2.0.2)
Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (3.1.2)
Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (4.15.0)
Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from gymnasium) (0.0.4)
Requirement already satisfied: pygame in /usr/local/lib/python3.12/dist-packages (2.6.1)

!pip install git+https://github.com/lusob/gym-ple

Collecting git+https://github.com/lusob/gym-ple
  Cloning https://github.com/lusob/gym-ple to /tmp/pip-req-build-nr2d_bu9
  Running command git clone --filter=blob:none --quiet https://github.com/lusob/gym-ple /tmp/pip-req-build-nr2d_bu9
  Resolved https://github.com/lusob/gym-ple to commit 7cedbf4e31be86f5ca2aae5c0dfd9d38825af64e
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: gym_ple
  Building wheel for gym_ple (setup.py) ... done
  Created wheel for gym_ple: filename=gym_ple-0.3-py3-none-any.whl size=5320 sha256=42d23c4118e80343877d8b2ae6b94eba8c0cf1167120e197eac409b01743221c
  Stored in directory: /tmp/pip-ephem-wheel-cache-09imi9f7/wheels/14/1c/c5/278fcc3f98d34804c20ec2a8ea18d87be4f3e4b8101bb996b5
Successfully built gym_ple
Installing collected packages: gym_ple
Successfully installed gym_ple-0.3

import numpy as np

from collections import deque

import matplotlib.pyplot as plt
%matplotlib inline

# PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical

# Gym
import gym
# import gym_pygame

# import gymnasium as gym
import pygame
import numpy as np

# If you are using PixelCopter specifically:
import gym_ple
from ple import PLE
from ple.games.pixelcopter import Pixelcopter

# Hugging Face Hub
from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.
import imageio

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

print(device)

cuda:0

env_id = "CartPole-v1"
# Create the env
env = gym.make(env_id)

# Create the evaluation env
eval_env = gym.make(env_id)

# Get the state space and action space
s_size = env.observation_space.shape[0]
a_size = env.action_space.n

/usr/local/lib/python3.12/dist-packages/gym/core.py:317: DeprecationWarning: WARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(
/usr/local/lib/python3.12/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: WARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.
  deprecation(

print("_____OBSERVATION SPACE_____ \n")
print("The State Space is: ", s_size)
print("Sample observation", env.observation_space.sample()) # Get a random observation

_____OBSERVATION SPACE_____ 

The State Space is:  4
Sample observation [-4.4762874e+00 -8.5093998e+37  9.1941938e-02  1.2201091e+38]

print("\n _____ACTION SPACE_____ \n")
print("The Action Space is: ", a_size)
print("Action Space Sample", env.action_space.sample()) # Take a random action

 _____ACTION SPACE_____ 

The Action Space is:  2
Action Space Sample 0

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, a_size)

    def forward(self, x):
         x = F.relu(self.fc1(x))
         x = self.fc2(x)
         return F.softmax(x, dim=1)

    def act(self, state):
        """
        Given a state, sample an action and return its log probability
        """
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()  #np.argmax(m)
        return action.item(), m.log_prob(action)

debug_policy = Policy(s_size, a_size, 64).to(device)
debug_policy.act(env.reset())

def reinforce(policy, optimizer, n_training_episodes, max_t, gamma, print_every):
    # Help us to calculate the score during the training
    scores_deque = deque(maxlen=100)
    scores = []
    # Line 3 of pseudocode
    for i_episode in range(1, n_training_episodes+1):
        saved_log_probs = []
        rewards = []
        state = env.reset()
        # Line 4 of pseudocode
        for t in range(max_t):
            action, log_prob = policy.act(state)
            saved_log_probs.append(log_prob)
            state, reward, done, _ = env.step(action)
            rewards.append(reward)
            if done:
                break
        scores_deque.append(sum(rewards))
        scores.append(sum(rewards))

        # Line 6 of pseudocode: calculate the return, G_t
        returns = deque(maxlen=max_t)
        n_steps = len(rewards)
        # Compute the discounted returns at each timestep,
        # as the sum of the gamma-discounted return at time t (G_t) + the reward at time t

        # In O(N) time, where N is the number of time steps
        # (this definition of the discounted return G_t follows the definition of this quantity
        # shown at page 44 of Sutton&Barto 2017 2nd draft)
        # G_t = r_(t+1) + r_(t+2) + ...

        # Given this formulation, the returns at each timestep t can be computed
        # by re-using the computed future returns G_(t+1) to compute the current return G_t
        # G_t = r_(t+1) + gamma*G_(t+1)
        # G_(t-1) = r_t + gamma* G_t
        # (this follows a dynamic programming approach, with which we memorize solutions in order
        # to avoid computing them multiple times)

        # This is correct since the above is equivalent to (see also page 46 of Sutton&Barto 2017 2nd draft)
        # G_(t-1) = r_t + gamma*r_(t+1) + gamma*gamma*r_(t+2) + ...


        ## Given the above, we calculate the returns at timestep t as:
        #               gamma[t] * return[t] + reward[t]
        #
        ## We compute this starting from the last timestep to the first, in order
        ## to employ the formula presented above and avoid redundant computations that would be needed
        ## if we were to do it from first to last.

        ## Hence, the queue "returns" will hold the returns in chronological order, from t=0 to t=n_steps
        ## thanks to the appendleft() function which allows to append to the position 0 in constant time O(1)
        ## a normal python list would instead require O(N) to do this.
        for t in range(n_steps)[::-1]:
            disc_return_t = (returns[0] if len(returns) > 0 else 0)
            returns.appendleft(rewards[t] + gamma * disc_return_t) # TODO: complete here

        ## standardization of the returns is employed to make training more stable
        eps = np.finfo(np.float32).eps.item()

        ## eps is the smallest representable float, which is
        # added to the standard deviation of the returns to avoid numerical instabilities
        returns = torch.tensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + eps)

        # Line 7:
        policy_loss = []
        for log_prob, disc_return in zip(saved_log_probs, returns):
            policy_loss.append(-log_prob * disc_return)
        policy_loss = torch.cat(policy_loss).sum()

        # Line 8: PyTorch prefers gradient descent
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        if i_episode % print_every == 0:
            print('Episode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))

    return scores

cartpole_hyperparameters = {
    "h_size": 16,
    "n_training_episodes": 1000,
    "n_evaluation_episodes": 10,
    "max_t": 1000,
    "gamma": 1.0,
    "lr": 1e-2,
    "env_id": env_id,
    "state_space": s_size,
    "action_space": a_size,
}

s_size, a_size

(4, 2)

# Create policy and place it to the device
cartpole_policy = Policy(cartpole_hyperparameters["state_space"], cartpole_hyperparameters["action_space"], cartpole_hyperparameters["h_size"]).to(device)
cartpole_optimizer = optim.Adam(cartpole_policy.parameters(), lr=cartpole_hyperparameters["lr"])

/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  return datetime.utcnow().replace(tzinfo=utc)

import numpy as np
# Add this line to fix the compatibility gap
np.bool8 = np.bool_

scores = reinforce(cartpole_policy,
                   cartpole_optimizer,
                   cartpole_hyperparameters["n_training_episodes"],
                   cartpole_hyperparameters["max_t"],
                   cartpole_hyperparameters["gamma"],
                   100)

Episode 100	Average Score: 57.10
Episode 200	Average Score: 315.52
Episode 300	Average Score: 356.00
Episode 400	Average Score: 152.03
Episode 500	Average Score: 231.00
Episode 600	Average Score: 320.78
Episode 700	Average Score: 353.43
Episode 800	Average Score: 424.83
Episode 900	Average Score: 496.80
Episode 1000	Average Score: 500.00

def evaluate_agent(env, max_steps, n_eval_episodes, policy):
  """
  Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.
  :param env: The evaluation environment
  :param n_eval_episodes: Number of episode to evaluate the agent
  :param policy: The Reinforce agent
  """
  episode_rewards = []
  for episode in range(n_eval_episodes):
    state = env.reset()
    step = 0
    done = False
    total_rewards_ep = 0

    for step in range(max_steps):
      action, _ = policy.act(state)
      new_state, reward, done, _ = env.step(action)
      total_rewards_ep += reward

      if done:
        break
      state = new_state
    episode_rewards.append(total_rewards_ep)
  mean_reward = np.mean(episode_rewards)
  std_reward = np.std(episode_rewards)

  return mean_reward, std_reward

# Quick check to ensure the environment steps correctly now
test_state = eval_env.reset()
try:
    s, r, d, i = eval_env.step(eval_env.action_space.sample())
    print("Success! env.step() is working with the patch.")
except AttributeError as e:
    print(f"Patch failed or not applied: {e}")

Success! env.step() is working with the patch.

evaluate_agent(eval_env,
               cartpole_hyperparameters["max_t"],
               cartpole_hyperparameters["n_evaluation_episodes"],
               cartpole_policy)

(np.float64(500.0), np.float64(0.0))

from huggingface_hub import HfApi, snapshot_download
from huggingface_hub.repocard import metadata_eval_result, metadata_save

from pathlib import Path
import datetime
import json
import imageio

import tempfile

import os

def record_video(env, policy, out_directory, fps=30):
  """
  Generate a replay video of the agent
  :param env
  :param Qtable: Qtable of our agent
  :param out_directory
  :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)
  """
  images = []
  done = False
  state = env.reset()
  img = env.render(mode='rgb_array')
  images.append(img)
  while not done:
    # Take the action (index) that have the maximum expected future reward given that state
    action, _ = policy.act(state)
    state, reward, done, info = env.step(action) # We directly put next_state = state for recording logic
    img = env.render(mode='rgb_array')
    images.append(img)
  imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)

def push_to_hub(repo_id,
                model,
                hyperparameters,
                eval_env,
                video_fps=30
                ):
  """
  Evaluate, Generate a video and Upload a model to Hugging Face Hub.
  This method does the complete pipeline:
  - It evaluates the model
  - It generates the model card
  - It generates a replay video of the agent
  - It pushes everything to the Hub

  :param repo_id: repo_id: id of the model repository from the Hugging Face Hub
  :param model: the pytorch model we want to save
  :param hyperparameters: training hyperparameters
  :param eval_env: evaluation environment
  :param video_fps: how many frame per seconds to record our video replay
  """

  _, repo_name = repo_id.split("/")
  api = HfApi()

  # Step 1: Create the repo
  repo_url = api.create_repo(
        repo_id=repo_id,
        exist_ok=True,
  )

  with tempfile.TemporaryDirectory() as tmpdirname:
    local_directory = Path(tmpdirname)

    # Step 2: Save the model
    torch.save(model, local_directory / "model.pt")

    # Step 3: Save the hyperparameters to JSON
    with open(local_directory / "hyperparameters.json", "w") as outfile:
      json.dump(hyperparameters, outfile)

    # Step 4: Evaluate the model and build JSON
    mean_reward, std_reward = evaluate_agent(eval_env,
                                            hyperparameters["max_t"],
                                            hyperparameters["n_evaluation_episodes"],
                                            model)
    # Get datetime
    eval_datetime = datetime.datetime.now()
    eval_form_datetime = eval_datetime.isoformat()

    evaluate_data = {
          "env_id": hyperparameters["env_id"],
          "mean_reward": mean_reward,
          "n_evaluation_episodes": hyperparameters["n_evaluation_episodes"],
          "eval_datetime": eval_form_datetime,
    }

    # Write a JSON file
    with open(local_directory / "results.json", "w") as outfile:
        json.dump(evaluate_data, outfile)

    # Step 5: Create the model card
    env_name = hyperparameters["env_id"]

    metadata = {}
    metadata["tags"] = [
          env_name,
          "reinforce",
          "reinforcement-learning",
          "custom-implementation",
          "deep-rl-class"
      ]

    # Add metrics
    eval = metadata_eval_result(
        model_pretty_name=repo_name,
        task_pretty_name="reinforcement-learning",
        task_id="reinforcement-learning",
        metrics_pretty_name="mean_reward",
        metrics_id="mean_reward",
        metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}",
        dataset_pretty_name=env_name,
        dataset_id=env_name,
      )

    # Merges both dictionaries
    metadata = {**metadata, **eval}

    model_card = f"""
  # **Reinforce** Agent playing **{env_id}**
  This is a trained model of a **Reinforce** agent playing **{env_id}** .
  To learn to use this model and train yours check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction
  """

    readme_path = local_directory / "README.md"
    readme = ""
    if readme_path.exists():
        with readme_path.open("r", encoding="utf8") as f:
          readme = f.read()
    else:
      readme = model_card

    with readme_path.open("w", encoding="utf-8") as f:
      f.write(readme)

    # Save our metrics to Readme metadata
    metadata_save(readme_path, metadata)

    # Step 6: Record a video
    video_path =  local_directory / "replay.mp4"
    record_video(env, model, video_path, video_fps)

    # Step 7. Push everything to the Hub
    api.upload_folder(
          repo_id=repo_id,
          folder_path=local_directory,
          path_in_repo=".",
    )

    print(f"Your model is pushed to the Hub. You can view your model here: {repo_url}")

# notebook_login()

!git config --global credential.helper store

# import os
# from kaggle_secrets import UserSecretsClient
# from huggingface_hub import HfApi, login


# # Fetch the secret
# user_secrets = UserSecretsClient()
# hf_token = user_secrets.get_secret("HF_TOKEN")


# # Force login
# login(token=hf_token, add_to_git_credential=True)

# # 2. Verify identity and permissions
# api = HfApi()
# user_info = api.whoami()
# print(f"✅ Successfully logged in as: {user_info['name']}")
# print(f"✅ Token has Write access: {user_info['auth']['type'] == 'WRITE'}")


import os
from huggingface_hub import HfApi, login

def authenticate_huggingface():
    """Robust authentication for Colab, Kaggle, and Local environments."""
    hf_token = None

    # 1. Try Google Colab Secrets
    try:
        from google.colab import userdata
        try:
            hf_token = userdata.get("HF_TOKEN")
        except userdata.SecretNotFoundError:
            print("💡 Instruction: Add 'HF_TOKEN' to the Secrets tab (key icon) and enable access.")
    except (ImportError, ModuleNotFoundError):
        pass

    # 2. Try Kaggle Secrets (if not on Colab)
    if not hf_token:
        try:
            from kaggle_secrets import UserSecretsClient
            hf_token = UserSecretsClient().get_secret("HF_TOKEN")
        except (ImportError, ModuleNotFoundError, Exception):
            pass

    # 3. Try Environment Variables (Local/Docker)
    if not hf_token:
        hf_token = os.getenv("HF_TOKEN")

    if not hf_token:
        print("⚠️ No token found. Please set HF_TOKEN in your environment/secrets.")
        return False

    # Perform Login
    try:
        login(token=hf_token, add_to_git_credential=True)
        api = HfApi()
        user_info = api.whoami()
        print(f"✅ Logged in as: {user_info['name']}")
        return True
    except Exception as e:
        print(f"❌ Login failed: {e}")
        return False

# Execute
authenticate_huggingface()

/usr/local/lib/python3.12/dist-packages/jupyter_client/session.py:203: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  return datetime.utcnow().replace(tzinfo=utc)

✅ Logged in as: Chiz

True

repo_id = "Chiz/Reinforce-CartPole-v2" #TODO Define your repo id {username/Reinforce-{model-id}}
push_to_hub(repo_id,
                cartpole_policy, # The model we want to save
                cartpole_hyperparameters, # Hyperparameters
                eval_env, # Evaluation environment
                video_fps=30
                )

import gym_ple
from ple import PLE
from ple.games.pixelcopter import Pixelcopter

# # Create environment directly without gym.make
# game = Pixelcopter()
# env = PLE(game, fps=30, display_screen=False)
# env.reset_game()

# s_size = env.getGameState().shape[0] if isinstance(env.getGameState(), np.ndarray) else len(env.getGameState())
# a_size = len(env.getActionSet())

# Training environment
game = Pixelcopter(width=48, height=48)
env = PLE(game, fps=30, display_screen=False)

# Evaluation environment
game_eval = Pixelcopter(width=48, height=48)
eval_env = PLE(game_eval, fps=30, display_screen=False)

state = env.getGameState()
s_size = len(state) if isinstance(state, dict) else state.shape[0]
a_size = len(env.getActionSet())

print("_____OBSERVATION SPACE_____ \n")
print("The State Space is: ", s_size)
print("Sample observation", env.getGameState()) # Get current game state

_____OBSERVATION SPACE_____ 

The State Space is:  7
Sample observation {'player_y': 24.0, 'player_vel': 0, 'player_dist_to_ceil': 7.0, 'player_dist_to_floor': 17.0, 'next_gate_dist_to_player': 34, 'next_gate_block_top': 15, 'next_gate_block_bottom': 24}

print("\n _____ACTION SPACE_____ \n")
print("The Action Space is: ", a_size)
actions = env.getActionSet()
print("Available Actions:", actions)
print("Action Space Sample", env.act(np.random.choice(actions)))

 _____ACTION SPACE_____ 

The Action Space is:  2
Available Actions: [119, None]
Action Space Sample 0.0

# env_id = "PixelCopter-PLE-v0"
# env = gym.make(env_id)
# eval_env = gym.make(env_id)
# s_size = env.observation_space.shape[0]
# a_size = env.action_space.n

# print("_____OBSERVATION SPACE_____ \n")
# print("The State Space is: ", s_size)
# print("Sample observation", env.observation_space.sample()) # Get a random observation

# print("\n _____ACTION SPACE_____ \n")
# print("The Action Space is: ", a_size)
# print("Action Space Sample", env.action_space.sample()) # Take a random action

class Policy(nn.Module):
    def __init__(self, s_size, a_size, h_size):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, h_size*2)
        self.fc3 = nn.Linear(h_size*2, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.softmax(x, dim=1)

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

env_id = "Pixelcopter-PLE-v0"

pixelcopter_hyperparameters = {
    "h_size": 64,
    "n_training_episodes": 50000,
    "n_evaluation_episodes": 10,
    "max_t": 10000,
    "gamma": 0.99,
    "lr": 1e-4,
    "env_id": env_id,
    "state_space": s_size,
    "action_space": a_size,
}

class PLEToGymWrapper:
    def __init__(self, ple_env):
        self.ple = ple_env
        self.action_set = self.ple.getActionSet()
        # Define action_space.n for the Policy class
        self.action_space = type('', (), {'n': len(self.action_set)})()

    def reset(self):
        self.ple.reset_game()
        state_dict = self.ple.getGameState()
        return np.array(list(state_dict.values()), dtype=np.float32)

    def step(self, action_idx):
        # Map 0/1 index to actual PLE command
        real_action = self.action_set[action_idx]
        reward = self.ple.act(real_action)
        done = self.ple.game_over()

        state_dict = self.ple.getGameState()
        obs = np.array(list(state_dict.values()), dtype=np.float32)

        return obs, reward, done, {} # Return 4 values for old Gym style

# Wrap your existing PLE environments
env = PLEToGymWrapper(env)
eval_env = PLEToGymWrapper(eval_env)

# Create policy and place it to the device
# torch.manual_seed(50)
pixelcopter_policy = Policy(pixelcopter_hyperparameters["state_space"], pixelcopter_hyperparameters["action_space"], pixelcopter_hyperparameters["h_size"]).to(device)
pixelcopter_optimizer = optim.Adam(pixelcopter_policy.parameters(), lr=pixelcopter_hyperparameters["lr"])

scores = reinforce(pixelcopter_policy,
                   pixelcopter_optimizer,
                   pixelcopter_hyperparameters["n_training_episodes"],
                   pixelcopter_hyperparameters["max_t"],
                   pixelcopter_hyperparameters["gamma"],
                   1000)

Episode 1000	Average Score: 3.87
Episode 2000	Average Score: 4.54
Episode 3000	Average Score: 6.35
Episode 4000	Average Score: 9.13
Episode 5000	Average Score: 10.89
Episode 6000	Average Score: 12.19
Episode 7000	Average Score: 13.28
Episode 8000	Average Score: 14.90
Episode 9000	Average Score: 15.24
Episode 10000	Average Score: 20.63
Episode 11000	Average Score: 17.43
Episode 12000	Average Score: 20.66
Episode 13000	Average Score: 15.15
Episode 14000	Average Score: 19.88
Episode 15000	Average Score: 20.19
Episode 16000	Average Score: 21.30
Episode 17000	Average Score: 24.13
Episode 18000	Average Score: 17.67
Episode 19000	Average Score: 27.18
Episode 20000	Average Score: 22.25
Episode 21000	Average Score: 24.40
Episode 22000	Average Score: 25.88
Episode 23000	Average Score: 34.55
Episode 24000	Average Score: 32.86
Episode 25000	Average Score: 24.67
Episode 26000	Average Score: 24.06
Episode 27000	Average Score: 23.00
Episode 28000	Average Score: 35.17
Episode 29000	Average Score: 32.25
Episode 30000	Average Score: 24.11
Episode 31000	Average Score: 23.31
Episode 32000	Average Score: 32.22
Episode 33000	Average Score: 37.62
Episode 34000	Average Score: 37.59
Episode 35000	Average Score: 17.34
Episode 36000	Average Score: 35.48
Episode 37000	Average Score: 29.94
Episode 38000	Average Score: 40.49
Episode 39000	Average Score: 35.80
Episode 40000	Average Score: 37.31
Episode 41000	Average Score: 34.92
Episode 42000	Average Score: 38.88
Episode 43000	Average Score: 40.18
Episode 44000	Average Score: 38.75
Episode 45000	Average Score: 42.44
Episode 46000	Average Score: 41.81
Episode 47000	Average Score: 38.52
Episode 48000	Average Score: 39.15
Episode 49000	Average Score: 47.77
Episode 50000	Average Score: 44.67

evaluate_agent(eval_env,
               pixelcopter_hyperparameters["max_t"],
               pixelcopter_hyperparameters["n_evaluation_episodes"],
               pixelcopter_policy)

(np.float64(33.6), np.float64(17.344739836619055))

class PLEToGymWrapper:
    def __init__(self, ple_env):
        self.ple = ple_env
        self.action_set = self.ple.getActionSet()
        # Define action_space.n for the Policy class
        self.action_space = type('', (), {'n': len(self.action_set)})()

    def reset(self):
        self.ple.reset_game()
        state_dict = self.ple.getGameState()
        return np.array(list(state_dict.values()), dtype=np.float32)

    def step(self, action_idx):
        # Map 0/1 index to actual PLE command
        real_action = self.action_set[action_idx]
        reward = self.ple.act(real_action)
        done = self.ple.game_over()

        state_dict = self.ple.getGameState()
        obs = np.array(list(state_dict.values()), dtype=np.float32)

        return obs, reward, done, {}

    def render(self, mode='rgb_array'):
        if mode == 'rgb_array':
            try:
                # Try to get screen surface from PLE game
                surface = self.ple.game.screen
                if surface is None:
                    return np.zeros((48, 48, 3), dtype=np.uint8)

                # Convert pygame surface to numpy array
                w, h = surface.get_size()
                pixel_array = pygame.surfarray.array3d(surface)
                # Transpose from (width, height, 3) to (height, width, 3)
                return np.transpose(pixel_array, (1, 0, 2)).astype(np.uint8)
            except Exception as e:
                print(f"Render error: {e}")
                return np.zeros((48, 48, 3), dtype=np.uint8)
        else:
            self.ple.display_screen = True
            return None

    def close(self):
        if hasattr(self.ple, 'close_game'):
            self.ple.close_game()

# Wrap the evaluation environment again so it has the .render() method
eval_env = PLEToGymWrapper(eval_env.ple)

env = PLEToGymWrapper(env.ple)

eval_env.reset()
frame = eval_env.render(mode='rgb_array')
print(f"Frame shape: {frame.shape}") # Should be something like (48, 48, 3) or (width, height, 3)

import matplotlib.pyplot as plt
plt.imshow(frame)
plt.title("Is this PixelCopter?")
plt.show()

Frame shape: (48, 48, 3)

repo_id = "Chiz/Reinforce-Pixelcopter-PLE-v0" #TODO Define your repo id {username/Reinforce-{model-id}}
push_to_hub(repo_id,
                pixelcopter_policy, # The model we want to save
                pixelcopter_hyperparameters, # Hyperparameters
                eval_env, # Evaluation environment
                video_fps=30
                )

Processing Files (0 / 0)      : |          |  0.00B /  0.00B

New Data Upload               : |          |  0.00B /  0.00B

  /tmp/tmpgwruj0mv/model.pt   : 100%|##########| 40.1kB / 40.1kB

Your model is pushed to the Hub. You can view your model here: https://huggingface.co/Chiz/Reinforce-Pixelcopter-PLE-v0

Unit 4: Code your first Deep Reinforcement Learning Algorithm with PyTorch: Reinforce. And test its robustness 💪¶

🎮 Environments:¶

📚 RL-Library:¶

Objectives of this notebook 🏆¶

This notebook is from the Deep Reinforcement Learning Course¶

Prerequisites 🏗️¶

Let's code Reinforce algorithm from scratch 🔥¶

An advice 💡¶

Set the GPU 💪¶

Create a virtual display 🖥¶

Install the dependencies 🔽¶

Import the packages 📦¶

Check if we have a GPU¶

First agent: Playing CartPole-v1 🤖¶

Create the CartPole environment and understand how it works¶

The environment 🎮¶

Why do we use a simple environment like CartPole-v1?¶

The CartPole-v1 environment¶

Let's build the Reinforce Architecture¶

Let's build the Reinforce Training Algorithm¶

Train it¶

Define evaluation method 📝¶

Evaluate our agent 📈¶

Publish our trained model on the Hub 🔥¶

Push to the Hub¶

Do not modify this code¶

.¶

Second agent: PixelCopter 🚁¶

Study the PixelCopter environment 👀¶

Define the new Policy 🧠¶

Define the hyperparameters ⚙️¶

Train it¶

Publish our trained model on the Hub 🔥¶

Some additional challenges 🏆¶

Keep Learning, stay awesome 🤗¶