Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo¶
In this notebook, you'll train a Deep Q-Learning agent playing Space Invaders using RL Baselines3 Zoo, a training framework based on Stable-Baselines3 that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.
We're using the RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.
⬇️ Here is an example of what you will achieve ⬇️
In [ ]:
%%html
<video controls autoplay><source src="https://huggingface.co/ThomasSimonini/ppo-SpaceInvadersNoFrameskip-v4/resolve/main/replay.mp4" type="video/mp4"></video>
🎮 Environments:¶
You can see the difference between Space Invaders versions here 👉 https://gymnasium.farama.org/environments/atari/space_invaders/#variants
📚 RL-Library:¶
Objectives of this notebook 🏆¶
At the end of the notebook, you will:
- Be able to understand deeper how RL Baselines3 Zoo works.
- Be able to push your trained agent and the code to the Hub with a nice video replay and an evaluation score 🔥.
This notebook is from Deep Reinforcement Learning Course¶
In this free course, you will:
- 📖 Study Deep Reinforcement Learning in theory and practice.
- 🧑💻 Learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.
- 🤖 Train agents in unique environments
And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course
Don’t forget to sign up to the course (we are collecting your email to be able to send you the links when each Unit is published and give you information about the challenges and updates).
The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5
Prerequisites 🏗️¶
Before diving into the notebook, you need to:
We're constantly trying to improve our tutorials, so if you find some issues in this notebook, please open an issue on the Github Repo.
Let's train a Deep Q-Learning agent playing Atari' Space Invaders 👾 and upload it to the Hub.¶
We strongly recommend students to use Google Colab for the hands-on exercises instead of running them on their personal computers.
By using Google Colab, you can focus on learning and experimenting without worrying about the technical aspects of setting up your environments.
To validate this hands-on for the certification process, you need to push your trained model to the Hub and get a result of >= 200.
To find your result, go to the leaderboard and find your model, the result = mean_reward - std of reward
For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process
An advice 💡¶
It's better to run this colab in a copy on your Google Drive, so that if it timeouts you still have the saved notebook on your Google Drive and do not need to fill everything from scratch.
To do that you can either do Ctrl + S or File > Save a copy in Google Drive.
Also, we're going to train it for 90 minutes with 1M timesteps. By typing !nvidia-smi will tell you what GPU you're using.
And if you want to train more such 10 million steps, this will take about 9 hours, potentially resulting in Colab timing out. In that case, I recommend running this on your local computer (or somewhere else). Just click on: File>Download.
Set the GPU 💪¶
- To accelerate the agent's training, we'll use a GPU. To do that, go to
Runtime > Change Runtime type
Hardware Accelerator > GPU
In [ ]:
!nvidia-smi
Thu Jan 29 14:33:45 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 45C P8 10W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Install RL-Baselines3 Zoo and its dependencies 📚¶
If you see ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. this is normal and it's not a critical error there's a conflict of version. But the packages we need are installed.
In [ ]:
!pip install git+https://github.com/DLR-RM/rl-baselines3-zoo
Collecting git+https://github.com/DLR-RM/rl-baselines3-zoo Cloning https://github.com/DLR-RM/rl-baselines3-zoo to /tmp/pip-req-build-ttt58_jn Running command git clone --filter=blob:none --quiet https://github.com/DLR-RM/rl-baselines3-zoo /tmp/pip-req-build-ttt58_jn Resolved https://github.com/DLR-RM/rl-baselines3-zoo to commit d29756c456caadbbebc15c35893674abb2453e0d Running command git submodule update --init --recursive -q Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting sb3_contrib<3.0,>=2.8.0a0 (from rl_zoo3==2.8.0a0) Downloading sb3_contrib-2.8.0a0-py3-none-any.whl.metadata (4.1 kB) Requirement already satisfied: gymnasium<1.3.0,>=0.29.1 in /usr/local/lib/python3.12/dist-packages (from rl_zoo3==2.8.0a0) (1.2.3) Collecting huggingface_sb3<4.0,>=3.0 (from rl_zoo3==2.8.0a0) Downloading huggingface_sb3-3.0-py3-none-any.whl.metadata (6.3 kB) Requirement already satisfied: tqdm in /usr/local/lib/python3.12/dist-packages (from rl_zoo3==2.8.0a0) (4.67.1) Requirement already satisfied: rich in /usr/local/lib/python3.12/dist-packages (from rl_zoo3==2.8.0a0) (13.9.4) Collecting optuna>=3.0 (from rl_zoo3==2.8.0a0) Downloading optuna-4.7.0-py3-none-any.whl.metadata (17 kB) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from rl_zoo3==2.8.0a0) (6.0.3) Collecting pytablewriter~=1.2 (from rl_zoo3==2.8.0a0) Downloading pytablewriter-1.2.1-py3-none-any.whl.metadata (38 kB) Collecting shimmy~=2.0 (from rl_zoo3==2.8.0a0) Downloading Shimmy-2.0.0-py3-none-any.whl.metadata (3.5 kB) Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium<1.3.0,>=0.29.1->rl_zoo3==2.8.0a0) (2.0.2) Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium<1.3.0,>=0.29.1->rl_zoo3==2.8.0a0) (3.1.2) Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium<1.3.0,>=0.29.1->rl_zoo3==2.8.0a0) (4.15.0) Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from gymnasium<1.3.0,>=0.29.1->rl_zoo3==2.8.0a0) (0.0.4) Requirement already satisfied: huggingface-hub~=0.8 in /usr/local/lib/python3.12/dist-packages (from huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (0.36.0) Requirement already satisfied: wasabi in /usr/local/lib/python3.12/dist-packages (from huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (1.1.3) Requirement already satisfied: alembic>=1.5.0 in /usr/local/lib/python3.12/dist-packages (from optuna>=3.0->rl_zoo3==2.8.0a0) (1.18.1) Collecting colorlog (from optuna>=3.0->rl_zoo3==2.8.0a0) Downloading colorlog-6.10.1-py3-none-any.whl.metadata (11 kB) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from optuna>=3.0->rl_zoo3==2.8.0a0) (25.0) Requirement already satisfied: sqlalchemy>=1.4.2 in /usr/local/lib/python3.12/dist-packages (from optuna>=3.0->rl_zoo3==2.8.0a0) (2.0.45) Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.12/dist-packages (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) (75.2.0) Collecting DataProperty<2,>=1.1.0 (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading DataProperty-1.1.0-py3-none-any.whl.metadata (11 kB) Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading mbstrdecoder-1.1.4-py3-none-any.whl.metadata (4.3 kB) Collecting pathvalidate<4,>=2.3.0 (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading pathvalidate-3.3.1-py3-none-any.whl.metadata (12 kB) Collecting tabledata<2,>=1.3.1 (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading tabledata-1.3.4-py3-none-any.whl.metadata (3.7 kB) Collecting tcolorpy<1,>=0.0.5 (from pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading tcolorpy-0.1.7-py3-none-any.whl.metadata (6.3 kB) Collecting typepy<2,>=1.3.2 (from typepy[datetime]<2,>=1.3.2->pytablewriter~=1.2->rl_zoo3==2.8.0a0) Downloading typepy-1.3.4-py3-none-any.whl.metadata (9.2 kB) Collecting stable_baselines3<3.0,>=2.8.0a0 (from sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) Downloading stable_baselines3-2.8.0a2-py3-none-any.whl.metadata (4.8 kB) Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich->rl_zoo3==2.8.0a0) (4.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich->rl_zoo3==2.8.0a0) (2.19.2) Requirement already satisfied: Mako in /usr/local/lib/python3.12/dist-packages (from alembic>=1.5.0->optuna>=3.0->rl_zoo3==2.8.0a0) (1.3.10) Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (3.20.3) Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (2025.3.0) Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (from huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (2.32.4) Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (1.2.0) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich->rl_zoo3==2.8.0a0) (0.1.2) Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.12/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter~=1.2->rl_zoo3==2.8.0a0) (5.2.0) Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy>=1.4.2->optuna>=3.0->rl_zoo3==2.8.0a0) (3.3.0) Requirement already satisfied: torch<3.0,>=2.3 in /usr/local/lib/python3.12/dist-packages (from stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (2.9.0+cu126) Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (2.2.2) Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (from stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.10.0) Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.12/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter~=1.2->rl_zoo3==2.8.0a0) (2.9.0.post0) Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.12/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter~=1.2->rl_zoo3==2.8.0a0) (2025.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil<3.0.0,>=2.8.0->typepy[datetime]<2,>=1.3.2->pytablewriter~=1.2->rl_zoo3==2.8.0a0) (1.17.0) Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (1.14.0) Requirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.6.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.1.6) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.77) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.77) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.6.80 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.80) Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (9.10.2.21) Requirement already satisfied: nvidia-cublas-cu12==12.6.4.1 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.4.1) Requirement already satisfied: nvidia-cufft-cu12==11.3.0.4 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (11.3.0.4) Requirement already satisfied: nvidia-curand-cu12==10.3.7.77 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (10.3.7.77) Requirement already satisfied: nvidia-cusolver-cu12==11.7.1.2 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (11.7.1.2) Requirement already satisfied: nvidia-cusparse-cu12==12.5.4.2 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.5.4.2) Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (0.7.1) Requirement already satisfied: nvidia-nccl-cu12==2.27.5 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (2.27.5) Requirement already satisfied: nvidia-nvshmem-cu12==3.3.20 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.3.20) Requirement already satisfied: nvidia-nvtx-cu12==12.6.77 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.77) Requirement already satisfied: nvidia-nvjitlink-cu12==12.6.85 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (12.6.85) Requirement already satisfied: nvidia-cufile-cu12==1.11.1.6 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (1.11.1.6) Requirement already satisfied: triton==3.5.0 in /usr/local/lib/python3.12/dist-packages (from torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.5.0) Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.12/dist-packages (from Mako->alembic>=1.5.0->optuna>=3.0->rl_zoo3==2.8.0a0) (3.0.3) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (1.3.3) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (4.61.1) Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (1.4.9) Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (11.3.0) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (3.3.1) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (2025.3) Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (3.4.4) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (3.11) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (2.5.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3<4.0,>=3.0->rl_zoo3==2.8.0a0) (2026.1.4) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch<3.0,>=2.3->stable_baselines3<3.0,>=2.8.0a0->sb3_contrib<3.0,>=2.8.0a0->rl_zoo3==2.8.0a0) (1.3.0) Downloading huggingface_sb3-3.0-py3-none-any.whl (9.7 kB) Downloading optuna-4.7.0-py3-none-any.whl (413 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 413.9/413.9 kB 17.7 MB/s eta 0:00:00 Downloading pytablewriter-1.2.1-py3-none-any.whl (91 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.1/91.1 kB 10.3 MB/s eta 0:00:00 Downloading sb3_contrib-2.8.0a0-py3-none-any.whl (92 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.0/93.0 kB 9.4 MB/s eta 0:00:00 Downloading Shimmy-2.0.0-py3-none-any.whl (30 kB) Downloading DataProperty-1.1.0-py3-none-any.whl (27 kB) Downloading mbstrdecoder-1.1.4-py3-none-any.whl (7.9 kB) Downloading pathvalidate-3.3.1-py3-none-any.whl (24 kB) Downloading stable_baselines3-2.8.0a2-py3-none-any.whl (187 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 187.3/187.3 kB 19.8 MB/s eta 0:00:00 Downloading tabledata-1.3.4-py3-none-any.whl (11 kB) Downloading tcolorpy-0.1.7-py3-none-any.whl (8.1 kB) Downloading typepy-1.3.4-py3-none-any.whl (31 kB) Downloading colorlog-6.10.1-py3-none-any.whl (11 kB) Building wheels for collected packages: rl_zoo3 Building wheel for rl_zoo3 (pyproject.toml) ... done Created wheel for rl_zoo3: filename=rl_zoo3-2.8.0a0-py3-none-any.whl size=78003 sha256=ee071607b953dc65683039d61c426b7464634910e781595d4bc7baee897c60fd Stored in directory: /tmp/pip-ephem-wheel-cache-ehguc35u/wheels/cd/09/5b/8130538724a187021e02afe53ce7a1269c516a78a487a1ba15 Successfully built rl_zoo3 Installing collected packages: tcolorpy, pathvalidate, mbstrdecoder, colorlog, typepy, shimmy, optuna, huggingface_sb3, stable_baselines3, DataProperty, tabledata, sb3_contrib, pytablewriter, rl_zoo3 Successfully installed DataProperty-1.1.0 colorlog-6.10.1 huggingface_sb3-3.0 mbstrdecoder-1.1.4 optuna-4.7.0 pathvalidate-3.3.1 pytablewriter-1.2.1 rl_zoo3-2.8.0a0 sb3_contrib-2.8.0a0 shimmy-2.0.0 stable_baselines3-2.8.0a2 tabledata-1.3.4 tcolorpy-0.1.7 typepy-1.3.4
In [ ]:
!apt-get install swig cmake ffmpeg
Reading package lists... Done Building dependency tree... Done Reading state information... Done cmake is already the newest version (3.22.1-1ubuntu1.22.04.2). ffmpeg is already the newest version (7:4.4.2-0ubuntu0.22.04.1). The following additional packages will be installed: swig4.0 Suggested packages: swig-doc swig-examples swig4.0-examples swig4.0-doc The following NEW packages will be installed: swig swig4.0 0 upgraded, 2 newly installed, 0 to remove and 41 not upgraded. Need to get 1,116 kB of archives. After this operation, 5,542 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 swig4.0 amd64 4.0.2-1ubuntu1 [1,110 kB] Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 swig all 4.0.2-1ubuntu1 [5,632 B] Fetched 1,116 kB in 1s (754 kB/s) Selecting previously unselected package swig4.0. (Reading database ... 121689 files and directories currently installed.) Preparing to unpack .../swig4.0_4.0.2-1ubuntu1_amd64.deb ... Unpacking swig4.0 (4.0.2-1ubuntu1) ... Selecting previously unselected package swig. Preparing to unpack .../swig_4.0.2-1ubuntu1_all.deb ... Unpacking swig (4.0.2-1ubuntu1) ... Setting up swig4.0 (4.0.2-1ubuntu1) ... Setting up swig (4.0.2-1ubuntu1) ... Processing triggers for man-db (2.10.2-1) ...
To be able to use Atari games in Gymnasium we need to install atari package. And accept-rom-license to download the rom files (games files).
In [ ]:
!pip install gymnasium[atari]
!pip install gymnasium[accept-rom-license]
Requirement already satisfied: gymnasium[atari] in /usr/local/lib/python3.12/dist-packages (1.2.3) Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[atari]) (2.0.2) Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[atari]) (3.1.2) Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[atari]) (4.15.0) Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from gymnasium[atari]) (0.0.4) Requirement already satisfied: ale_py>=0.9 in /usr/local/lib/python3.12/dist-packages (from gymnasium[atari]) (0.11.2) Requirement already satisfied: gymnasium[accept-rom-license] in /usr/local/lib/python3.12/dist-packages (1.2.3) WARNING: gymnasium 1.2.3 does not provide the extra 'accept-rom-license' Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[accept-rom-license]) (2.0.2) Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[accept-rom-license]) (3.1.2) Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from gymnasium[accept-rom-license]) (4.15.0) Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.12/dist-packages (from gymnasium[accept-rom-license]) (0.0.4)
Create a virtual display 🔽¶
During the notebook, we'll need to generate a replay video. To do so, with colab, we need to have a virtual screen to be able to render the environment (and thus record the frames).
Hence the following cell will install the librairies and create and run a virtual screen 🖥
In [ ]:
%%capture
!apt install python-opengl
!apt install xvfb
!pip3 install pyvirtualdisplay
In [ ]:
# Virtual display
from pyvirtualdisplay import Display
virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()
Out[ ]:
<pyvirtualdisplay.display.Display at 0x79d494e38f50>
Train our Deep Q-Learning Agent to Play Space Invaders 👾¶
To train an agent with RL-Baselines3-Zoo, we just need to do two things:
- Create a hyperparameter config file that will contain our training hyperparameters called
dqn.yml.
This is a template example:
SpaceInvadersNoFrameskip-v4:
env_wrapper:
- stable_baselines3.common.atari_wrappers.AtariWrapper
frame_stack: 4
policy: 'CnnPolicy'
n_timesteps: !!float 1e6
buffer_size: 100000
learning_rate: !!float 1e-4
batch_size: 32
learning_starts: 100000
target_update_interval: 1000
train_freq: 4
gradient_steps: 1
exploration_fraction: 0.1
exploration_final_eps: 0.01
# If True, you need to deactivate handle_timeout_termination
# in the replay_buffer_kwargs
optimize_memory_usage: False
In [ ]:
# Create the dqn.yml file with tuned hyperparameters for Atari
with open("dqn.yml", "w") as f:
f.write("""
SpaceInvadersNoFrameskip-v4:
env_wrapper:
- stable_baselines3.common.atari_wrappers.AtariWrapper
frame_stack: 4
policy: 'CnnPolicy'
n_timesteps: !!float 1e6
buffer_size: 100000
learning_rate: !!float 1.5e-4
batch_size: 64
learning_starts: 20000
target_update_interval: 1000
train_freq: 4
gradient_steps: 1
exploration_fraction: 0.1
exploration_final_eps: 0.01
optimize_memory_usage: False
""")
Here we see that:
- We use the
Atari Wrapperthat preprocess the input (Frame reduction ,grayscale, stack 4 frames) - We use
CnnPolicy, since we use Convolutional layers to process the frames - We train it for 10 million
n_timesteps - Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.
💡 My advice is to reduce the training timesteps to 1M, which will take about 90 minutes on a P100. !nvidia-smi will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: File>Download.
In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:
learning_ratebuffer_size (Experience Memory size)batch_size
As a good practice, you need to check the documentation to understand what each hyperparameters does: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters
- We start the training and save the models on
logsfolder 📁
- Define the algorithm after
--algo, where we save the model after-fand where the hyperparameter config is after-c.
In [ ]:
%load_ext tensorboard
%tensorboard --logdir logs/
In [ ]:
!python -m rl_zoo3.train --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/ -c dqn.yml --tensorboard-log logs/
Streaming output truncated to the last 5000 lines.
| learning_rate | 0.00015 |
| loss | 0.0276 |
| n_updates | 139103 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2500 |
| fps | 245 |
| time_elapsed | 2355 |
| total_timesteps | 578223 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0284 |
| n_updates | 139555 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2504 |
| fps | 245 |
| time_elapsed | 2360 |
| total_timesteps | 579298 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0215 |
| n_updates | 139824 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2508 |
| fps | 245 |
| time_elapsed | 2365 |
| total_timesteps | 580630 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0272 |
| n_updates | 140157 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2512 |
| fps | 245 |
| time_elapsed | 2370 |
| total_timesteps | 581966 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0178 |
| n_updates | 140491 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2516 |
| fps | 245 |
| time_elapsed | 2376 |
| total_timesteps | 583688 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0144 |
| n_updates | 140921 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.62e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2520 |
| fps | 245 |
| time_elapsed | 2382 |
| total_timesteps | 585148 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0495 |
| n_updates | 141286 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 521 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2524 |
| fps | 245 |
| time_elapsed | 2388 |
| total_timesteps | 586617 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0337 |
| n_updates | 141654 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2528 |
| fps | 245 |
| time_elapsed | 2392 |
| total_timesteps | 587805 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0194 |
| n_updates | 141951 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.65e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2532 |
| fps | 245 |
| time_elapsed | 2397 |
| total_timesteps | 589056 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0573 |
| n_updates | 142263 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 522 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2536 |
| fps | 245 |
| time_elapsed | 2401 |
| total_timesteps | 590108 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0114 |
| n_updates | 142526 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 522 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2540 |
| fps | 245 |
| time_elapsed | 2406 |
| total_timesteps | 591306 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.024 |
| n_updates | 142826 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 523 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2544 |
| fps | 245 |
| time_elapsed | 2411 |
| total_timesteps | 592744 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.019 |
| n_updates | 143185 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2548 |
| fps | 245 |
| time_elapsed | 2420 |
| total_timesteps | 595093 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0859 |
| n_updates | 143773 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 521 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2552 |
| fps | 245 |
| time_elapsed | 2425 |
| total_timesteps | 596434 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0217 |
| n_updates | 144108 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 525 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2556 |
| fps | 245 |
| time_elapsed | 2432 |
| total_timesteps | 598348 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0479 |
| n_updates | 144586 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 523 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2560 |
| fps | 245 |
| time_elapsed | 2436 |
| total_timesteps | 599117 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0237 |
| n_updates | 144779 |
----------------------------------
Eval num_timesteps=600000, episode_reward=588.00 +/- 264.53
Episode length: 3974.60 +/- 600.38
----------------------------------
| eval/ | |
| mean_ep_length | 3.97e+03 |
| mean_reward | 588 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 600000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0216 |
| n_updates | 144999 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2564 |
| fps | 244 |
| time_elapsed | 2454 |
| total_timesteps | 601165 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0213 |
| n_updates | 145291 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2568 |
| fps | 244 |
| time_elapsed | 2458 |
| total_timesteps | 602284 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0173 |
| n_updates | 145570 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 525 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2572 |
| fps | 244 |
| time_elapsed | 2464 |
| total_timesteps | 603893 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0194 |
| n_updates | 145973 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 524 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2576 |
| fps | 245 |
| time_elapsed | 2470 |
| total_timesteps | 605279 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0381 |
| n_updates | 146319 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 523 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2580 |
| fps | 245 |
| time_elapsed | 2478 |
| total_timesteps | 607343 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0175 |
| n_updates | 146835 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.74e+03 |
| ep_rew_mean | 524 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2584 |
| fps | 245 |
| time_elapsed | 2483 |
| total_timesteps | 608719 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0106 |
| n_updates | 147179 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.75e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2588 |
| fps | 245 |
| time_elapsed | 2490 |
| total_timesteps | 610650 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0184 |
| n_updates | 147662 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.75e+03 |
| ep_rew_mean | 528 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2592 |
| fps | 245 |
| time_elapsed | 2493 |
| total_timesteps | 611288 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.015 |
| n_updates | 147821 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.75e+03 |
| ep_rew_mean | 530 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2596 |
| fps | 245 |
| time_elapsed | 2497 |
| total_timesteps | 612219 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0209 |
| n_updates | 148054 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.74e+03 |
| ep_rew_mean | 529 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2600 |
| fps | 245 |
| time_elapsed | 2503 |
| total_timesteps | 614021 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0235 |
| n_updates | 148505 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 530 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2604 |
| fps | 245 |
| time_elapsed | 2509 |
| total_timesteps | 615530 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0139 |
| n_updates | 148882 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 530 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2608 |
| fps | 245 |
| time_elapsed | 2513 |
| total_timesteps | 616495 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0377 |
| n_updates | 149123 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 525 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2612 |
| fps | 245 |
| time_elapsed | 2519 |
| total_timesteps | 617980 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0294 |
| n_updates | 149494 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 522 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2616 |
| fps | 245 |
| time_elapsed | 2525 |
| total_timesteps | 619538 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0236 |
| n_updates | 149884 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2620 |
| fps | 245 |
| time_elapsed | 2531 |
| total_timesteps | 621162 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0482 |
| n_updates | 150290 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 526 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2624 |
| fps | 245 |
| time_elapsed | 2536 |
| total_timesteps | 622548 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0258 |
| n_updates | 150636 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.72e+03 |
| ep_rew_mean | 525 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2628 |
| fps | 245 |
| time_elapsed | 2541 |
| total_timesteps | 623819 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0204 |
| n_updates | 150954 |
----------------------------------
Eval num_timesteps=625000, episode_reward=306.00 +/- 120.22
Episode length: 2723.80 +/- 684.12
----------------------------------
| eval/ | |
| mean_ep_length | 2.72e+03 |
| mean_reward | 306 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 625000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0187 |
| n_updates | 151249 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 523 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2632 |
| fps | 244 |
| time_elapsed | 2553 |
| total_timesteps | 625199 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0346 |
| n_updates | 151299 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 521 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2636 |
| fps | 244 |
| time_elapsed | 2555 |
| total_timesteps | 625652 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0173 |
| n_updates | 151412 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2640 |
| fps | 244 |
| time_elapsed | 2560 |
| total_timesteps | 626762 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0173 |
| n_updates | 151690 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2644 |
| fps | 244 |
| time_elapsed | 2565 |
| total_timesteps | 628330 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0271 |
| n_updates | 152082 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 519 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2648 |
| fps | 244 |
| time_elapsed | 2572 |
| total_timesteps | 629942 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0217 |
| n_updates | 152485 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2652 |
| fps | 244 |
| time_elapsed | 2576 |
| total_timesteps | 631040 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0294 |
| n_updates | 152759 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2656 |
| fps | 244 |
| time_elapsed | 2581 |
| total_timesteps | 632190 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0205 |
| n_updates | 153047 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 517 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2660 |
| fps | 244 |
| time_elapsed | 2586 |
| total_timesteps | 633476 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0253 |
| n_updates | 153368 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2664 |
| fps | 244 |
| time_elapsed | 2590 |
| total_timesteps | 634663 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0412 |
| n_updates | 153665 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2668 |
| fps | 244 |
| time_elapsed | 2595 |
| total_timesteps | 635909 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0164 |
| n_updates | 153977 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2672 |
| fps | 245 |
| time_elapsed | 2600 |
| total_timesteps | 637284 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 154320 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 511 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2676 |
| fps | 245 |
| time_elapsed | 2604 |
| total_timesteps | 638147 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0395 |
| n_updates | 154536 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 508 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2680 |
| fps | 245 |
| time_elapsed | 2609 |
| total_timesteps | 639405 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.039 |
| n_updates | 154851 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2684 |
| fps | 245 |
| time_elapsed | 2613 |
| total_timesteps | 640557 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0309 |
| n_updates | 155139 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.66e+03 |
| ep_rew_mean | 507 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2688 |
| fps | 245 |
| time_elapsed | 2616 |
| total_timesteps | 641501 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0112 |
| n_updates | 155375 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.66e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2692 |
| fps | 245 |
| time_elapsed | 2621 |
| total_timesteps | 642619 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0257 |
| n_updates | 155654 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 508 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2696 |
| fps | 245 |
| time_elapsed | 2628 |
| total_timesteps | 644400 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0189 |
| n_updates | 156099 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2700 |
| fps | 245 |
| time_elapsed | 2631 |
| total_timesteps | 645223 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.027 |
| n_updates | 156305 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 519 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2704 |
| fps | 245 |
| time_elapsed | 2639 |
| total_timesteps | 647255 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0143 |
| n_updates | 156813 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.72e+03 |
| ep_rew_mean | 518 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2708 |
| fps | 245 |
| time_elapsed | 2645 |
| total_timesteps | 648726 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0111 |
| n_updates | 157181 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2712 |
| fps | 245 |
| time_elapsed | 2648 |
| total_timesteps | 649513 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0215 |
| n_updates | 157378 |
----------------------------------
Eval num_timesteps=650000, episode_reward=525.00 +/- 30.98
Episode length: 3786.80 +/- 537.81
----------------------------------
| eval/ | |
| mean_ep_length | 3.79e+03 |
| mean_reward | 525 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 650000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0107 |
| n_updates | 157499 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2716 |
| fps | 244 |
| time_elapsed | 2662 |
| total_timesteps | 650695 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.027 |
| n_updates | 157673 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.72e+03 |
| ep_rew_mean | 518 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2720 |
| fps | 244 |
| time_elapsed | 2668 |
| total_timesteps | 652262 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0244 |
| n_updates | 158065 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.72e+03 |
| ep_rew_mean | 519 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2724 |
| fps | 244 |
| time_elapsed | 2672 |
| total_timesteps | 653248 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0209 |
| n_updates | 158311 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2728 |
| fps | 244 |
| time_elapsed | 2677 |
| total_timesteps | 654676 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0208 |
| n_updates | 158668 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2732 |
| fps | 244 |
| time_elapsed | 2682 |
| total_timesteps | 656013 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 159003 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 511 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2736 |
| fps | 244 |
| time_elapsed | 2685 |
| total_timesteps | 656779 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0213 |
| n_updates | 159194 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2740 |
| fps | 244 |
| time_elapsed | 2692 |
| total_timesteps | 658676 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0242 |
| n_updates | 159668 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2744 |
| fps | 244 |
| time_elapsed | 2696 |
| total_timesteps | 659670 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0296 |
| n_updates | 159917 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2748 |
| fps | 244 |
| time_elapsed | 2702 |
| total_timesteps | 661125 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0219 |
| n_updates | 160281 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2752 |
| fps | 244 |
| time_elapsed | 2707 |
| total_timesteps | 662492 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0382 |
| n_updates | 160622 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 517 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2756 |
| fps | 244 |
| time_elapsed | 2711 |
| total_timesteps | 663534 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0331 |
| n_updates | 160883 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2760 |
| fps | 244 |
| time_elapsed | 2716 |
| total_timesteps | 664899 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0235 |
| n_updates | 161224 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2764 |
| fps | 244 |
| time_elapsed | 2724 |
| total_timesteps | 666738 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0267 |
| n_updates | 161684 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2768 |
| fps | 244 |
| time_elapsed | 2726 |
| total_timesteps | 667520 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0303 |
| n_updates | 161879 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.7e+03 |
| ep_rew_mean | 517 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2772 |
| fps | 244 |
| time_elapsed | 2734 |
| total_timesteps | 669433 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0353 |
| n_updates | 162358 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 515 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2776 |
| fps | 244 |
| time_elapsed | 2737 |
| total_timesteps | 670282 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0337 |
| n_updates | 162570 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.71e+03 |
| ep_rew_mean | 519 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2780 |
| fps | 244 |
| time_elapsed | 2741 |
| total_timesteps | 671351 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0161 |
| n_updates | 162837 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 515 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2784 |
| fps | 244 |
| time_elapsed | 2747 |
| total_timesteps | 672747 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0126 |
| n_updates | 163186 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 512 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2788 |
| fps | 244 |
| time_elapsed | 2751 |
| total_timesteps | 673766 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0191 |
| n_updates | 163441 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.66e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2792 |
| fps | 244 |
| time_elapsed | 2756 |
| total_timesteps | 674981 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0439 |
| n_updates | 163745 |
----------------------------------
Eval num_timesteps=675000, episode_reward=497.00 +/- 54.09
Episode length: 3865.40 +/- 422.10
----------------------------------
| eval/ | |
| mean_ep_length | 3.87e+03 |
| mean_reward | 497 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 675000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0196 |
| n_updates | 163749 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2796 |
| fps | 244 |
| time_elapsed | 2769 |
| total_timesteps | 675872 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0598 |
| n_updates | 163967 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2800 |
| fps | 244 |
| time_elapsed | 2775 |
| total_timesteps | 677362 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0167 |
| n_updates | 164340 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2804 |
| fps | 244 |
| time_elapsed | 2777 |
| total_timesteps | 678011 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0117 |
| n_updates | 164502 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2808 |
| fps | 244 |
| time_elapsed | 2781 |
| total_timesteps | 679014 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0101 |
| n_updates | 164753 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 505 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2812 |
| fps | 244 |
| time_elapsed | 2786 |
| total_timesteps | 680194 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0197 |
| n_updates | 165048 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 504 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2816 |
| fps | 244 |
| time_elapsed | 2789 |
| total_timesteps | 681055 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0133 |
| n_updates | 165263 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 505 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2820 |
| fps | 244 |
| time_elapsed | 2794 |
| total_timesteps | 682377 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0218 |
| n_updates | 165594 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 501 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2824 |
| fps | 244 |
| time_elapsed | 2798 |
| total_timesteps | 683481 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0266 |
| n_updates | 165870 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2828 |
| fps | 244 |
| time_elapsed | 2802 |
| total_timesteps | 684563 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0296 |
| n_updates | 166140 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2832 |
| fps | 244 |
| time_elapsed | 2810 |
| total_timesteps | 686624 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0253 |
| n_updates | 166655 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2836 |
| fps | 244 |
| time_elapsed | 2814 |
| total_timesteps | 687585 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0129 |
| n_updates | 166896 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2840 |
| fps | 244 |
| time_elapsed | 2818 |
| total_timesteps | 688669 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0185 |
| n_updates | 167167 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2844 |
| fps | 244 |
| time_elapsed | 2823 |
| total_timesteps | 689828 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0155 |
| n_updates | 167456 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 491 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2848 |
| fps | 244 |
| time_elapsed | 2828 |
| total_timesteps | 691224 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0466 |
| n_updates | 167805 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 488 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2852 |
| fps | 244 |
| time_elapsed | 2834 |
| total_timesteps | 692577 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0391 |
| n_updates | 168144 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 484 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2856 |
| fps | 244 |
| time_elapsed | 2837 |
| total_timesteps | 693423 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.039 |
| n_updates | 168355 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2860 |
| fps | 244 |
| time_elapsed | 2842 |
| total_timesteps | 694691 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0234 |
| n_updates | 168672 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 479 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2864 |
| fps | 244 |
| time_elapsed | 2848 |
| total_timesteps | 696323 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0196 |
| n_updates | 169080 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 474 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2868 |
| fps | 244 |
| time_elapsed | 2853 |
| total_timesteps | 697563 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0191 |
| n_updates | 169390 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 479 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2872 |
| fps | 244 |
| time_elapsed | 2859 |
| total_timesteps | 699268 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0314 |
| n_updates | 169816 |
----------------------------------
Eval num_timesteps=700000, episode_reward=494.00 +/- 228.28
Episode length: 3620.20 +/- 978.25
----------------------------------
| eval/ | |
| mean_ep_length | 3.62e+03 |
| mean_reward | 494 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 700000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.024 |
| n_updates | 169999 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 479 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2876 |
| fps | 243 |
| time_elapsed | 2872 |
| total_timesteps | 700191 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0261 |
| n_updates | 170047 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.47e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2880 |
| fps | 243 |
| time_elapsed | 2878 |
| total_timesteps | 701558 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0328 |
| n_updates | 170389 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2884 |
| fps | 243 |
| time_elapsed | 2884 |
| total_timesteps | 703269 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0349 |
| n_updates | 170817 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2888 |
| fps | 243 |
| time_elapsed | 2889 |
| total_timesteps | 704508 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0384 |
| n_updates | 171126 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.45e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2892 |
| fps | 243 |
| time_elapsed | 2895 |
| total_timesteps | 705885 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0202 |
| n_updates | 171471 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.44e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2896 |
| fps | 243 |
| time_elapsed | 2902 |
| total_timesteps | 707794 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0258 |
| n_updates | 171948 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.45e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2900 |
| fps | 243 |
| time_elapsed | 2907 |
| total_timesteps | 709061 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0283 |
| n_updates | 172265 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2904 |
| fps | 243 |
| time_elapsed | 2912 |
| total_timesteps | 710543 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0247 |
| n_updates | 172635 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2908 |
| fps | 243 |
| time_elapsed | 2919 |
| total_timesteps | 712130 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0175 |
| n_updates | 173032 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 482 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2912 |
| fps | 243 |
| time_elapsed | 2921 |
| total_timesteps | 712790 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0241 |
| n_updates | 173197 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2916 |
| fps | 243 |
| time_elapsed | 2924 |
| total_timesteps | 713503 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0277 |
| n_updates | 173375 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.41e+03 |
| ep_rew_mean | 469 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2920 |
| fps | 243 |
| time_elapsed | 2928 |
| total_timesteps | 714593 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0219 |
| n_updates | 173648 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.41e+03 |
| ep_rew_mean | 468 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2924 |
| fps | 244 |
| time_elapsed | 2933 |
| total_timesteps | 715908 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0273 |
| n_updates | 173976 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.41e+03 |
| ep_rew_mean | 469 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2928 |
| fps | 244 |
| time_elapsed | 2939 |
| total_timesteps | 717376 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0398 |
| n_updates | 174343 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.42e+03 |
| ep_rew_mean | 469 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2932 |
| fps | 244 |
| time_elapsed | 2943 |
| total_timesteps | 718391 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0289 |
| n_updates | 174597 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.42e+03 |
| ep_rew_mean | 469 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2936 |
| fps | 244 |
| time_elapsed | 2951 |
| total_timesteps | 720385 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0257 |
| n_updates | 175096 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.45e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2940 |
| fps | 244 |
| time_elapsed | 2954 |
| total_timesteps | 721326 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0213 |
| n_updates | 175331 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.44e+03 |
| ep_rew_mean | 467 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2944 |
| fps | 244 |
| time_elapsed | 2957 |
| total_timesteps | 722117 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.021 |
| n_updates | 175529 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.47e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2948 |
| fps | 244 |
| time_elapsed | 2965 |
| total_timesteps | 724094 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0135 |
| n_updates | 176023 |
----------------------------------
Eval num_timesteps=725000, episode_reward=634.00 +/- 187.49
Episode length: 4396.40 +/- 917.14
----------------------------------
| eval/ | |
| mean_ep_length | 4.4e+03 |
| mean_reward | 634 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 725000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0392 |
| n_updates | 176249 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.45e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2952 |
| fps | 243 |
| time_elapsed | 2983 |
| total_timesteps | 725746 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0178 |
| n_updates | 176436 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2956 |
| fps | 243 |
| time_elapsed | 2990 |
| total_timesteps | 727727 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0401 |
| n_updates | 176931 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2960 |
| fps | 243 |
| time_elapsed | 3001 |
| total_timesteps | 730527 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0335 |
| n_updates | 177631 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 479 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2964 |
| fps | 243 |
| time_elapsed | 3007 |
| total_timesteps | 732042 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0254 |
| n_updates | 178010 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.55e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2968 |
| fps | 243 |
| time_elapsed | 3013 |
| total_timesteps | 733708 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0173 |
| n_updates | 178426 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.53e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2972 |
| fps | 243 |
| time_elapsed | 3020 |
| total_timesteps | 735429 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0302 |
| n_updates | 178857 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2976 |
| fps | 243 |
| time_elapsed | 3026 |
| total_timesteps | 737099 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0289 |
| n_updates | 179274 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2980 |
| fps | 243 |
| time_elapsed | 3035 |
| total_timesteps | 739414 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0373 |
| n_updates | 179853 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2984 |
| fps | 243 |
| time_elapsed | 3038 |
| total_timesteps | 740169 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0513 |
| n_updates | 180042 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2988 |
| fps | 243 |
| time_elapsed | 3043 |
| total_timesteps | 741388 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0289 |
| n_updates | 180346 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2992 |
| fps | 243 |
| time_elapsed | 3052 |
| total_timesteps | 743734 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0305 |
| n_updates | 180933 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 2996 |
| fps | 243 |
| time_elapsed | 3054 |
| total_timesteps | 744422 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0285 |
| n_updates | 181105 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3000 |
| fps | 243 |
| time_elapsed | 3061 |
| total_timesteps | 746341 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0236 |
| n_updates | 181585 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3004 |
| fps | 243 |
| time_elapsed | 3066 |
| total_timesteps | 747593 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0281 |
| n_updates | 181898 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3008 |
| fps | 243 |
| time_elapsed | 3073 |
| total_timesteps | 749285 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0299 |
| n_updates | 182321 |
----------------------------------
Eval num_timesteps=750000, episode_reward=504.00 +/- 181.48
Episode length: 3352.60 +/- 547.52
----------------------------------
| eval/ | |
| mean_ep_length | 3.35e+03 |
| mean_reward | 504 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 750000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 182499 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.5e+03 |
| ep_rew_mean | 471 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3012 |
| fps | 243 |
| time_elapsed | 3084 |
| total_timesteps | 750042 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.025 |
| n_updates | 182510 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 471 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3016 |
| fps | 243 |
| time_elapsed | 3092 |
| total_timesteps | 752009 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0136 |
| n_updates | 183002 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 472 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3020 |
| fps | 243 |
| time_elapsed | 3099 |
| total_timesteps | 753845 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0148 |
| n_updates | 183461 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 475 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3024 |
| fps | 243 |
| time_elapsed | 3105 |
| total_timesteps | 755409 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0248 |
| n_updates | 183852 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3028 |
| fps | 243 |
| time_elapsed | 3111 |
| total_timesteps | 756901 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0203 |
| n_updates | 184225 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 479 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3032 |
| fps | 243 |
| time_elapsed | 3117 |
| total_timesteps | 758551 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0216 |
| n_updates | 184637 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 481 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3036 |
| fps | 243 |
| time_elapsed | 3120 |
| total_timesteps | 759410 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0713 |
| n_updates | 184852 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 484 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3040 |
| fps | 243 |
| time_elapsed | 3126 |
| total_timesteps | 760935 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0203 |
| n_updates | 185233 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 484 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3044 |
| fps | 243 |
| time_elapsed | 3130 |
| total_timesteps | 761877 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.02 |
| n_updates | 185469 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 481 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3048 |
| fps | 243 |
| time_elapsed | 3135 |
| total_timesteps | 763108 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0191 |
| n_updates | 185776 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 482 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3052 |
| fps | 243 |
| time_elapsed | 3140 |
| total_timesteps | 764401 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0224 |
| n_updates | 186100 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3056 |
| fps | 243 |
| time_elapsed | 3144 |
| total_timesteps | 765457 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0107 |
| n_updates | 186364 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 489 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3060 |
| fps | 243 |
| time_elapsed | 3151 |
| total_timesteps | 767388 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0168 |
| n_updates | 186846 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 487 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3064 |
| fps | 243 |
| time_elapsed | 3156 |
| total_timesteps | 768655 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 187163 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.62e+03 |
| ep_rew_mean | 488 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3068 |
| fps | 243 |
| time_elapsed | 3161 |
| total_timesteps | 769827 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0134 |
| n_updates | 187456 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3072 |
| fps | 243 |
| time_elapsed | 3168 |
| total_timesteps | 771697 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0299 |
| n_updates | 187924 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 493 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3076 |
| fps | 243 |
| time_elapsed | 3174 |
| total_timesteps | 773183 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0999 |
| n_updates | 188295 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.65e+03 |
| ep_rew_mean | 496 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3080 |
| fps | 243 |
| time_elapsed | 3179 |
| total_timesteps | 774522 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0192 |
| n_updates | 188630 |
----------------------------------
Eval num_timesteps=775000, episode_reward=431.00 +/- 249.59
Episode length: 3581.60 +/- 1447.41
----------------------------------
| eval/ | |
| mean_ep_length | 3.58e+03 |
| mean_reward | 431 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 775000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0219 |
| n_updates | 188749 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.65e+03 |
| ep_rew_mean | 496 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3084 |
| fps | 242 |
| time_elapsed | 3192 |
| total_timesteps | 775552 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.014 |
| n_updates | 188887 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 495 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3088 |
| fps | 242 |
| time_elapsed | 3199 |
| total_timesteps | 777103 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.023 |
| n_updates | 189275 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.66e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3092 |
| fps | 242 |
| time_elapsed | 3202 |
| total_timesteps | 777980 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0471 |
| n_updates | 189494 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.65e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3096 |
| fps | 242 |
| time_elapsed | 3206 |
| total_timesteps | 778968 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0249 |
| n_updates | 189741 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.66e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3100 |
| fps | 242 |
| time_elapsed | 3213 |
| total_timesteps | 780768 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0151 |
| n_updates | 190191 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.67e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3104 |
| fps | 243 |
| time_elapsed | 3217 |
| total_timesteps | 781910 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0375 |
| n_updates | 190477 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.68e+03 |
| ep_rew_mean | 508 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3108 |
| fps | 243 |
| time_elapsed | 3223 |
| total_timesteps | 783375 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0188 |
| n_updates | 190843 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 511 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3112 |
| fps | 243 |
| time_elapsed | 3227 |
| total_timesteps | 784467 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.016 |
| n_updates | 191116 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.74e+03 |
| ep_rew_mean | 525 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3116 |
| fps | 243 |
| time_elapsed | 3237 |
| total_timesteps | 786982 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0166 |
| n_updates | 191745 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.76e+03 |
| ep_rew_mean | 534 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3120 |
| fps | 243 |
| time_elapsed | 3244 |
| total_timesteps | 788976 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0232 |
| n_updates | 192243 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.74e+03 |
| ep_rew_mean | 533 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3124 |
| fps | 243 |
| time_elapsed | 3249 |
| total_timesteps | 790125 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0175 |
| n_updates | 192531 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.74e+03 |
| ep_rew_mean | 533 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3128 |
| fps | 243 |
| time_elapsed | 3255 |
| total_timesteps | 791662 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 192915 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.76e+03 |
| ep_rew_mean | 535 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3132 |
| fps | 243 |
| time_elapsed | 3260 |
| total_timesteps | 793011 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.102 |
| n_updates | 193252 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.76e+03 |
| ep_rew_mean | 533 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3136 |
| fps | 243 |
| time_elapsed | 3265 |
| total_timesteps | 794448 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0621 |
| n_updates | 193611 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.76e+03 |
| ep_rew_mean | 534 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3140 |
| fps | 243 |
| time_elapsed | 3270 |
| total_timesteps | 795624 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0205 |
| n_updates | 193905 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 535 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3144 |
| fps | 243 |
| time_elapsed | 3276 |
| total_timesteps | 797196 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0194 |
| n_updates | 194298 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 536 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3148 |
| fps | 243 |
| time_elapsed | 3281 |
| total_timesteps | 798517 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0258 |
| n_updates | 194629 |
----------------------------------
Eval num_timesteps=800000, episode_reward=696.00 +/- 203.14
Episode length: 4015.00 +/- 1252.83
----------------------------------
| eval/ | |
| mean_ep_length | 4.02e+03 |
| mean_reward | 696 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 800000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0188 |
| n_updates | 194999 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.79e+03 |
| ep_rew_mean | 536 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3152 |
| fps | 242 |
| time_elapsed | 3299 |
| total_timesteps | 800508 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0269 |
| n_updates | 195126 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.8e+03 |
| ep_rew_mean | 537 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3156 |
| fps | 242 |
| time_elapsed | 3306 |
| total_timesteps | 802293 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.014 |
| n_updates | 195573 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.81e+03 |
| ep_rew_mean | 540 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3160 |
| fps | 242 |
| time_elapsed | 3312 |
| total_timesteps | 803803 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0189 |
| n_updates | 195950 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.84e+03 |
| ep_rew_mean | 538 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3164 |
| fps | 242 |
| time_elapsed | 3317 |
| total_timesteps | 805161 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0245 |
| n_updates | 196290 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.84e+03 |
| ep_rew_mean | 537 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3168 |
| fps | 242 |
| time_elapsed | 3322 |
| total_timesteps | 806202 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.019 |
| n_updates | 196550 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.83e+03 |
| ep_rew_mean | 536 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3172 |
| fps | 242 |
| time_elapsed | 3328 |
| total_timesteps | 807838 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0211 |
| n_updates | 196959 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.9e+03 |
| ep_rew_mean | 554 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3176 |
| fps | 242 |
| time_elapsed | 3335 |
| total_timesteps | 809649 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0434 |
| n_updates | 197412 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 559 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3180 |
| fps | 242 |
| time_elapsed | 3341 |
| total_timesteps | 811173 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0408 |
| n_updates | 197793 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.94e+03 |
| ep_rew_mean | 567 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3184 |
| fps | 242 |
| time_elapsed | 3348 |
| total_timesteps | 812908 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0222 |
| n_updates | 198226 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.96e+03 |
| ep_rew_mean | 567 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3188 |
| fps | 242 |
| time_elapsed | 3352 |
| total_timesteps | 814224 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0147 |
| n_updates | 198555 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.95e+03 |
| ep_rew_mean | 566 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3192 |
| fps | 242 |
| time_elapsed | 3357 |
| total_timesteps | 815339 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0214 |
| n_updates | 198834 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.97e+03 |
| ep_rew_mean | 570 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3196 |
| fps | 242 |
| time_elapsed | 3365 |
| total_timesteps | 817436 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0251 |
| n_updates | 199358 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.94e+03 |
| ep_rew_mean | 565 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3200 |
| fps | 242 |
| time_elapsed | 3375 |
| total_timesteps | 820105 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0139 |
| n_updates | 200026 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 4e+03 |
| ep_rew_mean | 577 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3204 |
| fps | 242 |
| time_elapsed | 3379 |
| total_timesteps | 821130 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.062 |
| n_updates | 200282 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.98e+03 |
| ep_rew_mean | 573 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3208 |
| fps | 242 |
| time_elapsed | 3383 |
| total_timesteps | 822230 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0171 |
| n_updates | 200557 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.99e+03 |
| ep_rew_mean | 573 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3212 |
| fps | 243 |
| time_elapsed | 3389 |
| total_timesteps | 823861 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0607 |
| n_updates | 200965 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 4.01e+03 |
| ep_rew_mean | 575 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3216 |
| fps | 243 |
| time_elapsed | 3394 |
| total_timesteps | 824993 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0227 |
| n_updates | 201248 |
----------------------------------
Eval num_timesteps=825000, episode_reward=398.00 +/- 100.33
Episode length: 2806.80 +/- 273.90
----------------------------------
| eval/ | |
| mean_ep_length | 2.81e+03 |
| mean_reward | 398 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 825000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0182 |
| n_updates | 201249 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.99e+03 |
| ep_rew_mean | 574 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3220 |
| fps | 242 |
| time_elapsed | 3405 |
| total_timesteps | 826150 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0146 |
| n_updates | 201537 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.98e+03 |
| ep_rew_mean | 574 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3224 |
| fps | 242 |
| time_elapsed | 3409 |
| total_timesteps | 826806 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0214 |
| n_updates | 201701 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.91e+03 |
| ep_rew_mean | 566 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3228 |
| fps | 242 |
| time_elapsed | 3412 |
| total_timesteps | 827596 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0174 |
| n_updates | 201898 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.9e+03 |
| ep_rew_mean | 564 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3232 |
| fps | 242 |
| time_elapsed | 3416 |
| total_timesteps | 828679 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0234 |
| n_updates | 202169 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.9e+03 |
| ep_rew_mean | 564 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3236 |
| fps | 242 |
| time_elapsed | 3422 |
| total_timesteps | 830230 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0806 |
| n_updates | 202557 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 566 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3240 |
| fps | 242 |
| time_elapsed | 3428 |
| total_timesteps | 831983 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0207 |
| n_updates | 202995 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 562 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3244 |
| fps | 242 |
| time_elapsed | 3436 |
| total_timesteps | 833994 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0409 |
| n_updates | 203498 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 566 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3248 |
| fps | 242 |
| time_elapsed | 3443 |
| total_timesteps | 835769 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0211 |
| n_updates | 203942 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 561 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3252 |
| fps | 242 |
| time_elapsed | 3446 |
| total_timesteps | 836433 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0199 |
| n_updates | 204108 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 561 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3256 |
| fps | 242 |
| time_elapsed | 3451 |
| total_timesteps | 837830 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0262 |
| n_updates | 204457 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.94e+03 |
| ep_rew_mean | 565 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3260 |
| fps | 242 |
| time_elapsed | 3454 |
| total_timesteps | 838629 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0186 |
| n_updates | 204657 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 562 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3264 |
| fps | 242 |
| time_elapsed | 3459 |
| total_timesteps | 839823 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0218 |
| n_updates | 204955 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.91e+03 |
| ep_rew_mean | 560 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3268 |
| fps | 242 |
| time_elapsed | 3463 |
| total_timesteps | 840856 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0244 |
| n_updates | 205213 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 560 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3272 |
| fps | 242 |
| time_elapsed | 3467 |
| total_timesteps | 841869 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0445 |
| n_updates | 205467 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.91e+03 |
| ep_rew_mean | 558 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3276 |
| fps | 242 |
| time_elapsed | 3471 |
| total_timesteps | 842933 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0535 |
| n_updates | 205733 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 561 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3280 |
| fps | 242 |
| time_elapsed | 3476 |
| total_timesteps | 844160 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0236 |
| n_updates | 206039 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.89e+03 |
| ep_rew_mean | 555 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3284 |
| fps | 242 |
| time_elapsed | 3479 |
| total_timesteps | 845102 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 206275 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.89e+03 |
| ep_rew_mean | 556 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3288 |
| fps | 242 |
| time_elapsed | 3484 |
| total_timesteps | 846290 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0183 |
| n_updates | 206572 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.89e+03 |
| ep_rew_mean | 556 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3292 |
| fps | 242 |
| time_elapsed | 3490 |
| total_timesteps | 847848 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0293 |
| n_updates | 206961 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.89e+03 |
| ep_rew_mean | 554 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3296 |
| fps | 242 |
| time_elapsed | 3497 |
| total_timesteps | 849627 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0233 |
| n_updates | 207406 |
----------------------------------
Eval num_timesteps=850000, episode_reward=487.00 +/- 46.32
Episode length: 3533.40 +/- 222.29
----------------------------------
| eval/ | |
| mean_ep_length | 3.53e+03 |
| mean_reward | 487 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 850000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0633 |
| n_updates | 207499 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.87e+03 |
| ep_rew_mean | 548 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3300 |
| fps | 242 |
| time_elapsed | 3509 |
| total_timesteps | 850347 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0158 |
| n_updates | 207586 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.84e+03 |
| ep_rew_mean | 543 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3304 |
| fps | 242 |
| time_elapsed | 3513 |
| total_timesteps | 851516 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0291 |
| n_updates | 207878 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.84e+03 |
| ep_rew_mean | 540 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3308 |
| fps | 242 |
| time_elapsed | 3518 |
| total_timesteps | 852634 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0393 |
| n_updates | 208158 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.81e+03 |
| ep_rew_mean | 537 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3312 |
| fps | 242 |
| time_elapsed | 3522 |
| total_timesteps | 853676 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.03 |
| n_updates | 208418 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.83e+03 |
| ep_rew_mean | 537 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3316 |
| fps | 242 |
| time_elapsed | 3526 |
| total_timesteps | 854916 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0286 |
| n_updates | 208728 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.83e+03 |
| ep_rew_mean | 534 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3320 |
| fps | 242 |
| time_elapsed | 3532 |
| total_timesteps | 856376 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.056 |
| n_updates | 209093 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.85e+03 |
| ep_rew_mean | 538 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3324 |
| fps | 242 |
| time_elapsed | 3537 |
| total_timesteps | 857775 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0183 |
| n_updates | 209443 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.86e+03 |
| ep_rew_mean | 544 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3328 |
| fps | 242 |
| time_elapsed | 3543 |
| total_timesteps | 859107 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0127 |
| n_updates | 209776 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.84e+03 |
| ep_rew_mean | 540 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3332 |
| fps | 242 |
| time_elapsed | 3549 |
| total_timesteps | 860663 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0113 |
| n_updates | 210165 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.86e+03 |
| ep_rew_mean | 541 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3336 |
| fps | 242 |
| time_elapsed | 3552 |
| total_timesteps | 861644 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0325 |
| n_updates | 210410 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.87e+03 |
| ep_rew_mean | 540 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3340 |
| fps | 242 |
| time_elapsed | 3561 |
| total_timesteps | 863815 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0224 |
| n_updates | 210953 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 551 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3344 |
| fps | 242 |
| time_elapsed | 3568 |
| total_timesteps | 865642 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0167 |
| n_updates | 211410 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 549 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3348 |
| fps | 242 |
| time_elapsed | 3573 |
| total_timesteps | 866972 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0194 |
| n_updates | 211742 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.96e+03 |
| ep_rew_mean | 549 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3352 |
| fps | 242 |
| time_elapsed | 3583 |
| total_timesteps | 869519 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0191 |
| n_updates | 212379 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.94e+03 |
| ep_rew_mean | 548 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3356 |
| fps | 242 |
| time_elapsed | 3590 |
| total_timesteps | 871539 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0894 |
| n_updates | 212884 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 546 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3360 |
| fps | 242 |
| time_elapsed | 3596 |
| total_timesteps | 872790 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0405 |
| n_updates | 213197 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.94e+03 |
| ep_rew_mean | 550 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3364 |
| fps | 242 |
| time_elapsed | 3599 |
| total_timesteps | 873759 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0399 |
| n_updates | 213439 |
----------------------------------
Eval num_timesteps=875000, episode_reward=469.00 +/- 87.49
Episode length: 3551.00 +/- 406.00
----------------------------------
| eval/ | |
| mean_ep_length | 3.55e+03 |
| mean_reward | 469 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 875000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0171 |
| n_updates | 213749 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 549 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3368 |
| fps | 242 |
| time_elapsed | 3613 |
| total_timesteps | 875126 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0156 |
| n_updates | 213781 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.93e+03 |
| ep_rew_mean | 551 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3372 |
| fps | 242 |
| time_elapsed | 3620 |
| total_timesteps | 876641 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0313 |
| n_updates | 214160 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.92e+03 |
| ep_rew_mean | 551 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3376 |
| fps | 242 |
| time_elapsed | 3623 |
| total_timesteps | 877610 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0406 |
| n_updates | 214402 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.91e+03 |
| ep_rew_mean | 543 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3380 |
| fps | 242 |
| time_elapsed | 3626 |
| total_timesteps | 878472 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0295 |
| n_updates | 214617 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.89e+03 |
| ep_rew_mean | 542 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3384 |
| fps | 242 |
| time_elapsed | 3632 |
| total_timesteps | 879837 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.039 |
| n_updates | 214959 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.88e+03 |
| ep_rew_mean | 540 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3388 |
| fps | 242 |
| time_elapsed | 3636 |
| total_timesteps | 881014 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0168 |
| n_updates | 215253 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.79e+03 |
| ep_rew_mean | 520 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3392 |
| fps | 242 |
| time_elapsed | 3642 |
| total_timesteps | 882326 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0225 |
| n_updates | 215581 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 518 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3396 |
| fps | 242 |
| time_elapsed | 3646 |
| total_timesteps | 883360 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.02 |
| n_updates | 215839 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 517 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3400 |
| fps | 242 |
| time_elapsed | 3650 |
| total_timesteps | 884550 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0245 |
| n_updates | 216137 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3404 |
| fps | 242 |
| time_elapsed | 3656 |
| total_timesteps | 886046 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0149 |
| n_updates | 216511 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.79e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3408 |
| fps | 242 |
| time_elapsed | 3662 |
| total_timesteps | 887602 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0145 |
| n_updates | 216900 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.82e+03 |
| ep_rew_mean | 522 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3412 |
| fps | 242 |
| time_elapsed | 3667 |
| total_timesteps | 888898 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.024 |
| n_updates | 217224 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.81e+03 |
| ep_rew_mean | 522 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3416 |
| fps | 242 |
| time_elapsed | 3672 |
| total_timesteps | 890101 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0139 |
| n_updates | 217525 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.8e+03 |
| ep_rew_mean | 519 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3420 |
| fps | 242 |
| time_elapsed | 3676 |
| total_timesteps | 891289 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.078 |
| n_updates | 217822 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.79e+03 |
| ep_rew_mean | 516 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3424 |
| fps | 242 |
| time_elapsed | 3680 |
| total_timesteps | 892228 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0696 |
| n_updates | 218056 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.79e+03 |
| ep_rew_mean | 515 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3428 |
| fps | 242 |
| time_elapsed | 3685 |
| total_timesteps | 893512 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0175 |
| n_updates | 218377 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.78e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3432 |
| fps | 242 |
| time_elapsed | 3688 |
| total_timesteps | 894300 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0296 |
| n_updates | 218574 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.76e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3436 |
| fps | 242 |
| time_elapsed | 3693 |
| total_timesteps | 895552 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0212 |
| n_updates | 218887 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.75e+03 |
| ep_rew_mean | 511 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3440 |
| fps | 242 |
| time_elapsed | 3697 |
| total_timesteps | 896803 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0176 |
| n_updates | 219200 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 509 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3444 |
| fps | 242 |
| time_elapsed | 3702 |
| total_timesteps | 897931 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0304 |
| n_updates | 219482 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.73e+03 |
| ep_rew_mean | 507 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3448 |
| fps | 242 |
| time_elapsed | 3706 |
| total_timesteps | 899037 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0143 |
| n_updates | 219759 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3452 |
| fps | 242 |
| time_elapsed | 3710 |
| total_timesteps | 899936 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0171 |
| n_updates | 219983 |
----------------------------------
Eval num_timesteps=900000, episode_reward=476.00 +/- 150.31
Episode length: 3090.40 +/- 233.94
----------------------------------
| eval/ | |
| mean_ep_length | 3.09e+03 |
| mean_reward | 476 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 900000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0237 |
| n_updates | 219999 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 497 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3456 |
| fps | 242 |
| time_elapsed | 3725 |
| total_timesteps | 901751 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0384 |
| n_updates | 220437 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.69e+03 |
| ep_rew_mean | 496 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3460 |
| fps | 242 |
| time_elapsed | 3729 |
| total_timesteps | 902678 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0276 |
| n_updates | 220669 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.65e+03 |
| ep_rew_mean | 489 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3464 |
| fps | 242 |
| time_elapsed | 3732 |
| total_timesteps | 903537 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0147 |
| n_updates | 220884 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.64e+03 |
| ep_rew_mean | 487 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3468 |
| fps | 242 |
| time_elapsed | 3736 |
| total_timesteps | 904723 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0211 |
| n_updates | 221180 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.63e+03 |
| ep_rew_mean | 485 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3472 |
| fps | 242 |
| time_elapsed | 3739 |
| total_timesteps | 905531 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0135 |
| n_updates | 221382 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 478 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3476 |
| fps | 242 |
| time_elapsed | 3743 |
| total_timesteps | 906512 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0193 |
| n_updates | 221627 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3480 |
| fps | 242 |
| time_elapsed | 3747 |
| total_timesteps | 907561 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0642 |
| n_updates | 221890 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 471 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3484 |
| fps | 242 |
| time_elapsed | 3752 |
| total_timesteps | 908812 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0264 |
| n_updates | 222202 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 472 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3488 |
| fps | 242 |
| time_elapsed | 3757 |
| total_timesteps | 910111 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.00951 |
| n_updates | 222527 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 474 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3492 |
| fps | 242 |
| time_elapsed | 3763 |
| total_timesteps | 911592 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0197 |
| n_updates | 222897 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3496 |
| fps | 242 |
| time_elapsed | 3768 |
| total_timesteps | 912988 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0285 |
| n_updates | 223246 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3500 |
| fps | 242 |
| time_elapsed | 3774 |
| total_timesteps | 914383 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0186 |
| n_updates | 223595 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 474 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3504 |
| fps | 242 |
| time_elapsed | 3778 |
| total_timesteps | 915447 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0213 |
| n_updates | 223861 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 480 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3508 |
| fps | 242 |
| time_elapsed | 3782 |
| total_timesteps | 916633 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0774 |
| n_updates | 224158 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 480 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3512 |
| fps | 242 |
| time_elapsed | 3786 |
| total_timesteps | 917536 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0158 |
| n_updates | 224383 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 477 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3516 |
| fps | 242 |
| time_elapsed | 3789 |
| total_timesteps | 918232 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0123 |
| n_updates | 224557 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.53e+03 |
| ep_rew_mean | 470 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3520 |
| fps | 242 |
| time_elapsed | 3793 |
| total_timesteps | 919193 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.022 |
| n_updates | 224798 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 468 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3524 |
| fps | 242 |
| time_elapsed | 3799 |
| total_timesteps | 920986 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0202 |
| n_updates | 225246 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 470 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3528 |
| fps | 242 |
| time_elapsed | 3806 |
| total_timesteps | 922644 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0166 |
| n_updates | 225660 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.55e+03 |
| ep_rew_mean | 471 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3532 |
| fps | 242 |
| time_elapsed | 3812 |
| total_timesteps | 924233 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0133 |
| n_updates | 226058 |
----------------------------------
Eval num_timesteps=925000, episode_reward=427.00 +/- 102.99
Episode length: 3064.60 +/- 462.89
----------------------------------
| eval/ | |
| mean_ep_length | 3.06e+03 |
| mean_reward | 427 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 925000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0247 |
| n_updates | 226249 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3536 |
| fps | 241 |
| time_elapsed | 3823 |
| total_timesteps | 925256 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.019 |
| n_updates | 226313 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 469 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3540 |
| fps | 241 |
| time_elapsed | 3828 |
| total_timesteps | 926444 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0833 |
| n_updates | 226610 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 473 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3544 |
| fps | 241 |
| time_elapsed | 3832 |
| total_timesteps | 927277 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0319 |
| n_updates | 226819 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 476 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3548 |
| fps | 242 |
| time_elapsed | 3838 |
| total_timesteps | 928976 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0212 |
| n_updates | 227243 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 481 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3552 |
| fps | 242 |
| time_elapsed | 3843 |
| total_timesteps | 930173 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0224 |
| n_updates | 227543 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 480 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3556 |
| fps | 242 |
| time_elapsed | 3847 |
| total_timesteps | 931372 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0154 |
| n_updates | 227842 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 480 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3560 |
| fps | 242 |
| time_elapsed | 3850 |
| total_timesteps | 932077 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0154 |
| n_updates | 228019 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 485 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3564 |
| fps | 242 |
| time_elapsed | 3856 |
| total_timesteps | 933473 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0149 |
| n_updates | 228368 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 484 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3568 |
| fps | 242 |
| time_elapsed | 3860 |
| total_timesteps | 934540 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0161 |
| n_updates | 228634 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 485 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3572 |
| fps | 242 |
| time_elapsed | 3867 |
| total_timesteps | 936425 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0214 |
| n_updates | 229106 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 489 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3576 |
| fps | 242 |
| time_elapsed | 3872 |
| total_timesteps | 937770 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0366 |
| n_updates | 229442 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 488 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3580 |
| fps | 242 |
| time_elapsed | 3877 |
| total_timesteps | 938829 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0223 |
| n_updates | 229707 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.61e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3584 |
| fps | 242 |
| time_elapsed | 3881 |
| total_timesteps | 940050 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0268 |
| n_updates | 230012 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 491 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3588 |
| fps | 242 |
| time_elapsed | 3885 |
| total_timesteps | 941145 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0149 |
| n_updates | 230286 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 491 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3592 |
| fps | 242 |
| time_elapsed | 3892 |
| total_timesteps | 942800 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0343 |
| n_updates | 230699 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 493 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3596 |
| fps | 242 |
| time_elapsed | 3896 |
| total_timesteps | 943895 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0532 |
| n_updates | 230973 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3600 |
| fps | 242 |
| time_elapsed | 3902 |
| total_timesteps | 945387 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0125 |
| n_updates | 231346 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.6e+03 |
| ep_rew_mean | 496 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3604 |
| fps | 242 |
| time_elapsed | 3906 |
| total_timesteps | 946595 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0342 |
| n_updates | 231648 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 492 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3608 |
| fps | 242 |
| time_elapsed | 3913 |
| total_timesteps | 948346 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0267 |
| n_updates | 232086 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 493 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3612 |
| fps | 242 |
| time_elapsed | 3919 |
| total_timesteps | 949778 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.042 |
| n_updates | 232444 |
----------------------------------
Eval num_timesteps=950000, episode_reward=376.00 +/- 128.20
Episode length: 3299.80 +/- 443.50
----------------------------------
| eval/ | |
| mean_ep_length | 3.3e+03 |
| mean_reward | 376 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 950000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.00917 |
| n_updates | 232499 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 495 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3616 |
| fps | 241 |
| time_elapsed | 3932 |
| total_timesteps | 950879 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0146 |
| n_updates | 232719 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 486 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3620 |
| fps | 241 |
| time_elapsed | 3935 |
| total_timesteps | 951686 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.016 |
| n_updates | 232921 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.5e+03 |
| ep_rew_mean | 485 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3624 |
| fps | 241 |
| time_elapsed | 3941 |
| total_timesteps | 953246 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0175 |
| n_updates | 233311 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 486 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3628 |
| fps | 241 |
| time_elapsed | 3947 |
| total_timesteps | 954727 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0223 |
| n_updates | 233681 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3632 |
| fps | 241 |
| time_elapsed | 3953 |
| total_timesteps | 956200 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0166 |
| n_updates | 234049 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 480 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3636 |
| fps | 241 |
| time_elapsed | 3957 |
| total_timesteps | 957206 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0197 |
| n_updates | 234301 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3640 |
| fps | 241 |
| time_elapsed | 3961 |
| total_timesteps | 958380 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0151 |
| n_updates | 234594 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.46e+03 |
| ep_rew_mean | 483 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3644 |
| fps | 241 |
| time_elapsed | 3968 |
| total_timesteps | 960010 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0103 |
| n_updates | 235002 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.48e+03 |
| ep_rew_mean | 486 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3648 |
| fps | 241 |
| time_elapsed | 3974 |
| total_timesteps | 961537 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0134 |
| n_updates | 235384 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.47e+03 |
| ep_rew_mean | 485 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3652 |
| fps | 241 |
| time_elapsed | 3978 |
| total_timesteps | 962787 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0227 |
| n_updates | 235696 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 486 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3656 |
| fps | 241 |
| time_elapsed | 3982 |
| total_timesteps | 963813 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0245 |
| n_updates | 235953 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.49e+03 |
| ep_rew_mean | 487 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3660 |
| fps | 242 |
| time_elapsed | 3987 |
| total_timesteps | 964898 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0244 |
| n_updates | 236224 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.5e+03 |
| ep_rew_mean | 488 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3664 |
| fps | 242 |
| time_elapsed | 3991 |
| total_timesteps | 966140 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0193 |
| n_updates | 236534 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 486 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3668 |
| fps | 242 |
| time_elapsed | 3997 |
| total_timesteps | 967680 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.018 |
| n_updates | 236919 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.53e+03 |
| ep_rew_mean | 491 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3672 |
| fps | 242 |
| time_elapsed | 4005 |
| total_timesteps | 969840 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0188 |
| n_updates | 237459 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 501 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3676 |
| fps | 242 |
| time_elapsed | 4011 |
| total_timesteps | 971203 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0196 |
| n_updates | 237800 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3680 |
| fps | 242 |
| time_elapsed | 4016 |
| total_timesteps | 972668 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0118 |
| n_updates | 238166 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.56e+03 |
| ep_rew_mean | 503 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3684 |
| fps | 242 |
| time_elapsed | 4024 |
| total_timesteps | 974528 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0198 |
| n_updates | 238631 |
----------------------------------
Eval num_timesteps=975000, episode_reward=499.00 +/- 106.41
Episode length: 4057.00 +/- 643.55
----------------------------------
| eval/ | |
| mean_ep_length | 4.06e+03 |
| mean_reward | 499 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 975000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0112 |
| n_updates | 238749 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3688 |
| fps | 241 |
| time_elapsed | 4038 |
| total_timesteps | 975443 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0107 |
| n_updates | 238860 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 499 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3692 |
| fps | 241 |
| time_elapsed | 4043 |
| total_timesteps | 976663 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.046 |
| n_updates | 239165 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3696 |
| fps | 241 |
| time_elapsed | 4047 |
| total_timesteps | 977823 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.018 |
| n_updates | 239455 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 501 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3700 |
| fps | 241 |
| time_elapsed | 4054 |
| total_timesteps | 979553 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0314 |
| n_updates | 239888 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.52e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3704 |
| fps | 241 |
| time_elapsed | 4061 |
| total_timesteps | 981264 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0774 |
| n_updates | 240315 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.53e+03 |
| ep_rew_mean | 498 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3708 |
| fps | 241 |
| time_elapsed | 4065 |
| total_timesteps | 982296 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.013 |
| n_updates | 240573 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 495 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3712 |
| fps | 241 |
| time_elapsed | 4068 |
| total_timesteps | 983258 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0184 |
| n_updates | 240814 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 495 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3716 |
| fps | 241 |
| time_elapsed | 4073 |
| total_timesteps | 984503 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0141 |
| n_updates | 241125 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.51e+03 |
| ep_rew_mean | 495 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3720 |
| fps | 241 |
| time_elapsed | 4078 |
| total_timesteps | 985601 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0203 |
| n_updates | 241400 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.53e+03 |
| ep_rew_mean | 501 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3724 |
| fps | 241 |
| time_elapsed | 4084 |
| total_timesteps | 987403 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.034 |
| n_updates | 241850 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 500 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3728 |
| fps | 241 |
| time_elapsed | 4092 |
| total_timesteps | 989365 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0771 |
| n_updates | 242341 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.54e+03 |
| ep_rew_mean | 501 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3732 |
| fps | 241 |
| time_elapsed | 4096 |
| total_timesteps | 990397 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0264 |
| n_updates | 242599 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3736 |
| fps | 241 |
| time_elapsed | 4103 |
| total_timesteps | 992146 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.038 |
| n_updates | 243036 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 506 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3740 |
| fps | 241 |
| time_elapsed | 4107 |
| total_timesteps | 993197 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0208 |
| n_updates | 243299 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.58e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3744 |
| fps | 241 |
| time_elapsed | 4112 |
| total_timesteps | 994415 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0142 |
| n_updates | 243603 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 510 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3748 |
| fps | 241 |
| time_elapsed | 4116 |
| total_timesteps | 995708 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0323 |
| n_updates | 243926 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.59e+03 |
| ep_rew_mean | 514 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3752 |
| fps | 241 |
| time_elapsed | 4120 |
| total_timesteps | 996641 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0293 |
| n_updates | 244160 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.57e+03 |
| ep_rew_mean | 513 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3756 |
| fps | 241 |
| time_elapsed | 4128 |
| total_timesteps | 998625 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0294 |
| n_updates | 244656 |
----------------------------------
----------------------------------
| rollout/ | |
| ep_len_mean | 3.55e+03 |
| ep_rew_mean | 509 |
| exploration_rate | 0.01 |
| time/ | |
| episodes | 3760 |
| fps | 241 |
| time_elapsed | 4131 |
| total_timesteps | 999385 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0849 |
| n_updates | 244846 |
----------------------------------
Eval num_timesteps=1000000, episode_reward=520.00 +/- 197.53
Episode length: 3888.00 +/- 765.39
----------------------------------
| eval/ | |
| mean_ep_length | 3.89e+03 |
| mean_reward | 520 |
| rollout/ | |
| exploration_rate | 0.01 |
| time/ | |
| total_timesteps | 1000000 |
| train/ | |
| learning_rate | 0.00015 |
| loss | 0.0165 |
| n_updates | 244999 |
----------------------------------
Saving to logs//dqn/SpaceInvadersNoFrameskip-v4_1
Solution¶
In [ ]:
!python -m rl_zoo3.train --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/ -c dqn.yml
Let's evaluate our agent 👀¶
- RL-Baselines3-Zoo provides
enjoy.py, a python script to evaluate our agent. In most RL libraries, we call the evaluation scriptenjoy.py. - Let's evaluate it for 5000 timesteps 🔥
In [ ]:
!python -m rl_zoo3.enjoy --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/
2026-01-29 15:57:33.105033: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1769702253.124913 22552 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1769702253.130988 22552 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered W0000 00:00:1769702253.146613 22552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702253.146640 22552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702253.146644 22552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702253.146647 22552 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. 2026-01-29 15:57:33.151230: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality. Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade. See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information. Loading latest experiment, id=1 Loading logs/dqn/SpaceInvadersNoFrameskip-v4_1/SpaceInvadersNoFrameskip-v4.zip A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138) [Powered by Stella] Stacking 4 frames Atari Episode Score: 200.00 Atari Episode Length 2187 Atari Episode Score: 460.00 Atari Episode Length 3797 Atari Episode Score: 755.00 Atari Episode Length 3428 Atari Episode Score: 200.00 Atari Episode Length 2187 Atari Episode Score: 480.00 Atari Episode Length 4363
Solution¶
In [ ]:
!python -m rl_zoo3.enjoy --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/
Publish our trained model on the Hub 🚀¶
Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.
In [47]:
import os
import glob
# This should now return a path!
video_files = glob.glob('logs/videos/**/*.mp4', recursive=True)
print(f"Videos found: {video_files}")
Videos found: []
In [48]:
glob.glob("logs/dqn/SpaceInvadersNoFrameskip-v4_1/*.zip")
Out[48]:
['logs/dqn/SpaceInvadersNoFrameskip-v4_1/best_model.zip', 'logs/dqn/SpaceInvadersNoFrameskip-v4_1/SpaceInvadersNoFrameskip-v4.zip']
In [49]:
import gymnasium as gym
import ale_py
from stable_baselines3.common.atari_wrappers import AtariWrapper
from stable_baselines3 import DQN
from gymnasium.wrappers import RecordVideo
from collections import deque
import numpy as np
import os
import glob
# 1. Initialize
gym.register_envs(ale_py)
model_path = glob.glob("logs/dqn/SpaceInvadersNoFrameskip-v4_1/*.zip")[0]
model = DQN.load(model_path)
# 2. Setup Recording Env
temp_folder = "./temp_video"
env = gym.make("SpaceInvadersNoFrameskip-v4", render_mode="rgb_array")
env = AtariWrapper(env)
env = RecordVideo(env, video_folder=temp_folder, name_prefix="eval")
# 3. Prediction Loop with Shape Correction
obs, info = env.reset()
# Initialize stack with the first frame (removing the last channel dim)
stacked_obs = deque([obs.squeeze(-1) for _ in range(4)], maxlen=4)
done = False
while not done:
# Convert deque to numpy array: shape becomes (4, 84, 84)
input_obs = np.array(stacked_obs)
# Predict action
action, _ = model.predict(input_obs, deterministic=True)
# Step environment
obs, reward, terminated, truncated, info = env.step(action)
# Remove the (1) from (84, 84, 1) before adding to stack
stacked_obs.append(obs.squeeze(-1))
done = terminated or truncated
env.close()
# 4. Shipment Move
os.makedirs("logs/videos", exist_ok=True)
generated_file = glob.glob(f"{temp_folder}/*.mp4")[0]
os.rename(generated_file, "logs/videos/replay.mp4")
print("Success! The observation shape is now exactly (4, 84, 84).")
/usr/local/lib/python3.12/dist-packages/gymnasium/wrappers/rendering.py:293: UserWarning: WARN: Overwriting existing videos at /content/temp_video folder (try specifying a different `video_folder` for the `RecordVideo` wrapper if this is not desired)
logger.warn(
Success! The observation shape is now exactly (4, 84, 84).
In [50]:
import os
print(f"Replay video exists: {os.path.exists('logs/videos/replay.mp4')}")
Replay video exists: True
In [51]:
import os
import glob
# This should now return a path!
video_files = glob.glob('logs/videos/**/*.mp4', recursive=True)
print(f"Videos found: {video_files}")
Videos found: ['logs/videos/replay.mp4']
In [52]:
# import os
# import shutil
# # Define the target folder (make sure this matches your experiment folder name)
# target_dir = "logs/dqn/SpaceInvadersNoFrameskip-v4_1/videos/"
# os.makedirs(target_dir, exist_ok=True)
# # Move the replay video
# shutil.move("logs/videos/replay.mp4", target_dir + "replay.mp4")
# print(f"Video moved to: {target_dir}")
In [53]:
from huggingface_hub import upload_file
# Define your details
HF_USERNAME = "Chiz" # Replace with your HF username
REPO_ID = f"{HF_USERNAME}/dqn-SpaceInvadersNoFrameskip-v4"
# Path to the video we successfully created earlier
local_video_path = "logs/videos/replay.mp4"
try:
upload_file(
path_or_fileobj=local_video_path,
path_in_repo="replay.mp4",
repo_id=REPO_ID,
repo_type="model"
)
print(f"Success! Video uploaded to {REPO_ID}")
except Exception as e:
print(f"Upload failed. Error: {e}")
print("Make sure you have created the repo first using the push_to_hub command (without the video flag).")
Success! Video uploaded to Chiz/dqn-SpaceInvadersNoFrameskip-v4
By using rl_zoo3.push_to_hub you evaluate, record a replay, generate a model card of your agent and push it to the hub.
This way:
- You can showcase our work 🔥
- You can visualize your agent playing 👀
- You can share with the community an agent that others can use 💾
- You can access a leaderboard 🏆 to see how well your agent is performing compared to your classmates 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard
To be able to share your model with the community there are three more steps to follow:
1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join
2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.
- Create a new token (https://huggingface.co/settings/tokens) with write role
- Copy the token
- Run the cell below and past the token
In [ ]:
from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.
notebook_login()
!git config --global credential.helper store
If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: huggingface-cli login
3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥
Let's run push_to_hub.py file to upload our trained agent to the Hub.
--repo-name : The name of the repo
-orga: Your Hugging Face username
-f: Where the trained model folder is (in our case logs)
In [20]:
!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga Chiz -f logs/
2026-01-29 16:09:57.194811: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1769702997.214144 25784 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1769702997.220077 25784 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered W0000 00:00:1769702997.235316 25784 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702997.235346 25784 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702997.235350 25784 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769702997.235353 25784 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. 2026-01-29 16:09:57.239813: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality. Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade. See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information. Loading latest experiment, id=1 Loading logs/dqn/SpaceInvadersNoFrameskip-v4_1/SpaceInvadersNoFrameskip-v4.zip A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138) [Powered by Stella] Stacking 4 frames Wrapping the env in a VecTransposeImage. Uploading to Chiz/dqn-SpaceInvadersNoFrameskip-v4, make sure to have the rights ℹ This function will save, evaluate, generate a video of your agent, create a model card and push everything to the hub. It might take up to some minutes if video generation is activated. This is a work in progress: if you encounter a bug, please open an issue. Fetching 14 files: 0% 0/14 [00:00<?, ?it/s] dqn-SpaceInvadersNoFrameskip-v4.zip: 0% 0.00/27.2M [00:00<?, ?B/s] data: 205kB [00:00, 220MB/s] Fetching 14 files: 7% 1/14 [00:00<00:05, 2.52it/s] args.yml: 1.27kB [00:00, 6.40MB/s] README.md: 2.79kB [00:00, 17.9MB/s] dqn-SpaceInvadersNoFrameskip-v4/policy.o(…): 0% 0.00/13.5M [00:00<?, ?B/s] config.yml: 100% 549/549 [00:00<00:00, 4.74MB/s] _stable_baselines3_version: 100% 7.00/7.00 [00:00<00:00, 79.8kB/s] dqn-SpaceInvadersNoFrameskip-v4/policy.p(…): 0% 0.00/13.5M [00:00<?, ?B/s] train_eval_metrics.zip: 0% 0.00/35.6k [00:00<?, ?B/s] results.json: 100% 152/152 [00:00<00:00, 1.17MB/s] system_info.txt: 100% 246/246 [00:00<00:00, 2.83MB/s] env_kwargs.yml: 100% 23.0/23.0 [00:00<00:00, 238kB/s] train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<00:00, 60.0kB/s] dqn-SpaceInvadersNoFrameskip-v4/policy.p(…): 100% 13.5M/13.5M [00:01<00:00, 10.2MB/s] dqn-SpaceInvadersNoFrameskip-v4/policy.o(…): 100% 13.5M/13.5M [00:01<00:00, 8.73MB/s] dqn-SpaceInvadersNoFrameskip-v4.zip: 100% 27.2M/27.2M [00:01<00:00, 14.7MB/s] Fetching 14 files: 100% 14/14 [00:02<00:00, 6.63it/s] Saving model to: hub/dqn-SpaceInvadersNoFrameskip-v4/dqn-SpaceInvadersNoFrameskip-v4 Saving video to /tmp/tmpw0yytxq0/-step-0-to-step-1000.mp4 Moviepy - Building video /tmp/tmpw0yytxq0/-step-0-to-step-1000.mp4. Moviepy - Writing video /tmp/tmpw0yytxq0/-step-0-to-step-1000.mp4 Moviepy - Done ! Moviepy - video ready /tmp/tmpw0yytxq0/-step-0-to-step-1000.mp4 ✘ 'DummyVecEnv' object has no attribute 'video_recorder' ✘ We are unable to generate a replay of your agent, the package_to_hub process continues ✘ Please open an issue at https://github.com/huggingface/huggingface_sb3/issues ℹ Pushing repo dqn-SpaceInvadersNoFrameskip-v4 to the Hugging Face Hub Processing Files (0 / 0) : | | 0.00B / 0.00B New Data Upload : | | 0.00B / 0.00B ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...nvadersNoFrameskip-v4.zip: 61% 16.6M/27.2M [00:00<?, ?B/s] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] Processing Files (4 / 5) : 80% 43.6M/54.3M [00:00<00:00, 212MB/s, ???B/s ] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] Processing Files (4 / 5) : 100% 54.1M/54.3M [00:00<00:00, 120MB/s, 52.8MB/s ] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...nvadersNoFrameskip-v4.zip: 99% 27.1M/27.2M [00:00<00:00, 26.5MB/s] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] Processing Files (5 / 5) : 100% 54.3M/54.3M [00:00<00:00, 45.2MB/s, 17.8MB/s ] New Data Upload : 100% 151k/151k [00:00<00:00, 188kB/s, 252kB/s ] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...nvadersNoFrameskip-v4.zip: 100% 27.2M/27.2M [00:00<00:00, 17.8MB/s] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] Processing Files (5 / 5) : 100% 54.3M/54.3M [00:01<00:00, 54.1MB/s, 13.4MB/s ] New Data Upload : 100% 151k/151k [00:01<00:00, 150kB/s, 189kB/s ] ...v4/train_eval_metrics.zip: 100% 35.6k/35.6k [00:00<?, ?B/s] ...-v4/pytorch_variables.pth: 100% 1.26k/1.26k [00:00<?, ?B/s] ...NoFrameskip-v4/policy.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...p-v4/policy.optimizer.pth: 100% 13.5M/13.5M [00:00<?, ?B/s] ...nvadersNoFrameskip-v4.zip: 100% 27.2M/27.2M [00:00<00:00, 13.3MB/s] ℹ Your model is pushed to the hub. You can view your model here: https://huggingface.co/Chiz/dqn-SpaceInvadersNoFrameskip-v4
Solution¶
In [ ]:
!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga ThomasSimonini -f logs/
.¶
Congrats 🥳 you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:
- See a video preview of your agent at the right.
- Click "Files and versions" to see all the files in the repository.
- Click "Use in stable-baselines3" to get a code snippet that shows how to load the model.
- A model card (
README.mdfile) which gives a description of the model and the hyperparameters you used.
Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.
Compare the results of your agents with your classmates using the leaderboard 🏆
Load a powerful trained model 🔥¶
- The Stable-Baselines3 team uploaded more than 150 trained Deep Reinforcement Learning agents on the Hub.
You can find them here: 👉 https://huggingface.co/sb3
Some examples:
- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4
- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4
- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4
- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4
Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4
In [21]:
%%html
<video controls autoplay><source src="https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4/resolve/main/replay.mp4" type="video/mp4"></video>
- We download the model using
rl_zoo3.load_from_hub, and place it in a new folder that we can callrl_trained
In [22]:
# Download model and save it into the logs/ folder
!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/
2026-01-29 16:12:28.918324: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1769703148.937981 26477 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1769703148.943987 26477 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered W0000 00:00:1769703148.959199 26477 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703148.959224 26477 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703148.959228 26477 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703148.959231 26477 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. 2026-01-29 16:12:28.963837: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality. Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade. See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information. Downloading from https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4 dqn-BeamRiderNoFrameskip-v4.zip: 100% 27.2M/27.2M [00:01<00:00, 23.1MB/s] config.yml: 100% 548/548 [00:00<00:00, 5.14MB/s] No normalization file args.yml: 100% 887/887 [00:00<00:00, 9.47MB/s] env_kwargs.yml: 100% 3.00/3.00 [00:00<00:00, 31.3kB/s] train_eval_metrics.zip: 100% 244k/244k [00:00<00:00, 906kB/s] Saving to rl_trained/dqn/BeamRiderNoFrameskip-v4_1
- Let's evaluate if for 5000 timesteps
In [23]:
!python -m rl_zoo3.enjoy --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000 -f rl_trained/ --no-render
2026-01-29 16:12:41.256578: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1769703161.275752 26549 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered E0000 00:00:1769703161.281815 26549 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered W0000 00:00:1769703161.296256 26549 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703161.296284 26549 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703161.296288 26549 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. W0000 00:00:1769703161.296293 26549 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once. 2026-01-29 16:12:41.300665: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality. Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade. See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information. Loading latest experiment, id=1 Loading rl_trained/dqn/BeamRiderNoFrameskip-v4_1/BeamRiderNoFrameskip-v4.zip A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138) [Powered by Stella] Stacking 4 frames /usr/local/lib/python3.12/dist-packages/stable_baselines3/common/save_util.py:167: UserWarning: Could not deserialize object exploration_schedule. Consider using `custom_objects` argument to replace this object. Exception: code expected at least 16 arguments, got 15 warnings.warn( /usr/local/lib/python3.12/dist-packages/stable_baselines3/common/vec_env/patch_gym.py:95: UserWarning: You loaded a model that was trained using OpenAI Gym. We strongly recommend transitioning to Gymnasium by saving that model again. warnings.warn( /usr/local/lib/python3.12/dist-packages/stable_baselines3/common/base_class.py:773: UserWarning: You are probably loading a DQN model saved with SB3 < 2.4.0, we truncated the optimizer state so you can save the model again to avoid issues in the future (see https://github.com/DLR-RM/stable-baselines3/pull/1963 for more info). Original error: loaded state dict contains a parameter group that doesn't match the size of optimizer's group Note: the model should still work fine, this only a warning. warnings.warn( Atari Episode Score: 3028.00 Atari Episode Length 14816
Why not trying to train your own Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? 🏆.
If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters in the model card, you have the hyperparameters of the trained agent.
But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can use Optuna for optimizing the Hyperparameters 🔥.
Some additional challenges 🏆¶
The best way to learn is to try things by your own!
In the Leaderboard you will find your agents. Can you get to the top?
Here's a list of environments you can try to train your agent with:
- BeamRiderNoFrameskip-v4
- BreakoutNoFrameskip-v4
- EnduroNoFrameskip-v4
- PongNoFrameskip-v4
Also, if you want to learn to implement Deep Q-Learning by yourself, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py
Congrats on finishing this chapter!
If you’re still feel confused with all these elements...it's totally normal! This was the same for me and for all people who studied RL.
Take time to really grasp the material before continuing and try the additional challenges. It’s important to master these elements and having a solid foundations.
In the next unit, we’re going to learn about Optuna. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.
This is a course built with you 👷🏿♀️¶
Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9
We're constantly trying to improve our tutorials, so if you find some issues in this notebook, please open an issue on the Github Repo.
See you on Bonus unit 2! 🔥
Keep Learning, Stay Awesome 🤗¶
In [ ]: