predpreygrass.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "Ovnzh88NsJ2u",
      "metadata": {
        "id": "Ovnzh88NsJ2u"
      },
      "source": [
        "# Predator-Prey-Grass MARL\n",
        "\n",
        "A Predator-Prey-Grass multi-agent gridworld environment. Featuring dynamic spawning and deletion and partial observability of agents.\n",
        "\n",
        "https://github.com/doesburg11/PredPreyGrass\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Environment dynamics\n",
        "Learning agents Predators (red) and Prey (blue) both sequentially expend energy moving around, and replenish it by eating. Prey eat Grass (green), and Predators eat Prey if they end up on the same grid cell. The agents obtain all the energy from the eaten resource. Predators die of starvation when their energy is run out, Prey die either of starvation or when being eaten by a Predator. Both learning agents asexually reproduce when energy levels exceed a certain threshold (by eating). In the base configuration, newly created agents are placed at random over the entire gridworld. Learning agents learn to move based on their partial observations (transparent red and blue squares) of the environment."
      ],
      "metadata": {
        "id": "F_IqaQow-n8O"
      },
      "id": "F_IqaQow-n8O"
    },
    {
      "cell_type": "markdown",
      "id": "ab7f2ffc",
      "metadata": {
        "id": "ab7f2ffc"
      },
      "source": [
        "## Step 1: Clone the PredPreyGrass repository from GitHub"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YxckAE2D604P",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YxckAE2D604P",
        "outputId": "2108a6bd-ad19-45ee-c2a0-31749661b8e6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'PredPreyGrass'...\n",
            "remote: Enumerating objects: 20895, done.\u001b[K\n",
            "remote: Counting objects: 100% (726/726), done.\u001b[K\n",
            "remote: Compressing objects: 100% (205/205), done.\u001b[K\n",
            "remote: Total 20895 (delta 668), reused 541 (delta 520), pack-reused 20169 (from 2)\u001b[K\n",
            "Receiving objects: 100% (20895/20895), 523.69 MiB | 15.19 MiB/s, done.\n",
            "Resolving deltas: 100% (12836/12836), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/doesburg11/PredPreyGrass.git # > /dev/null 2>&1"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ijd03JF48uCL",
      "metadata": {
        "id": "ijd03JF48uCL"
      },
      "source": [
        "## Step 2: Install the PredPreyGrass package"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "846b9364",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "846b9364",
        "outputId": "5eab4923-8ada-4dc3-fb30-fc3163f179a7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/PredPreyGrass\n"
          ]
        }
      ],
      "source": [
        "%cd PredPreyGrass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ILYF_q_11qFn",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ILYF_q_11qFn",
        "outputId": "c4e7b73c-1991-4230-8afe-4881536ea95c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Obtaining file:///content/PredPreyGrass\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pettingzoo (from predpreygrass==0.1)\n",
            "  Downloading pettingzoo-1.24.3-py3-none-any.whl.metadata (8.5 kB)\n",
            "Collecting stable_baselines3 (from predpreygrass==0.1)\n",
            "  Downloading stable_baselines3-2.5.0-py3-none-any.whl.metadata (4.8 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.11/dist-packages (from predpreygrass==0.1) (1.26.4)\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.11/dist-packages (from predpreygrass==0.1) (2.6.1)\n",
            "Requirement already satisfied: gymnasium>=0.28.0 in /usr/local/lib/python3.11/dist-packages (from pettingzoo->predpreygrass==0.1) (1.1.1)\n",
            "Collecting gymnasium>=0.28.0 (from pettingzoo->predpreygrass==0.1)\n",
            "  Downloading gymnasium-1.0.0-py3-none-any.whl.metadata (9.5 kB)\n",
            "Requirement already satisfied: torch<3.0,>=2.3 in /usr/local/lib/python3.11/dist-packages (from stable_baselines3->predpreygrass==0.1) (2.6.0+cu124)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.11/dist-packages (from stable_baselines3->predpreygrass==0.1) (3.1.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from stable_baselines3->predpreygrass==0.1) (2.2.2)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.11/dist-packages (from stable_baselines3->predpreygrass==0.1) (3.10.0)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium>=0.28.0->pettingzoo->predpreygrass==0.1) (4.12.2)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.11/dist-packages (from gymnasium>=0.28.0->pettingzoo->predpreygrass==0.1) (0.0.4)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (3.17.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (3.1.6)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (2024.10.0)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.4.127 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.4.127 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.4.127 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cublas-cu12==12.4.5.8 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cufft-cu12==11.2.1.3 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-curand-cu12==10.3.5.147 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cusolver-cu12==11.6.1.9 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusparse-cu12==12.3.1.170 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (0.6.2)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (2.21.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (12.4.127)\n",
            "Collecting nvidia-nvjitlink-cu12==12.4.127 (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1)\n",
            "  Downloading nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Requirement already satisfied: triton==3.2.0 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (3.2.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from sympy==1.13.1->torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (1.3.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (4.56.0)\n",
            "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (1.4.8)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (24.2)\n",
            "Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (11.1.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (3.2.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable_baselines3->predpreygrass==0.1) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->stable_baselines3->predpreygrass==0.1) (2025.1)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->stable_baselines3->predpreygrass==0.1) (2025.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.7->matplotlib->stable_baselines3->predpreygrass==0.1) (1.17.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/dist-packages (from jinja2->torch<3.0,>=2.3->stable_baselines3->predpreygrass==0.1) (3.0.2)\n",
            "Downloading pettingzoo-1.24.3-py3-none-any.whl (847 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m847.8/847.8 kB\u001b[0m \u001b[31m58.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading stable_baselines3-2.5.0-py3-none-any.whl (183 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.9/183.9 kB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gymnasium-1.0.0-py3-none-any.whl (958 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m958.1/958.1 kB\u001b[0m \u001b[31m66.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl (363.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (13.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m114.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (24.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.6/24.6 MB\u001b[0m \u001b[31m85.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (883 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 kB\u001b[0m \u001b[31m58.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl (211.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl (56.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl (127.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl (207.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m92.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: nvidia-nvjitlink-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, gymnasium, pettingzoo, nvidia-cusparse-cu12, nvidia-cudnn-cu12, nvidia-cusolver-cu12, stable_baselines3, predpreygrass\n",
            "  Attempting uninstall: nvidia-nvjitlink-cu12\n",
            "    Found existing installation: nvidia-nvjitlink-cu12 12.5.82\n",
            "    Uninstalling nvidia-nvjitlink-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-nvjitlink-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-curand-cu12\n",
            "    Found existing installation: nvidia-curand-cu12 10.3.6.82\n",
            "    Uninstalling nvidia-curand-cu12-10.3.6.82:\n",
            "      Successfully uninstalled nvidia-curand-cu12-10.3.6.82\n",
            "  Attempting uninstall: nvidia-cufft-cu12\n",
            "    Found existing installation: nvidia-cufft-cu12 11.2.3.61\n",
            "    Uninstalling nvidia-cufft-cu12-11.2.3.61:\n",
            "      Successfully uninstalled nvidia-cufft-cu12-11.2.3.61\n",
            "  Attempting uninstall: nvidia-cuda-runtime-cu12\n",
            "    Found existing installation: nvidia-cuda-runtime-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-runtime-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-runtime-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cuda-nvrtc-cu12\n",
            "    Found existing installation: nvidia-cuda-nvrtc-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-nvrtc-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-nvrtc-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cuda-cupti-cu12\n",
            "    Found existing installation: nvidia-cuda-cupti-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-cupti-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-cupti-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cublas-cu12\n",
            "    Found existing installation: nvidia-cublas-cu12 12.5.3.2\n",
            "    Uninstalling nvidia-cublas-cu12-12.5.3.2:\n",
            "      Successfully uninstalled nvidia-cublas-cu12-12.5.3.2\n",
            "  Attempting uninstall: gymnasium\n",
            "    Found existing installation: gymnasium 1.1.1\n",
            "    Uninstalling gymnasium-1.1.1:\n",
            "      Successfully uninstalled gymnasium-1.1.1\n",
            "  Attempting uninstall: nvidia-cusparse-cu12\n",
            "    Found existing installation: nvidia-cusparse-cu12 12.5.1.3\n",
            "    Uninstalling nvidia-cusparse-cu12-12.5.1.3:\n",
            "      Successfully uninstalled nvidia-cusparse-cu12-12.5.1.3\n",
            "  Attempting uninstall: nvidia-cudnn-cu12\n",
            "    Found existing installation: nvidia-cudnn-cu12 9.3.0.75\n",
            "    Uninstalling nvidia-cudnn-cu12-9.3.0.75:\n",
            "      Successfully uninstalled nvidia-cudnn-cu12-9.3.0.75\n",
            "  Attempting uninstall: nvidia-cusolver-cu12\n",
            "    Found existing installation: nvidia-cusolver-cu12 11.6.3.83\n",
            "    Uninstalling nvidia-cusolver-cu12-11.6.3.83:\n",
            "      Successfully uninstalled nvidia-cusolver-cu12-11.6.3.83\n",
            "  Running setup.py develop for predpreygrass\n",
            "Successfully installed gymnasium-1.0.0 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-nvjitlink-cu12-12.4.127 pettingzoo-1.24.3 predpreygrass-0.1 stable_baselines3-2.5.0\n"
          ]
        }
      ],
      "source": [
        "!pip install -e . # > /dev/null 2>&1"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1cl0jQi38asS",
      "metadata": {
        "id": "1cl0jQi38asS"
      },
      "source": [
        "## Step 3: Install additional requirements"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YT-fCJUcAfdR",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YT-fCJUcAfdR",
        "outputId": "2521fa78-ff7c-4e91-9dfb-997eff196841"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting supersuit==3.9.3\n",
            "  Downloading SuperSuit-3.9.3-py3-none-any.whl.metadata (3.2 kB)\n",
            "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.11/dist-packages (from supersuit==3.9.3) (1.26.4)\n",
            "Requirement already satisfied: gymnasium>=0.28.1 in /usr/local/lib/python3.11/dist-packages (from supersuit==3.9.3) (1.0.0)\n",
            "Collecting tinyscaler>=1.2.6 (from supersuit==3.9.3)\n",
            "  Downloading tinyscaler-1.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium>=0.28.1->supersuit==3.9.3) (3.1.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium>=0.28.1->supersuit==3.9.3) (4.12.2)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.11/dist-packages (from gymnasium>=0.28.1->supersuit==3.9.3) (0.0.4)\n",
            "Downloading SuperSuit-3.9.3-py3-none-any.whl (50 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.2/50.2 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tinyscaler-1.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (563 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m563.3/563.3 kB\u001b[0m \u001b[31m40.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: tinyscaler, supersuit\n",
            "Successfully installed supersuit-3.9.3 tinyscaler-1.2.8\n",
            "Requirement already satisfied: tensorboard==2.18.0 in /usr/local/lib/python3.11/dist-packages (2.18.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (1.71.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (3.7)\n",
            "Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (1.26.4)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (24.2)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (4.25.6)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (1.17.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard==2.18.0) (3.1.3)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard==2.18.0) (3.0.2)\n",
            "Requirement already satisfied: stable-baselines3[extra] in /usr/local/lib/python3.11/dist-packages (2.5.0)\n",
            "Requirement already satisfied: gymnasium<1.1.0,>=0.29.1 in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (1.0.0)\n",
            "Requirement already satisfied: numpy<3.0,>=1.20 in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (1.26.4)\n",
            "Requirement already satisfied: torch<3.0,>=2.3 in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (2.6.0+cu124)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (3.1.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (2.2.2)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (3.10.0)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (4.11.0.86)\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (2.6.1)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (2.18.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (5.9.5)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (4.67.1)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (13.9.4)\n",
            "Requirement already satisfied: ale-py>=0.9.0 in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (0.10.2)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.11/dist-packages (from stable-baselines3[extra]) (11.1.0)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium<1.1.0,>=0.29.1->stable-baselines3[extra]) (4.12.2)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.11/dist-packages (from gymnasium<1.1.0,>=0.29.1->stable-baselines3[extra]) (0.0.4)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.71.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.7)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (24.2)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (4.25.6)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.17.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.1.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (3.17.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (3.1.6)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (2024.10.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.127)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.127)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.127)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (9.1.0.70)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.5.8)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (11.2.1.3)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (10.3.5.147)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (11.6.1.9)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.3.1.170)\n",
            "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (0.6.2)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (2.21.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.127)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (12.4.127)\n",
            "Requirement already satisfied: triton==3.2.0 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (3.2.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.11/dist-packages (from torch<3.0,>=2.3->stable-baselines3[extra]) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from sympy==1.13.1->torch<3.0,>=2.3->stable-baselines3[extra]) (1.3.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (4.56.0)\n",
            "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (1.4.8)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (3.2.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.11/dist-packages (from matplotlib->stable-baselines3[extra]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->stable-baselines3[extra]) (2025.1)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->stable-baselines3[extra]) (2025.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/dist-packages (from rich->stable-baselines3[extra]) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from rich->stable-baselines3[extra]) (2.18.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/dist-packages (from markdown-it-py>=2.2.0->rich->stable-baselines3[extra]) (0.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]) (3.0.2)\n"
          ]
        }
      ],
      "source": [
        "!pip install supersuit==3.9.3 # >  /dev/null 2>&1\n",
        "!pip install tensorboard==2.18.0 # > /dev/null 2>&1\n",
        "!pip install stable-baselines3[extra] # > /dev/null 2>&1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zbryBHjp46qa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zbryBHjp46qa",
        "outputId": "97e162c7-c7da-4398-fa0e-3a24fd32c6c6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "env: XDG_RUNTIME_DIR=/tmp/runtime-dir\n"
          ]
        }
      ],
      "source": [
        "%env XDG_RUNTIME_DIR=/tmp/runtime-dir\n",
        "!mkdir -p /tmp/runtime-dir"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Predator-Prey-Grass MARL with SB3 PPO\n",
        "The MARL environment is implemented using PettingZoo, and the agents are trained using Stable Baselines3 (SB3) PPO. Essentially this solution demonstrates how SB3 can be adapted for MARL using parallel environments and centralized training."
      ],
      "metadata": {
        "id": "IHiEhncr_QZB"
      },
      "id": "IHiEhncr_QZB"
    },
    {
      "cell_type": "markdown",
      "id": "gFh0qUmP9ttl",
      "metadata": {
        "id": "gFh0qUmP9ttl"
      },
      "source": [
        "## Step 4: Evaluate the environment with a random policy"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "80nHZzUVi0Yg",
      "metadata": {
        "id": "80nHZzUVi0Yg"
      },
      "source": [
        "The configuration of the environment can be adjusted in:  /content/PredPreyGrass/predpreygrass/single_objective/config/config_predpreygrass.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0t-mlI7SB2jT",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0t-mlI7SB2jT",
        "outputId": "803c7400-785e-4c6a-dcae-86405812f8e3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "self.motion_range: [[-2, -2], [-2, -1], [-2, 0], [-2, 1], [-2, 2], [-1, -2], [-1, -1], [-1, 0], [-1, 1], [-1, 2], [0, -2], [0, -1], [0, 0], [0, 1], [0, 2], [1, -2], [1, -1], [1, 0], [1, 1], [1, 2], [2, -2], [2, -1], [2, 0], [2, 1], [2, 2]]\n",
            "len(self.motion_range): 25\n",
            "self.action_range: 5\n",
            "seed: 42 set in predpreygrass_aec.py seed\n",
            "Seed set to 42\n",
            "agent: prey_43, reward: 10.0\n",
            "agent: prey_44, reward: 10.0\n",
            "agent: prey_47, reward: 10.0\n",
            "agent: prey_99, reward: 10.0\n",
            "agent: prey_96, reward: 10.0\n"
          ]
        }
      ],
      "source": [
        "!python predpreygrass/pettingzoo/eval/evaluate_random_policy.py"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qbQ00o25-8C5",
      "metadata": {
        "id": "qbQ00o25-8C5"
      },
      "source": [
        "## Step 5: Train model and save to file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5YHXfKfEzHwh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5YHXfKfEzHwh",
        "outputId": "75e85a43-79c4-48f0-c141-f39e92e4a3b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2025-03-14 18:40:12.182364: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
            "E0000 00:00:1741977612.522755    4054 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "E0000 00:00:1741977612.625761    4054 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2025-03-14 18:40:13.353350: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "-----------------------------------------------------------------------------\n",
            "Destination source code: /content/PredPreyGrass/output/2025-03-14_18:40:21\n",
            "Configuration file directory: /content/PredPreyGrass/output/2025-03-14_18:40:21/config/\n",
            "-----------------------------------------------------------------------------\n",
            "Number of CPU cores utilized: 2\n",
            "self.motion_range: [[-2, -2], [-2, -1], [-2, 0], [-2, 1], [-2, 2], [-1, -2], [-1, -1], [-1, 0], [-1, 1], [-1, 2], [0, -2], [0, -1], [0, 0], [0, 1], [0, 2], [1, -2], [1, -1], [1, 0], [1, 1], [1, 2], [2, -2], [2, -1], [2, 0], [2, 1], [2, 2]]\n",
            "len(self.motion_range): 25\n",
            "self.action_range: 5\n",
            "seed: 0 set in predpreygrass_aec.py seed\n",
            "Seed set to 0\n",
            "Starting training on predpreygrass_aec_v0.\n",
            "\u001b[2KTraceback (most recent call last):\n",
            "\u001b[2K  File \n",
            "\"/content/PredPreyGrass/predpreygrass/pettingzoo/train/train_sb3_ppo_parallel_wrapped_aec_env.py\", \n",
            "line 88, in <module>\n",
            "\u001b[2K    trainer.train(is_wrapped=True)\n",
            "\u001b[2K  File \"/content/PredPreyGrass/predpreygrass/pettingzoo/train/utils/trainer.py\", line 113, in train\n",
            "\u001b[2K    self._train_model(env, total_timesteps)\n",
            "\u001b[2K  File \"/content/PredPreyGrass/predpreygrass/pettingzoo/train/utils/trainer.py\", line 74, in \n",
            "_train_model\n",
            "\u001b[2K    model.learn(\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/stable_baselines3/ppo/ppo.py\", line 311, in learn\n",
            "\u001b[2K    return super().learn(\n",
            "\u001b[2K           ^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/stable_baselines3/common/on_policy_algorithm.py\", \n",
            "line 323, in learn\n",
            "\u001b[2K    continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, \n",
            "n_rollout_steps=self.n_steps)\n",
            "\u001b[2K                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/stable_baselines3/common/on_policy_algorithm.py\", \n",
            "line 218, in collect_rollouts\n",
            "\u001b[2K    new_obs, rewards, dones, infos = env.step(clipped_actions)\n",
            "\u001b[2K                                     ^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/stable_baselines3/common/vec_env/base_vec_env.py\", \n",
            "line 207, in step\n",
            "\u001b[2K    return self.step_wait()\n",
            "\u001b[2K           ^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/supersuit/vector/sb3_vector_wrapper.py\", line 26, in\n",
            "step_wait\n",
            "\u001b[2K    observations, rewards, terminations, truncations, infos = self.venv.step_wait()\n",
            "\u001b[2K                                                              ^^^^^^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/supersuit/vector/multiproc_vec.py\", line 205, in \n",
            "step_wait\n",
            "\u001b[2K    compressed_infos = self._receive_info()\n",
            "\u001b[2K                       ^^^^^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/local/lib/python3.11/dist-packages/supersuit/vector/multiproc_vec.py\", line 196, in \n",
            "_receive_info\n",
            "\u001b[2K    data = cin.recv()\n",
            "\u001b[2K           ^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 250, in recv\n",
            "\u001b[2K    buf = self._recv_bytes()\n",
            "\u001b[2K          ^^^^^^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 430, in _recv_bytes\n",
            "\u001b[2K    buf = self._recv(4)\n",
            "\u001b[2K          ^^^^^^^^^^^^^\n",
            "\u001b[2K  File \"/usr/lib/python3.11/multiprocessing/connection.py\", line 395, in _recv\n",
            "\u001b[2K    chunk = read(handle, remaining)\n",
            "\u001b[2K            ^^^^^^^^^^^^^^^^^^^^^^^\n",
            "\u001b[2KKeyboardInterrupt\n",
            "\u001b[2K\u001b[35m   3%\u001b[0m \u001b[91m━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m128,600/4,096,000 \u001b[0m [ \u001b[33m0:01:19\u001b[0m < \u001b[36m0:40:11\u001b[0m , \u001b[31m1,646 it/s\u001b[0m ]Exception ignored in: <function tqdm.__del__ at 0x7aaba9875c60>\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/tqdm/std.py\", line 1148, in __del__\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/tqdm/rich.py\", line 121, in close\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/progress.py\", line 1191, in __exit__\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/progress.py\", line 1177, in stop\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/live.py\", line 155, in stop\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/console.py\", line 1131, in line\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/console.py\", line 1679, in print\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/console.py\", line 1536, in _collect_renderables\n",
            "  File \"/usr/local/lib/python3.11/dist-packages/rich/protocol.py\", line 28, in rich_cast\n",
            "ImportError: sys.meta_path is None, Python is likely shutting down\n"
          ]
        }
      ],
      "source": [
        "!python predpreygrass/pettingzoo/train/train_sb3_ppo_parallel_wrapped_aec_env.py"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Q4gpfPJzrLS3",
      "metadata": {
        "id": "Q4gpfPJzrLS3"
      },
      "source": [
        "## Step 6: Evaluation trained model from file"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "E-3xTQrArh74",
      "metadata": {
        "id": "E-3xTQrArh74"
      },
      "source": [
        "The trained model is now saved to a zip file on Colab, along with the file structure and configuration. To evaluate this model and the asociated configuration, manually fill in the correct timestamp below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "opdq0PDXsA9w",
      "metadata": {
        "id": "opdq0PDXsA9w"
      },
      "outputs": [],
      "source": [
        "# adjust time stamp accordingly\n",
        "timestamp=\"2025-03-16_18:38:46\"\n",
        "evaluation_script = \"/content/PredPreyGrass/output/\"+timestamp+\"/eval/evaluate_ppo_from_file_aec_env.py\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5PgtBezKswLx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5PgtBezKswLx",
        "outputId": "14965546-b39f-4339-de44-818553c9c2d8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2025-01-05 19:31:08.033478: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2025-01-05 19:31:08.082660: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2025-01-05 19:31:08.098206: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2025-01-05 19:31:08.138313: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2025-01-05 19:31:10.913195: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "-----------------------------------------------------------------------------\n",
            "Evaluation_directory:  /content/PredPreyGrass/output/2025-01-05_18:38:46/eval\n",
            "Destination_source_code_dir:  /content/PredPreyGrass/output/2025-01-05_18:38:46\n",
            "-----------------------------------------------------------------------------\n",
            "Start evaluation on: /content/PredPreyGrass/output/2025-01-05_18:38:46\n",
            "-----------------------------------------------------------------------------\n",
            "Evaluation results:\n",
            "-----------------------------------------------------------------------------\n",
            "Date and Time: 2025-01-05_19:31:13\n",
            "Environment: predpreygrass_aec_v0\n",
            "Grid transformation: bounded\n",
            "Learning algorithm: PPO\n",
            "loaded_policy: /content/PredPreyGrass/output/2025-01-05_18:38:46/output/predpreygrass_aec_v0_steps_11_468_800\n",
            "Training steps: 11_468_800\n",
            "Evaluation directory: /content/PredPreyGrass/output/2025-01-05_18:38:46\n",
            "-----------------------------------------------------------------------------\n",
            "\n",
            "Eps 0 Lngth = 10000 Strv Prd/cycl = 0.031 Strv Pry/cycl = 0.163 Eatn Pry/cycl = 0.276 Eatn Gra/cycl = 1.502 Brn Prd/cycl = 0.03 Brn Pry/cycle = 0.442 Mn age Prd = 183.4 Mn age Pry = 84.6\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"/content/PredPreyGrass/output/2025-01-05_18:38:46/eval/evaluate_ppo_from_file_aec_env.py\", line 61, in <module>\n",
            "    evaluator.parallel_wrapped_aec_env_training_aec_evaluation()\n",
            "  File \"/content/PredPreyGrass/output/2025-01-05_18:38:46/eval/utils/evaluator.py\", line 709, in parallel_wrapped_aec_env_training_aec_evaluation\n",
            "    env.step(action)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/pettingzoo/utils/wrappers/order_enforcing.py\", line 96, in step\n",
            "    super().step(action)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/pettingzoo/utils/wrappers/base.py\", line 47, in step\n",
            "    self.env.step(action)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/pettingzoo/utils/wrappers/assert_out_of_bounds.py\", line 26, in step\n",
            "    super().step(action)\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/pettingzoo/utils/wrappers/base.py\", line 47, in step\n",
            "    self.env.step(action)\n",
            "  File \"/content/PredPreyGrass/predpreygrass/single_objective/envs/base_env/predpreygrass_aec.py\", line 61, in step\n",
            "    def step(self, action):\n",
            "KeyboardInterrupt\n"
          ]
        }
      ],
      "source": [
        "!python {evaluation_script}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KFSR462e9i4R",
      "metadata": {
        "id": "KFSR462e9i4R"
      },
      "source": [
        "## Step 7: Save project to Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "j-A3FSQ5tiEw",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j-A3FSQ5tiEw",
        "outputId": "fd0ef4d6-4c13-4243-96b9-162226175ced"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ED-Zlj5E-UYb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ED-Zlj5E-UYb",
        "outputId": "de6d9fe1-abaf-43b5-cbc5-02b0c70b4504"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Directory copied to /content/drive/My Drive/PredPreyGrass\n"
          ]
        }
      ],
      "source": [
        "import shutil\n",
        "\n",
        "# Source directory in Colab\n",
        "source_dir = '/content/PredPreyGrass'\n",
        "\n",
        "# Destination directory in Google Drive\n",
        "destination_dir = '/content/drive/My Drive/PredPreyGrass'\n",
        "\n",
        "# Copy the directory\n",
        "shutil.copytree(source_dir, destination_dir)\n",
        "\n",
        "print(f\"Directory copied to {destination_dir}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Uk7xaoALu9U0",
      "metadata": {
        "id": "Uk7xaoALu9U0"
      },
      "source": [
        "## Step 8: Display population developments per episode\n",
        "\n",
        "The population developments per episode are saved to google drive into pdf's:\n",
        "\n",
        "\"/content/drive/MyDrive/PredPreyGrass/output/\"{timestamp}\"/output/population_charts/PredPreyPopulation_episode_{episode number}.pdf"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The RLlib solution with decentralized traing"
      ],
      "metadata": {
        "id": "lX_rmkd9Ahqa"
      },
      "id": "lX_rmkd9Ahqa"
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install ray[rllib]==2.43.0"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jwMiXKnOBtIV",
        "outputId": "9fc52b08-7426-4c81-d0f4-2155271e6fdb"
      },
      "id": "jwMiXKnOBtIV",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting ray==2.43.0 (from ray[rllib]==2.43.0)\n",
            "  Downloading ray-2.43.0-cp311-cp311-manylinux2014_x86_64.whl.metadata (19 kB)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (8.1.8)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (3.17.0)\n",
            "Requirement already satisfied: jsonschema in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (4.23.0)\n",
            "Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (1.1.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (24.2)\n",
            "Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (4.25.6)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (6.0.2)\n",
            "Requirement already satisfied: aiosignal in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (1.3.2)\n",
            "Requirement already satisfied: frozenlist in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (1.5.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.11/dist-packages (from ray==2.43.0->ray[rllib]==2.43.0) (2.32.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (2.2.2)\n",
            "Collecting tensorboardX>=1.9 (from ray[rllib]==2.43.0)\n",
            "  Downloading tensorboardX-2.6.2.2-py2.py3-none-any.whl.metadata (5.8 kB)\n",
            "Requirement already satisfied: pyarrow>=9.0.0 in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (18.1.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (2024.10.0)\n",
            "Requirement already satisfied: dm_tree in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (0.1.9)\n",
            "Requirement already satisfied: gymnasium==1.0.0 in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (1.0.0)\n",
            "Collecting lz4 (from ray[rllib]==2.43.0)\n",
            "  Downloading lz4-4.4.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
            "Collecting ormsgpack==1.7.0 (from ray[rllib]==2.43.0)\n",
            "  Downloading ormsgpack-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.5/43.5 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.11/dist-packages (from ray[rllib]==2.43.0) (1.14.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium==1.0.0->ray[rllib]==2.43.0) (1.26.4)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium==1.0.0->ray[rllib]==2.43.0) (3.1.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.11/dist-packages (from gymnasium==1.0.0->ray[rllib]==2.43.0) (4.12.2)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.11/dist-packages (from gymnasium==1.0.0->ray[rllib]==2.43.0) (0.0.4)\n",
            "Requirement already satisfied: absl-py>=0.6.1 in /usr/local/lib/python3.11/dist-packages (from dm_tree->ray[rllib]==2.43.0) (1.4.0)\n",
            "Requirement already satisfied: attrs>=18.2.0 in /usr/local/lib/python3.11/dist-packages (from dm_tree->ray[rllib]==2.43.0) (25.2.0)\n",
            "Requirement already satisfied: wrapt>=1.11.2 in /usr/local/lib/python3.11/dist-packages (from dm_tree->ray[rllib]==2.43.0) (1.17.2)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.11/dist-packages (from jsonschema->ray==2.43.0->ray[rllib]==2.43.0) (2024.10.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.11/dist-packages (from jsonschema->ray==2.43.0->ray[rllib]==2.43.0) (0.36.2)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.11/dist-packages (from jsonschema->ray==2.43.0->ray[rllib]==2.43.0) (0.23.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas->ray[rllib]==2.43.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->ray[rllib]==2.43.0) (2025.1)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->ray[rllib]==2.43.0) (2025.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests->ray==2.43.0->ray[rllib]==2.43.0) (3.4.1)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests->ray==2.43.0->ray[rllib]==2.43.0) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests->ray==2.43.0->ray[rllib]==2.43.0) (2.3.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests->ray==2.43.0->ray[rllib]==2.43.0) (2025.1.31)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->pandas->ray[rllib]==2.43.0) (1.17.0)\n",
            "Downloading ray-2.43.0-cp311-cp311-manylinux2014_x86_64.whl (67.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.7/67.7 MB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ormsgpack-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (220 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m220.6/220.6 kB\u001b[0m \u001b[31m20.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tensorboardX-2.6.2.2-py2.py3-none-any.whl (101 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m101.7/101.7 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lz4-4.4.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m64.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: tensorboardX, ormsgpack, lz4, ray\n",
            "Successfully installed lz4-4.4.3 ormsgpack-1.7.0 ray-2.43.0 tensorboardX-2.6.2.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!python predpreygrass/rllib/train.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G5WNcpuLSpDU",
        "outputId": "ab3be8fd-0b03-4c6b-9296-512102d57168"
      },
      "id": "G5WNcpuLSpDU",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
            "E0000 00:00:1741984979.499294   17298 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "E0000 00:00:1741984979.505578   17298 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2025-03-14 20:43:09,403\tINFO worker.py:1841 -- Started a local Ray instance.\n",
            "2025-03-14 20:43:11,602\tWARNING deprecation.py:50 -- DeprecationWarning: `AlgorithmConfig.rl_module(model_config_dict=..)` has been deprecated. Use `AlgorithmConfig.rl_module(model_config=..)` instead. This will raise an error in the future!\n",
            "╭────────────────────────────────────────────────────────────╮\n",
            "│ Configuration for experiment     PPO_2025-03-14_20-43-11   │\n",
            "├────────────────────────────────────────────────────────────┤\n",
            "│ Search algorithm                 BasicVariantGenerator     │\n",
            "│ Scheduler                        FIFOScheduler             │\n",
            "│ Number of trials                 1                         │\n",
            "╰────────────────────────────────────────────────────────────╯\n",
            "\n",
            "View detailed results here: /root/ray_results/PPO_2025-03-14_20-43-11\n",
            "To visualize your results with TensorBoard, run: `tensorboard --logdir /tmp/ray/session_2025-03-14_20-43-06_846913_17298/artifacts/2025-03-14_20-43-11/PPO_2025-03-14_20-43-11/driver_artifacts`\n",
            "2025-03-14 20:43:11,725\tWARNING algorithm_config.py:4703 -- You are running PPO on the new API stack! This is the new default behavior for this algorithm. If you don't want to use the new API stack, set `config.api_stack(enable_rl_module_and_learner=False,enable_env_runner_and_connector_v2=False)`. For a detailed migration guide, see here: https://docs.ray.io/en/master/rllib/new-api-stack-migration-guide.html\n",
            "\n",
            "Trial status: 1 PENDING\n",
            "Current time: 2025-03-14 20:43:11. Total running time: 0s\n",
            "Logical resource usage: 0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   PENDING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "\u001b[36m(pid=17615)\u001b[0m WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
            "\u001b[36m(pid=17615)\u001b[0m E0000 00:00:1741984993.411272   17615 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "\u001b[36m(pid=17615)\u001b[0m E0000 00:00:1741984993.417744   17615 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "\u001b[36m(PPO pid=17615)\u001b[0m 2025-03-14 20:43:20,657\tWARNING algorithm_config.py:4703 -- You are running PPO on the new API stack! This is the new default behavior for this algorithm. If you don't want to use the new API stack, set `config.api_stack(enable_rl_module_and_learner=False,enable_env_runner_and_connector_v2=False)`. For a detailed migration guide, see here: https://docs.ray.io/en/master/rllib/new-api-stack-migration-guide.html\n",
            "\u001b[36m(pid=17724)\u001b[0m WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
            "\u001b[36m(pid=17724)\u001b[0m E0000 00:00:1741985006.441517   17724 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "\u001b[36m(pid=17725)\u001b[0m E0000 00:00:1741985006.385767   17725 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "Trial status: 1 PENDING\n",
            "Current time: 2025-03-14 20:43:41. Total running time: 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   PENDING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17725)\u001b[0m 2025-03-14 20:43:57,235\tWARNING deprecation.py:50 -- DeprecationWarning: `RLModule(config=[RLModuleConfig object])` has been deprecated. Use `RLModule(observation_space=.., action_space=.., inference_only=.., model_config=.., catalog_class=..)` instead. This will raise an error in the future!\n",
            "\u001b[36m(pid=17800)\u001b[0m WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\u001b[32m [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)\u001b[0m\n",
            "\u001b[36m(pid=17800)\u001b[0m E0000 00:00:1741985006.903669   17800 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
            "\u001b[36m(pid=17800)\u001b[0m E0000 00:00:1741985006.929410   17800 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
            "\n",
            "Trial PPO_PredPreyGrass_f1cf2_00000 started with configuration:\n",
            "╭───────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial PPO_PredPreyGrass_f1cf2_00000 config                                │\n",
            "├───────────────────────────────────────────────────────────────────────────┤\n",
            "│ _disable_action_flattening                                          False │\n",
            "│ _disable_execution_plan_api                                            -1 │\n",
            "│ _disable_initialize_loss_from_dummy_batch                           False │\n",
            "│ _disable_preprocessor_api                                           False │\n",
            "│ _dont_auto_sync_env_runner_states                                   False │\n",
            "│ _enable_rl_module_api                                                  -1 │\n",
            "│ _env_to_module_connector                                                  │\n",
            "│ _fake_gpus                                                          False │\n",
            "│ _is_atari                                                                 │\n",
            "│ _learner_class                                                            │\n",
            "│ _learner_connector                                                        │\n",
            "│ _model_config/conv_filters                           ... [64, [3, 3], 1]] │\n",
            "│ _model_config/fcnet_activation                                       relu │\n",
            "│ _model_config/fcnet_hiddens                                    [256, 256] │\n",
            "│ _module_to_env_connector                                                  │\n",
            "│ _prior_exploration_config/type                         StochasticSampling │\n",
            "│ _rl_module_spec                                                           │\n",
            "│ _tf_policy_handles_more_than_one_loss                               False │\n",
            "│ _torch_grad_scaler_class                                                  │\n",
            "│ _torch_lr_scheduler_classes                                               │\n",
            "│ _train_batch_size_per_learner                                             │\n",
            "│ _use_msgpack_checkpoints                                            False │\n",
            "│ _validate_config                                                     True │\n",
            "│ action_mask_key                                               action_mask │\n",
            "│ action_space                                                              │\n",
            "│ actions_in_input_normalized                                         False │\n",
            "│ add_default_connectors_to_env_to_module_pipeline                     True │\n",
            "│ add_default_connectors_to_learner_pipeline                           True │\n",
            "│ add_default_connectors_to_module_to_env_pipeline                     True │\n",
            "│ always_attach_evaluation_results                                       -1 │\n",
            "│ auto_wrap_old_gym_envs                                                 -1 │\n",
            "│ batch_mode                                              truncate_episodes │\n",
            "│ callbacks                                            ..._.EpisodeReturn'> │\n",
            "│ callbacks_on_algorithm_init                                               │\n",
            "│ callbacks_on_checkpoint_loaded                                            │\n",
            "│ callbacks_on_env_runners_recreated                                        │\n",
            "│ callbacks_on_environment_created                                          │\n",
            "│ callbacks_on_episode_created                                              │\n",
            "│ callbacks_on_episode_end                                                  │\n",
            "│ callbacks_on_episode_start                                                │\n",
            "│ callbacks_on_episode_step                                                 │\n",
            "│ callbacks_on_evaluate_end                                                 │\n",
            "│ callbacks_on_evaluate_start                                               │\n",
            "│ callbacks_on_sample_end                                                   │\n",
            "│ callbacks_on_train_result                                                 │\n",
            "│ checkpoint_trainable_policies_only                                  False │\n",
            "│ clip_actions                                                        False │\n",
            "│ clip_param                                                            0.3 │\n",
            "│ clip_rewards                                                              │\n",
            "│ compress_observations                                               False │\n",
            "│ count_steps_by                                                  env_steps │\n",
            "│ create_env_on_driver                                                False │\n",
            "│ custom_async_evaluation_function                                       -1 │\n",
            "│ custom_eval_function                                                      │\n",
            "│ dataset_num_iters_per_learner                                             │\n",
            "│ delay_between_env_runner_restarts_s                                   60. │\n",
            "│ disable_env_checking                                                False │\n",
            "│ eager_max_retraces                                                     20 │\n",
            "│ eager_tracing                                                        True │\n",
            "│ enable_async_evaluation                                                -1 │\n",
            "│ enable_connectors                                                      -1 │\n",
            "│ enable_env_runner_and_connector_v2                                   True │\n",
            "│ enable_rl_module_and_learner                                         True │\n",
            "│ enable_tf1_exec_eagerly                                             False │\n",
            "│ entropy_coeff                                                          0. │\n",
            "│ entropy_coeff_schedule                                                    │\n",
            "│ env                                                         PredPreyGrass │\n",
            "│ env_runner_cls                                                            │\n",
            "│ env_runner_health_probe_timeout_s                                     30. │\n",
            "│ env_runner_restore_timeout_s                                        1800. │\n",
            "│ env_task_fn                                                            -1 │\n",
            "│ episode_lookback_horizon                                                1 │\n",
            "│ episodes_to_numpy                                                    True │\n",
            "│ evaluation_config                                                         │\n",
            "│ evaluation_duration                                                    10 │\n",
            "│ evaluation_duration_unit                                         episodes │\n",
            "│ evaluation_force_reset_envs_before_iteration                         True │\n",
            "│ evaluation_interval                                                       │\n",
            "│ evaluation_num_env_runners                                              0 │\n",
            "│ evaluation_parallel_to_training                                     False │\n",
            "│ evaluation_sample_timeout_s                                          120. │\n",
            "│ explore                                                              True │\n",
            "│ export_native_model_files                                           False │\n",
            "│ fake_sampler                                                        False │\n",
            "│ framework                                                           torch │\n",
            "│ gamma                                                                0.99 │\n",
            "│ grad_clip                                                                 │\n",
            "│ grad_clip_by                                                  global_norm │\n",
            "│ gym_env_vectorize_mode                                               SYNC │\n",
            "│ ignore_env_runner_failures                                          False │\n",
            "│ in_evaluation                                                       False │\n",
            "│ input                                                             sampler │\n",
            "│ input_compress_columns                                 ['obs', 'new_obs'] │\n",
            "│ input_filesystem                                                          │\n",
            "│ input_read_batch_size                                                     │\n",
            "│ input_read_episodes                                                 False │\n",
            "│ input_read_method                                            read_parquet │\n",
            "│ input_read_sample_batches                                           False │\n",
            "│ input_spaces_jsonable                                                True │\n",
            "│ keep_per_episode_custom_metrics                                     False │\n",
            "│ kl_coeff                                                              0.2 │\n",
            "│ kl_target                                                            0.01 │\n",
            "│ lambda                                                                 1. │\n",
            "│ local_gpu_idx                                                           0 │\n",
            "│ local_tf_session_args/inter_op_parallelism_threads                      8 │\n",
            "│ local_tf_session_args/intra_op_parallelism_threads                      8 │\n",
            "│ log_gradients                                                        True │\n",
            "│ log_level                                                            WARN │\n",
            "│ log_sys_usage                                                        True │\n",
            "│ logger_config                                                             │\n",
            "│ logger_creator                                                            │\n",
            "│ lr                                                                 0.0003 │\n",
            "│ lr_schedule                                                               │\n",
            "│ materialize_data                                                    False │\n",
            "│ materialize_mapped_data                                              True │\n",
            "│ max_num_env_runner_restarts                                          1000 │\n",
            "│ max_requests_in_flight_per_aggregator_actor                             3 │\n",
            "│ max_requests_in_flight_per_env_runner                                   1 │\n",
            "│ max_requests_in_flight_per_learner                                      3 │\n",
            "│ metrics_episode_collection_timeout_s                                  60. │\n",
            "│ metrics_num_episodes_for_smoothing                                    100 │\n",
            "│ min_sample_timesteps_per_iteration                                      0 │\n",
            "│ min_time_s_per_iteration                                                  │\n",
            "│ min_train_timesteps_per_iteration                                       0 │\n",
            "│ minibatch_size                                                        128 │\n",
            "│ model/_disable_action_flattening                                    False │\n",
            "│ model/_disable_preprocessor_api                                     False │\n",
            "│ model/_time_major                                                   False │\n",
            "│ model/_use_default_native_models                                       -1 │\n",
            "│ model/always_check_shapes                                           False │\n",
            "│ model/attention_dim                                                    64 │\n",
            "│ model/attention_head_dim                                               32 │\n",
            "│ model/attention_init_gru_gate_bias                                    2.0 │\n",
            "│ model/attention_memory_inference                                       50 │\n",
            "│ model/attention_memory_training                                        50 │\n",
            "│ model/attention_num_heads                                               1 │\n",
            "│ model/attention_num_transformer_units                                   1 │\n",
            "│ model/attention_position_wise_mlp_dim                                  32 │\n",
            "│ model/attention_use_n_prev_actions                                      0 │\n",
            "│ model/attention_use_n_prev_rewards                                      0 │\n",
            "│ model/conv_activation                                                relu │\n",
            "│ model/conv_bias_initializer                                               │\n",
            "│ model/conv_bias_initializer_config                                        │\n",
            "│ model/conv_filters                                                        │\n",
            "│ model/conv_kernel_initializer                                             │\n",
            "│ model/conv_kernel_initializer_config                                      │\n",
            "│ model/conv_transpose_bias_initializer                                     │\n",
            "│ model/conv_transpose_bias_initializer_config                              │\n",
            "│ model/conv_transpose_kernel_initializer                                   │\n",
            "│ model/conv_transpose_kernel_initializer_config                            │\n",
            "│ model/custom_action_dist                                                  │\n",
            "│ model/custom_model                                                        │\n",
            "│ model/custom_preprocessor                                                 │\n",
            "│ model/dim                                                              84 │\n",
            "│ model/encoder_latent_dim                                                  │\n",
            "│ model/fcnet_activation                                               tanh │\n",
            "│ model/fcnet_bias_initializer                                              │\n",
            "│ model/fcnet_bias_initializer_config                                       │\n",
            "│ model/fcnet_hiddens                                            [256, 256] │\n",
            "│ model/fcnet_weights_initializer                                           │\n",
            "│ model/fcnet_weights_initializer_config                                    │\n",
            "│ model/framestack                                                     True │\n",
            "│ model/free_log_std                                                  False │\n",
            "│ model/grayscale                                                     False │\n",
            "│ model/log_std_clip_param                                             20.0 │\n",
            "│ model/lstm_bias_initializer                                               │\n",
            "│ model/lstm_bias_initializer_config                                        │\n",
            "│ model/lstm_cell_size                                                  256 │\n",
            "│ model/lstm_use_prev_action                                          False │\n",
            "│ model/lstm_use_prev_action_reward                                      -1 │\n",
            "│ model/lstm_use_prev_reward                                          False │\n",
            "│ model/lstm_weights_initializer                                            │\n",
            "│ model/lstm_weights_initializer_config                                     │\n",
            "│ model/max_seq_len                                                      20 │\n",
            "│ model/no_final_linear                                               False │\n",
            "│ model/post_fcnet_activation                                          relu │\n",
            "│ model/post_fcnet_bias_initializer                                         │\n",
            "│ model/post_fcnet_bias_initializer_config                                  │\n",
            "│ model/post_fcnet_hiddens                                               [] │\n",
            "│ model/post_fcnet_weights_initializer                                      │\n",
            "│ model/post_fcnet_weights_initializer_config                               │\n",
            "│ model/use_attention                                                 False │\n",
            "│ model/use_lstm                                                      False │\n",
            "│ model/vf_share_layers                                               False │\n",
            "│ model/zero_mean                                                      True │\n",
            "│ normalize_actions                                                    True │\n",
            "│ num_aggregator_actors_per_learner                                       0 │\n",
            "│ num_consecutive_env_runner_failures_tolerance                         100 │\n",
            "│ num_cpus_for_main_process                                               2 │\n",
            "│ num_cpus_per_env_runner                                                 1 │\n",
            "│ num_cpus_per_learner                                                 auto │\n",
            "│ num_env_runners                                                         4 │\n",
            "│ num_envs_per_env_runner                                                 4 │\n",
            "│ num_epochs                                                             30 │\n",
            "│ num_gpus                                                                1 │\n",
            "│ num_gpus_per_env_runner                                                 0 │\n",
            "│ num_gpus_per_learner                                                    0 │\n",
            "│ num_learners                                                            0 │\n",
            "│ observation_filter                                               NoFilter │\n",
            "│ observation_fn                                                            │\n",
            "│ observation_space                                                         │\n",
            "│ offline_data_class                                                        │\n",
            "│ offline_sampling                                                    False │\n",
            "│ ope_split_batch_by_episode                                           True │\n",
            "│ output                                                                    │\n",
            "│ output_compress_columns                                ['obs', 'new_obs'] │\n",
            "│ output_filesystem                                                         │\n",
            "│ output_max_file_size                                             67108864 │\n",
            "│ output_max_rows_per_file                                                  │\n",
            "│ output_write_episodes                                                True │\n",
            "│ output_write_method                                         write_parquet │\n",
            "│ output_write_remaining_data                                         False │\n",
            "│ placement_strategy                                                   PACK │\n",
            "│ policies/predator_policy                             ... Discrete(5), {}) │\n",
            "│ policies/prey_policy                                 ... Discrete(5), {}) │\n",
            "│ policies_to_train                                                         │\n",
            "│ policy_map_cache                                                       -1 │\n",
            "│ policy_map_capacity                                                   100 │\n",
            "│ policy_mapping_fn                                    ...t 0x7e49b7a59d00> │\n",
            "│ policy_states_are_swappable                                         False │\n",
            "│ postprocess_inputs                                                  False │\n",
            "│ prelearner_buffer_class                                                   │\n",
            "│ prelearner_class                                                          │\n",
            "│ prelearner_module_synch_period                                         10 │\n",
            "│ preprocessor_pref                                                deepmind │\n",
            "│ remote_env_batch_wait_ms                                                0 │\n",
            "│ remote_worker_envs                                                  False │\n",
            "│ render_env                                                          False │\n",
            "│ replay_sequence_length                                                    │\n",
            "│ restart_failed_env_runners                                           True │\n",
            "│ restart_failed_sub_environments                                     False │\n",
            "│ rollout_fragment_length                                              auto │\n",
            "│ sample_collector                                     ...leListCollector'> │\n",
            "│ sample_timeout_s                                                      600 │\n",
            "│ sampler_perf_stats_ema_coef                                               │\n",
            "│ seed                                                                      │\n",
            "│ sgd_minibatch_size                                                     -1 │\n",
            "│ shuffle_batch_per_epoch                                              True │\n",
            "│ shuffle_buffer_size                                                     0 │\n",
            "│ simple_optimizer                                                       -1 │\n",
            "│ sync_filters_on_rollout_workers_timeout_s                             10. │\n",
            "│ synchronize_filters                                                    -1 │\n",
            "│ tf_session_args/allow_soft_placement                                 True │\n",
            "│ tf_session_args/device_count/CPU                                        1 │\n",
            "│ tf_session_args/gpu_options/allow_growth                             True │\n",
            "│ tf_session_args/inter_op_parallelism_threads                            2 │\n",
            "│ tf_session_args/intra_op_parallelism_threads                            2 │\n",
            "│ tf_session_args/log_device_placement                                False │\n",
            "│ torch_compile_learner                                               False │\n",
            "│ torch_compile_learner_dynamo_backend                             inductor │\n",
            "│ torch_compile_learner_dynamo_mode                                         │\n",
            "│ torch_compile_learner_what_to_compile                ...ile.FORWARD_TRAIN │\n",
            "│ torch_compile_worker                                                False │\n",
            "│ torch_compile_worker_dynamo_backend                                onnxrt │\n",
            "│ torch_compile_worker_dynamo_mode                                          │\n",
            "│ torch_skip_nan_gradients                                            False │\n",
            "│ train_batch_size                                                     1024 │\n",
            "│ update_worker_filter_stats                                           True │\n",
            "│ use_critic                                                           True │\n",
            "│ use_gae                                                              True │\n",
            "│ use_kl_loss                                                          True │\n",
            "│ use_worker_filter_stats                                              True │\n",
            "│ validate_env_runners_after_construction                              True │\n",
            "│ vf_clip_param                                                         10. │\n",
            "│ vf_loss_coeff                                                          1. │\n",
            "│ vf_share_layers                                                        -1 │\n",
            "│ worker_cls                                                             -1 │\n",
            "╰───────────────────────────────────────────────────────────────────────────╯\u001b[36m(PPO pid=17615)\u001b[0m Trainable.setup took 38.983 seconds. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.\n",
            "\u001b[36m(PPO pid=17615)\u001b[0m Install gputil for GPU system monitoring.\n",
            "\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m Episode 1: R=10.0 Global SUM=10.0\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Prey: Total Reward = 10.00, Avg Reward = 1.25\n",
            "\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:44:11. Total running time: 1min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:44:42. Total running time: 1min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:45:12. Total running time: 2min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:45:42. Total running time: 2min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:46:12. Total running time: 3min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭──────────────────────────────────────────╮\n",
            "│ Trial name                      status   │\n",
            "├──────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING  │\n",
            "╰──────────────────────────────────────────╯\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17725)\u001b[0m Episode 2: R=20.0 Global SUM=80.0\u001b[32m [repeated 9x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17725)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 9x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17725)\u001b[0m   - Prey: Total Reward = 20.00, Avg Reward = 1.25\u001b[32m [repeated 9x across cluster]\u001b[0m\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:46:42. Total running time: 3min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17800)\u001b[0m Episode 5: R=40.0 Global SUM=260.0\u001b[32m [repeated 7x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17800)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 7x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17800)\u001b[0m   - Prey: Total Reward = 40.00, Avg Reward = 3.64\u001b[32m [repeated 7x across cluster]\u001b[0m\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:47:12. Total running time: 4min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:47:42. Total running time: 4min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:48:12. Total running time: 5min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:48:42. Total running time: 5min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:49:12. Total running time: 6min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:49:42. Total running time: 6min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         1            157.715                        1                     1032                      nan │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m Episode 6: R=10.0 Global SUM=250.0\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Prey: Total Reward = 10.00, Avg Reward = 0.45\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m Episode 7: R=150.0 Global SUM=400.0\u001b[32m [repeated 6x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 6x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Prey: Total Reward = 150.00, Avg Reward = 8.33\u001b[32m [repeated 6x across cluster]\u001b[0m\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:50:12. Total running time: 7min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:50:42. Total running time: 7min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:51:12. Total running time: 8min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:51:42. Total running time: 8min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:52:12. Total running time: 9min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:52:42. Total running time: 9min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:53:12. Total running time: 10min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:53:42. Total running time: 10min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:54:12. Total running time: 11min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:54:42. Total running time: 11min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         2            352.961                        1                     2061                  5.26882 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m Episode 8: R=30.0 Global SUM=430.0\u001b[32m [repeated 4x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 4x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17724)\u001b[0m   - Prey: Total Reward = 30.00, Avg Reward = 1.15\u001b[32m [repeated 4x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17767)\u001b[0m Episode 11: R=180.0 Global SUM=1050.0\u001b[32m [repeated 8x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17767)\u001b[0m   - Predators: Total Reward = 0.00, Avg Reward = 0.00\u001b[32m [repeated 8x across cluster]\u001b[0m\n",
            "\u001b[36m(MultiAgentEnvRunner pid=17767)\u001b[0m   - Prey: Total Reward = 180.00, Avg Reward = 7.20\u001b[32m [repeated 8x across cluster]\u001b[0m\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:55:12. Total running time: 12min 0s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         3            658.545                        1                     3091                  3.37003 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
            "Trial status: 1 RUNNING\n",
            "Current time: 2025-03-14 20:55:42. Total running time: 12min 30s\n",
            "Logical resource usage: 6.0/8 CPUs, 0/1 GPUs (0.0/1.0 accelerator_type:T4)\n",
            "╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n",
            "│ Trial name                      status       iter     total time (s)     ...lls_per_iteration     ..._sampled_lifetime     ...fetime_throughput │\n",
            "├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤\n",
            "│ PPO_PredPreyGrass_f1cf2_00000   RUNNING         3            658.545                        1                     3091                  3.37003 │\n",
            "╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.14"
    },
    "vscode": {
      "interpreter": {
        "hash": "ac2eaa0ea0ebeafcc7822e65e46aa9d4f966f30b695406963e145ea4a91cd4fc"
      }
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}