GPU-power-energy-optimization.v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# GPU power and energy optimization interactive walkthrough\n",
        "\n",
        "*Alan Gray, NVIDIA, February 2025*\n",
        "\n",
        "In this exercise, you will be guided to optimize the energy usage of a benchmark by reducing the GPU clock frequency (which in turn reduces power draw).\n",
        "\n",
        "We will use a [QCD (Physics) benchmark](https://repository.prace-ri.eu/git/UEABS/ueabs/-/tree/master/qcd/part_1) here, but the same techniques can be used for any code.\n",
        "\n",
        "This notebook is designed for T4 GPUs, but can be run on other GPUs with some minor adjustments. The commands can also be copied and run directly on any other GPU server.\n",
        "\n",
        "If running this notebook on Google Colab: to get started, change the runtime type to \"T4 GPU\" via the \"Additional connection options\" button in the top right of this notebook, and click \"Connect\". (*If no T4 GPUs are available at the current time, or you get a message about usage limits, you can try again later*.)\n",
        "\n",
        "Press the \"play\" button on each code cell in turn following the instructions. When it has completed a tick will appear, and you can move on to the next cell."
      ],
      "metadata": {
        "id": "BM65UllRPavq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Setup\n",
        "\n",
        "Test an interactive prompt, using nvidia-smi to print details of the available GPU:"
      ],
      "metadata": {
        "id": "YBh16HncQFqP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! nvidia-smi"
      ],
      "metadata": {
        "id": "0KWCNdk8rl5I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get the benchmark code:"
      ],
      "metadata": {
        "id": "Xlgaxf2Mrqyd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! git clone --depth=1 https://repository.prace-ri.eu/git/UEABS/ueabs.git"
      ],
      "metadata": {
        "id": "H632s2yKcYFT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a config file to allow the benchmark to be built (the details of this are not important for this exercise):"
      ],
      "metadata": {
        "id": "abCrTJKDQZvU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('ueabs/qcd/part_1/config.mk', 'w') as f:\n",
        "    f.write(\"\"\"\n",
        "MPIDIR=/usr/lib/x86_64-linux-gnu/openmpi\n",
        "GPUS_PER_NODE=1\n",
        "NVARCH=sm_75 # For T4 GPU, adjust for any other GPU\n",
        "CFLAGS = $(DEFINES) -O2 -DARCH=0 -w -I $(MPIDIR)/include\n",
        "LDFLAGS = -lm  -arch=$(NVARCH) -L./targetDP -ltarget -L$(MPIDIR)/lib -lmpi -lm -lgomp\n",
        "CC=mpicc\n",
        "TARGETCC=nvcc\n",
        "TARGETCFLAGS=-x cu -arch=$(NVARCH) -I. -DCUDA -DVVL=1 -DSoA -DGPUSPN=$(GPUS_PER_NODE) -dc -c $(CFLAGS)\n",
        "    \"\"\")\n"
      ],
      "metadata": {
        "id": "RcX3d2AKdbz9"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Build the \"targetDP\" library, a dependency of the benchmark:"
      ],
      "metadata": {
        "id": "sRIGjo5rQPKP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pushd ueabs/qcd/part_1/targetDP; make clean; make; popd"
      ],
      "metadata": {
        "id": "Z-QDBbQEe_BW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Build the benchmark:"
      ],
      "metadata": {
        "id": "cowit-RcQjWN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pushd ueabs/qcd/part_1/src; make clean; make; popd"
      ],
      "metadata": {
        "id": "9HXJyUErfRH5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Adjust the input file to specify 500 iterations, which large enough for representative power measurements:"
      ],
      "metadata": {
        "id": "oGiZ7Fi6QpC5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! sed -i \"s/max_cg_iters 1/max_cg_iters 500/g\" ueabs/qcd/part_1/src/kernel_E.input"
      ],
      "metadata": {
        "id": "THULjiD2hAhP"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a script to run the code while monitoring GPU power and clock frequency. See the inline comments in the script."
      ],
      "metadata": {
        "id": "UaNZSwCqQ09I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('run.sh', 'w') as f:\n",
        "    f.write(\"\"\"\n",
        "# start nvidia-smi looping in the background, writing power and clock measurements to a CSV file every 100 ms.\n",
        "nvidia-smi --query-gpu=index,power.draw,clocks.gr --format=csv --loop-ms 100 > GPU_readings.csv 2>&1 &\n",
        "# run the benchmark\n",
        "pushd ueabs/qcd/part_1/src/\n",
        "mpirun --allow-run-as-root -np 1 ./bench\n",
        "popd\n",
        "# stop nvidia-smi\n",
        "pkill nvidia-smi\n",
        "    \"\"\")"
      ],
      "metadata": {
        "id": "GeHKBEioXlEs"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Run at default clock\n",
        "\n",
        "**Run the script you just created (will take around 1 minute on a T4 GPU, ignore the warning about a missing reference file):**\n",
        "\n",
        "(the **bold** text here and below is to highlight parts that will be repeated in this exercise)"
      ],
      "metadata": {
        "id": "c7Cz7h69R92e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! bash ./run.sh 2>&1 | tee run.log"
      ],
      "metadata": {
        "id": "fAdAmFH9hkYc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's have a look at the first few lines of the GPU_readings.csv file. You will see that it consists of a header, plus power and clock measurements:"
      ],
      "metadata": {
        "id": "3J_kftzPSCti"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! head -5 GPU_readings.csv"
      ],
      "metadata": {
        "id": "md-0WT1nyCTX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Extract the power readings into a separate file:**"
      ],
      "metadata": {
        "id": "7r4g5rz6TgrF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! cat GPU_readings.csv | grep -v index | awk '{ print $2}'  > GPU_power.csv"
      ],
      "metadata": {
        "id": "_Oi6WOkdTyQ1"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a graph of power measurements. You will see that the power ramps up to the maximum (70W for T4) as the benchmark starts up.\n",
        "\n",
        "(Optional: Create a similar graph of clock frequency measurements - see end of notebook.)"
      ],
      "metadata": {
        "id": "7VfnLEM_BV1m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "data = pd.read_csv('GPU_power.csv', header=None)\n",
        "column = data.iloc[:, 0]\n",
        "plt.figure(figsize=(5, 3))\n",
        "plt.plot(column)\n",
        "plt.xlabel('Sample')\n",
        "plt.ylabel('GPU Power (W)')\n",
        "plt.show()\n"
      ],
      "metadata": {
        "id": "4IC2NIwTmJTc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Strip the first 200 and last 100 entries from the power readings file, to remove startup and shutdown parts (which won't be relevant for a real-world longer run):**"
      ],
      "metadata": {
        "id": "ffNc6oeQ_Hlf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! sed -i '1,200d' GPU_power.csv; sed -i \"$(($(wc -l < GPU_power.csv)-100)),\\$d\" GPU_power.csv"
      ],
      "metadata": {
        "id": "NYJi2M1Imlmx"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a script to extract average power, time and energy (power multiplied by time) readings from our data:"
      ],
      "metadata": {
        "id": "G53ZDBsYBxkl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('write_stats.sh', 'w') as f:\n",
        "    f.write(\"\"\"\n",
        "# Extract average power across the samples\n",
        "power=`cat GPU_power.csv | awk 'BEGIN{sum=0;count=0}{ sum += $1; count+=1}END{print sum/count}'`;\n",
        "# Extract time from benchmark output\n",
        "time=`grep \"BENCHMARK TIME \" run.log | awk '{ print $3 }'`;\n",
        "# Calculate Energy = Power x Time\n",
        "energy=$(awk -v p=\"$power\" -v t=\"$time\" 'BEGIN {print (p*t/1000)}');\n",
        "printf \"Power = %.1f W Time = %.1f s Energy = %.1f kJ\" $power $time $energy\n",
        "    \"\"\")"
      ],
      "metadata": {
        "id": "gR-G47tF5IBu"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Run our script to write the power, time and energy readings:**"
      ],
      "metadata": {
        "id": "3eQxDCNg-1r-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! bash ./write_stats.sh"
      ],
      "metadata": {
        "id": "5HmpMPGn7Ijd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Record Results\n",
        "\n",
        "Double click on this cell to edit it and take a note of your measurements at default clock:\n",
        "\n",
        "Default clock:\n",
        "Power = ?? W Time = ?? s Energy = ?? kJ\n",
        "\n",
        "Reduced clock:\n",
        "Power = ?? W Time = ?? s Energy = ?? kJ"
      ],
      "metadata": {
        "id": "_Laz2tvA81Z3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Repeat with reduced clock\n",
        "\n",
        "Reduce the maximum GPU clock frequency to 990 GHz (for T4 GPU) by setting \"Application Clocks\". (Note that 5001 is the memory clock for the T4 GPU, and we are not changing that in this exercise.)\n",
        "\n",
        "\n",
        "(Optional: query which clocks are supported choose any different maximum frequency - see end of notebook.)"
      ],
      "metadata": {
        "id": "MuY2GckedATp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! nvidia-smi -ac 5001,990"
      ],
      "metadata": {
        "id": "iyUF7TPXjsoY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now re-run the benchmark with the reduced clock by going back and clicking play on all the cells denoted by a **bold** text description above, and update the \"Results\" cell with the \"Reduced clock\" values. You will see the power is reduced with minimal impact on the benchmark time meaning that the GPU energy of the benchmark decreases 😀.\n",
        "\n",
        "You can now try some of the optional commands below, to\n",
        "\n",
        "*   analyse the clock frequency behaviour\n",
        "*   re-run at different maximum clock frequencies\n",
        "*   run using a power limit instead of a maximum clock frequency\n",
        "\n",
        "This benchmark is mainly sensitive to GPU memory bandwidth. While behavior varies across different codes and GPUs, there usually exists similar potential for energy savings. For more details, see my GTC presentations:\n",
        "\n",
        "https://www.nvidia.com/en-us/on-demand/session/gtc24-s62419/\n",
        "\n",
        "https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s52087/  \n",
        ".\n",
        "\n"
      ],
      "metadata": {
        "id": "aO3LN-N5e-V1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Optional extra commands"
      ],
      "metadata": {
        "id": "HKPyPl4rdRkx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Extract the clock readings into a separate file:"
      ],
      "metadata": {
        "id": "BrjBuYqhT2Jm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! cat GPU_readings.csv | grep -v index | awk '{ print $4}'  > GPU_clock.csv"
      ],
      "metadata": {
        "id": "sgHMkMjbxbdW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a graph of clock measurements:"
      ],
      "metadata": {
        "id": "1MaMqzSTBfTJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "data = pd.read_csv('GPU_clock.csv', header=None)\n",
        "column = data.iloc[:, 0]\n",
        "plt.figure(figsize=(5, 3))\n",
        "plt.plot(column)\n",
        "plt.xlabel('Sample')\n",
        "plt.ylabel('SM Clock (MHz)')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "n15JX2JiV4F3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Query the supported clocks on the GPU (to allow repeated runs with any supported clock):"
      ],
      "metadata": {
        "id": "kD0HeolKdWbC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! nvidia-smi -q -d SUPPORTED_CLOCKS"
      ],
      "metadata": {
        "id": "wWCbicl_lXcj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Set a power limit instead of a maximum clock, for comparison of these techniques. First, reset the aplication clocks to the highest supported, and then set the power limit e.g. to the power you measured above at reduced clock, and re-run the benchmark. The resulting benchmark time will tell you if there is any benefit of either technique."
      ],
      "metadata": {
        "id": "lGZa2XI-dahR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! ! nvidia-smi -ac 5001,1590; nvidia-smi -pl 60"
      ],
      "metadata": {
        "id": "amDOFgMCeA-U"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}