Trends report (#765)

Trends report is available to collaborators at https://llm-exp.oss-fuzz.com/trend-reports/index.html This PR * Adds trends report web page * Adds a cloud run function to update `trend-reports/index.json` in GCS * Adds a cloud run function to update web page files (html, js, css) * Adds a script to deploy cloud run functions (this script needs to be run manually after every change to the code) * Moves trends report summary generation tool to a subdirectory so everything related to trends report is together @DavidKorczynski @DonggeLiu please let me know if you want anything else added to the trends report
google · Feb 2, 2025 · 24b121a · 24b121a
1 parent 48fa001
commit 24b121a
Show file tree

Hide file tree

Showing 13 changed files with 1,225 additions and 3 deletions.
diff --git a/report/README.md b/report/README.md
@@ -0,0 +1,31 @@
+# Reports
+
+## Experiment Report
+
+*  While the experiment is running, `upload_report.sh` periodically generates
+   an experiment report and uploads it to
+   `gs://oss-fuzz-gcb-experiment-run-logs/Result-reports/`.
+*  After the experiment a final report is generated and uploaded to GCS.
+*  These reports are accessible to collaborators via
+   `https://llm-exp.oss-fuzz.com/Result-reports/{experiment_name}`
+
+## Trends Report
+
+1. After each experiment is finished, `docker_run.sh` uploads a summary json
+   file to `gs://oss-fuzz-gcb-experiment-run-logs/trend-reports/`.
+2. Upload of the summary json triggers a
+   [Cloud Run Function](https://pantheon.corp.google.com/functions/details/us-central1/llm-trends-report-index?env=gen1&project=oss-fuzz)
+   which updates
+   `gs://oss-fuzz-gcb-experiment-run-logs/trend-reports/index.json`.
+3. The
+   [trends report web page](https://llm-exp.oss-fuzz.com/trend-reports/index.html)
+   loads the index and discovers available summary json files.
+
+# Updating the Code
+
+*  The Cloud Run Functions are updated manually by running
+   `deploy_functions.sh`.
+*  The web page files in `gs://oss-fuzz-gcb-experiment-run-logs/trend-reports/`
+   are updated via a
+   [Cloud Run Function](https://pantheon.corp.google.com/functions/details/us-central1/llm-trends-report-web?env=gen1&project=oss-fuzz).
+
diff --git a/report/__init__.py b/report/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2024 Google LLC
+# Copyright 2025 Google LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

diff --git a/report/docker_run.sh b/report/docker_run.sh
@@ -194,7 +194,7 @@ fi
 # Wait for the report process to finish uploading.
 wait $pid_report
 
-$PYTHON -m report.trends_report \
+$PYTHON -m report.trends_report.upload_summary \
   --results-dir ${LOCAL_RESULTS_DIR:?} \
   --output-path "gs://oss-fuzz-gcb-experiment-run-logs/trend-reports/${GCS_TREND_REPORT_PATH:?}" \
   --name ${EXPERIMENT_NAME:?} \

diff --git a/report/trends_report/deploy_functions.sh b/report/trends_report/deploy_functions.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+UPDATE_INDEX_TOPIC=llm-trends-report
+# gcs notification
+UPDATE_INDEX_BUCKET='gs://oss-fuzz-gcb-experiment-run-logs'
+UPDATE_INDEX_PREFIX='trend-reports/'
+# function
+UPDATE_INDEX_FUNCTION=llm-trends-report-index
+UPDATE_INDEX_FILENAME=update_index.py
+UPDATE_INDEX_ENTRY_POINT=trends_report_index
+
+UPDATE_WEB_TOPIC=llm-trends-report-web
+# function
+UPDATE_WEB_FUNCTION=llm-trends-report-web
+UPDATE_WEB_FILENAME=update_web.py
+UPDATE_WEB_ENTRY_POINT=trends_report_web
+# scheduler
+UPDATE_WEB_SCHEDULER=llm-trends-report-web
+UPDATE_WEB_CRON='0 * * * *'
+UPDATE_WEB_MESSAGE='update'
+
+function deploy_pubsub_topic {
+  topic=$1
+  project=$2
+
+  echo "gcloud pubsub topics describe $topic --project $project"
+  if ! gcloud pubsub topics describe $topic --project $project;
+    then
+      gcloud pubsub topics create $topic --project $project
+  fi
+}
+
+function deploy_scheduler {
+  scheduler_name=$1
+  schedule="$2"
+  topic=$3
+  message="$4"
+  project=$5
+
+  if gcloud scheduler jobs describe $scheduler_name --project $project ;
+    then
+      gcloud scheduler jobs update pubsub $scheduler_name \
+        --project $project \
+        --schedule "$schedule" \
+        --topic $topic \
+        --message-body "$message"
+    else
+      gcloud scheduler jobs create pubsub $scheduler_name \
+        --project $project \
+        --schedule "$schedule" \
+        --topic $topic \
+        --message-body "$message"
+  fi
+}
+
+function deploy_cloud_function {
+  name=$1
+  filename=$2
+  entry_point=$3
+  topic=$4
+  project=$5
+
+  gcloud functions deploy $name \
+    --entry-point $entry_point \
+    --trigger-topic $topic \
+    --runtime python312 \
+    --project $project \
+    --timeout 540 \
+    --region us-central1 \
+    --max-instances 1 \
+    --memory 4096MB \
+    --set-build-env-vars=GOOGLE_FUNCTION_SOURCE=$filename
+}
+
+if [ $# == 1 ]; then
+  PROJECT_ID=$1
+else
+  echo -e "\n Usage ./deploy_functions.sh <project-name>"; exit;
+fi
+
+deploy_pubsub_topic $UPDATE_INDEX_TOPIC $PROJECT_ID
+
+gcloud storage buckets notifications create \
+  $UPDATE_INDEX_BUCKET \
+  --topic=$UPDATE_INDEX_TOPIC \
+  --event-types=OBJECT_FINALIZE \
+  --object-prefix="$UPDATE_INDEX_PREFIX"
+
+deploy_cloud_function $UPDATE_INDEX_FUNCTION \
+  $UPDATE_INDEX_FILENAME \
+  $UPDATE_INDEX_ENTRY_POINT \
+  $UPDATE_INDEX_TOPIC \
+  $PROJECT_ID
+
+deploy_pubsub_topic $UPDATE_WEB_TOPIC $PROJECT_ID
+
+deploy_scheduler $UPDATE_WEB_SCHEDULER \
+  "$UPDATE_WEB_CRON" \
+  $UPDATE_WEB_TOPIC \
+  "$UPDATE_WEB_MESSAGE" \
+  $PROJECT_ID
+
+deploy_cloud_function $UPDATE_WEB_FUNCTION \
+  $UPDATE_WEB_FILENAME \
+  $UPDATE_WEB_ENTRY_POINT \
+  $UPDATE_WEB_TOPIC \
+  $PROJECT_ID
diff --git a/report/trends_report/requirements.txt b/report/trends_report/requirements.txt
@@ -0,0 +1 @@
+google-cloud-storage==2.9.*
diff --git a/report/trends_report/update_index.py b/report/trends_report/update_index.py
@@ -0,0 +1,57 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Cloud Run Function to update the trends report index."""
+
+import json
+import sys
+
+from google.cloud import storage
+
+
+def trends_report_index(event, context):
+  """Read all the trends reports in GCS and write an index at the root."""
+  # Don't trigger on changes to index.json or other top level files
+  if len(event['attributes']['objectId'].split('/')) < 3:
+    return ''
+
+  index = {}
+  bucket = storage.Client().bucket('oss-fuzz-gcb-experiment-run-logs')
+  for b in bucket.list_blobs(prefix='trend-reports/'):
+    # Skip reading index.json or other top level files
+    if len(b.name.split('/')) < 3:
+      continue
+
+    print(f'Reading {b.name}')
+    try:
+      # e.g. trend-reports/scheduled/2024-11-02-weekly-all.json -> scheduled
+      directory = b.name.split('/')[1]
+      report = json.loads(b.download_as_text())
+      index[report['name']] = {
+          'directory': directory,
+          'name': report['name'],
+          'url': report['url'],
+          'date': report['date'],
+          'benchmark_set': report['benchmark_set'],
+          'llm_model': report['llm_model'],
+          'tags': report['tags'],
+      }
+    except:
+      print('****************************', file=sys.stderr)
+      print(f'Issue when reading {b.name}', file=sys.stderr)
+      print('****************************', file=sys.stderr)
+
+  bucket.blob('trend-reports/index.json').upload_from_string(
+      json.dumps(index), content_type='application/json')
+
+  return ''
diff --git a/report/trends_report/update_web.py b/report/trends_report/update_web.py
@@ -0,0 +1,50 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Cloud Run Function to update trends report web page from GitHub."""
+
+import io
+import os
+import tempfile
+import urllib.request
+import zipfile
+
+from google.cloud import storage
+
+REPO_ZIP_LINK = 'https://github.com/google/oss-fuzz-gen/archive/refs/heads/main.zip'
+ZIP_DIR = 'oss-fuzz-gen-trends-report'
+
+
+def trends_report_web(event, context):
+  """Update trends report web page files from GitHub."""
+  bucket = storage.Client().bucket('oss-fuzz-gcb-experiment-run-logs')
+
+  with urllib.request.urlopen(REPO_ZIP_LINK) as response:
+    zip_contents = response.read()
+
+  with tempfile.TemporaryDirectory() as temp:
+    with zipfile.ZipFile(io.BytesIO(zip_contents)) as zip_file:
+      zip_file.extractall(temp)
+      for path in zip_file.namelist():
+        parts = path.split('/report/trends_report_web/')
+
+        # Upload files under report/trends_report_web/
+        if len(parts) > 1 and parts[1] != '':
+          fname = parts[1]
+          print(f'uploading {path} to trend-reports/{fname}')
+          blob = bucket.blob(f'trend-reports/{fname}')
+          blob.upload_from_filename(os.path.join(temp, path))
+
+
+if __name__ == "__main__":
+  trends_report_web(None, None)
diff --git a/report/trends_report.py → report/trends_report/upload_summary.py b/report/trends_report.py → report/trends_report/upload_summary.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright 2024 Google LLC
+# Copyright 2025 Google LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

diff --git a/report/trends_report_web/d3.min.js b/report/trends_report_web/d3.min.js
diff --git a/report/trends_report_web/index.html b/report/trends_report_web/index.html
@@ -0,0 +1,94 @@
+<!--
+Copyright 2025 Google LLC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>OSS Fuzz Gen Trends Report</title>
+    <link href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined" rel="stylesheet" />
+    <link href="https://fonts.googleapis.com/css2?family=Noto+Sans:ital,wght@0,400..600;1,400..600&display=swap" rel="stylesheet" />
+    <link rel="stylesheet" href="style.css">
+  </head>
+  <body>
+    <h1>Trends Report</h1>
+    <section>
+      <button id="filters-toggle" class="filters-toggle">
+        <span class="material-symbols-outlined">filter_alt</span>
+        Filters
+      </button>
+      <div id="filters" class="filters" style="display: none;">
+        <div>
+          <h3>Date Range</h3>
+          <select id="date-range-filter">
+            <option value="all" selected>all</option>
+            <option value="7">7 days</option>
+            <option value="14">14 days</option>
+            <option value="30">30 days</option>
+            <option value="60">60 days</option>
+            <option value="90">90 days</option>
+            <option value="120">120 days</option>
+            <option value="180">180 days</option>
+          </select>
+        </div>
+        <div>
+          <h3>LLM models</h3>
+          <div id="llm-filter">
+            <input type="text" placeholder="Start typing to filter" />
+          </div>
+        </div>
+        <div>
+          <h3>Benchmark sets</h3>
+          <div id="benchmark-filter">
+            <input type="text" placeholder="Start typing to filter" />
+          </div>
+        </div>
+        <div>
+          <h3>Tags</h3>
+          <div id="tag-filter">
+            <input type="text" placeholder="Start typing to filter" />
+          </div>
+        </div>
+      </div>
+    </section>
+    <section>
+      <h2>Overview</h2>
+      <div id="overview-chart" class="chart"></div>
+      <div id="overview-table"></div>
+      <h2>Projects</h2>
+      <div id="overview-coverage-chart" class="chart"></div>
+      <div id="projects" class="projects"></div>
+      <h3 id="project-header"></h3>
+      <h4>Coverage</h4>
+      <div id="project-coverage-chart" class="chart"></div>
+      <div id="project-coverage-table"></div>
+      <h4>Build Rate</h4>
+      <div id="project-build-chart" class="chart"></div>
+      <h4>Crash Rate</h4>
+      <div id="project-crash-chart" class="chart"></div>
+    </section>
+    <section>
+      <h1>Links</h1>
+      <ul id="links"></ul>
+    </section>
+    <div class="loading-shadow-box" id="loading">
+      <span class="material-symbols-outlined">progress_activity</span>
+    </div>
+    <script defer src="d3.min.js"></script>
+    <script defer src="plot.min.js"></script>
+    <script defer src="index.js"></script>
+  </body>
+</html>