Add HiPerGator scripts for automated dataset release validation and reporting

[Copilot]

This PR implements a complete solution for periodically validating MillionTrees dataset releases on HiPerGator infrastructure. The scripts automate running release tests and generate comprehensive reports with dataset size information to confirm that releases are well-structured and accessible.

What's Added

Core Scripts

• submit_test_release.sh - SLURM submission script configured for HiPerGator with appropriate resource allocation (32GB RAM, 24hr time limit, GPU partition)
• generate_dataset_report.py - Standalone Python script that generates comprehensive dataset reports with version history and size tracking
• HIPERGATOR_TESTING.md - Complete documentation covering setup, usage, and scheduling
• test_hipergator_scripts.py - Validation test suite to ensure scripts work correctly

Key Features

Automated Dataset Validation:

• Runs all tests in tests/test_release.py with full pytest output capture
• Validates dataset downloads, structure, format, and PyTorch dataloader functionality
• Proper error handling and exit code propagation for SLURM job status

Comprehensive Reporting:

• Extracts dataset size information from all versions (current total: ~77.86 GB)
• Generates markdown reports with download URLs, compressed sizes, and test results
• Tracks dataset growth over time for infrastructure planning

Robust Logging:

• Creates timestamped log directories with organized output files
• Maintains latest symlinks for easy access to recent results
• Captures both test results and report generation logs

Dataset Size Summary

Dataset	Latest Version	Compressed Size
TreePolygons	0.2	70.24 GB
TreePoints	0.2	1.36 GB
TreeBoxes	0.2	6.26 GB
Total	-	77.86 GB

Usage

On HiPerGator

sbatch submit_test_release.sh

Generate Reports Locally

python generate_dataset_report.py
python generate_dataset_report.py --test-exit-code 0  # include test results

Scheduling

Can be scheduled via cron for weekly validation:

0 2 * * 0 cd /home/b.weinstein/MillionTrees && sbatch submit_test_release.sh

Benefits

• Quality Assurance: Ensures dataset releases remain accessible and properly formatted
• Infrastructure Planning: Tracks dataset growth for storage and bandwidth planning
• Automated Monitoring: Provides regular validation of the MillionTrees benchmark integrity
• Documentation: Generates up-to-date reports with current dataset information

The scripts are ready for deployment on HiPerGator and provide a comprehensive solution for maintaining dataset quality assurance through automated testing and reporting.

Fixes #46.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• data.rc.ufl.edu
  • Triggering command: python -m pytest tests/test_release.py -v --tb=short (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[bw4sz]

@ethanwhite do you see this?

[ethanwhite]

@bw4sz - yes, I see it