Skip to content

mlcommons/mlperf_inference_test_submissions_v5.0

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ec0a702 · Feb 20, 2025
Feb 6, 2025
Feb 20, 2025
Feb 20, 2025
Feb 20, 2025
Oct 10, 2024
Jan 4, 2025
Feb 20, 2025
Feb 20, 2025
Feb 20, 2025
Feb 20, 2025

Repository files navigation

Please download summary.xlsx to view the most recent results.

[2025-02-20 19:48:40,344 submission_checker1.py:3263 INFO] Results=19, NoResults=0, Power Results=0
[2025-02-20 19:48:40,344 submission_checker1.py:3270 INFO] ---
[2025-02-20 19:48:40,345 submission_checker1.py:3271 INFO] Closed Results=0, Closed Power Results=0

[2025-02-20 19:48:40,345 submission_checker1.py:3276 INFO] Open Results=19, Open Power Results=0

[2025-02-20 19:48:40,345 submission_checker1.py:3281 INFO] Network Results=0, Network Power Results=0

[2025-02-20 19:48:40,345 submission_checker1.py:3286 INFO] ---
[2025-02-20 19:48:40,345 submission_checker1.py:3288 INFO] Systems=14, Power Systems=0
[2025-02-20 19:48:40,345 submission_checker1.py:3292 INFO] Closed Systems=0, Closed Power Systems=0
[2025-02-20 19:48:40,345 submission_checker1.py:3297 INFO] Open Systems=14, Open Power Systems=0
[2025-02-20 19:48:40,345 submission_checker1.py:3302 INFO] Network Systems=0, Network Power Systems=0
[2025-02-20 19:48:40,345 submission_checker1.py:3307 INFO] ---
[2025-02-20 19:48:40,345 submission_checker1.py:3312 INFO] SUMMARY: submission looks OK
[2025-02-20 19:48:41,304 module.py:5481 INFO] -        ! call "postprocess" from /home/runner/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-submission-checker/customize.py

Organization Availability Division SystemType SystemName Platform Model MlperfModel Scenario Result Accuracy number_of_nodes host_processor_model_name host_processors_per_node host_processor_core_count accelerator_model_name accelerators_per_node Location framework operating_system notes compliance errors version inferred has_power Units weight_data_types
0 MLCommons available open edge gh_macos-latest x86 gh_macos-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config resnet50 resnet Offline 22.6142 acc: 76.000 1 undefined 1 3 nan 0 open/MLCommons/results/gh_macos-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config/resnet50/offline onnxruntime v1.20.1 (darwin-23.6.0) nan 1 0 v5.0 0 False Samples/s fp32
1 MLCommons available open edge gh_macos-latest x86 gh_macos-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config retinanet retinanet Offline 0.32368 mAP: 76.951 1 undefined 1 3 nan 0 open/MLCommons/results/gh_macos-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config/retinanet/offline onnxruntime v1.20.1 (darwin-23.6.0) nan 1 0 v5.0 0 False Samples/s fp32
2 MLCommons available open edge gh_ubuntu-latest gh_ubuntu-latest-reference-cpu-tvm-onnx_v1.19.2-default_config resnet50 resnet Offline 1.71838 acc: 80.000 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest-reference-cpu-tvm-onnx_v1.19.2-default_config/resnet50/offline tvm-onnx v1.19.2 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
3 MLCommons available open edge gh_macos-latest_x86 gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config bert-99 bert-99 Offline 7.26681 F1: 80.0 1 undefined 1 3 nan 0 open/MLCommons/results/gh_macos-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline deepsparse v1.8.0 (darwin-23.6.0) nan 1 0 v5.0 0 False Samples/s fp32
4 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config resnet50 resnet Offline 23.0058 acc: 76.000 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/resnet50/offline onnxruntime Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
5 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config retinanet retinanet Offline 0.417437 mAP: 76.951 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/retinanet/offline onnxruntime Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
6 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config resnet50 resnet Offline 23.1816 acc: 76.000 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config/resnet50/offline onnxruntime v1.20.1 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
7 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config retinanet retinanet Offline 0.433605 mAP: 76.951 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-onnxruntime_v1.20.1-default_config/retinanet/offline onnxruntime v1.20.1 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
8 MLCommons available open edge gh_ubuntu-latest x86 gh_ubuntu-latest_x86-reference-cpu-tf_v2.18.0-default_config resnet50 resnet Offline 20.5941 acc: 76.000 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-tf_v2.18.0-default_config/resnet50/offline tf v2.18.0 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
9 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.6.0-default_config retinanet retinanet Offline 0.412006 mAP: 76.951 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.6.0-default_config/retinanet/offline pytorch v2.6.0 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
10 MLCommons available open datacenter gh_action gh_action-reference-cpu-pytorch_v2.6.0-default_config llama2-70b-99 llama2-70b-99 Offline 0.385187 ROUGE1: 61.7021 ROUGE2: 37.9679 ROUGEL: 39.3617 TOKENS_PER_SAMPLE: 610.0 1 AMD Ryzen 9 7950X 16-Core Processor 1 16 nan 0 open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline pytorch v2.6.0 Ubuntu 22.04 (linux-6.8.0-52-generic-glibc2.35) nan 1 0 v5.0 0 False Tokens/s fp32
11 MLCommons available open datacenter gh_action gh_action-reference-cpu-pytorch_v2.6.0-default_config mixtral-8x7b mixtral-8x7b Offline 0.0148696 ROUGE1: 31.4286 ROUGE2: 5.8824 ROUGEL: 28.5714 TOKENS_PER_SAMPLE: 152.0 gsm8k_accuracy: 100.0 mbxp_accuracy: 0.0 1 AMD Ryzen 9 7950X 16-Core Processor 1 16 nan 0 open/MLCommons/results/gh_action-reference-cpu-pytorch_v2.6.0-default_config/mixtral-8x7b/offline pytorch v2.6.0 Ubuntu 22.04 (linux-6.8.0-52-generic-glibc2.35) nan 1 0 v5.0 0 False Tokens/s fp32
12 MLCommons available open datacenter gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.0-default_config rgat rgat Offline 9.77098 acc: 75.800 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-pytorch_v2.4.0-default_config/rgat/offline pytorch v2.4.0 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
13 MLCommons available open edge gh_macos-latest_x86 gh_macos-latest_x86-reference-cpu-pytorch_v2.6.0-default_config retinanet retinanet Offline 0.255097 mAP: 76.951 1 undefined 1 3 nan 0 open/MLCommons/results/gh_macos-latest_x86-reference-cpu-pytorch_v2.6.0-default_config/retinanet/offline pytorch v2.6.0 (darwin-23.6.0) nan 1 0 v5.0 0 False Samples/s fp32
14 MLCommons available open edge gh_macos-latest x86 gh_macos-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config resnet50 resnet Offline 8.9953 acc: 76.000 1 undefined 1 3 nan 0 open/MLCommons/results/gh_macos-latest_x86-mlcommons_cpp-cpu-onnxruntime-default_config/resnet50/offline onnxruntime (darwin-23.6.0) nan 1 0 v5.0 0 False Samples/s fp32
15 MLCommons available open edge default default-mlcommons_cpp-cpu-onnxruntime-default_config retinanet retinanet Offline 0.433643 mAP: 49.593 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/default-mlcommons_cpp-cpu-onnxruntime-default_config/retinanet/offline onnxruntime Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32
16 MLCommons available open edge gh_action gh_action-reference-gpu-pytorch_v2.6.0-cu124 gptj-99 gptj-99 Offline 47.3803 ROUGE1: 32.2581 ROUGE2: 6.6667 ROUGEL: 22.5806 GEN_LEN: 264 1 Intel(R) Xeon(R) w7-2495X 1 24 NVIDIA GeForce RTX 4090 2 open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.6.0-cu124/gptj-99/offline pytorch v2.6.0 Ubuntu 22.04 (linux-6.8.0-52-generic-glibc2.35) nan 1 0 v5.0 0 False Tokens/s fp32
17 MLCommons available open edge gh_action gh_action-reference-gpu-pytorch_v2.6.0-cu124 stable-diffusion-xl stable-diffusion-xl Offline 0.358519 CLIP_SCORE: 16.36890298128128 FID_SCORE: 237.82578979650336 1 Intel(R) Xeon(R) w7-2495X 1 24 NVIDIA GeForce RTX 4090 2 open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.6.0-cu124/stable-diffusion-xl/offline pytorch v2.6.0 Ubuntu 22.04 (linux-6.8.0-52-generic-glibc2.35) nan 1 0 v5.0 0 False Samples/s fp32
18 MLCommons available open edge gh_ubuntu-latest_x86 gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config bert-99 bert-99 Offline 6.23624 F1: 80.0 1 AMD EPYC 7763 64-Core Processor 1 2 nan 0 open/MLCommons/results/gh_ubuntu-latest_x86-reference-cpu-deepsparse_v1.8.0-default_config/bert-99/offline deepsparse v1.8.0 Ubuntu 24.04 (linux-6.8.0-1021-azure-glibc2.39) nan 1 0 v5.0 0 False Samples/s fp32