Rearrange driver prints #4158

eddieliao · 2025-07-21T23:44:42Z

Swaps the read and compile prints so that read will come first. Also adds additional prints for quantization and for precompiled programs.
e.g. Old:

Running [ MIGraphX Version: 2.13.0.fc8f6ffee-dirty ]: ./build/bin/driver run /code/nas/migraphx/models/resnet50-v2-7.onnx
[2025-07-21 23:35:05]
Compiling ...
Reading: /code/nas/migraphx/models/resnet50-v2-7.onnx

module: "main"
@0 = check_context::migraphx::gpu::context -> float_type, {}, {}
@1 = hip::hip_allocate_memory[shape=int8_type, {268443648}, {1},id=main:scratch] -> int8_type, {268443648}, {1}
@2 = hip::hip_copy_literal[id=main:@literal:72] -> float_type, {64, 3, 7, 7}, {147, 49, 7, 1}
@3 = hip::hip_copy_literal[id=main:@literal:71] -> float_type, {1, 64, 112, 112}, {1, 12544, 112, 1}
...

New:

Running [ MIGraphX Version: 2.13.0.fc8f6ffee-dirty ]: ./build/bin/driver run /code/nas/migraphx/models/resnet50-v2-7.onnx
[2025-07-21 23:35:05]
Reading: /code/nas/migraphx/models/resnet50-v2-7.onnx
Compiling ... 

module: "main"
@0 = check_context::migraphx::gpu::context -> float_type, {}, {}
@1 = hip::hip_allocate_memory[shape=int8_type, {268443648}, {1},id=main:scratch] -> int8_type, {268443648}, {1}
@2 = hip::hip_copy_literal[id=main:@literal:72] -> float_type, {64, 3, 7, 7}, {147, 49, 7, 1}
@3 = hip::hip_copy_literal[id=main:@literal:71] -> float_type, {1, 64, 112, 112}, {1, 12544, 112, 1}

For precompiled models:
Old:

Running [ MIGraphX Version: 2.13.0.fc8f6ffee-dirty ]: ./build/bin/driver run mnist.mxr --fp16
[2025-07-21 23:38:16]
Reading: mnist.mxr
Allocating params ... 
module: "main"
@0 = check_context::migraphx::gpu::context -> float_type, {}, {}
@1 = hip::hip_allocate_memory[shape=int8_type, {31360}, {1},id=main:scratch] -> int8_type, {31360}, {1}
@2 = hip::hip_copy_literal[id=main:@literal:3] -> float_type, {8, 1, 1}, {1, 1, 1}
...

New:

Running [ MIGraphX Version: 2.13.0.fc8f6ffee-dirty ]: ./build/bin/driver run mnist.mxr --fp16
[2025-07-21 23:38:16]
Reading: mnist.mxr
Program is already compiled, skipping compilation ...
[WARNING]: Quantization options are ignored as program is already compiled.
Allocating params ... 
module: "main"
@0 = check_context::migraphx::gpu::context -> float_type, {}, {}
@1 = hip::hip_allocate_memory[shape=int8_type, {31360}, {1},id=main:scratch] -> int8_type, {31360}, {1}
@2 = hip::hip_copy_literal[id=main:@literal:3] -> float_type, {8, 1, 1}, {1, 1, 1}
...

Copilot

Pull Request Overview

This PR improves the logging output order and messaging for the MIGraphX driver by reorganizing print statements to show a more logical sequence of operations. The changes ensure that "Reading" messages appear before compilation messages and add informative output for quantization operations and precompiled program handling.

Moved compilation print statements from command functions into the compiler logic for better flow
Added quantization progress messages for each quantization type (fp16, bf16, int8, fp8, int4)
Added informative messages for precompiled programs including skip notification and quantization warnings

src/driver/main.cpp

pfultz2 · 2025-07-22T14:28:04Z

I think this looks good, when are you going to move it from draft to open?

eddieliao · 2025-07-22T16:28:47Z

I think this looks good, when are you going to move it from draft to open?

Just moved to open, wanted to make sure none of it was failing CI first.

lakhinderwalia

Approved. Minor nit-picking that you can optionally address. Thanks.

src/driver/main.cpp

migraphx-bot · 2025-07-22T23:30:37Z

Test	Batch	Rate new 729f63	Rate old 408b98	Diff	Compare
torchvision-resnet50	64	3,250.28	3,249.84	0.01%	✅
torchvision-resnet50_fp16	64	6,931.91	6,927.96	0.06%	✅
torchvision-densenet121	32	2,450.14	2,450.72	-0.02%	✅
torchvision-densenet121_fp16	32	4,180.45	4,199.41	-0.45%	✅
torchvision-inceptionv3	32	1,636.90	1,638.39	-0.09%	✅
torchvision-inceptionv3_fp16	32	2,752.07	2,755.49	-0.12%	✅
cadene-inceptionv4	16	771.65	772.10	-0.06%	✅
cadene-resnext64x4	16	814.61	818.89	-0.52%	✅
slim-mobilenet	64	7,467.73	7,467.98	-0.00%	✅
slim-nasnetalarge	64	211.17	211.20	-0.02%	✅
slim-resnet50v2	64	3,340.57	3,342.30	-0.05%	✅
bert-mrpc-onnx	8	1,148.98	1,148.63	0.03%	✅
bert-mrpc-tf	1	457.20	457.81	-0.13%	✅
pytorch-examples-wlang-gru	1	337.95	342.34	-1.28%	✅
pytorch-examples-wlang-lstm	1	475.68	466.93	1.87%	✅
torchvision-resnet50_1	1	795.81	793.10	0.34%	✅
cadene-dpn92_1	1	414.87	413.32	0.38%	✅
cadene-resnext101_1	1	388.52	392.78	-1.08%	✅
onnx-taau-downsample	1	396.77	396.46	0.08%	✅
dlrm-criteoterabyte	1	33.76	33.77	-0.04%	✅
dlrm-criteoterabyte_fp16	1	51.21	51.15	0.12%	✅
agentmodel	1	10,231.47	10,131.48	0.99%	✅
unet_fp16	2	60.74	60.68	0.10%	✅
resnet50v1_fp16	1	1,032.61	1,037.89	-0.51%	✅
resnet50v1_int8	1	1,047.83	1,046.81	0.10%	✅
bert_base_cased_fp16	64	1,170.17	1,169.82	0.03%	✅
bert_large_uncased_fp16	32	361.35	361.33	0.01%	✅
bert_large_fp16	1	203.75	203.50	0.12%	✅
distilgpt2_fp16	16	2,240.04	2,239.25	0.04%	✅
yolov5s	1	534.46	540.44	-1.11%	✅
tinyllama	1	43.96	44.01	-0.11%	✅
vicuna-fastchat	1	45.15	45.32	-0.38%	✅
whisper-tiny-encoder	1	419.23	419.11	0.03%	✅
whisper-tiny-decoder	1	411.92	402.75	2.28%	✅
llama2_7b	1	19.18	19.18	0.01%	✅
qwen1.5-7b	1	23.68	23.65	0.12%	✅
phi3-3.8b	1	26.86	26.86	0.00%	✅
mask-rcnn	1	12.79	12.82	-0.19%	✅
llama3-8b	1	21.84	21.84	0.02%	✅
whisper-large-encoder	1	10.22	10.22	0.01%	✅
whisper-large-decoder	1	104.51	103.52	0.96%	✅
mistral-7b	1	23.86	23.83	0.09%	✅
FLUX.1-schnell	1	766.71	767.72	-0.13%	✅
nan	nan	nan	nan	nan%	❌

This build is not recommended to merge 🔴

migraphx-bot · 2025-07-22T23:30:39Z

✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

❌bert-mrpc-tf: ERROR - check error output

error: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

error: unknown warning option '-Wnrvo' [-Werror,-Wunknown-warning-option]

2025-07-22 16:55:50.306639: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1753221355.764048 181826 gpu_device.cc:2022] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 62973 MB memory: -> device: 0, name: AMD Instinct MI250X/MI250, pci bus id: 0000:b3:00.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1753221356.664830 181826 mlir_graph_optimization_pass.cc:401] MLIR V1 optimization pass is not enabled
2025-07-22 16:56:05.510017: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510069: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510118: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510167: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510217: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510266: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510316: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
2025-07-22 16:56:05.510346: E external/local_xla/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:250] bitcode module is required by this HLO module but was not found at ./opencl.bc
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
error: Failure when generating HSACO
2025-07-22 16:56:05.511506: E tensorflow/compiler/mlir/tools/kernel_gen/tf_framework_c_interface.cc:228] INTERNAL: Generating device code failed.
2025-07-22 16:56:05.512650: W tensorflow/core/framework/op_kernel.cc:1829] UNKNOWN: JIT compilation failed.
2025-07-22 16:56:05.512672: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
2025-07-22 16:56:05.512682: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
2025-07-22 16:56:05.512700: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 11217777527359497193
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1407, in _do_call
return fn(*args)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1390, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1483, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 335, in main
y_out = sess.run(y, feed_dict=tf_dict)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 977, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1220, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1400, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/client/session.py", line 1426, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:

Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
Detected at node 'import/bert/embeddings/LayerNorm/moments/SquaredDifference' defined at (most recent call last):
Node: 'import/bert/embeddings/LayerNorm/moments/SquaredDifference'
2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
[[import/loss/output/_21]]
(1) UNKNOWN: JIT compilation failed.
[[{{node import/bert/embeddings/LayerNorm/moments/SquaredDifference}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'import/bert/embeddings/LayerNorm/moments/SquaredDifference':

✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

✅ agentmodel: PASSED: MIGraphX meets tolerance

🔴unet: FAILED: MIGraphX is not within tolerance - check verbose output

✅ resnet50v1: PASSED: MIGraphX meets tolerance

✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

✅ bert_large: PASSED: MIGraphX meets tolerance

✅ yolov5s: PASSED: MIGraphX meets tolerance

✅ tinyllama: PASSED: MIGraphX meets tolerance

✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

✅ llama2_7b: PASSED: MIGraphX meets tolerance

✅ qwen1.5-7b: PASSED: MIGraphX meets tolerance

✅ phi3-3.8b: PASSED: MIGraphX meets tolerance

🔴mask-rcnn: FAILED: MIGraphX is not within tolerance - check verbose output

✅ llama3-8b: PASSED: MIGraphX meets tolerance

✅ whisper-large-decoder: PASSED: MIGraphX meets tolerance

✅ mistral-7b: PASSED: MIGraphX meets tolerance

✅ FLUX.1-schnell: PASSED: MIGraphX meets tolerance

eddieliao added 2 commits July 21, 2025 23:32

Swap read and compile prints, add additional prints for quantization

be6eefc

Formatting

ff8d26d

eddieliao self-assigned this Jul 21, 2025

eddieliao added enhancement New feature or request simple small or simple changes labels Jul 21, 2025

eddieliao requested a review from Copilot July 21, 2025 23:45

Copilot AI reviewed Jul 21, 2025

View reviewed changes

src/driver/main.cpp Show resolved Hide resolved

eddieliao marked this pull request as ready for review July 22, 2025 16:27

eddieliao requested a review from causten as a code owner July 22, 2025 16:27

eddieliao requested review from pfultz2 and lakhinderwalia July 22, 2025 16:27

lakhinderwalia approved these changes Jul 22, 2025

View reviewed changes

src/driver/main.cpp Outdated Show resolved Hide resolved

src/driver/main.cpp Outdated Show resolved Hide resolved

src/driver/main.cpp Outdated Show resolved Hide resolved

eddieliao added 2 commits July 22, 2025 17:06

Change wording in response to feedback

cc634b8

Formatting

729f63b

pfultz2 approved these changes Jul 22, 2025

View reviewed changes

causten merged commit 01aa6aa into develop Jul 23, 2025
47 of 50 checks passed

causten deleted the rearrange_driver_prints branch July 23, 2025 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rearrange driver prints #4158

Rearrange driver prints #4158

Uh oh!

eddieliao commented Jul 21, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

pfultz2 commented Jul 22, 2025

Uh oh!

eddieliao commented Jul 22, 2025

Uh oh!

lakhinderwalia left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

migraphx-bot commented Jul 22, 2025

Uh oh!

migraphx-bot commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Rearrange driver prints #4158

Rearrange driver prints #4158

Uh oh!

Conversation

eddieliao commented Jul 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

pfultz2 commented Jul 22, 2025

Uh oh!

eddieliao commented Jul 22, 2025

Uh oh!

lakhinderwalia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

migraphx-bot commented Jul 22, 2025

Uh oh!

migraphx-bot commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!