[tests] Fix Windows test summary always reporting 'All tests passed'#24772
[tests] Fix Windows test summary always reporting 'All tests passed'#24772rolfbjarne merged 3 commits intomainfrom
Conversation
Fix a copy-paste bug in create-windows-html-report.cs where both the success and failure branches wrote '# 🎉 All N tests passed 🎉'. The failure branch now generates a proper summary with <details>/<summary> tags listing failed tests, matching the format TestResults.psm1 expects. Also fix TestResults.psm1 to emit <details> before <summary> (correct HTML nesting order), and avoid including misleading success-format file content in the failure section when the result file has no failure details. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Fixes the Windows test report summary generation so failures are correctly reported (instead of always saying “All tests passed”), and aligns summary formatting with what the PowerShell report aggregator expects.
Changes:
- Update
create-windows-html-report.csto produce a failure summary using<details>/<summary>and list failed tests (or failed TRX files). - Fix
TestResults.psm1to emit valid HTML nesting (<details>before<summary>) and avoid echoing misleading “all tests passed” content in failure sections. - Update PowerShell tests to match the corrected output format.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tools/devops/automation/scripts/TestResults.psm1 | Fixes <details>/<summary> emission order and improves failure-section content when the summary file lacks failure details. |
| tools/devops/automation/scripts/TestResults.Tests.ps1 | Adjusts expected outputs to match the corrected HTML nesting and new failure messaging. |
| scripts/create-windows-html-report/create-windows-html-report.cs | Generates a proper failure-mode markdown summary with <details>/<summary> and failed-test listings for Windows TRX parsing. |
Comments suppressed due to low confidence (1)
scripts/create-windows-html-report/create-windows-html-report.cs:222
- The failure summary writes
trxName,test.Name, and the first line oftest.Messagedirectly into the markdown/HTML output. These values can contain characters like<,>,&, or newlines (\r\n) which can break the rendered report (and allow unintended HTML/markdown injection). Consider escaping/sanitizing these fields (and trimming\r) before appending them tosummaryContents.
foreach (var (trxName, test) in allFailedTests) {
var msg = string.IsNullOrEmpty (test.Message) ? "" : $": {test.Message.Split ('\n') [0]}";
summaryContents.AppendLine ($" * {trxName}/{test.Name}: {test.Outcome}{msg}");
}
scripts/create-windows-html-report/create-windows-html-report.cs
Outdated
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ [CI Build #303ae0d] Build passed (Build packages) ✅Pipeline on Agent |
✅ [PR Build #303ae0d] Build passed (Detect API changes) ✅Pipeline on Agent |
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
✅ [CI Build #303ae0d] Build passed (Build macOS tests) ✅Pipeline on Agent |
🔥 [CI Build #303ae0d] Test results 🔥Test results❌ Tests failed on VSTS: test results 0 tests crashed, 12 tests failed, 144 tests passed. Failures❌ monotouch tests (iOS)11 tests failed, 0 tests passed.Failed tests
|
🚀 [CI Build #303ae0d] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 156 tests passed 🎉 Tests counts✅ cecil: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download Pipeline on Agent |
Fix a copy-paste bug in create-windows-html-report.cs where both the
success and failure branches wrote '# 🎉 All N tests passed 🎉'.
The failure branch now generates a proper summary with
<details>/<summary>tags listing failed tests, matching the format TestResults.psm1 expects.
Also fix TestResults.psm1 to emit
<details>before<summary>(correctHTML nesting order), and avoid including misleading success-format file
content in the failure section when the result file has no failure details.