[ui, ci] retain artifacts from test runs including test timing #24555

philrenaud · 2024-11-26T15:10:12Z

Description

Our ember-test-audit action is currently failing, and I'm using that as an excuse to improve our front-end testing story generally.

This will be a two-step approach, and this PR is the first step:

Save artifacts, including test timing, from test-ui test results on merge to main
Compare to a rolling window of these test results on subsequent PRs

This PR adds artifact uploads on the back of test-ui. Because those are partitioned/run concurrently 4x, part of this is combining those.

The artifact generated looks like this:

{
  "timestamp": "2024-12-02T18:20:59.531Z",
  "sha": "7fd10531ef3e12672a19a7a9aa7a6e4f4f874614",
  "summary": {
    "total": 1606,
    "passed": 1606,
    "failed": 0
  },
  "duration": 841403,
  "tests": [
    {
      "name": "Acceptance | allocation fs: it passes an accessibility audit",
      "partition": 1,
      "browser": "Chrome 131.0",
      "passed": true,
      "duration": 1331,
      "error": null,
      "logs": []
    },
    {
      "name": "Acceptance | allocation fs: visiting filesystem root",
      "partition": 1,
      "browser": "Chrome 131.0",
      "passed": true,
      "duration": 505,
      "error": null,
      "logs": []
    },
    {
      "name": "Acceptance | allocation fs: visiting filesystem paths",
      "partition": 1,
      "browser": "Chrome 131.0",
      "passed": true,
      "duration": 542,
      "error": null,
      "logs": []
    },
...

In phase 2 of this, we'll be able to do test-by-test duration averages and comparisons, and tease out those tests with the greatest deltas, find flakes, etc.

.github/workflows/test-ui.yml

philrenaud · 2024-11-26T15:38:05Z

ui/app/index.html

@@ -24,6 +24,7 @@
    {{content-for "body"}}

    <script src="{{rootURL}}assets/vendor.js"></script>
+


no-op to trigger the

pull_request: paths: - "ui/**"

part of the action.

github-actions · 2024-11-26T15:49:07Z

Ember Test Audit comparison

	main	`41c9725`	change
passes	1580	1580	0
failures	5	5	0
flaky	0	0	0
duration	000ms	000ms	-000ms

.github/workflows/test-ui.yml

tgross

LGTM

.github/workflows/test-ui.yml

tgross

LGTM!

DingoEatingFuzz

Super cool to see this work get picked up again 🤩

DingoEatingFuzz · 2024-12-03T18:06:20Z

scripts/combine-ui-test-results.js

+  for (let i = 1; i <= NUM_PARTITIONS; i++) {
+    try {
+      const data = JSON.parse(
+        fs.readFileSync(`../test-results/test-results-${i}/test-results.json`).toString()
+      );


Will there be anything else in the test-results directory other than subdirectories for each partition?

I wonder if you can walk the dir tree instead of relying on synchronizing NUM_PARTITIONS and the GHA matrix config.

Yeah, I left a pretty verbose dir trail here with the hope that I'd populate this with further artifacts / test-run metadata. No explicit plans yet, though!

DingoEatingFuzz · 2024-12-03T18:07:14Z

scripts/combine-ui-test-results.js

+    tests: results
+  };
+
+  fs.writeFileSync('../ui/combined-test-results.json', JSON.stringify(output, null, 2));


Doing a combine step that takes json and emits json is really nice. Should prove to be more flexible than trying to do all the collating, aggregating, and analysis in one script.

* retain artifacts from test runs including test timing * Pinning commit hashes for action helpers * trigger for ui-test run * Trying to isolate down to a simple upload * Once more with mkdir * What if we just wrote our own test reporter tho * Let the partitioned runs handle placement * Filter out common token logs, add a summary at the end, and note failures in logtime * Custom reporter cannot also have an output file, he finds out two days late * Aggregate summary, duration, and removing failure case * Conditional test report generation * Timeouts are errors * Trying with un-partitioned input json file * Remove the commented-out lines for main-only runs * combine-ui-test-results as its own script

…release/1.9.x (#25177) * no-op commit due to failed cherry-picking * [ui, ci] retain artifacts from test runs including test timing (#24555) * retain artifacts from test runs including test timing * Pinning commit hashes for action helpers * trigger for ui-test run * Trying to isolate down to a simple upload * Once more with mkdir * What if we just wrote our own test reporter tho * Let the partitioned runs handle placement * Filter out common token logs, add a summary at the end, and note failures in logtime * Custom reporter cannot also have an output file, he finds out two days late * Aggregate summary, duration, and removing failure case * Conditional test report generation * Timeouts are errors * Trying with un-partitioned input json file * Remove the commented-out lines for main-only runs * combine-ui-test-results as its own script * Remove the Ember Test Audit workflow (#24637) * [ui, tests] Various acceptance test fixups (v main) (#25031) * Add factory hooks for jobs to have previously stable versions and stopped status * Since #24973 node-read isn't presupposed and so should regex match only on the common url parts * Job detail tests for title buttons are now bimodal and default to having previously-stable version in history * prettier plz * Breaking a thing on purpose to see if my other broken thing is broken * continue-on-error set to false to get things red when appropriate * OK what if continue-on-error=true but we do a separate failure reporting after the fact * fail-fast are you the magic incantation that I need? * Re-fix my test now that fast-fail is off * Fix to server-leader by adding a region first, and always()-append to uploading partition results * Express failure step lists failing tests so you don't have to click back into ember-exam step * temporary snapshot and logging for flakey test in service job detail * Bunch of region and tasklogs test fixups * using allocStatusDistribution to ensure service job always has a non-queued alloc * Manually remove ember test audit (re-do #24637) --------- Co-authored-by: temp <[email protected]> Co-authored-by: Phil Renaud <[email protected]> Co-authored-by: Phil Renaud <[email protected]>

github-actions · 2025-04-03T02:18:07Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

github-advanced-security bot found potential problems Nov 26, 2024

View reviewed changes

.github/workflows/test-ui.yml Fixed Show fixed Hide fixed

.github/workflows/test-ui.yml Fixed Show fixed Hide fixed

.github/workflows/test-ui.yml Fixed Show fixed Hide fixed

retain artifacts from test runs including test timing

acdd210

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 73d86fa to acdd210 Compare November 26, 2024 15:15

vercel bot deployed to Preview – nomad-ui November 26, 2024 15:16 View deployment

Pinning commit hashes for action helpers

243162b

vercel bot deployed to Preview – nomad-ui November 26, 2024 15:34 View deployment

philrenaud marked this pull request as ready for review November 26, 2024 15:36

philrenaud requested review from a team as code owners November 26, 2024 15:36

philrenaud commented Nov 26, 2024

View reviewed changes

.github/workflows/test-ui.yml Outdated Show resolved Hide resolved

philrenaud commented Nov 26, 2024

View reviewed changes

trigger for ui-test run

71b073d

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 52e500b to 71b073d Compare November 26, 2024 16:06

vercel bot deployed to Preview – nomad-ui November 26, 2024 16:07 View deployment

philrenaud marked this pull request as draft November 26, 2024 18:56

Trying to isolate down to a simple upload

77ce4ef

vercel bot deployed to Preview – nomad-ui November 26, 2024 18:59 View deployment

vercel bot deployed to Preview – nomad-ui November 26, 2024 19:59 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 0729338 to 3391750 Compare November 26, 2024 20:05

vercel bot deployed to Preview – nomad-ui November 26, 2024 20:06 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 3391750 to fef9738 Compare November 26, 2024 20:11

vercel bot deployed to Preview – nomad-ui November 26, 2024 20:12 View deployment

github-advanced-security bot found potential problems Nov 26, 2024

View reviewed changes

.github/workflows/test-ui.yml Fixed Show fixed Hide fixed

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from fef9738 to 73f3a32 Compare November 26, 2024 20:21

vercel bot deployed to Preview – nomad-ui November 26, 2024 20:22 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 73f3a32 to 8f358e6 Compare November 26, 2024 20:26

vercel bot deployed to Preview – nomad-ui November 26, 2024 20:27 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 8f358e6 to 699ed1d Compare November 26, 2024 20:36

vercel bot deployed to Preview – nomad-ui November 26, 2024 20:38 View deployment

tgross previously approved these changes Dec 2, 2024

View reviewed changes

.github/workflows/test-ui.yml Outdated Show resolved Hide resolved

philrenaud dismissed tgross’s stale review via c4dc4c8 December 2, 2024 19:47

vercel bot deployed to Preview – nomad-ui December 2, 2024 19:48 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from c4dc4c8 to 8f81bf8 Compare December 2, 2024 20:02

vercel bot deployed to Preview – nomad-ui December 2, 2024 20:03 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 8f81bf8 to 0603cab Compare December 2, 2024 20:57

vercel bot deployed to Preview – nomad-ui December 2, 2024 20:58 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 0603cab to 040060f Compare December 2, 2024 21:17

vercel bot deployed to Preview – nomad-ui December 2, 2024 21:19 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from 040060f to e9bf68a Compare December 2, 2024 21:34

vercel bot deployed to Preview – nomad-ui December 2, 2024 21:35 View deployment

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from e9bf68a to ebfcad6 Compare December 2, 2024 21:46

vercel bot deployed to Preview – nomad-ui December 2, 2024 21:47 View deployment

combine-ui-test-results as its own script

2adc472

philrenaud force-pushed the b-ui/ember-audit-test-artifacts branch from ebfcad6 to 2adc472 Compare December 2, 2024 22:04

vercel bot deployed to Preview – nomad-ui December 2, 2024 22:05 View deployment

philrenaud requested a review from tgross December 3, 2024 14:28

tgross approved these changes Dec 3, 2024

View reviewed changes

philrenaud merged commit 4b91c17 into main Dec 3, 2024
34 of 37 checks passed

philrenaud deleted the b-ui/ember-audit-test-artifacts branch December 3, 2024 14:56

DingoEatingFuzz reviewed Dec 3, 2024

View reviewed changes

This was referenced Dec 4, 2024

[ui, ci] Ember Exam failures should cause overall test failures #24603

Closed

[ui, ci] UI test look-back and comparison for PRs #24604

Draft

Remove the Ember Test Audit workflow #24637

Merged

philrenaud mentioned this pull request Feb 7, 2025

[ui, tests] Various acceptance test fixups (v main) #25031

Merged

1 task

hc-github-team-nomad-core mentioned this pull request Feb 20, 2025

Backport of [ui, tests] Various acceptance test fixups (v main) into release/1.9.x #25177

Merged

1 task

github-actions bot locked as resolved and limited conversation to collaborators Apr 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ui, ci] retain artifacts from test runs including test timing #24555

[ui, ci] retain artifacts from test runs including test timing #24555

Uh oh!

philrenaud commented Nov 26, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

philrenaud Nov 26, 2024

Uh oh!

github-actions bot commented Nov 26, 2024 •

edited

Loading

Uh oh!

Uh oh!

tgross left a comment

Uh oh!

Uh oh!

tgross left a comment

Uh oh!

Uh oh!

DingoEatingFuzz left a comment

Uh oh!

DingoEatingFuzz Dec 3, 2024

Uh oh!

philrenaud Dec 3, 2024

Uh oh!

DingoEatingFuzz Dec 3, 2024

Uh oh!

github-actions bot commented Apr 3, 2025

Uh oh!

Uh oh!

		@@ -24,6 +24,7 @@
		{{content-for "body"}}

		<script src="{{rootURL}}assets/vendor.js"></script>

[ui, ci] retain artifacts from test runs including test timing #24555

[ui, ci] retain artifacts from test runs including test timing #24555

Uh oh!

Conversation

philrenaud commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

philrenaud Nov 26, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ember Test Audit comparison

Uh oh!

Uh oh!

tgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DingoEatingFuzz left a comment

Choose a reason for hiding this comment

Uh oh!

DingoEatingFuzz Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

philrenaud Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

DingoEatingFuzz Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 3, 2025

Uh oh!

Uh oh!

philrenaud commented Nov 26, 2024 •

edited

Loading

github-actions bot commented Nov 26, 2024 •

edited

Loading