Skip build and test on OSes that have no changes

[jonahgraham]

Many of the PRs on SWT are OS specific, this change only runs the build on OSes if there are changes that can affect that OS.

This is going to be more important with #2714 because potentially a number of new jobs will be run per PR and avoiding some of the extra time waiting will be a benefit to all.

The junit.yml has been updated to record tests per platform in addition to all tests. When not all platforms are run in a PR, only the respective platform results are reported as a comment. If all platforms are run, the cumulative results are reported as a comment.

Part of #2714

[github-actions]

Test Results

  147 files  ±0    147 suites  ±0   20m 9s ⏱️ - 1m 18s
4 670 tests ±0  4 648 ✅ ±0  22 💤 ±0  0 ❌ ±0 
  411 runs  ±0    406 ✅ ±0   5 💤 ±0  0 ❌ ±0 

Results for commit df9e5bb. ± Comparison against base commit 13d7370.

♻️ This comment has been updated with latest results.

[HeikoKlare]

I totally like the idea of skipping tests for OSes that are not affected by a change. I am just a bit concerned about the condensed test results posted by the bot being rendered less useful. Maybe we could check if we can adapt that the results to be separated by OS, such that you can easily and properly compare for the OS(es) of interest for the current PR.

Edit: just saw that Christoph already made a similar comment: #2762 (comment)

[jonahgraham]

I am going to split this PR out from the rest of them for #2741

[laeubi]

It shows number of tests/runs down compared to master because so many tests were skipped, but it still shows 0 failures as expected. Further down you can see the non-win32 runs are skipped

As mentioned on the other PR (sorry for confusion!) I think we then better should have three check runs for all three oss to prevent the problem you see with the test drops and then have one comment for each os e.g.

• Test Results GTK
• Test Results Win32
• Test Results Cocoa

beside that I'm wondering if it really pay off much to skip the tests, the Jenkins if often the slowest one and Github has completed most often much earlier anyways.

Also I'm wondering what happens with changes to the common parts of SWT that can change behavior on other OS of course as well.

[HeikoKlare]

Also I'm wondering what happens with changes to the common parts of SWT that can change behavior on other OS of course as well.

From my understanding, there is a classification for those changes that enables all OS builds via needs.changes.outputs.other == 'true'

beside that I'm wondering if it really pay off much to skip the tests, the Jenkins if often the slowest one and Github has completed most often much earlier anyways.

Even though this is often the case, in my experience "often" rather means 60% of the cases than 90%, so avoiding unnecessary GHA runs will my in opinion still increase the average roundtrip time. And I see at least two further reasons to do this:

• Flakyness: we repeatedly run into the situation that some builds start to get flaky for different reasons; reducing the number of (unnecessary) builds that can prevent you from making progress (both in moving actual changes forward as well as in investigating the reason for builds not working properly) is beneficial.
• Resources: why should we utilize resources when we definitely know that the results are useless? It consumers material and energy without any benefit. You can argue that SWT builds are comparably small such that this does not pay off, but it's at least an additional point in combination with the other arguments.

[jonahgraham]

@laeubi / @HeikoKlare thanks for your input - I agree and will see if I have time to bring this to a conclusion.

Also I'm wondering what happens with changes to the common parts [...]

From my understanding, [...]

Yes, that is correct, anything that isn't in the platform specific changes (e.g **/win32/**) will mean everything gets run. What this means is any PR that includes/updates a test, snippet, or example change will cause all platforms to run.

[jonahgraham]

I have now implemented test result reporting based on what has run. You can see what it will look like once this change is merged in the 4 examples I provide in my fork:

Single platform only

• Conditional run example - win32 only edit jonahgraham/eclipse.platform.swt#26

Only the platform that was run generates a comment.

Common code (or any change that triggers all the platforms)

• Conditional run example - common code edit jonahgraham/eclipse.platform.swt#27

Nothing is skipped in this case. Since linux + macos are matrix jobs, the total number of matrix jobs run is 6 (will be more once #2768 is merged). The comment added to the PR is the summary of all the jobs.

If you click through the results you can also see the results of the individual platforms. This is especially useful when reviewing skipped jobs and a few of the other annotations.

2 platforms change

• Conditional run example - win32 and macos code edit jonahgraham/eclipse.platform.swt#28

In this case the 3rd platform SWT Matrix job is skipped. This means that two comments are created, for the two platforms that did run.

Only markdown changed (or anything in a category with no associated jobs)

• Conditional run example - markdown only edit jonahgraham/eclipse.platform.swt#29

In this case all the SWT Matrix jobs are skipped and no test results are reported.

[jonahgraham]

4 examples I provide in my fork:

Actually 5 - I realized I missed an important case, what failures look like.

How errors report

In this example PR I have a ridiculous failures that throws an exception in the Display constructor meaning almost everything fails. This is what that looks like.

• Conditional run example - win32 only with test failures jonahgraham/eclipse.platform.swt#30

Test report diff to base commit

When I made the screenshots in #2764 (comment) the base commit had not finished running yet, so they were not showing diffs to base commit. I updated all the linked example PRs, but I am only redoing this screenshot:

[jonahgraham]

This is now ready for review.

[jonahgraham]

I plan to merge this soon if there are no objections.

[HeikoKlare]

Thank you for these improvements, @jonahgraham!

I had no chance to dig deeper into this change yet, unfortunately. But as mentioned earlier already, I really like the concept for multiple reasons (focused validation results with reduced impact of flaky tests, reduced resource consumption, etc.).
And it's great to have all your examples to understand how it will look like and to be confident that it will work as expected. I also like the isall mode in case every platform is affected, such that you will not get three reports but just a single one then.

I just scanned the PR and at a glance it looks fine. And there is also no risk in this, as in the important case that tests fails, the GitHub Actions executing them will fail as well. So in case there is something that does not work perfectly fine yet, we will probably notice soon and can easily adapt.

Looking forward to see this in production.

[jonahgraham]

This is now merged. Thanks @HeikoKlare for the review - and hopefully it will just work so you never need to dig in deeper than you have already.

Once the master builds and existing PRs rebase to include this change then we should see the fruits of this labour. FWIW I did check to make sure there was likely a benefit, and there were plenty of commits to SWT that only changed one platform.