feat(package-tests): add skip input

[williamboman]

No description provided.

[williamboman]

@mason-org/triage Sorry for the ping but what do yall think? This plus mason-org/mason-registry#9861 would enable mason-org/mason-registry#9774 which in turn would unlock the Mason v2 release

[chrisgrieser]

There are a few PRs which are stuck due to tests wrongly failing, so having a method to skip tests if need be makes sense. The only potential issue is people abusing it to get their PRs skip the tests, but I think in the rare case that happens, we would notice it in the "Files changed" overview, so I don't think it's a big concern.

[williamboman]

There are a few PRs which are stuck due to tests wrongly failing, so having a method to skip tests if need be makes sense.

Hm in my experience test failures actually mean something did go wrong. There are cases where installation requires external system dependencies (especially true for build sources which are semi-deprecated for this reason), but it's kind of rare. A common issue is that Renovate creates PRs before assets have been fully uploaded upstream, in which case re-running the workflow fixes things. This could probably be fixed by automating the retry mechanism, for example labeling on first failure and have a hourly scheduled job that retries labeled PRs that are older than x minutes.

The only potential issue is people abusing it to get their PRs skip the tests

Not sure if things have changed but afaik only maintainers can add labels to PRs, no?

[chrisgrieser]

yeah, there were some stray cases where we couldn't really figure out what's causing the tests to fail, e.g. mason-org/mason-registry#9598

Not sure if things have changed but afaik only maintainers can add labels to PRs, no?

oh, I see, thought this was about adding some key to a yaml file like with ci-skip. GitHub actions do not seem to be very verbose in that matter, wasn't apparent to me that those are labels. In that case, my minor concern is moot of course.

[williamboman]

yeah, there were some stray cases where we couldn't really figure out what's causing the tests to fail, e.g. mason-org/mason-registry#9598

Hmm.. I recognize that error, I think the issue is that the branch is too out-of-date with the base branch. The error doesn't make much sense to me but updating the branch should fix it

oh, I see, thought this was about adding some key to a yaml file like with ci-skip. GitHub actions do not seem to be very verbose in that matter, wasn't apparent to me that those are labels. In that case, my minor concern is moot of course.

No no skipping CI permanently like that is a last resort. This is the reusable workflow which now allows an optional skip input. Registries that uses this workflow can then decide how they want to use this input (doesn't necessarily have to be through labels, but that's how it's implemented in the other PR)