[pipeline/e2e] end-to-end-tests.ps1 should fail with failure test

[WangWeiLin-MV]

Fix end-to-end-tests.ps1, which should fail and stop with failure test case.

Change

• Script end-to-end-tests.ps1
  • Check test $LASTEXITCODE
• Script end-to-end-tests-prelude.ps1
  • Reset $LASTEXITCODE for Throw-IfNotFailed
• Cases
  • autocomplete.ps1
    • Rename to autocomplete-command.ps1
  • env-tools.ps1
    • Add Throw-IfNotFailed for non-windows
  • ci-verify-versions.ps1
    • Remove-Problem-Matchers for malformed port
• Pipeline
  • Update to action/checkout@4, actions/setup-node@v4, lukka/[email protected]
    • https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
  • Import module Pester for e2e tests

[WangWeiLin-MV]

Test

Example of un-expected pass before fixed

• https://github.com/microsoft/vcpkg-tool/actions/runs/11435724127/job/31811675237?pr=1518#annotation:14:22
  • 7755480

Example of failed as expected after fixed

• https://github.com/microsoft/vcpkg-tool/actions/runs/11455844701/job/31872711589#annotation:14:33
  • d9d0cba

Example of error message Unexpected character; expected property name in GitHub action page

• https://github.com/microsoft/vcpkg-tool/actions/runs/11450810448
• Now there is no other annotations except the deprecation warning for macos-12.

[BillyONeal]

Fix end-to-end-tests.ps1, which should fail and stop with failure test case.

As far as I can tell it already does this. Do you have a repro for it not doing so?

(EDIT: I see you have an example in a comment here but it's not clear how you got that to happen)

[BillyONeal]

It looks like the repro might only happen with something like this "Pester" module ( as in #1512 ) because all other existing e2e tests fail by throwing an exception, thus stopping further test runs.

[WangWeiLin-MV]

Fix end-to-end-tests.ps1, which should fail and stop with failure test case.

As far as I can tell it already does this. Do you have a repro for it not doing so?

(EDIT: I see you have an example in a comment here but it's not clear how you got that to happen)

Sorry I forgot to attach the test commit 7755480

Run this script with $ErrorActionPreference = "Stop", will get status $? to False and $LASTEXITCODE to 2.

BeforeAll {
    Import-Module $PSScriptRoot/not-exist/posh-vcpkg
}
Describe 'Module posh-vcpkg tests' {
    It 'Expect fail' {}
}

This issue has been fixed by:

• Check $lastTextExitCode = $LASTEXITCODE
• Reset $LASTEXITCODE for Throw-IfNotFailed

[WangWeiLin-MV]

I'm now trying to fix the error message Unexpected character; expected property name in GitHub action page, for example https://github.com/microsoft/vcpkg-tool/actions/runs/11450810448

Update: Fixed by adding Remove-Problem-Matchers and Restore-Problem-Matchers

---

P.S.

The command vcpkg x-ci-verify-versions return 0 and output error message to stdout instead of stderr for azure-pipelines\e2e-assets\ci-verify-versions-registry\ports\malformed\vcpkg.json.

[WangWeiLin-MV]

Unable change runner machine runs-on from macos-12 to macos-14, even there aren't any build macos-12 string in the repository.

It hangs with PR check:

builds / build (macos-12, macos-ci) Expected — Waiting for status to be reported

---

Update:

It might be waiting the branch protection Rulesets - Require status checks to pass, changing machine caused the job name to change.

As a transitional, add arm64 macos-14 and x86_64 macos-14-large job as an additional job instead of renaming the existing one.

https://github.com/actions/runner-images?tab=readme-ov-file#available-images

---

Update 2:

• [macOS] The macOS 12 Actions runner image will begin deprecation on 10/7/24 and will be fully unsupported by 12/3/24 for GitHub and ADO actions/runner-images#10721
• Change to macOS CI runs from macOS-12 to macOS-13 #1536

[WangWeiLin-MV]

Note

vmImage: macOS-12 is deprecated in azure-pipelines/signing.yml:208

https://devblogs.microsoft.com/devops/upcoming-deprecation-of-macos-12-hosted-pipeline-image/