slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1265196	dotnet/maui	Controls CollectionView	Log
1265455	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65213
1265300	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#123569
1265250	dotnet/runtime	osx-arm64 release Installer_Build_And_Test	Log	dotnet/runtime#120769
1265342	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123313
1265269	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
2886365	dotnet-sdk	Darwin	Log	#57132
2886504	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 8.0	Log
1265232	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#123568
1265195	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log
2886464	dotnet-maui	Controls (v18.4) WebView
1265063	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#123412
2886463	dotnet-maui	RunOniOS_MauiRelease macOS-arm64	Log
2886454	dotnet-maui	Controls (v18.4) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible
1265143	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log	dotnet/maui#33693
2886391	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 9.0	Log
2886378	dotnet-performance	Performance scenarios ubuntu 2204 x64 Tiger 10.0	Log
2886366	dotnet-sdk	Darwin	Log	#57133
1264989	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2886282	dotnet-maui	Controls Layout	Log
1264843	dotnet/sdk	🍎 dotnetup tests: macOS (osx.15.amd64.open)	Log	dotnet/sdk#52649
2886270	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger main	Log
1264758	dotnet/maui	Controls Layout	Log
1264720	dotnet/maui	Build macOS-arm64		dotnet/maui#33152
1264727	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log
2886099	dotnet-maui	Controls (v18.4) Page,Performance,Picker,ProgressBar
1264714	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1264560	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log
1264310	dotnet/maui	Controls CollectionView	Log
1264235	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1264453	dotnet/maui	Controls Layout	Log
2886042	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger main	Log
1264338	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log	dotnet/maui#33432
1264336	dotnet/maui	Controls WebView	Log	dotnet/maui#33590
1264309	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log
1264325	dotnet/maui	Controls Layout	Log	dotnet/maui#27847
2885819	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1263931	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#121343
1263932	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123521
1263787	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
1263830	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log	dotnet/maui#33552
1263397	dotnet/maui	Controls Layout	Log
2885708	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger main	Log
1263409	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2885681	dotnet-maui	Controls (v18.4) Accessibility,ActionSheet,ActivityIndicator,Animation,AppLinks
1263623	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#63813
1263525	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2885643	dotnet-aspire	osx_arm64	Log
2885607	dotnet-maui	Controls (v18.4) Shell
1263011	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33632
1263427	dotnet/aspnetcore	Tests: macOS	Log
2885574	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger main	Log
1263388	dotnet/maui	Controls Layout	Log
1256005	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65082
1262925	dotnet/maui	Controls CollectionView	Log
2885485	dotnet-performance	Performance maui_scenarios_android NativeAOT CoreCLR_Debug windows 22H2 x64 Pixel main	Log
1262921	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33552
1263219	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#122373
1262765	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123458
1262768	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123462
1263087	dotnet/aspnetcore	Tests: macOS	Log
1262945	dotnet/maui	Controls Layout	Log
1262966	dotnet/maui	Build macOS	Log	dotnet/maui#33673
1262982	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65184
1262934	dotnet/maui	Controls Layout	Log
1262971	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64751
1262941	dotnet/maui	RunOniOS_MauiRelease macOS	Log	dotnet/maui#33616
1262924	dotnet/maui	RunOniOS_MauiRelease macOS	Log
2885229	dotnet-maui	RunOniOS_MauiDebug macOS	Log
2885212	dotnet-maui	Controls (v18.4) ViewBaseTests,Window
1262914	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65185
2885205	dotnet-maui	macOS Unit Tests	Log
1262844	dotnet/maui	RunOniOS_BlazorDebug macOS	Log
1261175	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64964
1262851	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123265
1262795	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log	dotnet/maui#33656
1262723	dotnet/maui	RunOniOS_BlazorRelease macOS-arm64	Log	dotnet/maui#33671
1262752	dotnet/aspnetcore	Tests: macOS	Log
2885122	dotnet-performance	Performance maui_scenarios_android NativeAOT CoreCLR windows 22H2 x64 Pixel main	Log
1262529	dotnet/maui	Controls Layout	Log	dotnet/maui#33662
2885016	dotnet-maui	RunOniOS_MauiRelease macOS	Log
1262568	dotnet/maui	RunOniOS_MauiRelease macOS	Log
1262577	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1262516	dotnet/aspnetcore	Tests: macOS	Log
1262162	dotnet/maui	Controls CollectionView	Log
2884907	dotnet-performance	Performance maui_scenarios_android R2RComposite CoreCLR windows 22H2 x64 Pixel main	Log
1262264	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123471
1262258	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123469
1262248	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123468
1262330	dotnet/maui	RunOnAndroid macOS	Log	dotnet/maui#33328
1262268	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65151
1260040	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64408
1262151	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64851
1262121	dotnet/aspnetcore	Tests: macOS	Log
1262054	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123462
1262021	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#123458
1262012	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log
1261503	dotnet/maui	Controls CollectionView	Log
1261772	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2884625	dotnet-performance	Performance maui_scenarios_android NativeAOT CoreCLR windows 22H2 x64 Pixel main	Log
Displaying 100 of 576 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
36	226	576

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.