dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
871563	dotnet/sdk	Run TestBuild Tests	Log	#44786
871339	dotnet/sdk	Run TestBuild Tests	Log	#44899
871038	dotnet/sdk	Run TestBuild Tests	Log	#44834
871083	dotnet/sdk	Run TestBuild Tests	Log	#44802
870714	dotnet/sdk	Run TestBuild Tests	Log	#44802
870458	dotnet/sdk	Run TestBuild Tests	Log	#44691
870523	dotnet/sdk	Run TestBuild Tests	Log	#44882
869600	dotnet/sdk	Run TestBuild Tests	Log	#44540
869535	dotnet/sdk	Run TestBuild Tests	Log	#44810
869404	dotnet/sdk	Run TestBuild Tests	Log	#44786
869322	dotnet/sdk	Run TestBuild Tests	Log	#44834
868628	dotnet/sdk	Run TestBuild Tests	Log	#44810
867420	dotnet/sdk	Run TestBuild Tests	Log	#44764
867379	dotnet/sdk	Run TestBuild Tests	Log	#44802
867301	dotnet/sdk	Run FullFramework Tests	Log	#44807
867060	dotnet/sdk	Run TestBuild Tests	Log	#44802
866984	dotnet/sdk	Run TestBuild Tests	Log	#44730
866133	dotnet/sdk	Run TestBuild Tests	Log	#44540
865871	dotnet/sdk	Run TestBuild Tests	Log	#44691
865846	dotnet/sdk	Run TestBuild Tests	Log	#44774
865756	dotnet/sdk	Run TestBuild Tests	Log	#44603
865016	dotnet/sdk	Run FullFramework Tests	Log	#44730
864884	dotnet/sdk	Run TestBuild Tests	Log	#44691
864568	dotnet/sdk	Run FullFramework Tests	Log	#44745
864433	dotnet/sdk	Run TestBuild Tests	Log	#44721
864173	dotnet/sdk	Run FullFramework Tests	Log	#44699
864106	dotnet/sdk	Run TestBuild Tests	Log	#44691
864032	dotnet/sdk	Run FullFramework Tests	Log	#44730
863986	dotnet/sdk	Run TestBuild Tests	Log	#44603
863759	dotnet/sdk	Run TestBuild Tests	Log	#44603
863268	dotnet/sdk	Run TestBuild Tests	Log	#44701
862957	dotnet/sdk	Run TestBuild Tests	Log	#44603
863215	dotnet/sdk	Run TestBuild Tests	Log	#44696
863133	dotnet/sdk	Run TestBuild Tests	Log	#44683
863131	dotnet/sdk	Run TestBuild Tests	Log	#44695
863105	dotnet/sdk	Run TestBuild Tests	Log	#44685
863078	dotnet/sdk	Run Tests in Helix	Log	#44693
863049	dotnet/sdk	Run TestBuild Tests	Log	#44570
863033	dotnet/sdk	Run Tests in Helix	Log	#43085
862878	dotnet/sdk	Run FullFramework Tests	Log	#44691
862030	dotnet/sdk	Run Tests in Helix	Log
861894	dotnet/sdk	Run TestBuild Tests	Log	#44633
860813	dotnet/sdk	Run TestBuild Tests	Log	#43015
860456	dotnet/sdk	Run TestBuild Tests	Log	#44603
860751	dotnet/sdk	Run TestBuild Tests	Log	#44630
860566	dotnet/sdk	Run TestBuild Tests	Log	#44627
859434	dotnet/sdk	Run TestBuild Tests	Log	#44598
859239	dotnet/sdk	Run FullFramework Tests	Log	#44603
858952	dotnet/sdk	Run TestBuild Tests	Log	#44589
858589	dotnet/sdk	Run TestBuild Tests	Log	#44588
857871	dotnet/sdk	Run TestBuild Tests	Log	#44455
856177	dotnet/sdk	Run TestBuild Tests	Log	#44518
855890	dotnet/sdk	Run TestBuild Tests	Log	#44511
855384	dotnet/sdk	Run FullFramework Tests	Log	#44501
854330	dotnet/sdk	Run TestBuild Tests	Log	#43015
853730	dotnet/sdk	Run TestBuild Tests	Log	#43060
853549	dotnet/sdk	Run FullFramework Tests	Log	#44308
851579	dotnet/sdk	Run TestBuild Tests	Log	#44308
851247	dotnet/sdk	Run Tests in Helix	Log	#44276
851195	dotnet/sdk	Run TestBuild Tests	Log	#44400
851087	dotnet/sdk	Run TestBuild Tests	Log	#44394
851011	dotnet/sdk	Run FullFramework Tests	Log	#44390
850414	dotnet/sdk	Run TestBuild Tests	Log	#44348
2566073	dotnet-sdk	Run Tests in Helix	Log	#44078
849300	dotnet/sdk	Run TestBuild Tests	Log	#44014
848426	dotnet/sdk	Run TestBuild Tests	Log	#44159
847837	dotnet/sdk	Run TestBuild Tests	Log	#44310

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	18	67

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.