RCA for Creator property is required errors that broke helix-machines pipelines on 10 March

[dougbu]

after a series of problems w/ Matrix of Truth data (see Teams conversations #1 and #2, we started seeing problems w/ the Validate on-prem Queues and Validate stages in dotnet-helix-machines-pr and (later) dotnet-helix-machines-ci. see Teams conversation for thread about the validation problems

this issue is a request to document the root cause of the validation failures, especially why the production Helix service seemed fine while staging definitely wasn't and how validation in pipelines deploying the database and app missed the problem until it broke validation in helix-machines pipelines. if something was manually changed between deployments, that would be good to know as well

Details

Errors included

##[error]D:\a\_work\1\s\ValidateHelix.dll(,): error : pr-azurelinux.3.amd64-dougbu-restore.helix-t: unhandled exception System.ArgumentException: The request is invalid
---> Microsoft.DotNet.Helix.Client.RestApiException`1[Microsoft.DotNet.Helix.Client.Models.ApiError]: The response contained an invalid status code 400 Bad Request

Body: {"Message":"The request is invalid","ActivityId":"439183a306c23dc3d967527fb8bf132f","Errors":["The Creator property is required.","The Creator property is required."]}
   --- End of inner exception stack trace ---
   at Microsoft.DotNet.Helix.Client.HelixApi.HandleFailedRequest(RestApiException ex) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/HelixApi.cs:line 46
   at Microsoft.DotNet.Helix.Client.Job.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 165
   at Microsoft.DotNet.Helix.Client.Job.NewAsync(JobCreationRequest body, String idempotencyKey, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 128
   at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 239
   at EndToEndTests.Program.TestQueue(KeyValuePair`2 queueDefPair) in D:\a\_work\1\s\src\ValidateHelix\Program.cs:line 337

and

/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : ArgumentException: The request is invalid [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Client.HelixApi.HandleFailedRequest(RestApiException ex) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/HelixApi.cs:line 46 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Client.Job.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 165 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Client.Job.NewAsync(JobCreationRequest body, String idempotencyKey, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 128 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 239 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs:line 267 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :    at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 58 [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : RestApiException`1: The response contained an invalid status code 400 Bad Request [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :  [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : Body: {"Message":"The request is invalid","ActivityId":"b5a45924cebcbcb83468d4470b73ce5c","Errors":["The Creator property is required.","The Creator property is required."]} [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :  [/home/vsts/work/1/s/eng/common/helixpublish.proj]
/home/vsts/.nuget/packages/microsoft.dotnet.helix.sdk/8.0.0-beta.25114.5/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error :  [/home/vsts/work/1/s/eng/common/helixpublish.proj]

Discussions

one summary of the Teams conversation on this topic is "confused". I originally thought it sounded very much like service authentication problems that occurred w/ the Helix service on 20 February. rotating a token resolved that earlier situation. we spent a while trying to do the rotation, then discovered problem related to disabling some SQL authentication settings. @hoyosjs also found

bottom line, it took a while to find the actual problem and it would be good understand why and how to avoid similar issues in the future. do we need to improve the response message for cases like this❓ is something else missing in our processes or code❓

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[dougbu]

see @engyebrahim's doc