-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Description
New issue checklist
- I searched for existing GitHub issues
- I read pipeline troubleshooting guide
- I checked how to collect logs
Task name
AzurePowershell
Task version
5.257.0
Issue Description
We're seeing an issue with our AzurePowershell@5 ADO tasks exiting prematurely when the task fails to get a Workload Identity Federation token on the first attempt. We're hitting this quite frequently, and it's fairly sporadic (probably due to the nature of this being retry based). It's hitting multiple times per week, and at worst we've seen this 50+ times in a week.
Given the error snippet below and looking at the code:
VERBOSE: Removing assemlby resolver.
VERBOSE: SYSTEM_PLANID: '394eeaf6-2faa-4b94-a012-4767dfd29c6c'
VERBOSE: SYSTEM_JOBID: '339dd970-ef3f-5f5c-e5b6-c56c83629183'
VERBOSE: SYSTEM_HOSTTYPE: 'build'
VERBOSE: SYSTEM_TEAMPROJECTID: 'f91058ab-8cee-4376-891c-9bf45b71733c'
VERBOSE: Failed to fetch federated token. Remaining retries count = '2'
VERBOSE: Leaving Initialize-AzModule.
An error occurred in Initialize-AzModule
I'm assuming that we're failing to get the token on the first attempt, then succeeding on the second (since the log doesn't show any other retries, nor does the Failed to create OIDC token.
line appear.)
azure-pipelines-tasks/Tasks/Common/VstsAzureHelpers_/Utility.ps1
Lines 248 to 270 in 000e8ee
$timeToWait = 4000 | |
for (($retryAttempt = 1), ($retryLimit = 3); $retryAttempt -le $retryLimit; $retryAttempt++) { | |
$tokenResponse = $taskHttpClient.CreateOidcTokenAsync( | |
$projectId, | |
$hub, | |
$planId, | |
$jobId, | |
$connectedServiceNameARM, | |
$null | |
).Result | |
$federatedToken = $tokenResponse.OidcToken | |
if ($null -ne $federatedToken) { | |
return $federatedToken | |
} | |
if ($retryAttempt -lt $retryLimit) { | |
Write-Verbose "Failed to fetch federated token. Remaining retries count = '$($retryLimit - $retryAttempt)'" | |
Start-Sleep -m $timeToWait * $retryAttempt | |
} | |
} | |
Write-Verbose "Failed to create OIDC token." | |
throw (New-Object System.Exception(Get-VstsLocString -Key AZ_CouldNotGenerateOidcToken)) |
Given that we should have a token at this point, we'd expect the log to continue with the lines Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
, but instead it just leaves the module and throws an error.
azure-pipelines-tasks/Tasks/Common/VstsAzureHelpers_/InitializeAzModuleFunctions.ps1
Lines 382 to 388 in 000e8ee
$clientAssertionJwt = Get-VstsFederatedToken -serviceConnectionId $connectedServiceNameARM -vstsAccessToken $vstsAccessToken ` | |
-azAccountsModuleVersion $azAccountsModuleVersion -isPSCore $isPSCore | |
Write-Host "##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue" | |
$null = Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue | |
Write-Host "##[command]Clear-AzContext -Scope Process" | |
$null = Clear-AzContext -Scope Process |
I can't see anything obvious that would be causing this, but we could use some help with this.
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows Server 2025
Relevant log output
VERBOSE: Removing assemlby resolver.
VERBOSE: SYSTEM_PLANID: '394eeaf6-2faa-4b94-a012-4767dfd29c6c'
VERBOSE: SYSTEM_JOBID: '339dd970-ef3f-5f5c-e5b6-c56c83629183'
VERBOSE: SYSTEM_HOSTTYPE: 'build'
VERBOSE: SYSTEM_TEAMPROJECTID: 'f91058ab-8cee-4376-891c-9bf45b71733c'
VERBOSE: Failed to fetch federated token. Remaining retries count = '2'
VERBOSE: Leaving Initialize-AzModule.
An error occurred in Initialize-AzModule
Full task logs with system.debug enabled
Repro steps
Bit tough to reproduce
Setup a Workload Identity Federation ARM Service Connection
Have a pipeline use an AzurePowershell@5 task using the above service connection
The task can run anything (i.e. `Write-Host 'hello world'`)
Have the token acquisition in `Get-VstsFederatedToken` fail on the first attempt, then succeed on the second attempt