Remediation Task could not be created

[LarsVidingSE]

Hi, I really appreciate all help with this error.

We have EPAC up and running in our lab environment which is based on Azure DevOps (ADO) pipelines and single tenant.
We have synced in policies from ALZ and AMBA to an ALZ MG hierarchy.

Pipeline [ecap-dev] ( epac-dev-pipeline.yml) and [epac-tenant] (epac-tenant-pipeline.yml) are working.
To check remediation we have created an Azure storageacount with configuration that is not compatible with policies which are assigned via EPAC.

When manually running the pipepline [epac-remediation] (epac-remediation-pipeline.yml), it will success. But the pipeline find 29 number of non-compilant resources and it start to creating 18 number of remediation tasks.
The process is ending with bellow message.

`Remediation Task Status

18 needed
18 failed to create
0 created
0 succeded `

The error message of all 18 number of tasks is

"Remediation Task could not be created"

I really appreciate any suggestions of where we shall look for the reason why the pipeline [epac-remediation] (epac-remediation-pipeline.yml) canot create Remediation Tasks.

Thanks in advance

[brianmooremsft]

Hi - For the service principal that is running the pipeline, does it have the necessary Azure RBAC roles/permissions for the Azure scope that EPAC is applying policy to? Specifically wondering whether it has the required role "Role Based Access Control Administrator" From the documentation https://azure.github.io/enterprise-azure-policy-as-code/ci-cd-app-registrations/ The following Service Principals & Role assignments would be created to support this structure: Service Principal Azure Role Assignment Assignment Scope spn-epac-plan Reader Tenant Root Group spn-epac-dev Resource Policy Contributor Role Based Access Control Administrator epac-contoso spn-epac-tenant-deploy Resource Policy Contributor Contoso spn-epac-tenant-roles Role Based Access Control Administrator Contoso From: Lars Viding ***@***.***> Sent: Friday, August 9, 2024 5:20 AM To: Azure/enterprise-azure-policy-as-code ***@***.***> Cc: Subscribed ***@***.***> Subject: [Azure/enterprise-azure-policy-as-code] Remediation Task could not be created (Discussion #722) Hi, I really appreciate all help with this error. We have EPAC up and running in our lab environment which is based on Azure DevOps (ADO) pipelines and single tenant. We have synced in policies from ALZ and AMBA to an ALZ MG hierarchy. Pipeline [ecap-dev] ( epac-dev-pipeline.yml) and [epac-tenant] (epac-tenant-pipeline.yml) are working. To check remediation we have created an Azure storageacount with configuration that is not compatible with policies which are assigned via EPAC. When manually running the pipepline [epac-remediation] (epac-remediation-pipeline.yml), it will success. But the pipeline find 29 number of non-compilant resources and it start to creating 18 number of remediation tasks. The process is ending with bellow message. `Remediation Task Status 18 needed 18 failed to create 0 created 0 succeded ` The error message of all 18 number of tasks is "Remediation Task could not be created" I really appreciate any suggestions of where we shall look for the reason why the pipeline [epac-remediation] (epac-remediation-pipeline.yml) canot create Remediation Tasks. Thanks in advance - Reply to this email directly, view it on GitHub<#722>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASWPRU3SXUESMMUNGA336WLZQSXX3AVCNFSM6AAAAABMILSPUGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGAZDSNRVGA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[LarsVidingSE]

Thanks, brianmooremsft for helping me with this.

Well, I have been sick with covid and not been able to answer until now.

I have controlled and doublecheck everything.
The pipeline for remediation is using a separate ADO Service connection which is connected to the spn-epac-tenant-roles. And this spn has [Role Based Access Control Administrator] from the production root which is the root of our production MG.

For test purposes I added [Resource Policy Contributor] from the production root MG.
Just to see if the only service connector that is running this pipeline need to have more Az roles.
But this did not help.

I have also run the pipelines inline command manually.
New-AzRemediationTasks.ps1 -PacEnvironmentSelector "prod" -OnlyCheckManagedAssignments
I connected with a user account who has owner on like everything in the environment.
And I got the exact same results. “Remediation Task could not be created”.

[LarsVidingSE]

Add more testing that I have done.

As I understand it is en raw 256 in New-AzRemediationTasks.ps1 that is where this command is EQ to $null.
$newPolicyRemediationTask = Start-AzPolicyRemediation @parameters -ErrorAction SilentlyContinue -WhatIf:$WhatIfPreference
I think I must figure out a way to print out some status or files just so see what is in the @parameters.

[LarsVidingSE]

With some help from colleague, we run the remediation pipeline with checked the box of Enable system diagnostics. Which did give use debug and the verbose output.
Therefore, we saw the parameters value of each remediations task that could not be created.
Then we manually run the Start-AzPolicyRemediation with exact parameters and got the actual error messages because the pipeline cloud not create a remediation task.

Start-AzPolicyRemediation: Operation returned an invalid status code 'Conflict' (ManagementGroupNotRegistered: The management group: 'alecta' or any of its ancestors are not registered to 'Microsoft.PolicyInsights'. Creating remediation tasks for large number of resources in management group scopes requires that the management group is registered to the 'Microsoft.PolicyInsights' resource provider. Please register the management group and try again. 'See https://aka.ms/ResourceProviderMGRegistration for more details.)”
The reason was the service provider 'Microsoft.PolicyInsights' was not registered at the management group level.
With help from
[Providers - Register At Management Group Scope - REST API (Azure Resource Management) | Microsoft Learn](url)

We registered the service provider to all management group within our environment in each pacSelector deploymentRootScope and throughout each hierarchy.

Then the remediation pipeline did managed to create remediation tasks.

:)