M365DSC_Data issue. UsedWorkloads in Environment#Generic.psd1 is not currently used

[martincaddick]

It looks like UsedWorkloads is not currently used at this stage.

UsedWorkloads = @{
Azure = $false # Specifies if this workload is used in the configuration
AzureAD = $true # Specifies if this workload is used in the configuration
AzureDevOps = $false # Specifies if this workload is used in the configuration

[ykuijs]

Hi @martincaddick,

The UsedWorkloads section is used in the M365Configuration.ps1 file:

M365DSC_CICD/Scripts/M365Configuration.ps1

Line 24 in 9035794

if ($null -ne $azureAppCreds -and $ConfigurationData.NonNodeData.Environment.UsedWorkloads.Azure -eq $true)

In there, the config checks for each workload if the value is set to True. If that is not the case, that specific workload is ignored. That way you can disable the workloads that you are not using.

[martincaddick]

I think that it ignores the true/false for doing the prebuild tests and merge but when it gets to compile the MOF then it respects the true/false. I'll pay more attention to my next couple of runs.

btw, I needed to increase the MaxEnvelopeSize to 16M.

Side note, I'll create an issue for SCRoleMemberGroups in M365DSC. Each SCRoleMemberGroup takes 20 seconds to evaluate pushing the whole Compliance test > 1 hr. Without SCRoleMemberGroup the test is completed in 20 minutes.
(Wow!. Just seen microsoft/Microsoft365DSC#5615)

[ykuijs]

That is correct. The PreBuild step does not take the UsedWorkloads setting into account. Only during compilation, this setting is used.

Now that you mention this: Have been brainstorming about also using the setting in the PreBuild step. We could exclude that workload in the merge step, but the challenge there is that the setting is Environment/Tenant specific. Since the Basic layer applies to all tenants, we will have to merge all files there meaning the settings for that workload in the Basic layer still end up in the merged data.

We could simply remove the settings of the specific workload after merge. That means the unit tests won't be executed on those settings, which will save some processing time.

But at the same time: How often will you exclude a single workload for a single environment? If you only want to use this solution for AAD/Entra ID and Teams for example, none of the layers will contain much settings for other workloads anyways.

Am still a bit undecided how much work this will require to implement versus the benefit is will bring.

[martincaddick]

At this stage probably just document the current functionality. Could be as simple a one or two lines of comment in the #Generic.psd1 just above UsedWorkloads =

[martincaddick]

Hi @ykuijs
Couple of things I've been working on, some complete, others in progress.

1. Complete. Option to include/exclude PesterTests in the BuildMOF.

The wrapped all the PreBuild.ps1 checks with an if like this.

# Quality checks
if($PesterTest){
    Write-Log -Object ' '
    Write-Log -Object '---------------------------------------------------------'
    Write-Log -Object ' Running quality checks '
    Write-Log -Object '---------------------------------------------------------'
    Write-Log -Object ' '
    $qaTestResults = & $qaTestPath
    if ($qaTestResults.Result -ne 'Passed')
    {
        Write-Log -Object "[ERROR] $($qaTestResults.FailedCount) QA checks failed! Exiting!"
        Write-Host '##vso[task.complete result=Failed;]Failed'
        exit -1
    }
}

2. Complete (but I'm not sure how it fits in with all of the other pipelines yet. Option to select a tenant which then uses a tenantcode to pre-select a bunch of parameters for use in scripts and pipelines. Screenshot above.

3. In Progress Test-Compliance (and this is where it relates to this issue)
   MOF files can be pretty big and if I only want to check the compliance for one workload then I need the mof to be small.

I've been seeing some errors at mof compilation time and it was frustrating me that I couldn't identify them earlier, hence, I put this script together. It loads the full.psd1 and then modify the userworkloads on the fly to then generate a mof just for that workload. Assuming it works I get a localhost.mof in a workload folder. If it fails I get a localhost.error.mof in the folder and an entry in the log file.

QuickMOF.ps1

# param(
#     $environ
# )

$workingDirectory = "C:\temp"
$pipelinesDirectory = "C:\Path\M365DSC_CICD"
. $pipelinesDirectory\Scripts\M365Configuration.ps1
$environ = "TenantName"
$outputFolder = "C:\Temp\MOFTest\$environ"
$errLog = "$outputFolder\errors.log"
$environmentFile = "$workingDirectory\$environ.psd1"
$environment = Import-PSDataFile -Path $environmentFile
$keys = $environment.NonNodeData.Environment.UsedWorkloads.Keys
foreach ($key in $keys)
{
    # Create a copy of the current setup
    $testworkloads = @{}
    foreach ($innerkey in $keys)
    {
        $testworkloads[$innerkey] = $false
    }
    # Set the current workload to true
    $testworkloads[$key] = $true
    $environment.NonNodeData.Environment.UsedWorkloads = $testworkloads
    $used = $environment.NonNodeData.Environment.UsedWorkloads.GetEnumerator() | Sort-Object | Out-String
    Write-Log -Object $used
    # This ^ needs to modify the environment object
    # This is where we will just create a mof for the specific workload
    if($environment.NonNodeData[$key])
    {
        try {
            Write-Host "Creating MOF file for $key"
            M365Configuration -ConfigurationData $environment -OutputPath "$outputFolder\$key" -Verbose
        }
        catch {
            $errorMessage = $_.Exception.Message
            $errorDetails = $_.Exception.StackTrace
            [string]$workload = $key
            $logmessage = "[$(Get-Date)] Workload: $workload`nError: $errorMessage`nDetails: $errorDetails`n"
            Add-Content -Path $errLog -Value $logmessage
        }
    }
}

I was thinking that maybe I could use the same principle and pass the required workload(s) in the test-compliance pipeline, then modify the usedworkloads accordingly, compile a new mof and the run the compliance with the new mof file.