Improve workloads versioning

[marcpopMSFT]

Is your feature request related to a problem? Please describe.

There is no way to specify a specific version of the SDK for workloads. Rollback files but you have to know individual workload manifest versions. We've discussed having the concept of grouped versioning where say 6.0.100 has a set of manifest versions that match that so a customer could say they wanted to install 6.0.100 or 6.0.101 versions of those manifests/packs (think of a workloads version of releases.json).

Describe the solution you'd like

TBD

Additional context

This could be really confusing. Would this go into a global.json or as a command line argument?

[marcpopMSFT]

@mhutch I believe you had mentioned some ideas in this space previously as well. We'll be assigning this to a PM soon to gather more requirements.

[marcpopMSFT]

CC @baronfel since I can't assign this yet. We can chat about it when you're ready.

[sandyarmstrong]

Thanks for filing, I also found this very confusing. It has not been clear to me how to pick a specific workload version, or how to know ahead of time what version will be used. This information is not available in the dotnet workload commands.

When using workloads, the most obvious approach that seems useful to me would be to pin versions in global.json, just like with any other dotnet/msbuild SDK.

[cwensley]

This is causing major issues for us, especially for the macos workload. We figured that the specific version of a workload would be pinned to the version of the SDK, but that is not the case.

Is there a way to install a specific version of a workload via command line or other? At the very least that would get our developers back on track and we can update the workload for our builds after we have done sufficient testing.

This is a major problem for us, and waiting until 6.0.3xx is not really an option here.

[sandyarmstrong]

Is there a way to install a specific version of a workload via command line or other?

@cwensley yeah, here's the approach I've been taking:

1. Create a rollback.json file that looks something like {"microsoft.net.sdk.macos":"12.0.101-preview.10.251"}.
2. Ensure you have sdk.version set properly in global.json.
3. Run sudo dotnet workload install macos --from-rollback-file rollback.json --ignore-failed-sources --source https://aka.ms/dotnet6/nuget/index.json --source https://api.nuget.org/v3/index.json

You'll probably have to redo this if you install any security updates. I know I had to after using 6.0.100 and then installing 6.0.101.

If you also need to check what workload version is installed, I suggest reading the entirety of #22882.

[cwensley]

Hey @sandyarmstrong, thank you for the very quick response! I'll try it out.

Is there any documentation about rollback.json? I can't find anything and it isn't shown in dotnet workload install --help

[sandyarmstrong]

Is there any documentation about rollback.json? I can't find anything and it isn't shown in dotnet workload install --help

They recently added https://github.com/dotnet/sdk/blob/main/documentation/general/workload-rollback.md. I still think it would be nice if even advanced options like these were in the inline help.

[mattleibow]

You could use some specifically constructed URL for the rollbacks. For example, we do this for maui: https://aka.ms/dotnet/maui/6.0.200/preview.13.json

That way we can pick the "blessed" versions for a particular version of maui.

I could see the same for dotnet itself where it has a url for each band (or locally from something) and then it just uses that. This way users do not need to care about individual manifest versions.

[sandyarmstrong]

Making workload versioning usable

Allow specifying version in workload install without rollback files

Rollback files are overly complex for the simple scenario of installing a single workload. They should install (and resolve) similar to nuget packages.

It should be possible to run:

sudo dotnet workload install macos --version 12.1.301-preview.13.4

Standardize specifying workloads (and optionally, versions) in global.json

CI and other tooling would be easier if we established a standardized practice of listing workload info in the same place where we specify other SDK info.

Then we could run dotnet workload install (or perhaps dotnet workload install --from-global-json) and have it automatically pick up workloads, versions, and feeds from global.json.

Samples

With version/sources specified

{
  "sdk": {
    "version": "6.0.200"
  },
  "workloads": {
    "macos": {
      "version": "12.1.301-preview.13.4",
      "sources": [
        "https://aka.ms/dotnet6/nuget/index.json",
        "https://api.nuget.org/v3/index.json"
      ]
    }
  }
}

Without specifying version

{
  "sdk": {
    "version": "6.0.200"
  },
  "workloads": {
    "macos": {}
  }
}

Helper script for using this today

❓ What is the relationship between short workload names used by dotnet workload install and long package names used by rollback files? This script assumes that the long name used in rollback files is always the short name prepended by "microsoft.net.sdk".

Click to reveal script

param(
    [string]$globalJsonPath,
    [string]$dotnetPath = "dotnet"
)

# If it's just 'dotnet', it's being resolved from PATH. Otherwise, resolve it.
if (Test-Path $dotnetPath) {
    $dotnetPath = (Resolve-Path $dotnetPath).Path
}

function New-TemporaryFilePath
{
    # Neither New-TemporaryFile nor New-Guid can be resolved when running these
    # scripts via cmd scripts from a PowerShell 7 terminal.
    return Join-Path ([System.IO.Path]::GetTempPath()) ([System.Guid]::NewGuid())
}

function Get-WorkloadName
(
    [string]$shortName
)
{
    return "microsoft.net.sdk.$shortName"
}

function Get-InstalledWorkloads
{
    $installedWorkloadsWithVersions = @{}

    # Shows versions of workloads that would be installed if a rollback were
    # done. Includes uninstalled workloads, though, so these version numbers
    # are only useful to us if we've confirmed elsewhere that the workload is
    # actually installed.
    $workloadUpdateOutput = & $dotnetPath workload update --print-rollback

    $foundJson = $false
    $rollback = $null
    foreach ($line in $workloadUpdateOutput) {
        if ($foundJson) {
            $rollback = $line | ConvertFrom-Json
            break
        }

        if ($line -eq "==workloadRollbackDefinitionJsonOutputStart==") {
            $foundJson = $true
        }
    }

    if (-not $rollback) {
        return $installedWorkloadsWithVersions
    }

    # Shows what workloads are actually installed. Does not show installed
    # version information, unless an update happens to be available.
    # https://github.com/dotnet/sdk/issues/22882
    $workloadListOutput = & $dotnetPath workload list --machine-readable

    $foundJson = $false
    $workloadList = $null
    foreach ($line in $workloadListOutput) {
        if ($foundJson) {
            $workloadList = $line | ConvertFrom-Json
            break
        }

        if ($line -eq "==workloadListJsonOutputStart==") {
            $foundJson = $true
        }
    }

    foreach ($shortWorkloadName in $workloadList.installed) {
        $workloadName = Get-WorkloadName -shortName $shortWorkloadName
        if ($workloadName -in $rollback.PSObject.Properties.Name) {
            $installedWorkloadsWithVersions.$workloadName = $rollback.$workloadName
        }
    }

    return $installedWorkloadsWithVersions
}

function Install-Workload
(
    [string]$shortName,
    [string]$version = $null,
    [string[]]$sources = $null
)
{
    $workloadName = Get-WorkloadName -shortName $shortName

    $installedVersion = (Get-InstalledWorkloads)[$workloadName]

    if ($installedVersion) {
        Write-Host "Workload $workloadName ($shortName) $installedVersion already installed"

        if (-not $version -or $version -eq $installedVersion) {
            continue
        }
    }

    Write-Host "Preparing to install workload $workloadName ($shortName) $version..."

    # Safest to uninstall the existing version first
    if ($installedVersion) {
        Write-Host "  Uninstalling $shortName..."

        # NOTE: This requires elevated access if using a system dotnet install
        & $dotnetPath workload uninstall $shortName
    }

    $rollbackArgs = @()
    $rollbackFile = $null
    if ($version) {
        $rollbackJson = @{}
        $rollbackJson[$workloadName] = $version

        $rollbackFile = New-TemporaryFilePath
        $rollbackJson | ConvertTo-Json | Set-Content -Path $rollbackFile

        $rollbackArgs += "--from-rollback-file"
        $rollbackArgs += $rollbackFile
    }

    $sourcesArgs = @()
    foreach ($source in $sources) {
        $sourcesArgs += "--source"
        $sourcesArgs += $source
    }

    Write-Host "  Installing $shortName..."

    # NOTE: This requires elevated access if using a system dotnet install
    & $dotnetPath workload install $shortName $rollbackArgs $sourcesArgs

    if ($rollbackFile) {
        Remove-Item $rollbackFile
    }
}

$GlobalJson = Get-Content $globalJsonPath | ConvertFrom-Json

foreach ($workload in $GlobalJson.workloads.PSObject.Properties) {
    Install-Workload `
        -shortName $workload.Name `
        -version $workload.Value.version `
        -sources $workload.Value.sources
}

[sandyarmstrong]

I've made some updates to my proposal text, and improvements to the script.

[sandyarmstrong]

I've been trying out this approach with dotnet/roslyn#59429.

[mattleibow]

@sandyarmstrong I know the script is just for now, but any solution we make should also consider Tizen, which uses a samsung prefix.

On a separate note, how does this handle the case where I say have preview 13 in one directory and preview 14 in another - and then try build both? Will workloads support side-by-side?

global.json usually feels like a "we have multiple versions installed, so use this one". I also know that (somewhere) we are looking to make the version in the global.json be a min version. So an sdk 6.0.100 could also select 6.0.200. Is this something we need to consider for workload versions?

Additionally, since we can install 100 and 200 of things, I am wondering if we need to worry about preview versions? Although, I also recall (somewhere) that we may be adding preview version bands too?

A fair bunch of evolving bits here and new things :)

[sandyarmstrong]

@mattleibow

I know the script is just for now, but any solution we make should also consider Tizen, which uses a samsung prefix.

Yeah, I don't really understand why we use short workload names during uninstall/reinstall, but long names in rollback files and elsewhere. It's pretty confusing. I'd need help from somebody who understands workloads better to determine what would make the most sense in global.json.

On a separate note, how does this handle the case where I say have preview 13 in one directory and preview 14 in another - and then try build both? Will workloads support side-by-side?

A workload release is tied to a specific SDK/runtime "feature band" (6.0.100 and 6.0.101 share a feature band, 6.0.200 is a separate feature band). As I understand it, you can only have one workload version installed for a given feature band. My proposal does not change the fact that to build two projects with differing workload versions in the same feature band, you either need to repeatedly uninstall/reinstall workloads, or you need separate base SDK installations. It would be great if this would change, but I'm mostly looking at how workloads are installed today.

Perhaps the workloads property should be under sdk to reinforce this relationship.

global.json usually feels like a "we have multiple versions installed, so use this one". I also know that (somewhere) we are looking to make the version in the global.json be a min version. So an sdk 6.0.100 could also select 6.0.200. Is this something we need to consider for workload versions?

Are you thinking of the rollForward option? I suppose we could support that here, too.

[mattleibow]

@joeloff has been the person I bug most about workload things, he may have comments or know who else might be interested in workload-y and global-y things,

[cosminstirbu]

Are there any news on this? We're also interested in a way to pin the workloads versions. Is the combination of global.json and rollback.json the currently recommended workaround?

[jamescrosswell]

Any news on this?

Our CI Builds have currently stopped working because we can't pin Workload versions.

[baronfel]

Going to close this in favor of the work to install workloads side-by-side and allow pinning workload versions in global.json.