Feat: Support for downloading releases based on some version constraint

[ProbstDJakob]

I've tried to use the docker image ghcr.io/opentofu/opentofu:1 in order to have the latest version of OpenTofu with all bug fixes but without any breaking changes. This approach had some disadvantages which led me to open the issue opentofu/opentofu#1931 and the result of it was that the image should not be used for my (specific) purpose. Throughout searching for a solution, we came up with the idea to add version resolving to get.opentofu.org (opentofu/opentofu#1931 (comment)).

The idea is that there would be an URL which accepts partial versions and expands them to the latest matching release. For example when requesting https://get.opentofu.org/download/plattform/linux/amd64/1.tar.gz it would redirect to https://get.opentofu.org/download/plattform/linux/amd64/1.8.2.tar.gz which in turn would redirect to https://github.com/opentofu/opentofu/releases/download/v1.8.2/tofu_1.8.2_linux_amd64.tar.gz. Or to put it in a more technical example (pseudo configuration):

match /download/binary/(darwin|freebsd|linux|openbsd|solaris|windows)/(386|amd64|arm|arm64)/1((?:\.tar\.gz|\.zip)(?:\.gpgsig|\.pem|\.sig)?) {
  redirect $host/download/binary/$1/$2/1.8.2$3;
}

match /download/binary/(darwin|freebsd|linux|openbsd|solaris|windows)/(386|amd64|arm|arm64)/1\.8((?:\.tar\.gz|\.zip)(?:\.gpgsig|\.pem|\.sig)?) {
  redirect $host/download/binary/$1/$2/1.8.2$3;
}

match /download/binary/(darwin|freebsd|linux|openbsd|solaris|windows)/(386|amd64|arm|arm64)/1\.7((?:\.tar\.gz|\.zip)(?:\.gpgsig|\.pem|\.sig)?) {
  redirect $host/download/binary/$1/$2/1.7.3$3;
}

// ...

match /download/binary/(darwin|freebsd|linux|openbsd|solaris|windows)/(386|amd64|arm|arm64)/(\d+\.\d+\.\d+(?:-rc\d+|-beta\d+)?)((?:\.tar\.gz|\.zip)(?:\.gpgsig|\.pem|\.sig)?) {
  redirect https://github.com/opentofu/opentofu/releases/download/v$3/tofu_$3_$1_$2$4;
}


match /download/packaged/(386|amd64|arm|arm64)/1((?:\.apk|\.deb|\.rpm)(?:\.gpgsig|\.pem|\.sig)?) {
  redirect $host/download/packaged/$1/1.8.2$2;
}

// ...

Yes not all combinations of the RegEx are valid assets, but this is just an example and also GitHub would just return 404.

This could also be achieved by dynamically evaluation to which version the URL https://get.opentofu.org/download/plattform/linux/amd64/1.tar.gz should resolve to, but then the web server would then need to either query the GitHub API or keep an updated Git Repo clone to query that one. In either case the web server is more likely to struggle with handling a huge amount of requests. On the other hand the approach via a hardcoded configuration would need to be updated on every release. If there is an automatic release pipeline, this would be not much of a problem, but doing this manually is cumbersome.

[abstractionfactory]

Thanks for raising this @ProbstDJakob. A direct download URL would require a web worker, which would be good to avoid. However, we could use the releases generator present in this repository to pre-generate a simple text and/or JSON file for each major and minor version that would contain the artifacts. This would make it feasible to pass simple version constraints to the POSIX/PowerShell scripts.

I raised this issue for the core team for discussion.

[ProbstDJakob]

I am not quite sure if I am understanding you correctly. To rephrase you in my own words, you do not host a web server like nginx or caddy to serve the content and thus can not write custom redirect rules. Instead you use a static file hosting service. If this is the case, a directory structure like the following could be generated on rollouts based on the current releases:

/
├── install-opentofu.sh
└── version-resolver
    ├── 1            # content: 1.8.2
    ├── 1.7          # content: 1.7.3
    ├── 1.7.3        # content: 1.7.3
    ├── 1.8          # content: 1.8.2
    ├── 1.8.0-beta1  # content: 1.8.0-beta1
    ├── 1.8.0-rc1    # content: 1.8.0-rc1
    ├── 1.8.1        # content: 1.8.1
    └── 1.8.2        # content: 1.8.2

This way no need for something like jq or other tools to parse the json is required and a simple request like curl "https://get.opentofu.org/version-resolver/$version" would return the expanded version (or 404).

[abstractionfactory]

We use CloudFlare to host static files. While we can technically use workers to create dynamic content, their use costs money and it would be good to avoid that. Other than that, yes, this would roughly be the approach.

[abstractionfactory]

Hey @ProbstDJakob we discussed this with the core team and we decided that this would be worth implementing in some form (probably by providing simple to parse download information in a file).

[ProbstDJakob]

That sounds great, thanks.