Skip to content

UpdateCLI scripts for versions.yml #1602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

theletterf
Copy link
Contributor

@theletterf theletterf commented Jul 22, 2025

Fixes #1498

+CC @reakaleek @bmorelli25

@v1v I've adapted the work you did for opentelemetry, would this be sufficient? Do we need different tokens, etc.?

@reakaleek
Copy link
Member

For stack version we can use https://artifacts.elastic.co/releases/stack.json

Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Early review, need to look it at again in my laptop

…tic/docs-builder into theletterf-add-updatecli-automation
@theletterf
Copy link
Contributor Author

theletterf commented Jul 23, 2025

@reakaleek Switched to the yaml kind for all targets and added source and target for stack using the JSON artifact as source. Any way we could dry-run this locally?

@theletterf theletterf requested review from v1v and reakaleek July 23, 2025 09:05
@reakaleek
Copy link
Member

@reakaleek Switched to the yaml kind for all targets and added source and target for stack using the JSON artifact as source. Any way we could dry-run this locally?

You should be able to run something like updatecli diff --config manifest.yaml.

But there might be some prequisites to make it work. I haven't worked with updatecli locally in a while.

However, I remember you need to provide all those requiredEnv variables to make it work.

@theletterf
Copy link
Contributor Author

@reakaleek If the syntax and content looks legit (are we covering all that need to be updated in versions.yml?) then I would crank this one manually once the required infra PRs are merged.

@reakaleek
Copy link
Member

@reakaleek If the syntax and content looks legit (are we covering all that need to be updated in versions.yml?) then I would crank this one manually once the required infra PRs are merged.

SGTM

@theletterf
Copy link
Contributor Author

+CC @colleenmcginnis

@v1v
Copy link
Member

v1v commented Jul 23, 2025

$ gh pr checkout 1602
$ GITHUB_TOKEN=$(gh auth token) \
   GITHUB_ACTOR=v1v \
   BRANCH_NAME=main \
   updatecli diff \
           --config updatecli/updatecli.d/versions.yml \
           --values updatecli/values.d/scm.yml --debug

It failed with:

+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "updatecli/updatecli.d/versions.yml"
DEBUG: pipelineid undefined, we'll try to generate one
DEBUG: using pipeline name to generate the pipelineid

SCM repository retrieved: 1
DEBUG: cloning git repository: https://github.com/v1v/docs-builder.git in /var/folders/t7/ghqdh8cx2g12pwb_w0ncmw900000gn/T/updatecli/github/v1v/docs-builder
Enumerating objects: 15121, done.
Counting objects: 100% (640/640), done.
Compressing objects: 100% (459/459), done.
Total 15121 (delta 428), reused 238 (delta 172), pack-reused 14481 (from 2)
DEBUG: Fetching remote branches for resetting local ones


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++
+ PIPELINE +
++++++++++++



####################################################
# BUMP RELEASE VERSIONS IN THE CONFIG/VERSIONS.YML #
####################################################

source: source#latest-apm-agent-dotnet-version
--------------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4900 (reset at 2025-07-23T14:22:40Z)
DEBUG: 58 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v1.32.2" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v1.32.2" to "1.32.2"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-edot-java-version
-------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4899 (reset at 2025-07-23T14:22:40Z)
DEBUG: 14 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v1.4.1" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v1.4.1" to "1.4.1"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-edot-node-version
-------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4898 (reset at 2025-07-23T14:22:40Z)
DEBUG: 26 releases found
Searching for version matching pattern "^v(\\d*).(\\d*).(\\d*)$"
✔ GitHub release version "v1.1.1" found matching pattern "^v(\\d*).(\\d*).(\\d*)$" of kind "regex"
[transformers]
✔ Result correctly transformed from "v1.1.1" to "1.1.1"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-edot-android-version
----------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4897 (reset at 2025-07-23T14:22:40Z)
DEBUG: 16 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v1.1.0" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v1.1.0" to "1.1.0"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-edot-python-version
---------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4896 (reset at 2025-07-23T14:22:40Z)
DEBUG: 16 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v1.4.0" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v1.4.0" to "1.4.0"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-apm-agent-node-version
------------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4895 (reset at 2025-07-23T14:22:40Z)
DEBUG: 52 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v4.13.0" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v4.13.0" to "4.13.0"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-apm-agent-ruby-version
------------------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
DEBUG: GitHub Api credit used 1, remaining 4894 (reset at 2025-07-23T14:22:40Z)
DEBUG: 4 releases found
Searching for version matching pattern "latest"
✔ GitHub release version "v4.8.0" found matching pattern "latest" of kind "latest"
[transformers]
✔ Result correctly transformed from "v4.8.0" to "4.8.0"
DEBUG: configuration "key" not set, defaulting to "tagname"

source: source#latest-ece-version
-------------------------
DEBUG: configuration "key" not set, defaulting to "tagname"
ERROR: 	Could not resolve to a Repository with the name 'elastic/cloud-enterprise'.
ERROR: ✗ Could not resolve to a Repository with the name 'elastic/cloud-enterprise'.
DEBUG: configuration "key" not set, defaulting to "tagname"

ERROR: something went wrong in "target#update-docs-docset-ece" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-dotnet" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-java" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-node" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-android" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-python" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-node" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-ruby" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-curator" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-php" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-collector" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-ecctl" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-java" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-php" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-stack" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-dotnet" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-rum" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-ios" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-go" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-eck" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed


ERROR: something went wrong in "target#update-docs-docset-apm-agent-python" : update pipeline: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed

Pipeline "Bump release versions in the config/versions.yml" failed
Skipping due to:
	something went wrong during target execution


ACTIONS
========

ERROR: Pipeline "Bump release versions in the config/versions.yml" failed to update: template: cfg:380:48: executing "cfg" at <source "latest-ece-version">: error calling source: parent source "latest-ece-version" failed

=============================

SUMMARY:



✗ Bump release versions in the config/versions.yml:
	Source:
		✔ [latest-apm-agent-dotnet-version] Get latest release version for APM .NET agent
		- [latest-apm-agent-go-version] 
		- [latest-apm-agent-java-version] 
		✔ [latest-apm-agent-node-version] Get latest release version for APM Node.js agent
		- [latest-apm-agent-php-version] 
		- [latest-apm-agent-python-version] 
		✔ [latest-apm-agent-ruby-version] Get latest release version for APM Ruby agent
		- [latest-apm-agent-rum-version] 
		- [latest-curator-version] 
		- [latest-ecctl-version] 
		✗ [latest-ece-version] Get latest release version for Elastic Cloud Enterprise
		- [latest-eck-version] 
		✔ [latest-edot-android-version] Get latest release version for the apm-agent-android
		- [latest-edot-collector-version] 
		- [latest-edot-dotnet-version] 
		- [latest-edot-ios-version] 
		✔ [latest-edot-java-version] Get latest release version for the elastic-otel-java
		✔ [latest-edot-node-version] Get latest release version for the elastic-otel-node
		- [latest-edot-php-version] 
		✔ [latest-edot-python-version] Get latest release version for the elastic-otel-python
		- [latest-stack-version] 
	Target:
		- [update-docs-docset-android] 
		- [update-docs-docset-apm-agent-dotnet] 
		- [update-docs-docset-apm-agent-go] 
		- [update-docs-docset-apm-agent-java] 
		- [update-docs-docset-apm-agent-node] 
		- [update-docs-docset-apm-agent-php] 
		- [update-docs-docset-apm-agent-python] 
		- [update-docs-docset-apm-agent-ruby] 
		- [update-docs-docset-apm-agent-rum] 
		- [update-docs-docset-collector] 
		- [update-docs-docset-curator] 
		- [update-docs-docset-dotnet] 
		- [update-docs-docset-ecctl] 
		- [update-docs-docset-ece] 
		- [update-docs-docset-eck] 
		- [update-docs-docset-ios] 
		- [update-docs-docset-java] 
		- [update-docs-docset-node] 
		- [update-docs-docset-php] 
		- [update-docs-docset-python] 
		- [update-docs-docset-stack] 

Likely, I don't have access to some of those GitHub repositories; hence this implementation requires a few things:

  1. Use our read-only GH app
  2. Use the GH token for accessing GH repositories
  3. Use the other GH token for writing the PR.

@theletterf
Copy link
Contributor Author

@v1v Removed ECE following advice from @florent-leborgne and @shainaraskas.

@theletterf
Copy link
Contributor Author

@v1v @reakaleek The script now extracts the stack version from the Elasticsearch repo because UpdateCLI uses Dasel v1 for JSON query, which lacks tail notation ([-1]). It also writes the stack version using regex because we're using a YAML anchor, and UpdateCLI doesn't know how to handle those. Could you give it a look?

@v1v
Copy link
Member

v1v commented Jul 23, 2025

The script now extracts the stack version from the Elasticsearch repo because UpdateCLI uses

do you know if https://github.com/elastic/docs-builder/pull/1602/files#diff-1f9aac33c46980c8e7615cefd22096957b1dfa1d377ebb8eef03f103d1953b6dR38-R40 is ordered by semver? Just wanna be sure it's not the first match.

@theletterf
Copy link
Contributor Author

@v1v I think so? It's the same tactic we use for the Collector to avoid catching 8.x versions.

Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this exhaustive for all versions.yml?

Will it open a single PR or multiple?

@v1v
Copy link
Member

v1v commented Jul 23, 2025

Will it open a single PR or multiple?

One single PR

@theletterf
Copy link
Contributor Author

@Mpdreamz only version we're missing is ECE, as explained above, I think, but I'd love it if @shainaraskas could validate it.

Copy link
Contributor

@shainaraskas shainaraskas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would nice to see an example output of this by manually downgrading the version #s in versions.yml and then running the workflow.

does this open PRs? is there boilerplate PR content?

as far as I see, docs eng are the codeowners for the config file ... probably not something to change right this second if the decoupling work is still in flight, but the people who have the context to block/approve this change might not be correctly positioned to do so right now

Copy link
Contributor

@shainaraskas shainaraskas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would nice to see an example output of this by manually downgrading the version #s in versions.yml and then running the workflow.

does this open PRs? is there boilerplate PR content?

as far as I see, docs eng are the codeowners for the config file ... probably not something to change right this second if the decoupling work is still in flight, but the people who have the context to block/approve this change might not be correctly positioned to do so right now

@theletterf
Copy link
Contributor Author

@shainaraskas Example PR: elastic/opentelemetry#330

@theletterf
Copy link
Contributor Author

@v1v @reakaleek Seems like we're ready for primetime? Let's merge once Victor confirms everything is in place on the Robots' side. :)

@theletterf theletterf requested a review from Mpdreamz July 24, 2025 09:30
@v1v
Copy link
Member

v1v commented Jul 24, 2025

Seems like we're ready for primetime? Let's merge once Victor confirms everything is in place on the Robots' side. :)

There are a couple of pending tasks:

  • Waiting to merge another PR to gain access to the secret management. GH secrets are now available.
  • Waiting for IT to install the GH app

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Automate versions.yml updates using updatecli
6 participants