8.x builds are failing

[ycombinator]

I just manually triggered the 8.x buildkite pipeline — https://buildkite.com/elastic/fleet-server/builds/5446 — and it fails one of the test cases for the TestAgentInstallSuite (complete logs) like so:

=== RUN   TestAgentInstallSuite/TestAPMInstrumentationPolicy
--
  | agent_install_test.go:416:
  | Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1730158375640266840/elastic/fleet-server/testing/e2e/agent_install_test.go:416
  | Error:      	Received unexpected error:
  | exit status 1
  | Test:       	TestAgentInstallSuite/TestAPMInstrumentationPolicy
  | Messages:   	elastic-agent install failed. command: /usr/bin/sudo /tmp/e2e-agent_install_test/elastic-agent-8.17.0-SNAPSHOT-linux-x86_64/elastic-agent install --fleet-server-es=http://localhost:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuMTo5ZEJEQjQzNVR4NmZxWVpXbko0MDR3 --fleet-server-insecure-http=true --fleet-server-host=0.0.0.0 --fleet-server-policy=fleet-server-apm --non-interactive, exit_code: 1, output: Installing in non-interactive mode.
  | [=== ] Service Started  [6s] Elastic Agent successfully installed, starting enrollment.
  | [=== ] Waiting For Enroll...  [6s] {"log.level":"info","@timestamp":"2024-10-28T23:47:10.048Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":483},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
  | {"log.level":"info","@timestamp":"2024-10-28T23:47:12.061Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":826},"message":"Fleet Server - Error - failed version compatibility check with elasticsearch (Agent: 8.17.0-SNAPSHOT, Elasticsearch: 8.16.0): unsupported version","ecs.version":"1.6.0"}
  | Error: fleet-server failed: timed out waiting for Fleet Server to start after 2m0s
  | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.17/fleet-troubleshooting.html

The notable error is this:

Fleet Server - Error - failed version compatibility check with elasticsearch (Agent: 8.17.0-SNAPSHOT, Elasticsearch: 8.16.0): unsupported version

I believe this is happening because of this line, which should point to 8.17.0-xxxxxxxx-SNAPSHOT but is instead pointing to 8.16.0-xxxxxxxx-SNAPSHOT:

fleet-server/dev-tools/integration/.env

Line 3 in fc71697

ELASTICSEARCH_VERSION=8.16.0-02af53da-SNAPSHOT

That line is supposed to be kept up to date by automation PRs such as this example (this one is targeting the main branch, which points to 9.0.0). However, we have not seen such automation PRs targeting the 8.x branch. I believe this might be happening because the list returned by

fleet-server/.github/workflows/bump-elastic-stack-snapshot.yml

Lines 13 to 20 in a3fd826

	filter:
	runs-on: ubuntu-latest
	timeout-minutes: 1
	outputs:
	matrix: ${{ steps.generator.outputs.matrix }}
	steps:
	- id: generator
	uses: elastic/oblt-actions/elastic/active-branches@v1

does not contain 8.x. @v1v could you help look into this please? Thanks!

[v1v]

. @v1v could you help look into this please? Thanks!

This is now fixed. 8.x was considered a valid minor version and was removed as we only supported up to 2 active 8 minor versions.

This might not be true anymore with the new release cadence model - but that's another thing we have not planned yet in our end to be supported - release team might provide an official source of truth for the current active branches (still waiting for their feedback)