feat: support PGO file

[v1v]

As long as the default.pgo file exists in https://github.com/elastic/apm-managed-service/ then this shared policy can be used to create the relevant PRs in the MIS projects.

Part of https://github.com/elastic/observability-robots/issues/2563

Producer

https://github.com/elastic/apm-managed-service/ needs to create the PGO file in the main branch.

Consumers

They need to create the file updatecli-compose.yaml that includes

# Config file for `updatecli compose ...`.
# https://www.updatecli.io/docs/core/compose/
policies:
  - name: Handle pgo files
    policy: ghcr.io/elastic/oblt-updatecli-policies/apm-managed-service/pgo:0.1.0
    values:
      - .ci/updatecli/values.d/scm.yml
      - .ci/updatecli/values.d/pgo.yml

and the relevant GitHub action running this.

For instance, see https://github.com/elastic/apm-server/blob/main/updatecli-compose.yaml and https://github.com/elastic/apm-server/blob/main/.github/workflows/update-compose.yml

Why

This approach has been done for the APM agent specs also:

• https://github.com/elastic/apm-data
• https://github.com/elastic/apm
• https://github.com/elastic/apm-agent-java/blob/main/updatecli-compose.yaml

Follow ups

• Provide PGO diff

[1pkg]

@v1v thank you for working on this, I think this pull based approach should work for PGO in MIS. Just to double check with you that I understand it correctly, to target an individual CPU profile PGO file from apm-managed-service we will need to use a similar configuration for updatecli, see an example for apm-ingest-service.

# Config file for `updatecli compose ...`.
# https://www.updatecli.io/docs/core/compose/
policies:
  - name: Handle pgo files
    policy: ghcr.io/elastic/oblt-updatecli-policies/apm-managed-service/pgo:0.1.0
    values:
      - .ci/updatecli/values.d/pgo.yml

scm:
  enabled: true
  owner: elastic
  repository: oblt-updatecli-policies
  username: obltmachine
  branch: main
  commitusingapi: true

# use one existing file until the pgo file is in the repository.
pgo_file: default.pgo
pgo_target_path: .
pgo_source_path: pgo/apm-ingest-service/default.pgo

[1pkg]

For description we actually need something more fancy, we want to print a small diff summary between old version of the PGO file and the new version that is getting raised in the PR. So it's easy to quickly preview what has changed and sanity check the differences. Right now it's done with the go tool pprof in https://github.com/elastic/apm-managed-service/pull/1365/files#diff-fc41f706e66d7e247dec9daa0e605237c1abcb49144cc0685892d82ab165279bR7, we will need to do something similar here as well.

[v1v]

We use gh api to collect the PR reference, so the PRs in the MIS projects will contain a reference in the PR description. I was thinking we could use the git commit title, but IIUC, go tool pprof needs to run for each specific MIS project.

Therefore, I need to figure out whether we can use something in the updatecli to run the diff, let me figure out if I can support that.

Do you happen to have two pgo files I can use to run go tool pprof and test if my implementation with the diff works as expected?

[1pkg]

Here few PGO files from the apm-server

new.pgo.log
old.pgo.log

go tool pprof -top -base=old.pgo.log new.pgo.log

[v1v]

to target an individual CPU profile PGO file from apm-managed-service we will need to use a similar configuration for updatecli

Correct

[1pkg]

@v1v could we move forward with merging these changes?

[v1v]

@v1v could we move forward with merging these changes?

I want to run some manual testing using the PGO files you sent me, I plan to work on it today

[v1v]

I've tested the pgo diff and it's not working as I wanted, there are some failures when adding the diff:

----
✔ shell command executed successfully
DEBUG: empty changelog found for the source
DEBUG: empty source detected

ERROR: something went wrong in "target#pgo" : update pipeline: template: cfg:9: unexpected "\\" in operand
ERROR: something went wrong in "source#pgo-compare" : update pipeline: template: cfg:9: unexpected "\\" in operand

Pipeline "automation: update PGO" failed
Skipping due to:
	something went wrong during target execution

Even though the content of the diff is found if I don't use it for creating the PR:

source: source#pgo-compare
------------------

The shell 🐚 command "/bin/sh /var/folders/t7/ghqdh8cx2g12pwb_w0ncmw900000gn/T/updatecli/bin/d5d381ffb4baed868b70565e2baa9ce56f10e6ad5d535f5ddd15a10ab73819e4.sh" ran successfully with the following output:
----
File: apm-server
Type: cpu
Time: Dec 1, 2024 at 6:16am (CET)
Duration: 590.34s, Total samples = 0 
Showing nodes accounting for 0, 0% of 0 total
      flat  flat%   sum%        cum   cum%
----
DEBUG: command stderr output was:
----
----

I guess something is not right in the content to be interpolated...

---

However, if no diff, then PRs are created:

• https://github.com/elastic/observability-robots-playground/pull/189 (that's one of my playground for testing things out)

If no strong opinion, I'd remove the pgo diff in this PR, so we can use as is and then figure out the pgo diff in a follow-up (either finding the root issue, or maybe adding a GH workflow in charge of adding a comment with the diff)

What do you think?

[1pkg]

@v1v I think, it's fine to add PGO diff in a followup. I have no concern against merging this change as-is for now. Thanks for experimenting with this.