[feature] e2e tests for new provenance-registry option

[laurentsimon]

Once #2956 is merged, let's add an e2e tests in the https://github.com/slsa-framework/example-package/tree/main/.github/workflows

Need to figure out which registry / account we use for the second registry.

/cc @saisatishkarra

[saisatishkarra]

@laurentsimon The #2962 is merged. I can work on the e2e test workflow. Is there a consensus on the second registry for the e2e?

[laurentsimon]

Let's try to use docker (our existing tests use GitHub registry, I think). I'll find a way to create an account for testing. Let me know if you're able to make sense of the e2e tests. We only need to add one test, you can use https://github.com/slsa-framework/example-package/blob/bf2a15594ee748d53bdea8e6b5471c77fe6135dd/.github/workflows/e2e.container.schedule.main.default.slsa3.yml#L86 as a starting point and update the names / options.

Verification starts in https://github.com/slsa-framework/example-package/blob/main/.github/workflows/scripts/e2e.container.default.verify.sh. As a first step, it's probably enough to export the COSIGN_X if the value is set in the config file (you can detect that by using a proper name for the workflow, see some examples for Go https://github.com/slsa-framework/example-package/blob/main/.github/workflows/scripts/e2e.go.default.verify.sh#L44-L46).

Once the slsa-verifier has a dedicated option, we can update the workflow. But let's start with COSIGN_X env variable, first?

Let me know if that makes sense or not.

Thanks again for the help, much appreciated!

[saisatishkarra]

@laurentsimon I looked at the tests and they make sense. I would start with e2e.container.schedule.main.provenance-registry.slsa3.yml and reuse most of the GHCR registry for pushing the image and add the new docker registry to push provenance to a separate registry.

Would you mind sharing the secrets / env values for the new docker repository for hosting provenance?

[laurentsimon]

I'l add the secrets into the settings. Will let you know this week when done.

[laurentsimon]

I've created a secret named E2E_DOCKER_HUB_TOKEN, username laurentsimon. Let's use this account for the container images, and the gcr.io for the provenance. Let's also delete the container when it's used.
Please cc me on the PR you send. Thank you!

[saisatishkarra]

@laurentsimon The current pipeline uses, Github registry (GHCR not gcr) for container image. I was hoping to use docker.io (new registry) for provenance. Did you mean to swap them as per your comment in #2981 (comment)?

[laurentsimon]

Correct: if possible we can swap them., unless it complicates things too much. My reasoning was that I'd like to delete the images after each run to avoid docker's complaining. I'm not sure how to delete the attestations only... but it's that possible let's do that instead. Maybe cosign has a command to delete all attestations for a given image?

[saisatishkarra]

Okay. If I get this right you would like to:

1. Publish Container image (workflow name) to docker.io instead of existing GHCR.io. If so,

• Does the repository exist in dockerhub?
• What is the secret / value of the repository name to publish image (NOT provenance) that I can leverage with user laurentsimon and token E2E_DOCKER_HUB_TOKEN
• Delete docker image at the end of workflow

2. Publish Container provenance image (attestation) to GHCR under slsa-framework/workflowname repository using GH_TOKEN and github.actor.

• Does this provenance need to be deleted as well?

[saisatishkarra]

@laurentsimon Here is the PR: slsa-framework/example-package#294.

Storage registries:

• Build container image - Dockerhub
• Container image Provenance - GHCR

LMK if anything needs changing. Can you confirm if you expect to delete the container image (workflow name) in the docker hub vs. the provenance package in GHCR vs. both? If so, do the e2e_docker_thub_oken and/or e2e_container_token (GHCR) have the delete permissions?

[laurentsimon]

deleting the image from docker should be enough. The token should have the right permissions. Thanks

[saisatishkarra]

@laurentsimon After some digging seems like docker is deprecating the Advanced Image management API and hub-tool (experimental) doesn't yet support providing password from env / doesn't obey docker login creds, and is pending.

Here are the 3 issues for deleting from dockerhub:

• Deprecation of Advanced Image Management docker/roadmap#534
• https://docs.docker.com/docker-hub/api/latest/#tag/images/operation/PostNamespacesDeleteImages
• Enable logging in via environment variables docker/hub-tool#198

2 Solution alternatives:

• Option 1: Push container image to GHCR and provenance image to dockerhub (i.e. Swap token permissions and registries for storing image and provenance). Leverage https://github.com/marketplace/actions/delete-ghcr-io-package-tag to delete docker container image from GHCR and leave out provenance in a docker registry

• Option 2: Push container image to dockerhub and provenance to GHCR. Skip deletion/pruning of images and document reasons.

Clarifications needed:

• LMK what option to proceed with?
• If Option 1 is chosen, confirm that the GH_TOKEN (E2E_CONTAINER_TOKEN) has permission to delete the image from GHCR / and What repository to push the provenance in docker hub (Currently Using workflow name as docker repo).

[saisatishkarra]

PR slsa-framework/example-package#294 merged for e2e

[saisatishkarra]

@ianlewis / @laurentsimon let me know if this feature is ready (i.e. the e2e test is working???) to be used and when the tag/release is made?

Also is there something that needs to be worked on on the verifier end?

[ianlewis]

Looks like it failed for some reason. I'll take a look on Monday.
#3024

[laurentsimon]

It's working now.