Deploy artifact to public.ecr.aws

[oscarbc96]

There is already an unused repository in the public AWS registry: aquasecurity/trivy-checks.

This PR follows the same approach as aquasecurity/trivy-java-db and aquasecurity/trivy-db, enabling publishing to both the GitHub registry and AWS.

Notes for reviewers:
Are the ECR_ACCESS_KEY_ID and ECR_SECRET_ACCESS_KEY secrets already available for this repo as organization secrets? If not, they will need to be added.

[simar7]

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

[oscarbc96]

@simar7 Thank you for the quick review!

While the embedded misconfiguration checks do provide a fallback, I think we should still consider publishing to AWS registries. In certain network scenarios, users may only have connectivity to private AWS registries. Publishing to public AWS allows users to set up a pull-through cache for easy access, as outlined here: AWS Pull-Through Cache.

Additionally, if users don’t frequently upgrade the Trivy binary —such as in environments with limited capacity to maintain up-to-date binaries— they may miss critical checks. Relying solely on the embedded checks could leave gaps if they aren’t retrieving the latest updates. They would be forced to use by default --skip-check-update.

[jlamande]

Hi,

I met similar ghcr rate limits on trivy-checks as trivy-db and trivy-java-db.

However while this bundle is not available on ECR Gallery, I have tried to publish it to a trivy-checks repository in my ECR Public registry using oras cp and oras push. But I always get a 500 Internal Server Error response from ECR.

$ oras cp "ghcr.io/aquasecurity/trivy-checks:1" "$MY_PUBLIC_REGISTRY_URL/aquasecurity/trivy-checks:1"
...
Error response from registry: recognizable error message not found: PUT "https://public.ecr.aws/v2/xxxxxx/aquasecurity/trivy-checks/blobs/uploads/1c986cfc-0886-427f-9cfa-c43315af22f4?digest=sha256%3Ae3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": response status code 500: Internal Server Error

same behavior with oras pull/push :

$ oras pull ghcr.io/aquasecurity/trivy-checks:1
$ oras push "$MY_PUBLIC_REGISTRY_URL/aquasecurity/trivy-checks:1" \
              --config /dev/null:application/vnd.cncf.openpolicyagent.config.v1+js \
              bundle.tar.gz:application/vnd.cncf.openpolicyagent.layer.v1.tar+gzip
....
Error response from registry: recognizable error message not found: PUT "https://public.ecr.aws/v2/xxxxxx/aquasecurity/trivy-checks/blobs/uploads/43a73f5a-dab6-43cc-8ab2-8be20437da29?digest=sha256%3Ae3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855": response status code 500: Internal Server Error

Did you succeed in publishing trivy-checks to ECR Gallery ?

[oscarbc96]

Hi @jlamande,
We didn't try to clone it in our internal registry. We were waiting for the output of this PR :)

[oscarbc96]

Hi @simar7, just wanted to check if you could reconsider this PR when you have a moment. I believe publishing to AWS registries could still provide significant benefits for users in certain network scenarios, as mentioned earlier. Appreciate your time and input on this. Thank you!

[ravenolf]

This would be very useful! Especially since it's pretty easy to hit the GHCR pull limits and it would be consistent with other trivy artifacts (trivy-db and trivy-java-db). Looking forward to it being merged!

[raul-travelperk]

Hey @simar7! Could you please review this PR at your convenience? Merging it would greatly benefit us and others by enabling seamless integration of Trivy into the development workflow, which would be highly valuable. Thanks for considering it!

[gnadaban]

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

Hi @simar7 , unfortunately trivy-checks is also getting rate-limited when using the trivy-operator. This PR would help the community immensely, not to mention sparing everyone from having to set up a pullthrough cache that, you may have guessed, is also getting rate-limited.

[phyzical]

can confirm were getting hit by the rate limit on policy download for trivy-operator via a helm install on aws eks

[saidsef]

Hi, Suddenly, my GH actions workflows are failing due to rate limit:

err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-java-db/manifests/1: TOOMANYREQUESTS: retry-after: 98.586µs, allowed: 44000/minute\n\n"

This PR would really be useful for the community.

[aonan-wyze]

It will still fail when doing misconfig test even if we have embed checks; I think this ECR approach is necessary @simar7 simar7

[andersthorbeck]

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

trivy-checks is also being rate limited, however it can still function using the embedded checks, as you say.

I've experienced rate limiting on both https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:c2b4fe1cd51083ede5606a38fb24e7fafb06fd2632c9cf6d9c63f5a80a6c67dc:

Running Trivy with options: trivy config ./alerting
2024-11-07T13:47:55Z	INFO	Loaded	file_path="trivy.yaml"
2024-11-07T13:47:55Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-11-07T13:47:55Z	INFO	[misconfig] Need to update the built-in checks
2024-11-07T13:47:55Z	INFO	[misconfig] Downloading the built-in checks...
2024-11-07T13:47:55Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:c2b4fe1cd51083ede5606a38fb24e7fafb06fd2632c9cf6d9c63f5a80a6c67dc: TOOMANYREQUESTS: retry-after: 433.892µs, allowed: 44000/minute"
2024-11-07T13:47:56Z	INFO	[terraform scanner] Scanning root module	file_path="."
2024-11-07T13:47:56Z	INFO	Detected config files	num=1

and on https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1

2024-11-07T14:03:53Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1: TOOMANYREQUESTS: retry-after: 619.858µs, allowed: 44000/minute\n\n"

If it's possible to update the https://github.com/aquasecurity/trivy-action?tab=readme-ov-file#updating-caches-in-the-default-branch section to include how to retrieve the trivy-checks and put them in the cache, then we could cache many more requests against Trivy.

[ktzsolt]

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

@simar7 as others also noted, unfortunately trivy-checks is also rate limited so publishing it also to ecr would be much appreciated

2024-11-07T16:24:13Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1: TOOMANYREQUESTS: retry-after: 354.558µs, allowed: 44000/minute\n\n"

[simar7]

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

@simar7 as others also noted, unfortunately trivy-checks is also rate limited so publishing it also to ecr would be much appreciated

2024-11-07T16:24:13Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1: TOOMANYREQUESTS: retry-after: 354.558µs, allowed: 44000/minute\n\n"

AFAIK rate limit issues have only been experienced with the vuln-db and the java-db. Trivy already embeds misconfiguration checks into the binary today so there's always a fallback in the case checks bundle wasn't retrievable.

trivy-checks is also being rate limited, however it can still function using the embedded checks, as you say.

I've experienced rate limiting on both https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:c2b4fe1cd51083ede5606a38fb24e7fafb06fd2632c9cf6d9c63f5a80a6c67dc:

Running Trivy with options: trivy config ./alerting
2024-11-07T13:47:55Z	INFO	Loaded	file_path="trivy.yaml"
2024-11-07T13:47:55Z	INFO	[misconfig] Misconfiguration scanning is enabled
2024-11-07T13:47:55Z	INFO	[misconfig] Need to update the built-in checks
2024-11-07T13:47:55Z	INFO	[misconfig] Downloading the built-in checks...
2024-11-07T13:47:55Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-checks/blobs/sha256:c2b4fe1cd51083ede5606a38fb24e7fafb06fd2632c9cf6d9c63f5a80a6c67dc: TOOMANYREQUESTS: retry-after: 433.892µs, allowed: 44000/minute"
2024-11-07T13:47:56Z	INFO	[terraform scanner] Scanning root module	file_path="."
2024-11-07T13:47:56Z	INFO	Detected config files	num=1

and on https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1

2024-11-07T14:03:53Z	ERROR	[misconfig] Falling back to embedded checks	err="failed to download built-in policies: download error: OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-checks/manifests/1: TOOMANYREQUESTS: retry-after: 619.858µs, allowed: 44000/minute\n\n"

If it's possible to update the https://github.com/aquasecurity/trivy-action?tab=readme-ov-file#updating-caches-in-the-default-branch section to include how to retrieve the trivy-checks and put them in the cache, then we could cache many more requests against Trivy.

It will still fail when doing misconfig test even if we have embed checks; I think this ECR approach is necessary @simar7 simar7

I'm not seeing this behavior. The error message you've posted also describes the fact that it's falling back to using embedded checks. Are you running the latest version of trivy-action?

[andersthorbeck]

The error message you've posted also describes the fact that it's falling back to using embedded checks. Are you running the latest version of trivy-action?

Yes, version 0.28.0.
It is indeed falling back to using the embedded checks, but we should still strive to be able to update them from remote to catch also more recent additions, no?

[ktzsolt]

The error message you've posted also describes the fact that it's falling back to using embedded checks. Are you running the latest version of trivy-action?

I am not using trivy-action or github at all in this context, I am using the trivy image from dockerhub (aquasec/trivy:0.57.0) to do the scans (in our onprem gitLAB ci pipeline) overwriting the following env vars in the image/container:

ENV TRIVY_DB_REPOSITORY=public.ecr.aws/aquasecurity/trivy-db
ENV TRIVY_JAVA_DB_REPOSITORY=public.ecr.aws/aquasecurity/trivy-java-db
# ENV TRIVY_CHECKS_BUNDLE_REPOSITORY=public.ecr.aws/aquasecurity/trivy-checks

Last one is not working because trivy-checks is not populated on ecr, but using the ghcr still gives TOOMANYREQUESTS error, the other 2 works fine.

The fallback to the built in checks is good to have, but the "error" in our logs gives noise and "alert fatige"

[nikpivkin]

We seem to have found a workaround using mirror.gcr.io aquasecurity/trivy#7538 (comment) . PR to publish trivy-checks on Docker Hub.