Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: replace image pull and push library from ORAS to crane #3559

Draft
wants to merge 58 commits into
base: main
Choose a base branch
from

Conversation

AustinAbro321
Copy link
Contributor

@AustinAbro321 AustinAbro321 commented Mar 7, 2025

Description

This replaces the Crane library uses for image pulls and pushes with ORAS. This does not change zarf tools crane operations.

Proposal - ZEP-0019: Refactor Image operations #20

This changes the default value to --oci-concurrency from three to six. Crane pushes and pulls all layers concurrently always, and the replaced implementation pushed multiple images at a time. Therefore, no matter what number it will still be less than it was previously.

One unfortunate side effect of this PR is that when Syft or Steroscope reads images from the cache it adds the image layers back to the cache. It is expecting a cache format that Crane uses, but still works with a traditional OCI directory. I'll need to look into how this can be mitigated so we don't make things messier than necessary.

Related Issue

Fixes #3434
Fixes #3194
Relates to #3113
Relates to #2104

Checklist before merging

Sorry, something went wrong.

Copy link

netlify bot commented Mar 7, 2025

Deploy Preview for zarf-docs canceled.

Name Link
🔨 Latest commit 3106d02
🔍 Latest deploy log https://app.netlify.com/sites/zarf-docs/deploys/67d33d59f64d690008721370

Copy link

codecov bot commented Mar 7, 2025

Codecov Report

Attention: Patch coverage is 48.58871% with 255 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/internal/packager/images/pull.go 56.80% 93 Missing and 15 partials ⚠️
src/internal/packager/images/cache/cache.go 49.35% 31 Missing and 8 partials ⚠️
src/internal/packager/images/push.go 52.56% 26 Missing and 11 partials ⚠️
src/internal/packager/images/common.go 44.18% 20 Missing and 4 partials ⚠️
src/internal/packager2/mirror.go 0.00% 16 Missing ⚠️
src/pkg/cluster/tunnel.go 0.00% 9 Missing ⚠️
src/pkg/utils/image.go 0.00% 9 Missing ⚠️
src/internal/packager2/layout/create.go 0.00% 6 Missing ⚠️
src/internal/packager2/layout/package.go 0.00% 2 Missing ⚠️
src/internal/packager2/layout/sbom.go 0.00% 2 Missing ⚠️
... and 3 more
Files with missing lines Coverage Δ
src/cmd/viper.go 53.08% <100.00%> (ø)
src/cmd/package.go 42.81% <0.00%> (-0.07%) ⬇️
src/pkg/packager/creator/normal.go 7.12% <0.00%> (+0.18%) ⬆️
src/pkg/packager/deploy.go 5.40% <0.00%> (-0.01%) ⬇️
src/internal/packager2/layout/package.go 38.37% <0.00%> (-2.14%) ⬇️
src/internal/packager2/layout/sbom.go 51.02% <0.00%> (+0.62%) ⬆️
src/internal/packager2/layout/create.go 40.66% <0.00%> (+0.15%) ⬆️
src/pkg/cluster/tunnel.go 10.10% <0.00%> (-0.26%) ⬇️
src/pkg/utils/image.go 0.00% <0.00%> (ø)
src/internal/packager2/mirror.go 0.00% <0.00%> (ø)
... and 4 more
🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@RothAndrew
Copy link
Contributor

let's gooooooo! so excited for this lol

AustinAbro321 and others added 27 commits March 12, 2025 16:33
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
…3550)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
git push

Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
@AustinAbro321 AustinAbro321 force-pushed the move-image-operations-to-oras branch from a4f9e7a to 01c20ba Compare March 12, 2025 16:36
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
@@ -153,6 +154,15 @@ func (c *Cluster) ConnectToZarfRegistryEndpoint(ctx context.Context, registryInf
if tunnel, err = c.NewTunnel(ZarfNamespaceName, SvcResource, ZarfRegistryName, "", 0, ZarfRegistryPort); err != nil {
return "", tunnel, err
}
} else if dns.IsServiceURL(registryInfo.Address) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the packager2 mirror refactor the push logic was changed to take Kubernetes services as input, this brings this function in parity with that.

Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate off of crane for pulling and pushing OCI images image pull fail from corrupted Crane cache
2 participants