Roadmap for Lazy Syncing / Remote-only Repositories for Debian/Ubuntu

[hilariosantos91]

I am currently managing a large number of Ubuntu repositories in Uyuni, and the local storage requirements are becoming unsustainable. As we know, Debian-based repos tend to consume massive amounts of disk space because Uyuni mirrors every .deb file locally.

Is there any roadmap to implement Lazy Syncing (downloading packages only when requested by a client)?

Specifically, is there a way to configure a repository as 'Remote-only', where Uyuni would only parse the metadata (e.g., the Packages.gz or Release files) and proxy the actual package download from the upstream source on demand?

Any tips on how to optimize storage for Ubuntu without performing a full sync would be greatly appreciated.

Thanks!"

[rjmateus]

We did some research on that area and we have some ideas, but the implementation would be complex with a lot of corner cases. You should not expect any change in the near future (in a few months).
If you are using custom repositories you can use some expressions to exclude packages, if you know you will not need them.

[sbluhm]

Some work has already been done at openSUSE/mentoring#206

Discussion thread: #9120 (Research: ideas about lazy repo-sync project)

Feature Request: #3296 (Feature Request: thin-provisioned repositories / download policies)

Pull requests:
#9679 (Lzdownload service - download all packages of a given channel)
#9551 (Make the lzreposync loop over channels and synchronize them using a Status column)
#9101 (Parse and import both rpm and deb packages metadata)

@rjmateus @cbosdo @agraul , I am a fan of implementing things in stages to have something even partially usable instead of nothing at all. Especially when only parts of the features are required for some users.
That way, users that need the missing features can also contribute independently with an already existing direction. I believe this reduces the barrier to contribution.

Does it make sense to create a project here with the activities and can the code from openSUSE/mentoring#206 be shared/committed?

[cbosdo]

I still have on my todo list to review, adapt and merge those big PRs. The issue we have now is how to implement the download part: it's a bit trickier since multiple threads could try to get the same package that is still not downloaded on the server. Any implementation idea on that problem is more than welcomed.

[rjmateus]

One of the ideas of making it incremental was to implement the lazy repo-sync with the possibility to define different download policies.
This would allow us to provide some value to users until we have a proper solution for the issue in the download endpoint when the package is not present (it's more complex than it looks).

In the reposync code we could have it in steps, where first we process the channel metadata. As an after step, it would set which packages should be downloaded based on the defined policy. Examples of policies:

• All: Download all packages
• Latest: only the latest version of each package
• None: do not download packages

Then we would have a new service which would download the packages marked to download, and could be running in a higher priority.

The download endpoint used by the minion would be changed to fail the package was not available locally and "mark" it to be downloaded as soon as possible. This would not be the ideal solution for the download endpoint, but it would already bring a lot of value.

With this approach, if the download policy is defined to "all" the behaviour would be similar to what we have now. But this would give us flexibility to improve it in the future (or even allow users to create their own download policies).

From my point of view, the next step is not to merge the PR's but instead have a proper design of what we want to implement. That would make it clear which path we want to follow before transforming a crucial part of the application.

[sbluhm]

Adding the discussion thread: #9120 (Research: ideas about lazy repo-sync project)