ObjectStorageProvider: add write back support

[rressi-at-globus]

Description

When we succesfully restore a binary package from the cloud storage we should cache it locally to avoid downloading it again.

This feature works at these conditions:

• we have configured binary caches on the file system with write permissions (default,write, default,readwrite, files,...,write, files,...,readwrite)
• we have configured binary caches on the cloud storage with read permissions (x-aws,...,read, x-aws,...,readwrite, x-cos,...,read, x-cos,...,readwrite, x-gcs,...,read, x-gcs,...,readwrite)

To cache it, it just moves the restored zip to the local cache directory instead of deleting it.

Design

flowchart TD
    subgraph Dev["Developer Machine"]
        A[Checked-out C++ Code]
        B["VCPKG Build Process"]
        C["Local Binary Cache (.zip files)"]
    end

    subgraph AWS["Build of the external component (CMake)"]
        D[GitHub checkout]
        E[CMake config]
        F[CMake build]
    end

    A --> B
    B -->|Check| C
    C -->|Hit| BDone[Use from Local Cache]
    C -->|Miss| FCheck[Check S3 Cache]
    FCheck -->|Hit| FDownload[Download from S3]
    FDownload --> B
    FDownload -.-> CUpdate
    FCheck -->|Miss| D
    D --> E
    E --> F
    F --> CUpdate["Update Local Cache"]
    CUpdate --> B
    BDone --> B

    style BDone fill:#ccffcc,stroke:#333
    style FDownload fill:#ccffcc,stroke:#333
    style CUpdate fill:#ccffcc,stroke:#333
    linkStyle 6 stroke:red,stroke-width:3px,stroke-dasharray: 5, 5

Loading

Rational

We have our developers compiling with their machines inside our offices around the world and they also often work from home. To speed-up their job we store precompiled externals in archives on AWS S3.

For externals we have an internal solution that uses local hard-drive as first layer cache and S3 as second layer. When something is not found localy, it is downloaded and then left on the local hard-drive for a while.

A part of the compilation time optimization, this is also helping a lot into keeping egress costs under control:

This is an example of a month where costs were higher than usual because for some few days the feature was not working after an upgrade of our tool.

This small PR is implementing with fews lines of code a similar strategy to VCPKG.

[Osyotr]

Related: #1406

[rressi-at-globus]

There are 3 PRs trying to implement write back support to this tool.

This is a must-have feature in many circumstances.

What distinguish this solution is the following:

• it is in a mergeable status.
• it is very small.
• it doesn't require much testing (it is already working very nicely on our side).
• but it is limited to Cloud Storage backends (which also reduces the things to be tested).

What about taking in this now and then looking forward for a more robust solution based on the other proposals in the mid-long term?

At the end this is the most efficient solution possible on the backends we are targeting:

• just move/rename the packages on local binary caches after aving used them successfully.

No extra copies, just one fast, atomic I/O operation.

[rressi-at-globus]

@BillyONeal what do you think about my last comment?

[EvilMcStevil]

This looks like a very positive pr, that just does local caching where it can, why is it not accepted? I understand all of the issues about the general write back support problem, but this fixes 90% of my use case, such that I have a script that does this manually already, which I run regularly, but it pulls in way more than i need or want.

[rressi-at-globus]

• I have just resolved all merge conflicts.
• Please consider this as an intermediate solution, it would reduce EGRES costs and download time to many people. On our site it has been pretty succesful so far.