Storage migration (MinIO to PVC)

[aureq]

I currently have a Harbor setup that uses MinIO as the backend storage. This works fine, but I'd like to move away from MinIO. I use Longhorn as my PVC storage layer. I'm considering using a PVC to store all the container image blobs.

• Is there a (community) guide to perform such change?
• If I were to copy all the blogs from one backend to the other, will Harbor able to find what it needs to serve images?
• Do I need to patch anything in the database?
• Anything else I should be considering?

Context:
I can't use AWS or Cloud vendor as the cluster runs on baremetal (at home) and I'd rather keep things locally for cost and latency reasons. Harbor is deployed via the official Helm chart.

[chlins]

I think the issue is about how to migrate the minio based storage to the longhorn based PVC?, we have no experience for that but normally a safe way is you can setup a new harbor instance with the new storage and replicate the images from the old one.

[phin1x]

Maybe I'm a little late, but if anyone has the same question.
the file structure of the registry is the same on every storage backend (file, s3, gcp, ....). so just copy the files recursively from the s3 root to your pvc root (e.g. aws s3 sync s3://your-bucket-name /path/to/pvc) (or vice versa, if someone wants to migrate from pvc to s3). as the database is only used by the harbor components and not by the registry (distribution), the database does not need to be touched. we have migrated a 5TB harbor from pvc to s3 in this way to increase high availability.

[microyahoo]

Do you mean that only need to migrate the image storage data, regardless of the harbor database records? However, user, artifact, project member, repository etc. are all recorded in the database. @phin1x

[phin1x]

The storage of the oci artifacts is handle by the underlying distribution, formerly known as the docker registry. Since harbor only stores references to the artifacts in the distribution, you don't need to touch the database. Users, project member, ... are also things that are unknown to the distribution.

[microyahoo]

Replication allows user to replicate all projects and artifacts to another harbor.
https://goharbor.io/docs/edge/administration/configuring-replication/

[aureq]

After fiddling, I finally managed to set some time aside and I was able to migrate all the image blobs from MinIO to a Volume. Thanks in to @phin1x for their input. As noted by @chlins Harbor seems to recommend having 2 registries, so this method is likely unsupported.

Here are the steps I followed

1. Take note of the MinIO server URL, the bucket name, any bucket object prefix, the access key and the access secret key
2. Create a PVC of the desired size that will host the image blobs (harbor-storage-pvc) in Harbor's namespace
3. Ensure no Harbor registry operations would occur during the operation (best is probably to temporarily uninstall Harbor)
4. Create a MinIO client Pod as shown below, with the PVC attached
5. Open a shell in the MinIO Pod
6. Configure the MinIO client mc alias set src http://minio:9000 ACCESS_KEY ACCESS_SECRET_KEY
7. Mirror the bucket onto the volume mc mirror src/harbor/objectprefix /dest/

At this point, /dest/ should contain a directory named docker/ (And docker/ contains registry/).

8. Run the command sync to flush all volume operations
9. Delete the MinIO client Pod
10. Deploy Harbor but this time using the filesystem and the correct PVC reference as persistent storage.

Once you've been able to validate you're able to pull and push images, you may delete the relevant MinIO bucket or instance..

apiVersion: v1
kind: Pod
metadata:
  name: minio-client
spec:
  containers:
  - name: minio-client
    image: minio/mc:latest
    command: ["/bin/sleep"]
    args: ["86400"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - name: harbor-storage-vol
        mountPath: /longhorn
  volumes:
  - name: harbor-storage-vol
    persistentVolumeClaim:
      claimName: harbor-storage-pvc

[haiwu]

@aureq : May I ask why you want to migrate from MinIO to PVC? What are the cons of MinIO (or Pros of PVC) in your case that you would want to do this migration?

I am currently trying to see which way to go with, MinIO or PVC for harbor..

[aureq]

@haiwu Sure! For context, a run a small 3-nodes k8s cluster and it works really well (3x udoo-v8 with nvme). MinIO was simple and good enough for my initial setup. I liked the idea of an S3 clone. But some times ago, they introduce erasure coding (ref) which is a feature that 1) broke all my past MinIO deployment, 2) required a manual migration 3) the objects data wouldn't be stored in plain text since it would contain extra data for integrity and reconstruction and 5) part of erasure coding MinIO would regularly scan for silent data corruption on the stored objects driving additional activities on the cluster. And possibly 6, since I wasn't using the MinIO controller (multiple helm charts instead), having to upgrade my MinIO deployments on a regular basis was just a little too much.

Both 4 and 5 are problematic to my setup since I want to have minimal IOs on the cluster as a way to increase the nvmes lifespan, increase availability and have a fairly easy way to recover data when needed. And since my Lonhorn setup has volumes replicated across 3 nodes and with regular cloud backups, erasure coding wasn't necessary at all.

In what I'm saying above, it's important to note that my setup is a low budget, no sla and getting the best perf for watt kind of setup.