sidecar hard link block: function not implemented

[CooperWanng]

In my deployment configuration, I mount the storage of prometheus (prometheus-k8s-db) to the minio bucket through s3fs or geesefs, so that even if the prometheus program crashes or the container is deleted, the data of nearly two hours can be restored through the wal in the bucket after restart.
However, thanos-sidecar made an error when moving tsdb-block. It seems that thanos-sidecar wants to make a hard link to the block file, but s3fs, geesefs and other s3 mount implementations do not support hard links, which makes it impossible to move tsdb-block to the target storage of thanos. I tried geesefs and s3fs, and both had this problem.
I want to know if there is any way to avoid this situation. Is hard link a must for thanos-sidecar? Is there any way to avoid generating hard links for files?
I am using thanos version 0.35

[dosubot[bot]]

Hello @CooperWanng! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve this issue.

I found some relevant information regarding your issue with hard links in Thanos Sidecar:

1. Similar Issues and Discussions:

   • Thanos hardlink operation not permitted (Unsolved discussion) ^[1].
   • thanos-sidecar : hard link block : operation not permitted (Closed issue) - Suggested solution: Ensure Thanos and Prometheus run as the same user ^[2].
   • thanos-sidecar : link: can't create hardlink : Operation not supported (Closed issue) - Suggested solution: Run Thanos and Prometheus as the same user ^[3].
   • sidecar Azure blob storage: hard link block - operation not supported (Closed issue) - Issue due to Azure Files not supporting hard links ^[4].

2. Necessity of Hard Links:

   • Hard links are currently a must for thanos-sidecar when moving TSDB blocks. The upload function in pkg/shipper/shipper.go uses hard links to create a temporary upload directory to avoid interference from other operations in the TSDB directory ^[5].

3. Workarounds:

   • There are documented workarounds for environments where hard links are not supported. These include handling eventual consistency and partial uploads without relying on hard links. Key points include block immutability, sync delay, using meta.json as an integrity marker, and handling partial uploads and overlaps gracefully ^[6].

Unfortunately, avoiding the generation of hard links is not currently supported in thanos-sidecar. You might need to consider these workarounds or ensure that your storage system supports hard links.

To continue talking to Dosu, mention @dosu.