rsync accross FS with different protocol or credentials is broken

[adm271828]

Hi,

I'm completely new to fsspec and was extremely interested by the rsync utility (even if signaled as experimental) and gave it a try (version 2025.5.1).

My first use case is to sync a S3 source to a local directory.

Having a quick look at the source code, I tried something like this:

rsync('s3://bucket-name/key', './path/to/local/directory',
  inst_kwargs = {
    'default_method': 'options',
    'storage_options': {
      's3': { 'endpoint_url': 'xxxxxx', 'key': 'xxxxxxx', 'secret': 'xxxx', },
    },
  }
)

The idea being to let the GenericFileSystem instance used by rsync resolve both underlying filesystem behind source and destination.

Unfortunately this didn't work, because the GenericFileSystem:

• didn't override all methods rsync uses, namely isdir
• didn't resolve correctly URL

First problem can be solved by adding the following method to GenericFileSystem (I have absolutely no idea if this is correct, it simply mimics the way it is done for other methods):

    async def _isdir(
        self,
        url,
        **kwargs,
    ):
        fs = _resolve_fs(url, self.method, storage_options=self.st_opts)
        if fs.async_impl:
            return await fs._isdir(url, **kwargs)
        else:
            return fs.isdir(url, **kwargs)

Second problem was addressed by making sure every call to _resolve_fs uses the storage_options=self.st_opts argument (it is missing in many places).

Then it worked.

This is however not very efficient, because every call to _resolve_fs creates a new instance. rsync should probably call it only twice (once for the source, once for the destination, if they do not have the same protocol).

Then came the next question: what if I want to synchronise two S3 directories in differents buckets with different credentials? Also a use case I have.

Since _resolve_fs only uses the protocol to find a filesystem instance, this will not work.

Among the possible solutions I figured out:

• let _resolve_fs handle URL as URI and discriminate both on protocol and authority. At least the bucket name could be used to select different credentials in inst_kwargs. This would also probably work with other protocols
• make rsync take a source_fs and destination_fs argument instead of a unique fs

Hope this helps.

Best regards,

Antoine

[martindurant]

didn't override all methods rsync uses, namely isdir

This is surprising, since it ought to fall back to info() which in turn falls back to ls()[0]. It would be useful to form a test function in a PR, so that we can then fix it. Your code snippet looks reasonable.

didn't resolve correctly URL

Existing usage is via "default", "current" and "generic". I am not too surprised if the options are not being passed around correctly - again, a case where we need more testing. Perhaps we should implement the abstract FS bulk tests for generic.

[adm271828]

Thanks for your answer

didn't override all methods rsync uses, namely isdir

This is surprising, since it ought to fall back to info() which in turn falls back to ls()[0]. It would be useful to form a test function in a PR, so that we can then fix it. Your code snippet looks reasonable.

Not sure I can do more than the rsync call above. This is exactly as I did it (except for the true local path, true credential, and true S3 URL I can't obviously provide).

But you are right, I first tried to fix isdir, but the problem came in the first place because _resolve_fs was not called with storage_options everywhere.

I removed the redefinition of isdir and simply made sure every _resolve_fs call was given the storage_options=self.st_opts and this is sufficient (easy fix).

didn't resolve correctly URL

Existing usage is via "default", "current" and "generic". I am not too surprised if the options are not being passed around correctly - again, a case where we need more testing. Perhaps we should implement the abstract FS bulk tests for generic.

In fact I searched for rsync already open issues... 5 minutes after I made my post, and I found out explanations about 'current', 'generic'.

I'm not a great fan of such design, but anyway. The main concern I have is that in my test case, _resolve_fs is called 3 times for the S3 FS (source) and 2 times for the local FS (destination). If the config is stored in a module variable (or class variable such as current), do we have a warranty that all those calls will return the (same) good configuration accross multiple calls of _resolve_fs inside a unique rsync call, with async io?

[martindurant]

I removed the redefinition of isdir and simply made sure every _resolve_fs call was given the storage_options=self.st_opts and this is sufficient (easy fix).

Please push a fix, ideally with a test!

I'm not a great fan of such design, but anyway.

Happy to have alternatives. As might be obvious, GenericFS was conceived first, possibly as a general-purpose CLI handler; but actually rsync is by far the most useful and used case. Anything we can do to make rsync easier to use...

_resolve_fs is called 3 times for the S3 FS (source) and 2 times for the local FS (destination)

This wouldn't normally create new instances/connections, since fsspec caches them anyway - runtime isn't an issue, therefore. Also, rsync() itself is a blocking function, so you don't need to worry about async race conditions. However, the dict of filesystems could be updated in another thread, so you have a good point.