URL extension to include unambiguous (http) host information

[dsgibbons]

Is there any appetite for embedding host/endpoint information within fsspec URLs? Currently, protocols like s3:// or even hf:// can be ambiguous as they may refer to local servers as opposed to their more popular web counterparts.

I know I can configure local variables to point to the server of my choice (via AWS config file, environment variables, or even local Python variables) but I'd prefer to have this information captured in the URL itself. Some ideas for how this could be achieved:

• Embed in query parameters: s3://bucket/path?host=my-minio-server.com&scheme=https
• Add an @ character somewhere: s3://[email protected]/path
• Overload the chaining syntax (probably hacky, but was my first naive guess to see if fsspec already supported this): s3://bucket/path::https://my-minio-server.com

I haven't seen anyone else asking for this, so I doubt this will go anywhere, but I'd be interested in finding other people that also care about this problem.

[martindurant]

There are certainly example for some backends that do this kind of thing (ftp, ssh, github come to mind). In those cases, we are making use of established URL patterns - although with some invention also for the case of github.

How to formulate this kind of thing might be tricky - we'd be making a pattern that doesn't exist anywhere else. I think your idea for chaining isn't a bad one: passing that URL to fsspec.open would result in these kwargs going to S3FileSystem:

{'fo': 'https://my-minio-server.com', 'target_options': {}, 'target_protocol': 'https'}

which is essentially the information you are after.