[feature request] More global porcelain methods like `fsspec.listdir(...)` for basic usecases

[vadimkantorov]

Currently the user must instantiate a filesystem object manually (and thus know the filesystem name) to do analogue of os.listdir(...).

It would be nice to have more methods like existing fsspec.open(...) mimicking Python's core open(...). E.g. methods with similar invokation API as os.listdir(...), os.path.exists(...) etc (for simplifying porting existing scripts)

E.g. to be able to write generic code for simple usecases like:

# set up environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_ENDPOINT_URL
# python ls.py s3://mybucket/

import sys, fsspec

print(fsspec.listdir(sys.argv[1]))

My current crutch:

import sys, fsspec
path = sys.argv[1]
listdir = fsspec.filesystem(path.split('://')[0] if '://' in path else 'file').ls
# from os import listdir
print(listdir(path))

# python lsfsspec.py .
# python lsfsspec.py file://.
# python lsfsspec.py s3://mybucket/

---

A complementing idea could be creating fsspec.Path - an analogue of pathlib.Path object accepting arbitrary paths like s3:// or github:// which would instantiate/cache filesystem object automatically and use some global configuration like environment variables, and providing the methods which pathlib.Path currently supports

[martindurant]

Perhaps you want fsspec.generic? It has proven tricky to instruct users on how to best set up the defaults, though.

[vadimkantorov]

Maybe, but I also suggest that there should be more of basic stateless shortcuts with similar API as python builtin open / os.listdir/ os.path.exists / os.rename / os.remove etc (which could under the hood use the GenericFileSystem or instantiate file systems and set some of their settings from some global places like environment variables)

I think, this is a good solution for porting existing, simple I/O python codes

[martindurant]

It is an interesting idea, especially with a CLI on the front, but I think it would take some work.

[vadimkantorov]

well, my suggestion is that more of existing basic stateless I/O python code could be upgraded to using fsspec... fsspec.open(...) shortcut already exists, I'm proposing to have more of these mirroring the existing Python API's like os.listdir / os.rename / pathlib.Path etc... This would of course require fetching some config options from some global state (like env variables), and maybe even having some global config state (maybe fsspec.global_config or sth similar), but I think for easing of porting of existing code it's fine

I did not reflect on the CLI aspect, but I guess fsspec could provide some simple example reimpl of rclone or s3cmd, but working with all filesystems...

[martindurant]

Yes, fsspec.generic.rsync() already exists.
I think it would be reasonable to have fsspec.generic.* for all those functions, using the GenericFileSystem with "defaults".

Note, that the actual defaults for fsspec can be configured in a special file, see fsspec.config .

[vadimkantorov]

Also, maybe good would be to have a table in the docs mapping open / os.listdir / os.path.exists / os.path.join / os.rename / os.remove / etc function to the fsspec ones

E.g. then we can replace from os import rename to from fsspec import rename or something similar

[martindurant]

That's not a bad idea either!
There was talk about having a .path attribute on filesystems instances which emulates os.path for the specific implementation. That's obviously not the same thing, but related - but the effort to bring it about hasn't emerged.