From 65764ddadf66ec5745dfb9f119e89fd4186dbbd2 Mon Sep 17 00:00:00 2001 From: mxmlnkn Date: Thu, 10 Oct 2024 20:58:26 +0200 Subject: [PATCH 1/4] Add other known implementations under the fsspec Github namespace --- docs/source/api.rst | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/source/api.rst b/docs/source/api.rst index 9fdcd1f17..7248bb7f5 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -209,40 +209,46 @@ Built-in Implementations Other Known Implementations --------------------------- -- `abfs`_ for Azure Blob service -- `adl`_ for Azure DataLake storage +- `abfs`_ for Azure Blob service, with protocol "abfs://" +- `adl`_ for Azure DataLake storage, with protocol "adl://" - `alluxiofs`_ to access fsspec implemented filesystem with Alluxio distributed cache -- `boxfs`_ for access to Box file storage -- `dropbox`_ for access to dropbox shares +- `boxfs`_ for access to Box file storage, with protocol "box://" +- `dropbox`_ for access to dropbox shares, with protocol "dropbox://" - `dvc`_ to access DVC/Git repository as a filesystem -- `gcsfs`_ for Google Cloud Storage +- `gcsfs`_ for Google Cloud Storage, with protocol "gcs://" - `gdrive`_ to access Google Drive and shares (experimental) - `huggingface_hub`_ to access the Hugging Face Hub filesystem, with protocol "hf://" -- `lakefs`_ for lakeFS data lakes -- `ocifs`_ for access to Oracle Cloud Object Storage +- `ipfsspec`_ for the InterPlanetary File System (IPFS), with protocol "ipfs://" +- `lakefs`_ for lakeFS data lakes, with protocol "lakefs://" +- `ocifs`_ for access to Oracle Cloud Object Storage, with protocol "oci://" - `ocilake`_ for OCI Data Lake storage - `ossfs`_ for Alibaba Cloud (Aliyun) Object Storage System (OSS) - `p9fs`_ for 9P (Plan 9 Filesystem Protocol) servers -- `s3fs`_ for Amazon S3 and other compatible stores +- `s3fs`_ for Amazon S3 and other compatible stores, with protocol "s3://" +- `sshfs`_ for access to SSH servers, with protocol "ssh://" or "sftp://" +- `swiftspec`_ for OpenStack SWIFT, with protocol "swift://" - `wandbfs`_ to access Wandb run data (experimental) -- `webdav4`_ for WebDAV +- `webdav4`_ for WebDAV, with protocol "webdav://" or "dav://" - `xrootd`_ for xrootd, with protocol "root://" .. _abfs: https://github.com/dask/adlfs .. _adl: https://github.com/dask/adlfs .. _alluxiofs: https://github.com/fsspec/alluxiofs .. _boxfs: https://github.com/IBM/boxfs -.. _dropbox: https://github.com/MarineChap/intake_dropbox +.. _dropbox: https://github.com/fsspec/dropboxdrivefs .. _dvc: https://github.com/iterative/dvc .. _gcsfs: https://gcsfs.readthedocs.io/en/latest/ .. _gdrive: https://github.com/fsspec/gdrivefs .. _huggingface_hub: https://huggingface.co/docs/huggingface_hub/main/en/guides/hf_file_system +.. _ipfsspec: https://github.com/fsspec/ipfsspec .. _lakefs: https://github.com/appliedAI-Initiative/lakefs-spec -.. _ocifs: https://pypi.org/project/ocifs +.. _ocifs: https://ocifs.readthedocs.io/en/latest/ .. _ocilake: https://github.com/oracle/ocifs .. _ossfs: https://github.com/fsspec/ossfs .. _p9fs: https://github.com/pbchekin/p9fs-py .. _s3fs: https://s3fs.readthedocs.io/en/latest/ +.. _sshfs: https://github.com/fsspec/sshfs +.. _swiftspec: https://github.com/fsspec/swiftspec .. _wandbfs: https://github.com/jkulhanek/wandbfs .. _webdav4: https://github.com/skshetry/webdav4 .. _xrootd: https://github.com/CoffeaTeam/fsspec-xrootd From 19546109601b514facdd1a42dd79db99e278ccb7 Mon Sep 17 00:00:00 2001 From: mxmlnkn Date: Sat, 12 Oct 2024 22:42:17 +0200 Subject: [PATCH 2/4] Add more other known implementations --- docs/source/api.rst | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/source/api.rst b/docs/source/api.rst index 7248bb7f5..21058ad45 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -213,21 +213,31 @@ Other Known Implementations - `adl`_ for Azure DataLake storage, with protocol "adl://" - `alluxiofs`_ to access fsspec implemented filesystem with Alluxio distributed cache - `boxfs`_ for access to Box file storage, with protocol "box://" +- `csvbase`_ for access to csvbase.com hosted CSV files, with protocol "csvbase://" - `dropbox`_ for access to dropbox shares, with protocol "dropbox://" - `dvc`_ to access DVC/Git repository as a filesystem +- `fsspec-encrypted`_ for transparent encryption on top of other fsspec filesystems. - `gcsfs`_ for Google Cloud Storage, with protocol "gcs://" - `gdrive`_ to access Google Drive and shares (experimental) +- `git`_ to access Git repositories - `huggingface_hub`_ to access the Hugging Face Hub filesystem, with protocol "hf://" +- `hdfs-native`_ to access Hadoop filesystem, with protocol "hdfs://" +- `httpfs-sync`_ to access HTTP(s) files in a synchronous manner to offer an alternative to the aiohttp-based implementation. - `ipfsspec`_ for the InterPlanetary File System (IPFS), with protocol "ipfs://" +- `irods`_ for access to iRODS servers, with protocol "irods://" - `lakefs`_ for lakeFS data lakes, with protocol "lakefs://" +- `morefs`_ for `OverlayFileSystem`, `DictFileSystem`, and others - `ocifs`_ for access to Oracle Cloud Object Storage, with protocol "oci://" - `ocilake`_ for OCI Data Lake storage - `ossfs`_ for Alibaba Cloud (Aliyun) Object Storage System (OSS) - `p9fs`_ for 9P (Plan 9 Filesystem Protocol) servers +- `PyAthena`_ for S3 access to Amazon Athena, with protocol "s3://" or "s3a://" +- `PyDrive2`_ for Google Drive access - `s3fs`_ for Amazon S3 and other compatible stores, with protocol "s3://" - `sshfs`_ for access to SSH servers, with protocol "ssh://" or "sftp://" - `swiftspec`_ for OpenStack SWIFT, with protocol "swift://" - `wandbfs`_ to access Wandb run data (experimental) +- `wandbfsspec`_ to access Weights & Biases (experimental) - `webdav4`_ for WebDAV, with protocol "webdav://" or "dav://" - `xrootd`_ for xrootd, with protocol "root://" @@ -235,21 +245,31 @@ Other Known Implementations .. _adl: https://github.com/dask/adlfs .. _alluxiofs: https://github.com/fsspec/alluxiofs .. _boxfs: https://github.com/IBM/boxfs +.. _csvbase: https://github.com/calpaterson/csvbase-client .. _dropbox: https://github.com/fsspec/dropboxdrivefs .. _dvc: https://github.com/iterative/dvc +.. _fsspec-encrypted: https://github.com/thevgergroup/fsspec-encrypted .. _gcsfs: https://gcsfs.readthedocs.io/en/latest/ .. _gdrive: https://github.com/fsspec/gdrivefs +.. _git: https://github.com/iterative/scmrepo +.. _hdfs-native: https://github.com/Kimahriman/hdfs-native/blob/master/python/hdfs_native/fsspec.py +.. _httpfs-sync: https://github.com/moradology/httpfs-sync .. _huggingface_hub: https://huggingface.co/docs/huggingface_hub/main/en/guides/hf_file_system .. _ipfsspec: https://github.com/fsspec/ipfsspec -.. _lakefs: https://github.com/appliedAI-Initiative/lakefs-spec +.. _irods: https://github.com/xwcl/irods_fsspec +.. _lakefs: https://github.com/aai-institute/lakefs-spec +.. _morefs: https://github.com/iterative/morefs .. _ocifs: https://ocifs.readthedocs.io/en/latest/ .. _ocilake: https://github.com/oracle/ocifs .. _ossfs: https://github.com/fsspec/ossfs .. _p9fs: https://github.com/pbchekin/p9fs-py +.. _PyAthena: https://github.com/laughingman7743/PyAthena +.. _PyDrive2: https://github.com/iterative/PyDrive2 .. _s3fs: https://s3fs.readthedocs.io/en/latest/ .. _sshfs: https://github.com/fsspec/sshfs .. _swiftspec: https://github.com/fsspec/swiftspec .. _wandbfs: https://github.com/jkulhanek/wandbfs +.. _wandbfsspec: https://github.com/alvarobartt/wandbfsspec .. _webdav4: https://github.com/skshetry/webdav4 .. _xrootd: https://github.com/CoffeaTeam/fsspec-xrootd From ccb149b8a1a0c88c77c9d90a18353b45e4afeb7a Mon Sep 17 00:00:00 2001 From: Martin Durant Date: Thu, 14 Nov 2024 09:30:41 -0500 Subject: [PATCH 3/4] Update docs/source/api.rst --- docs/source/api.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/api.rst b/docs/source/api.rst index 21058ad45..b5912064a 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -209,6 +209,10 @@ Built-in Implementations Other Known Implementations --------------------------- + +Note that most of these projects are hosted outside of the `fsspec` organisation. Please read their +documentation carefully before using any particular package. + - `abfs`_ for Azure Blob service, with protocol "abfs://" - `adl`_ for Azure DataLake storage, with protocol "adl://" - `alluxiofs`_ to access fsspec implemented filesystem with Alluxio distributed cache From 92b93dad529d537cbaa6682fe13007eb654fd79e Mon Sep 17 00:00:00 2001 From: Martin Durant Date: Thu, 14 Nov 2024 09:48:32 -0500 Subject: [PATCH 4/4] lint --- docs/source/api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/api.rst b/docs/source/api.rst index d290f7ae1..5385accc7 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -211,7 +211,7 @@ Other Known Implementations Note that most of these projects are hosted outside of the `fsspec` organisation. Please read their -documentation carefully before using any particular package. +documentation carefully before using any particular package. - `abfs`_ for Azure Blob service, with protocol "abfs://" - `adl`_ for Azure DataLake storage, with protocol "adl://"