Description
What was promised:
I could call AbstractFileSystem.get_file(..., outfile=<some-file-handler>), and it would override lpath or serve in place of lpath.
What I found:
I could only call AbstractFileSystem.get_file(..., lpath=some/real/dir, outfile=fh, ...) to get the desired functionality.
Why:
This line: fsspec.spec:919
What it seems it should be:
def get_file(self, rpath, lpath, callback=DEFAULT_CALLBACK, outfile=None, **kwargs):
"""Copy single remote file to local"""
from .implementations.local import LocalFileSystem
if isfilelike(lpath) and outfile is None:
outfile = lpath
...
Second: The custom outfile is closed
The problem is I should be able to pass an open buffer to be written to, and close it myself. Instead, the outfile is always closed. That means I cannot buffer the file into memory without a wrapper that implements a no-op close.
In general, I should be able to write the following code:
fp = io.BytesIO()
FS.get_file(source, outfile=fp)
fp.seek(0)
# do something with the data
But instead: when it calls outfile.close() you release all the memory of the buffer.
Linked code
Here is the line: fsspec.spec:942
Solution
In order to be backwards compatible, the simplest thing I can think of is just add a new parameter with default value of True:
close_outfile: bool = True, ...):
...
if close_outfile and not isfilelike(lpath):
outfile.close()
-- and probably want to if-then off of something besides isfilelike(lpath), since that is (or should be) irrelevant once outfile is non-null.