Skip to content

Conversation

@ctoney
Copy link
Contributor

@ctoney ctoney commented Jan 31, 2026

Adds VSIS3_COPYFILE_USE_STREAMING_SOURCE in the list that appears under:
https://gdal.org/en/latest/user/virtual_file_systems.html#vsis3-aws-s3-files

I'm not sure how the master list in doc/source/user/configoptions.rst is generated or whether anything else is needed for it to appear there.

@rouault
Copy link
Member

rouault commented Jan 31, 2026

@ctoney Can you give more context on why you needed this option ? I see this was introduced in bb73f93 for testing purposes, but not sure if we want to surface that to users

@ctoney
Copy link
Contributor Author

ctoney commented Jan 31, 2026

Yes, understood.

I'm trying troubleshoot performance of file transfer from S3 in: firelab/gdalraster#890

When I first looked at

gdal/port/cpl_vsil_s3.cpp

Lines 3608 to 3612 in fbce9c0

if (STARTS_WITH(pszSource, osPrefix.c_str()) &&
CPLTestBool(CPLGetConfigOption(
"VSIS3_COPYFILE_USE_STREAMING_SOURCE", "YES")))
{
// Try to get a streaming path from the source path

I thought it made YES the effective default, but my first download attempt with setting that option explicitly was faster, so I was momentarily uncertain. Subsequent repeats of that first download, without explicitly setting the option, did not replicate the initial difference I saw. I see now that the code definitely makes YES the default, but I thought it could be worth documenting the behavior for VSICopyFile in that case. Otherwise, a user attempting to tune this may not realize that /vsis3_streaming/ is already in effect.

No problem if that's not a good idea to surface, and we can close this.

So far it's surprising to me that VSICopyFile in the test case referenced above is slower than the boto-based transfer, and especially that it's marginally slower than a manual read/write of 10 MB chunks using a VSIVirtualHandle. Major caveat is that those are initial results, and we need to test/experiment more to confirm whether the gdalraster code is implemented in an optimal way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants