Skip to content

gdal raster tile: speed-up generation of (max zoom) tiles in PNG format #12731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rouault
Copy link
Member

@rouault rouault commented Jul 8, 2025

By using an optimized code path, one can speed by 36% in some cases tile generation, when using libdeflate. If using zlib only, the gain is more modest, but still there.

e.g using https://nz-imagery.s3-ap-southeast-2.amazonaws.com/otago/otago_2025_0.2m/rgb/2193/CA14_5000_0304.tiff

Before:

$ time GDAL_RASTER_TILE_USE_PNG_OPTIM=NO gdal raster tile CA14_5000_0304.tiff CA14_5000_0304_tiled --progress
0...10...20...30...40...50...60...70...80...90...100 - done in 00:00:12.

real    0m13,234s
user    2m14,696s
sys     0m2,220s

Size of output directory: 373.356 MB

After:

$ time gdal raster tile CA14_5000_0304.tiff CA14_5000_0304_tiled --progress
0...10...20...30...40...50...60...70...80...90...100 - done in 00:00:08.

real    0m8,352s
user    1m26,253s
sys     0m1,929s

Size of output directory: 359.384 MB

@rouault rouault added this to the 3.12.0 milestone Jul 8, 2025
@rouault rouault added enhancement funded through GSP Work funded through the GDAL Sponsorship Program gdal_cli Anything related to the new 3.11 "gdal" CLI frontend labels Jul 8, 2025
@rouault rouault force-pushed the gdal_raster_tile_fast_png branch 2 times, most recently from e7d8db5 to 9799382 Compare July 8, 2025 11:31
github-advanced-security[bot]

This comment was marked as outdated.

@rouault rouault force-pushed the gdal_raster_tile_fast_png branch 3 times, most recently from f1b7859 to f45dbde Compare July 8, 2025 12:25
@rouault rouault force-pushed the gdal_raster_tile_fast_png branch from f45dbde to 8d6972f Compare July 8, 2025 13:36
@rouault
Copy link
Member Author

rouault commented Jul 9, 2025

FYI @jcphill

@jcphill
Copy link
Contributor

jcphill commented Jul 15, 2025

For my generic build (without libdeflate) on I see runtime decrease from 12:55 to 12:00. Adding libdeflate drops it further to 11:42. That's only a 10% drop overall but still an improvement.

Unfortunately, there is something broken in the tile generation as blocks that should be all one color (red in the left half of this example tile) show up as black.
215

@jcphill
Copy link
Contributor

jcphill commented Jul 15, 2025

It looks like if a highest-zoom tile is all one color it is drawn as black, although there are a few other examples where the highest-level tile is black but should be different colors. It is possible that there is one channel that is zero for the entire tile.

Copy link
Contributor

@jcphill jcphill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing I was seeing the black boxes because the blue channel was zero and it was being read as the alpha channel.

By using an optimized code path, one can speed by 36% in some cases tile
generation, when using libdeflate. If using zlib only, the gain is more
modest, but still there.

e.g using  https://nz-imagery.s3-ap-southeast-2.amazonaws.com/otago/otago_2025_0.2m/rgb/2193/CA14_5000_0304.tiff

Before:
```
$ time GDAL_RASTER_TILE_USE_PNG_OPTIM=NO gdal raster tile CA14_5000_0304.tiff CA14_5000_0304_tiled --progress
0...10...20...30...40...50...60...70...80...90...100 - done in 00:00:12.

real    0m13,234s
user    2m14,696s
sys     0m2,220s

Size of output directory: 373.356 MB
```

After:
```
$ time gdal raster tile CA14_5000_0304.tiff CA14_5000_0304_tiled --progress
0...10...20...30...40...50...60...70...80...90...100 - done in 00:00:08.

real    0m8,352s
user    1m26,253s
sys     0m1,929s

Size of output directory: 359.384 MB
```
@rouault rouault force-pushed the gdal_raster_tile_fast_png branch from 8d6972f to a99f508 Compare July 15, 2025 21:42
@coveralls
Copy link
Collaborator

Coverage Status

coverage: 71.091% (-0.003%) from 71.094%
when pulling a99f508 on rouault:gdal_raster_tile_fast_png
into 7b610c0 on OSGeo:master.

@jcphill
Copy link
Contributor

jcphill commented Jul 16, 2025

The tile problem is fixed and the runtime is down to 11:10 now, so 17% overall improvement.

@jcphill
Copy link
Contributor

jcphill commented Jul 16, 2025

The max zoom tile png files are 5% or so larger, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement funded through GSP Work funded through the GDAL Sponsorship Program gdal_cli Anything related to the new 3.11 "gdal" CLI frontend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants