Using `seek_points()` to obtain valid decompression ranges.

[forrestfwilliams]

I would like to use indexed_gzip to download only the relevant portions of a .gz file via a ranged GET request. I understand that you cannot use indexed_gzip to download and decompress from an arbitrary point (see #112). However I am hoping that it is possible to use the index generated by indexed_gzip and made accessible via the seek_points method to download and decompress a small portion of the larger file that contains the data I'm interested in. This is what I have so far:

import zlib

import indexed_gzip as igzip
import numpy as np


def get_data(gz_path: str, index1: int, index2: int) -> bytes:
    with igzip.IndexedGzipFile(gz_path) as f:
        f.build_full_index()
        seek_points = list(f.seek_points())

    array = np.array(seek_points)
    start = array[index1, 1]
    stop = array[index2, 1]

    # stand-in for a ranged GET request~~~~ #
    with open(gz_path, 'rb') as f:
        f.seek(start)
        compressed = f.read(stop - start)
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #

    decompressed = zlib.decompressobj(-1 * zlib.MAX_WBITS).decompress(compressed)
    return decompressed

This function performs as expected when passing the arguments get_data(file_path, 0, 1), but when not starting from the first index location (e.g., get_data(file_path, 1, 2)) the function fails in the zlib decompression step with the message: zlib.error: Error -3 while decompressing data: invalid block type.

I'm guessing that the root of this issue is that I do not fully understand how zlib decompression works and what the required data formatting is. If you have any suggestion on how to modify this function to achieve my goal, I'd appreciate it!

[forrestfwilliams]

@martindurant you may also be interested in this.

[martindurant]

I think the point is, that for gzip you must also have the compressor state at the seek point, which is why indexed_gzip stores the 32kB window along with every point.
However, if you open the file like

with fsspec.open("http://...") as f:

then the f file-like object can be used directly with indexed_gzip. You can export the index data to a file, and load it again with that file later to get the random access range requests you are after. If you have a .gz in mind, I can write down the whole workflow.

[forrestfwilliams]

Ah, OK. Here is a link to a sample gzipped image in S3. Do you think it would be possible to obtain the compressor state on the fly, say by always downloading the 32KB of data directly preceding the seek_point and initializing indexed_gzip with it? My goal here is to reduce the amount of index information we need to store as much as possible.

In my specific use case, the uncompressed byte ranges I want to obtain data for are known ahead of time. In this case I'm hoping that I don't need to store full index, just the index information I need to get that byte range.

[martindurant]

I have the same hope as you: that we can pick and choose which seek points get saved, immediately before points of interest, to keep the index file size down. I don't know if it's possible to simply read 32kB before the seek point and setting the state like that, or if the 32kB saved in the index currently is somehow different. I suspect the latter, else why save it?

To demonstrate that indexing does work for the remote file right now:

import indexed_gzip as igzip
import fsspec

h = fsspec.filesystem("https")
u = "https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz"
f = h.open(u)
i = igzip.IndexedGzipFile(f)
i.build_full_index()  # reads whole file, I think, or at least chunks all the way through
i.export_index("tiff.iindex")

fsspec.utils.setup_logging(logger_name="fsspec.http")
f = h.open(u)
2023-01-30 10:21:01,665 - fsspec.http - DEBUG - _file_info -- Retrieve file size for https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz

i2 = igzip.IndexedGzipFile(f)
i2.import_index("tiff.iindex")
i2.seek(100_000_000)
i2.read(20)
2023-01-30 10:22:25,659 - fsspec.http - DEBUG - async_fetch_range -- Fetch range for <File-like object HTTPFileSystem, https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz>: 62814387-68057268
2023-01-30 10:22:25,659 - fsspec.http - DEBUG - async_fetch_range -- https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz : bytes=62814387-68057267
2023-01-30 10:22:27,095 - fsspec.http - DEBUG - async_fetch_range -- Fetch range for <File-like object HTTPFileSystem, https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz>: 68057268-73300148
2023-01-30 10:22:27,095 - fsspec.http - DEBUG - async_fetch_range -- https://ffwilliams2-shenanigans.s3.us-west-2.amazonaws.com/bursts/s1a-iw2-slc-vv-20200604t022253-20200604t022318-032861-03ce65-005.tiff.gz : bytes=68057268-73300147
b'\x00\x14\x00T\x00D\x00;\x00B\x00G\x00K\x007\x00>\x00\xb9'

So in this case it needed two range requests to get any data with the standard readahead buffer size at the default 5MB.

[pauldmccarthy]

Hi @forrestfwilliams @martindurant, this is one of the main usage scenarios - pre-generate an index, and then use that index on subsequent reads to improve access speed. The IndexedGzipFile and zran modules can be used with Python file-likes, although with some limitations (e.g.).

The build_full_index method does a full pass through the data, creating seek points at approximately spacing intervals.

I have the same hope as you: that we can pick and choose which seek points get saved, immediately before points of interest, to keep the index file size down.

This isn't possible at the moment, but should be easy to implement as I suggested in #112

I don't know if it's possible to simply read 32kB before the seek point and setting the state like that, or if the 32kB saved in the index currently is somehow different. I suspect the latter, else why save it?

Seek points can't be located just anywhere - they need to be located at deflate block boundaries, which are usually somewhat arbitrarily placed throughout a deflate stream (although I have to confess that I'm not familiar with the different ways in which deflate streams are generated).

[martindurant]

Seek points can't be located just anywhere

Yes, understood, but is it necessary to store the 32kB per point in the index file, or can that in theory be re-read from the file?

[pauldmccarthy]

Yes, understood, but is it necessary to store the 32kB per point in the index file, or can that in theory be re-read from the file?

No - we need 32kb of uncompressed data to initialise for inflation - this is passed to the zlib inflateSetDictionary function.

[martindurant]

I thought so, thanks for the clarification. That makes it all the more important to be able to pick seek points, as well as is possible. This is on my long-term map; along with other block-based compression codecs for kerchunk, but gzip is by far the most common and therefor important.

[pauldmccarthy]

For more background (and for my own interest, as I only ever learned the bare minimum to get this library working), the details on why we need that 32kB are in the DEFLATE RFC section 3.2 - basically, the encoding dictionary used to compress a section of data is dynamically [re-]generated from the previous 32kB:

As noted above, encoded data blocks in the "deflate" format consist of sequences of symbols drawn from three conceptually distinct alphabets: either literal bytes, from the alphabet of byte values (0..255), or <length, backward distance> pairs, where the length is drawn from (3..258) and the distance is drawn from (1..32,768).

There is also the possibility of coming across deflate streams which use a pre-set dictionary, although I'm assuming that these are not very common.

[pauldmccarthy]

I may be able to dedicate some time to this, and to other required changes (specifically this), at some point in the near future. But I can't make any guarantees I'm afraid - this library is low priority for me at the moment, as it covers my own usage scenarios just fine. I'm happy to consult/suggest/review PRs though 👍

[forrestfwilliams]

Looking at the export_index format here would it be valid to:

1. build_full_index for a data file and export_index
2. Parse this file as binary, remove undesired seek_points (and associated windows) and update header
3. Save this altered index to a new file
4. Import the altered index file and use it to read from the data file

I'm not sure if removing undesired seek_points would affect the validity of the index.

[pauldmccarthy]

@forrestfwilliams I think that would actually work just fine! Although as I mentioned in #112, I don't think it would be particularly difficult to enhance zran.c to build an index with pre-specified seek points.

[forrestfwilliams]

@pauldmccarthy sounds good! I'll try developing the approach I developed since I don't have the C skills to confidently implement the approach you described in #112.

Reading through DEFLATE RFC section 3.2 it appears that there are special cases where you wouldn't need window data because there are no back-references in the DEFLATE block. Are these types of special cases identifiable via indexed_gzip, say by looking at whether the seek points have a data flag value of 1 or 0?

[pauldmccarthy]

Reading through DEFLATE RFC section 3.2 it appears that there are special cases where you wouldn't need window data because there are no back-references in the DEFLATE block. Are these types of special cases identifiable via indexed_gzip, say by looking at whether the seek points have a data flag value of 1 or 0?

@forrestfwilliams No, indexed_gzip is not privy to that information as it is not exposed by zlib; the seek point data flag in an exported index is only set to 0 for seek points at the beginning of streams. I would hazard a guess that the dymanic/adaptive format is probably much more common anyway.

[martindurant]

It occurs to me that, since the window data we store is uncompressed blocks and the largest component of index files, that index files should be very amenable to compression, at least as well as the original data was :)

the dymanic/adaptive format is probably much more common

wikipedia agrees on this. However "stored" (uncompressed) blocks might be common for some data like random floats and I don't suppose those strictly need the uncompressed window either. Obviously, data which has a lot of those should not be compressed in the first place!