dap4.core.util.DapException: Illegal chunk size

[Mikejmnez]

Hiya @lesserwhirls per our chat earlier:

An xarray/pydap user reported having trouble decoding the dap response from a NetCDF file using the TDS DAP4 service they are trying to set up. I took a look and became aware of two issues that seem TDS-specific

• Deserializing time_string variable. This is a 1D array of all string elements of size ~5400. The issue arises when the dap response contains a subset of time_string>=2730. Anything less, it works fine (I think). My insights into the dap response from TDS are limited and may not be 100% accurate.

• dap4.core.util.DapException: Illegal chunk size: 114574 when I attempt to download the entire DAP response from the TDS using the web browser.

Here is the public TDS endpoint (and you can replicate the 400 error and see the entire traceback when attempting to download the dap response):

https://thredds.atmohub.kit.edu/thredds/dap4/musica-iasi/Full_v030301/2021/IASIA_MUSICA_030301_L2_AllTargetProducts_20210101054454_73699.nc

Pydap can deserialize fine the dap response from this file produced by Hyrax, but that provides no insight into why it fails to deserialize the response from the TDS.

[DennisHeimbigner]

I can mostly verify the comment by Mikejmnez (re: https://github.com/Mikejmnez)](https://github.com/pydata/xarray/issues/10879#issuecomment-3457820223).

Some background. When a variable length string is sent, it is encoded as a count (an 8-byte integer) followed by 'count' characters. So if I have a 5400 element array of variable length strings, then there are 5400 (count, string) values.

From ncdump, I determined the following:

1. all of the counts are supposed to have the value 16
2. the data for each string is a time constant similar to 20210101T054909Z

It turns out that element 2730 has a count of 16, but the data appears to be only seven characters, namely "2021010". The missing data of course screws up the walking of subsequent variable length strings and produces the errors seen.

Mikejmnez says ...the TDS is serializing it as two chunks resulting in string data looking kind of funny at the edge of one of these chunks...
I do not see any evidence of this. He may be confusing the string length count for a chunk count.

I pulled down the underlying .nc file using the HTTPServer protocol and using URL "https://thredds.atmohub.kit.edu/thredds/catalog/musica-iasi/Full_v030301/2021/catalog.html?dataset=musica-iasi/Full_v030301/2021/IASIA_MUSICA_030301_L2_AllTargetProducts_20210101054454_73699.nc"

I dumped the "time_string" variable from the IASIA_MUSICA_030301_L2_AllTargetProducts_20210101054454_73699.nc dataset and it all appears to be correct.

So it is not clear to me why TDS is failing to properly serialize element 2730 of the time_string variable; it seems no different than any other element. It is possible, I suppose that there is some unprintable character in the string in the underlying .nc file, but that seem unlikely.

So at this point, I am baffled.

[FObersteiner]

Hi all, thanks for looking into this! Just wanted to mention, if you need any info /wrt to this specific TDS instance, let me know. You can find my email in the TDS Info section. I also sit nearby to the people who created the data mentioned above ;-)

[lesserwhirls]

Currently, MAXCHUNKSIZE is hard coded to be 64 KiB. If we have 2730 (count, string) values, where count is a 8-byte integer int and string is 16 characters, then we have 65,520 bytes which is awfully close MAXCHUNKSIZE. I will note that if I increase MAXCHUNKSIZE by 8 bytes (so there is one more (count,string) value worth of space in MAXCHUNKSIZE, as opposed to just string), I am able to use ncdump to dump the entire dataset without issues, but if I keep MAXCHUNKSIZE as-is, ncdump results in a segmentation fault. Not an answer, but it's a clue.

[lesserwhirls]

The ChunkWriter code has a bug where offsets into the source buffer are not getting tracked for cases where multiple chunks are required. What that means is that we are writing out the first 65536 bytes of the 129,600 byte response just fine, but not the remaining 64,064 bytes (those are just a copy of the first 64,064 bytes of source). Currently, the joining ends of the two chunks look like:

End of Chunk 1	Beginning of Chunk 2
[8-byte count]20210101	[8-byte count]20210101T054909Z[8-byte count]

when it should be:

End of Chunk 1	Beginning of Chunk 2
[8-byte count]20210101	T055451Z[8-byte count]20210101T055451Z[8-byte count]