Parse CDMS Soudan data

[zonca]

The plan is to test reading Soudan data with awkward-kaitai, references:

• Documentation about the data format
• Parsing of the data with construct

One concern is that the dataset are large and we want to make sure we do not load them in memory.

Dataset

On the SuperCDMS OSN:

aws s3 --profile osn ls s3://supercdms-data/CDMS/Soudan/R135/Raw/01150212_1819/01150212_1819_F0001.gz

we also have the parsed HDF5 generated via construct:

aws s3 --profile osn ls s3://supercdms-data/CDMS/Soudan/R135/Raw/01150212_1819/01150212_1819_F0001_parsed.hdf5

[zonca]

if data size makes it very difficult to parse the data, an option would be to try manually cut the file at the end, just keep the first 3 event and cut out the rest:

[zonca]

@maraMaraschino
I am testing your ksy file with the current version of awkward-kaitai:

https://github.com/det-lab/dataReaderWriter/blob/master/scdms_soudan/scdms_soudan.ksy

and I get a core dump even before going out of memory:

terminate called after throwing an instance of 'std::__ios_failure'
  what():  basic_ios::clear: iostream error
Aborted (core dumped) 

Is this the same issue you were getting?

[maraMaraschino]

terminate called after throwing an instance of 'std::__ios_failure'
  what():  basic_ios::clear: iostream error
Aborted (core dumped) 

Is this the same issue you were getting?

I don't believe so, but it's been a while since I've touched the file. I only remember having severe memory issues when trying to load a raw file in the web ide. None of the computers I've had access to so far have been able to load a full file before the browser would fully crash and force a reload.

[zonca]

@maraMaraschino thanks, do you have any test files with a smaller size?

[maraMaraschino]

@zonca I found a commit from when I was working on it here. It's 51MB, and then you should only have to gunzip it after downloading.
I'm also not 100% sure that was the best file to have been testing on. The other smallest file I know about can be rcloned from the OSN at:
supercdms-data/CDMS/Soudan/R135/Raw/01150211_1500/01150211_1500_F0026.gz
It's 52.3MB
Otherwise, the raw files are generally in the low 100mb range even unzipped, and the fastest computer I tried to load some on was a mac that had 64gb of ram and a 3.2GHz 8 core intel processor.

[zonca]

as @pibion was suggesting on the call, I (for I of course I mean Copilot and I) used your construct file definition to create a sample file which has 1 instance of all types of records, it is 624 bytes, that I call small! ;)

See https://github.com/det-lab/cdms_soudan/blob/main/scdms_soudan_minimal_file.ipynb

I tested that your notebook can parse this into an HDF5 file.

Even with this tiny file kaitai fails, I'll start debugging in detail and understand what is wrong.