POD5 has encountered an error: 'IOError: Invalid signature in file

[olawa]

Issue Description

pod5 merge crashes with this error when trying to merge old data into a single file. Dorado gives the same error but just skips the file.

Would be good if pod5 could continue and output a list of problematic files.

Specifications

• Pod5 Version: Pod5 version: 0.3.28
• Python Version: 3.6.8
• Platform: linux

[0x55555555]

Hi @olawa ,

We don't expect pod5 to crash when loading old data, can you provide a traceback, or example file which causes a crash?

Thanks,

• George

[marchoeppner]

Well, may as well hijack this issue since I have the same error. I wanted to test subsetting of pod5s for parallel processing.
This is the latest pod5 from pip; after downgrading polar. The summary command worked fine. The subset command did this:

pod5 subset /minknow/Listerien_Test02/ONT02/20250210_1215_MN47840_FBA73567_e8207421/pod5 -o split_by_channel -t 4 -s summary.tsv -c channel -r
/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/pod5_subset.py:237: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.
  .collect()
Parsed 15398 targets
Traceback (most recent call last):
  File "/home/mhoeppner/.local/bin/pod5", line 8, in <module>
    sys.exit(main())
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/main.py", line 61, in main
    return run_tool(parser)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/parsers.py", line 41, in run_tool
    raise exc
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/parsers.py", line 38, in run_tool
    return tool_func(**kwargs)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/parsers.py", line 578, in run
    return subset_pod5(**kwargs)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 63, in wrapper
    raise exc
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 60, in wrapper
    ret = func(*args, **kwargs)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/pod5_subset.py", line 658, in subset_pod5
    subset_pod5s_with_mapping(
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 63, in wrapper
    raise exc
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 60, in wrapper
    ret = func(*args, **kwargs)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/pod5_subset.py", line 603, in subset_pod5s_with_mapping
    sources_df = parse_sources(inputs, threads)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 63, in wrapper
    raise exc
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/utils.py", line 60, in wrapper
    ret = func(*args, **kwargs)
  File "/home/mhoeppner/.local/lib/python3.10/site-packages/pod5/tools/pod5_subset.py", line 337, in parse_sources
    raise item
RuntimeError: IOError: Invalid signature in file

[0x55555555]

How many files and reads are in the input set? Can you print the output of the summary command for each file?

Do you know what version of minknow (or other application) generated the files?

Thanks,

• George

[marchoeppner]

Thanks for the feedback. I actually went back to the archived run data, re-extraced it (on a different system) and there everything worked. I assume now that something about that other data directory got broken at some point.

[aforestsomewhere]

Hi, I also seem to be running into a similiar issue on old pod5 files I was trying to merge, as it reports an invalid signature in all.

Pod5 version: 0.3.33 (conda install)
Files: generated via MinKnow 24.06.14

pod5 inspect summary FAU29307_84ef9aaf_bed40475_71.pod5
Failed to open pod5 file: FAU29307_84ef9aaf_bed40475_71.pod5: IOError: Invalid signature in file

[0x55555555]

Hi @aforestsomewhere ,

What is the source of the pod5 data? has it been copied around, and could have become corrupted?

The error Invalid signature in file suggests the header at the beginning or end of the file is corrupt, and the file is not valid for reading.

Thanks,

• George

[aforestsomewhere]

Thanks for the reply @0x55555555

The files was transferred from the GridION last November and don't seem to have been touched since then (as far as I can tell). I did copy the files to rebasecall using the GPU locally and that did work, so it doesn't seem like the transfer process from the GridION corrupted them.

I don't have another backup of the pod5s unfortunately, so I might try to downgrade the pod5 software (it was a slightly older flowcell, in case it matters) or to convert to slow5.

[olawa]

Hi @0x55555555, it happened on limited number of files which I assume are corrupted in some way. They have likely been transferred between servers a few times.

I suppose I could manually remove the corrupted files but it would be better if pod5 had an option to skip invalid files.