POD5 has encountered an error: 'Invalid: flatbuffer size 1862616117 invalid. File offset: 9165526712, metadata length: 256'

[q1134269149]

Issue Description

Hi,
Recently, I split a large pod5 file by pod5 filter with read_id mapping file. However, one error occurs as follows:
"""
Parsed 18231 reads_ids from:
unique_readsName.txt
Found 4822098 read_ids from 1 inputs
Calculated 18020 transfers
Filtering: 0%| | 0/18020 [00:00<?, ?Read/s]
POD5 has encountered an error: 'Invalid: flatbuffer size 1862616117 invalid. File offset: 9165526712, metadata length: 256'

For detailed information set POD5_DEBUG=1'
Filtering: 0%| | 0/18020 [00:17<?, ?Read/s]
"""

cmd:

• nohup pod5 filter HeLa-NC-2_pass.pod5 -t 10 --ids unique_readsName.txt --output HeLa-NC-2.pod5 --missing-ok > log 2>&1 &

head -n 5 unique_readsName.txt:
"""
000188bb-e300-4d4d-9b4f-2bbfcca6fbf5
0007c1b2-e762-469d-9886-a4b293c7755c
0008220f-b810-4eb6-a49d-8b8686d76d86
000a83cd-543e-4f1a-b180-90bf2a5821a2
000d7ddd-5d84-4a9f-846a-df052cab7561
"""

And I also encountered a similar error when using pod5 subset on my other pod5 file, and the program automatically exited with this error message

May I ask if there is any solution?

Logs

Please provide any log files. These can be generated by setting the POD5_DEBUG environment variable e.g. POD5_DEBUG=1 pod5 view my.pod5

Specifications

• Pod5 Version: v0.3.28 (installed using pip)
• Python Version: v3.10.18
• Platform: Linux version 6.14.0-27-genetic (x86_64-linux-gnu-gcc-13 Ubuntu 13.3.3-6ubuntu2~24.04)

[0x55555555]

Hello @q1134269149,

Can I ask the origin of the file youre trying to split? How were they created? Have they been copied around after output from MinKNOW?

Most likely issue is a corrupted pod5 file - from moving the file around, does that sound possible?

Thanks,

• George

[q1134269149]

Hello，

I copied the pod5, and I checked it using md5. It is OK. And I tried to identify this issue using the pod5 inspect command, and I found that this error was caused by several reads. When Pod5 processes these reads, it reports the same error. Therefore, I am wondering if it is possible to set up pod5 to automatically skip these failed reads and not terminate when it encounters such an error in subsequent updates. The pod5 file is relatively large, and this problem is inevitable during the copying process. Many of my samples encountered a limited number of read copy errors during the copying process after being downloaded. Perhaps this is very useful.

Best,

hqin