bonito basecall model refinement preprocessing memory issues

[CodingKaiser]

Hello,

I am currently doing a basecalling model refinement and am running into some issues in the pre-processing basecalling step surrounding memory management.

Specifically, I have a folder of pod5 files totalling ~24GB I am passing to bonito basecaller in the following way:
bonito basecaller dna_r9.4.1_e8_hac@v3.3 --save-ctc --min-accuracy-save-ctc 0.9 -v --alignment-threads 10 --device 'cuda' --reference ~/Documents/genomes/T7_V01146.1.fasta ./T7/pod5s/ > ./T7/bonito_mapped_hac_ctc/basecalls_ctc.bam

However, after the initial basecalling, the process is killed due to a maxing out of available RAM on my machine.

> reading pod5
> outputting aligned bam
> loading model dna_r9.4.1_e8_hac@v3.3
> model basecaller params: {'batchsize': 512, 'chunksize': 10000, 'overlap': 500, 'quantize': None}
> loading reference
> calling: 1290710 reads [59:34, 361.04 reads/s]Killed

For now, I am attempting to subset the initial data, but this is obviously not ideal as I am discarding potentially useful signal from being used in the training step. It appears to me that bonito train only accepts a single --directory, so breaking up the basecalling by pod5 or similar would also not work. Is there an alternate approach?

Thanks in advance for your input.

All the best,
Falko Noé

[andrewgalbraith21]

Hello Falko,

You could run bonito basecalling on separate directories and then merge the .npy files after and then train. There's already a post with the code to merge .npy files.

[davidnewman02]

As suggested by @andrewgalbraith21 a good way forward here would be to split your data into smaller subsets.
We now provide some additional instructions for basecall model training here which should help get you going on understanding the input data format and how you can adapt it for your use case: https://github.com/nanoporetech/bonito/blob/master/documentation/training.md