Amenable to distributed computing?

[rchikhi]

Hi!
Would raptor be suitable in the following setting?

0. Consider n FASTQ samples where n is large (> 10k)
1. Construct 1 bloom filter per sample by running raptor on that sample (on a distributed cluster)
2. "Somehow" combine all n bloom filters into a (very large) iBF

best,
Rayan

[eseiler]

Hello Rayan,

thank you for your question!
We (colleagues and I) are indeed working on a general interface that allows distributing work among compute nodes; but I do not have an ETA on that yet.

It is possible to partition the IBF (raptor build [...] --parts N), it's certainly possible to use a similar idea for distributed computing. But, depending on how one partitions, this might generate quite some overhead.

However, I think your idea should work quite well:

• Divide the n samples among the available nodes. This should be done in a way that results in a multiple of 64 - this is the smallest (internal) step size for the number of samples/bins within one IBF. A smaller number (e.g. 100 instead of 128) would waste space.
• Each cluster builds an IBF using the same parameters (size, number of hashes, etc.). The size might be a bit tricky to know beforehand.

This results in many small IBFs that only represent a few samples each.
In theory, and I'll need to verify, one could now interleave the IBFs (with, let's say, 128 bins each) into one big IBF (approx. 10K bins).

Coming back to your question, currently your scenario is not handled by raptor, but it is manageable.
If you are interested in trying/testing above approach, we could certainly provide some raptor merge capability.
I am pretty certain that the approach works, but I cannot vouch for the efficiency. Also, I am currently finishing up some soon-expiring projects, so it might take a tad longer.

If you have time constraints or are rather looking for an existing/complete solution, I could also get back to once we have some general API that handles this use case.

Best
Enrico

[rchikhi]

Hi Enrico,
Thanks for the extensive answer! This seems potentially manageable but not quite ready indeed. The context of my question was to apply raptor on 10-100k samples in a very distributed fashion, think ~1000-2000 nodes (Serratus-scale). Happy to hear more when you have a general API.
best,
Rayan