Tracking SeqReader progress using pv

[vlad0x00]

It would be useful to be able to track what % of the input file has been read by SeqReader at any time. Since SeqReader may use external tools to decompress files, it doesn't always have direct access to the input file to tell progress. However, it's possible to track progress using pv. To do this, btllib should run pv on the input file and pipe that further into either the decompression tool or SeqReader directly.

Instead of %, we would want to have the total number of bytes of the input file and the number of bytes read so far. pv allows this using the -bn options. The rationale here is that if you have multiple SeqReaders at the same time, the actual progress % is the sum of read bytes over the sum of total bytes for all SeqReaders.

Using pv this way shouldn't affect performance, as it likely uses splicing (https://linux.die.net/man/2/splice) instead of reading/writing. However, if it does, then there is an optimization that can be done on Linux. Instead of piping, pv on Linux lets you track file read progress of a process given its PID. In this case, we would call pv to track the progress of whatever external tool SeqReader uses or on SeqReader's process if no external tool is used.

[parham-k]

Is there an easier way of doing this for a single SeqReader which reads sequences one by one? Like, after each iteration in the code below, can we have the total number of bytes we've read so far? In that case we can just compare that with the file size.

btllib::SeqReader reader(...);
for (const auto& record : reader) {
    process(record.seq);
    std::cout << reader.bytes_read() / filesize << std::endl;
}

[vlad0x00]

That'd be fairly straightforward, but would only work for uncompressed files.