Allow reading binary doc values as a RandomAccessInput

[iverase]

Following up this suggestion from @jpountz, here I propose to add a new method to the BinaryDocValues API that returns the content of the doc values as a RandomAccessInput. This will allow to have an off-heap version of the binary doc value whenever possible.

The main two changes here are:

1. Lucene90DocValuesProducer allocates the BytesRef for binary doc values lazily.

2. To make transparent when the binary doc value is already on heap, BytesRef implements RandomAccessInput so we can just return the result from binaryValue() in those cases. If that's not ok, we can create a RandomAccessInput wrapper for BytesRef instead.

relates #12459

[jpountz]

Using RandomAccessInput addresses problems with multiple callers wanting to consume the value, that helps.

In my opinion, we should not have both a binaryValue() and a randomAccessValue() getters in the long term. Can we remove the binaryValue() getter on the main branch and make it deprecated when we backport to 10.x?

we can create a RandomAccessInput wrapper for BytesRef instead.

I would like it better this way.

[jpountz]

In my experience, binary doc values are more often used to encode structured data, such as maps that help build scoring signals, geo shapes, etc. than actual binary content, so this change makes sense to me. I'm interested in having more opinions though.

Would be nice to extend AssertingBinaryDocValues to make sure that all reads in the input are within bounds.

[iverase]

I have a second iteration and I have realised that this change can be easily done if we introduce a new abstraction. I have called it RandomAccessInputRef and it is the equivalent to BytesRef but backed by a RandomAccessInput instead of a byte array. I am thinking of it as the off-heap version of BytesRef.

Using this abstraction makes the changes almost mechanical. The big difference between BytesRef and RandomAccessInputRef is that the latter cannot implement efficiently the compare interface so it cannot be use for comparisons and equality. In that case we need to read bytes on-heap.

With this new abstraction, we just need to add three new methods:

• BytesRefBuilder#copyBytes(RandomAccessInputRef in) throws IOException
• BytesRefArray#append(RandomAccessInputRef bytes) throws IOException
• ByteBlockPool#append(final RandomAccessInput bytes, long offset, int length) throws IOException: for supporting the method above)

And two helper classes:

• ByteArrayRandomAccessInput: To represent and byte array as a RandomAccessInput
• RandomAccessInputDataInput: to wrap a RandomAccessInput as a DataInput so we can use DataOutput#copyBytes(DataInput input, long numBytes)

There are few places that still needs to be adapted to take full advantage of the new API but they can be done in a follow up as they require a bit of work :

• MatchingFacetSetsCounts
• BaseShapeDocValuesQuery
• SerializedDVStrategy
• MonitorQuerySerializer

And there are two places that we will need to put the data on-heap as they are using equality:

• TermValComparator
• BinaryDocValueSelector

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!