Start big shards indexing with delay and load distribution

[sergey-safarov]

Summary

I have a large shard which needs to be indexed using a new view. When the indexing operation started, the same shard was indexed on several nodes.

This can be optimised using this logic.

1. If the shard has X changes, then the required delay is indexing for Y milliseconds. The delay may be larger on a node with a high CPU load and smaller on a node with less CPU load;
2. check the other cluster node started indexing for the same shard;
3. if the shard is already indexed on another node, then skip this shard for indexing now.

The same logic applied to all shards. At the end, all nodes are randomly distributed shards for indexing.
On the next view request, the same shard will be indexed on the other node.

Desired Behaviour

Nodes distribute the shards indexing task and do not index the same shard on several nodes at the same time.

Additional context

This allows for distributing CPU load on nodes and makes CPU spikes smoother.

[rnewson]

I don't see an optimisation here. Each node has (or should have) independent CPU's etc, there's no reason not to index all N copies at once in parallel. As long as you didn't query the new view the index building process runs at a background level, the ioq system vastly preferring interactive requests over the view build I/O.

[sergey-safarov]

Let me explain.
Please imagine that processing of 3,000,000 documents requires 24 hours. Like in my screenshot.
When you have a database q=12, a 3-node cluster, and each node has 4 core CPU.

In this case, on each node, processing will start for 3,000,000 docs for 12 shards.
When the logic described in the initial description is applied, then each node will be taken for indexing 1,000,000 docs, and 4 shards.
This will reduce the time to get a valid response from 24 hours to 8 hours.

Do you agree?

[rnewson]

I see what you are getting at, you'd ideally like couchdb to build just one copy of each of the q index shards first (and evenly divided across the nodes), so that you can query it sooner, and the other 2 copies should wait until afterward to start.

One suggestion for today is to configure the background indexing concurrency downward. See https://docs.couchdb.org/en/stable/config/indexbuilds.html for details. Ken unfortunately won't be aware of your preferred order.

[sergey-safarov]

Yes, when the number of changes is larger than max_incremental_updates, then it will be better to index the first shard copies and process the next copy when the previous is completed.
This also decreases the requirements for memory. A smaller number of documents processed simultaneously requires less memory, and this reduces the probability of a message

alarm_handler: {set,{system_memory_high_watermark,[]}}

The proposed mechanism looks simple to understand.

[rnewson]

We'd accept a pull request that implements this behaviour (as long as it's optional, as this mode will not work for everybody). Thank you for clarifying your idea.