Introduce peer checkpoints so CouchDB can safely remove deleted documents entirely

[rnewson]

Overview

Apache CouchDB retains some information (at minimum, doc id, doc revision tree and a deleted flag) for all deleted documents forever, in order that replication is guaranteed to converge. This is excessively pessimistic and we would like to improve matters.

This PR introduces a number of changes to achieve its goal;

1. database shards (.couch files under shards/ directory) gained an additional header property called drop_seq. Once set to a positive, non-negative integer, any deleted document with a lower update sequence is skipped entirely at next compaction.

2. The notion of a peer checkpoint document. These are all local docs and their ids must have prefix `_local/peer-checkpoint-'.

3. All indexers (mrview, search, nouveau) and the replicator have been taught to create and update peer checkpoints with the update sequence they have seen at appropriate times (i.e, after they have made every effort to commit the changes they've seen to durable storage).

4. A new endpoint POST /$dbname/_update_drop_seq which gathers information about the shards of the database, update sequences from all peer checkpoint documents, and the internal shard sync documents, and computes the drop_seq for each shard, and then sends RPC requests to those databases to update the drop_seq.

Testing recommendations

There are some simple tests in the eunit and elixir suites which will be run via the normal Makefile targets.

Additionally there is a stateful property-based test that exercises the code more comprehensively which can be started with make elixir-cluster. This will start a 3 node cluster with nouveau server running and perform random permutations of all relevant operations that could alter which deleted documents are dropped (making docs, deleting docs, creating indexes, creating and updating peer checkpoints, splitting shards).

Related Issues or Pull Requests

N/A

Checklist

• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• [TODO] Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches