Description
In p2p we use an "actor model" with several actors all running in-tandem. The tracker does most of the centralized dispatching and data-structure updating while the dialer, listener, spawner, and (multiple) peer actors contribute.
There are some "state-machine" patterns used, especially in the directory.rs and record.rs files. The transitions are often triggered by various actors sending messages asynchronously to the tracker actor, but are enforced by strict assert! macros as sanity-checks. While assert!s ensure a strict ordering and can help catch bugs, they can also serve as a severe source of production crashes. One thing that exacerbates the issue is that the assumptions about strict ordering are often 1) not clear or well documented and 2) difficult to reason about due to the asynchronous nature of the actor model. This can lead to difficulties in maintaining and testing the crate.
We should consider several options to fix this tension. One such option is to ignore "invalid" messages, however this can also serve as a source of uncaught bugs in the code. Another option is to make the assumptions about message-passing more explicit, especially as it pertains to ordering; however this can make the code more complex to use by increasing cognitive overhead.
Related: #1110, since the invariants and assumptions of this pattern should be well-documented, or trivial to understand. Currently they are neither