Handle Offline Operators to Maintain Domain Throughput

[jfrank-summit]

Problem

Inactive operators who remain staked but go offline continue to be eligible for bundle production elections based on their stake weight. Since they are offline, they fail to produce bundles when selected, which reduces the domain's overall throughput proportional to their stake percentage. This can significantly impact performance, especially for operators with large stakes, as the system continues to allocate slots to them without any output.

Key issues:

• Decreased bundle production rate
• Wasted election slots
• Potential for liveness attacks if adversaries stake without running nodes
• No automatic mechanism to detect and handle prolonged inactivity

This was partially mitigated in PR #2215, but a more complete solution is needed.

Proposed Solution

Implement a mechanism to automatically detect offline operators and temporarily exclude their stake from elections, while providing a way for them to signal their return to active status. We are researching various detection heuristics and exclusion algorithms to balance reliability, security, and operator experience.

Key Features

• Automatic detection of inactivity (e.g., probabilistic based on bundle production expectations over epochs)
• Temporary exclusion from elections without forced unstaking
• Simple reactivation process via extrinsic

Outcomes

• Mechanism prevents throughput degradation from offline operators
• False positives are minimized (active operators not mistakenly excluded)
• Security against gaming the system

Note: Replaces #2629

[teor2345]

Problem

Inactive operators who remain staked but go offline continue to be eligible for bundle production elections based on their stake weight. Since they are offline, they fail to produce bundles when selected, which reduces the domain's overall throughput proportional to their stake percentage. This can significantly impact performance, especially for operators with large stakes, as the system continues to allocate slots to them without any output.

As an initial quick fix, could we bound the ratio between the operator's stake, and the total stake?

For example, let's say we set the maximum operator stake at 50%.

So if an operator already had 40% of the stake on the domain, and they tried to stake another 40%, we would reject that transaction. (Or only take 20% and return the rest: 40+20 / 100+20 = 50%.)

[jfrank-summit]

Problem

Inactive operators who remain staked but go offline continue to be eligible for bundle production elections based on their stake weight. Since they are offline, they fail to produce bundles when selected, which reduces the domain's overall throughput proportional to their stake percentage. This can significantly impact performance, especially for operators with large stakes, as the system continues to allocate slots to them without any output.

As an initial quick fix, could we bound the ratio between the operator's stake, and the total stake?

For example, let's say we set the maximum operator stake at 50%.

So if an operator already had 40% of the stake on the domain, and they tried to stake another 40%, we would reject that transaction. (Or only take 20% and return the rest: 40+20 / 100+20 = 50%.)

I'd prefer to address the core problem rather than partially limiting the risk of a single operator. Also, the larger the operator the less incentive they have to go offline.

[teor2345]

Sure, fixing the core problem is much better here.

If we want to limit the risk/dominance of a single operator, that's a separate conversation.

[vedhavyas]

There are multiple things to handle here. There was a discussion with dariia sometime back. What we had then was
During epoch transtion, based on their stake share, check if they have missed any consecutive epochs till now and exclude them in next epoch.

From this spec

Based on operator stake percentage

• >= 10% , can miss at max 2 epoch including the latest
• >= 1% < 10%, can miss at max of 5 epochs including the latest
• rest default to 15 epochs

15 epochs as default seems little unfair for very small operators and we may end-up kicking them out more frequently. Also there is an assumption of 600 slots in the epoch with epoch at every 100 domain blocks and slot probability for consensus block is 1/6. While this maybe true but its no sure shot that every consensus block produces a domain block. It quiet possible where some consensus blocks may not produce any domain block at all. But 600 slots in an epoch maybe a min but could ideally be even less if some consensus blocks are produced with first few slots instead of 6 slots.

Another problem I see here with this the maintaining throughput. We ideally to maintain throughput, we want operators to continue producing bundles as and when they can. If we only consider an operator is active if they produced one bundle in an epoch, a malicious operator with large enough stake can overcome this by producing one bundle at any point in the epoch and no bundles are produced further. This will affect domain throughput since they did not produce enough number of bundles with in an epoch.

Ideally, we should have a way to probabilistically determine based on their stake how many bundles can they produce in a given epoch accumulated over multiple epochs and if the accumulator goes below the threshold, protocol kicks them out.

We can go with what is proposed as start but I dont think this will solve the problem of maintaining domain throughput like we wanted.

Maybe research team can pitch in here @jfrank-summit ?

[jfrank-summit]

@vedhavyas I think we discussed breaking this down into sub-issues a couple of weeks ago. Here are my thoughts:

1. Governance/Sudo extrinsic to exclude operator stake per domain. To be used when an operator is clearly offline from observation and has significant impact on domain liveness.

   • Add permissioned extrinsic to temporarily exclude an operator from elections for a given domain, without forcing unstake.
   • Reactivation extrinsic to include them again starting next epoch.

2. Permissionless inactivity evidence extrinsic with heuristics

   • Anyone can submit the extrinsic.
   • Storage: Maintain per-(domain, epoch, operator) tallies (bundles produced, staking % of total) for rolling N epoch window. We would need to think about efficient ways of handling this, my initial ideas above are probably not the best but the goal would be to be able to derive some metric of inactivity beyond some threshold
   • Potential triggers/thresholds:
     • If expected production over last N epochs ≥ X and actual < X * Y% (e.g., 50%), mark operator as inactive.
     • OR something like mentioned above:
       Based on operator stake percentage
       • >= 10% , can miss at max 2 epochs including the latest
       • >= 1% < 10%, can miss at max of 5 epochs including the latest
       • rest default to 15 epochs
   • On acceptance: move operator to exclusion set effective next epoch; allow reactivation (optionally require liveness signal and cooldown).

3. Protocol integration

   • On epoch transition, exclude operators in the exclusion set from current_operators and current_total_stake.
   • Ensure election params and thresholds (runtime API) use adjusted totals; events for exclusion/reactivation.

[teor2345]

Which reactivations should be automatic?

1. With forced deactivations, automatically reactivating at the next + 1 epoch would mean we'd have to manually re-submit the extrinsic until the operator is back up. This would be tricky and unusable.

• Instead, the deactivation extrinsic could have an optional end epoch? (For example, if the operator downtime is known to be 1 week.)

2. Operators that are online could regularly sign and send a liveness extrinsic. This avoids issues around race conditions and deactivation detection. If the operator is already in the active set, the extrinsic is considered invalid and skipped. If it is deactivated, the liveness extrinsic would be processed and reactivate it.

• for consistency, we should also require this extrinsic when the operator is first added to the operator set. This also improves our production code coverage.

Separately:
3. Operators that are excluded should not receive any rewards, so their share price would remain the same during exclusion. This might require some special handling for the final epoch where they are partially down:

• should they get any rewards for their final epoch? Is either full or no rewards for that epoch gameable?

[vedhavyas]

Reactivation extrinsic to include them again starting next epoch.

This should not be immediate but rather cooldown epoch, maybe 5 or 10, before them update their operator status else there will mostly situations where we kick them out and they immediately come back in next epoch. This totally defeats the purpose of kicking them to maintain throughtput

Anyone can submit the extrinsic.

I'm not so sure about that. If our approach is correct, then operators are kicked out at the end of every epoch based on the total number of expected bundles/slots they were supposed to produce during this epoch. If the a big whale decided to no produce bundle almost halting the chain, then sudo/governace can use the existing functions to kick them out and force transition epoch.

Maintain per-(domain, epoch, operator) tallies (bundles produced, staking % of total) for rolling N epoch window. We would need to think about efficient ways of handling this, my initial ideas above are probably not the best but the goal would be to be able to derive some metric of inactivity beyond some threshold

I have thought on this but did not finish yet. Will be submitting a forum post for wider discussion on this and gather some feedback.
But idea remains the same, to derive number of slot/bundles to be produced probabilistically for a given epoch and tally that with actual value and make a decision. For smaller operators, the tally can spread through couple of epochs.
Once decided to kick them out, the functionality should already exist.

OR something like mentioned above:
Based on operator stake percentage

IMHO, this will solve the problem like I have mentioned in my previous comment. It would be impossible to achive domain throughput with this approach and protocol can be tricked into thinking operator is not offline if they produce just one bundle. This may solve the situation where operators are really offline yes but we should also consider the griefing attack where operators simply decided to no produce any bundles to affect the domain block throughput

Overall my plan here is

• update the protocol spec
• Include the extrinsics to kick or reactivate operators through sudo
• Forum post on deriving number of bundles to produce probabilistically
• Implement the same with testing in chronos

[teor2345]

We also need to account for domains with no transactions, because they will have no blocks after a while.

[jfrank-summit]

Reactivation extrinsic to include them again starting next epoch.

To be clear, I meant at the start of the next epoch from the time the reactivation extrinsic is executed, not from the time of the deactivation. I don't think I worded that well

We also need to account for domains with no transactions, because they will have no blocks after a while.

The main idea is to only deactivate when an operator is producing the expected amount of bundles given their stake. If the expected amount of bundles is 0 because no blocks are produced, we should be fine.

Operators that are online could regularly sign and send a liveness extrinsic.

This still allows for an operator to show as online but still not produce the bundles they should be, impacting the liveness of the chain.

@vedhavyas your plan makes sense to me.