Allow SPs to move partitions between deadlines (FIP-0070)

[steven004]

This is a pre-FIP for discussion, inspired by discussion with @anorth and @nickle. Drafted by Venus team.

Summary

Add a new method for Miner actor to allow SPs to move a partition of sectors from one deadline to another, so that an SP could have control over their WindowedPoSt schedule.

Abstract

In WindowPoSt every 24-hour period is called a “proving period” and is broken down into a series of 30min, non-overlapping deadlines, making a total of 48 deadlines within any given 24-hour proving period. Each sector is assigned to one of these deadlines when proven to the chain by the network, grouped in partitions. The WindowPoSt of a particular partitions in a deadline has to be done in that particular 30min period every 24 hours.

This proposal is to add a method MovePartitions to move all sectors in one or more partitions from one deadline to another to allow SPs control the WindowPoSt time slot for sectors.

Change Motivation

When a Storage Provider (SP) maintains storage for Filecoin network, they often encounter various challenges such as system performance degradation, network issues, facility maintenance, and even changes in data center location. While dealing with these challenges, SPs require greater flexibility in controlling the period to prove the data integrity to the chain, while still adhering to the rule of proving once every 24 hours.

By implementing this proposal, several advantages can be realized:

• Creation of user-defined maintenance windows: SPs can create designated periods for maintenance activities without the risk of losing power.
• Definition of free hours for better duty roster: SPs can allocate specific hours of the day as free hours, enabling them to optimize their duty roster. This can be achieved by moving partitions out of those hours.
• Cost savings for large SPs on WindowPoSt hardware: Balancing the number of partitions across all the deadlines allows large SPs to optimize their hardware usage and reduce WindowPoSt hardware costs.
• Relieving access stress on storage nodes: In scenarios where an excessive number of partitions are concentrated within a single deadline on specific storage nodes, redistributing some partitions to other deadlines can help alleviate access stress and improve overall performance.
• Possibility of partition compaction: By moving partitions from different deadlines into a single deadline, it becomes possible to compact partitions that were originally distributed across multiple deadlines.

Specification

We propose adding the following method to the built-in Miner Actor.

// Moves sectors in partitions in one deadline to another
// Returns false if failed
// Any non-zero exit code must also be interpreted as a failure of moving
func (a Actor) MovePartitions(rt Runtime, params *MovePartitionsParams) *abi.EmptyValue

The params to this method indicate the partitions, their original deadline and destination deadline.

type MovePartitionsParams struct {
  OrigDeadline    uint64
  DestDeadline    uint64
  Partitions      bitfield.BitField
} 

To adhere to the requirement of conducting a WindowPoSt every 24 hours, the MovePartitions function only permits the movement of partitions to a DestDeadline whose next proving period is scheduled to occur within 24 hours after the OrigDeadline's last proving period. If the DestDeadline falls outside of this time frame, it will fail. This restriction ensures that the sector's period aligns with the required WindowPoSt interval.

Design Rationale

The primary objective is to introduce a straightforward mechanism for implementing flexible proving period settings for Storage Providers (SPs). Additionally, there is a consideration to offer greater flexibility by allowing the movement of any selected sectors from one deadline to another. This would enable an SP to align the expiration of sectors within a single partition. However, this approach poses certain risks, such as the potential increase in the cost of managing the bitfield due to non-sequential sector IDs. Moreover, the technical complexity involved in understanding the cost calculation may lead to unnecessary difficulties for SP operators.

Backwards Compatibility

This proposal introduced a new method to the built-in actors, which needs a network upgrade, but otherwise breaks no backwards compatibility

Test Cases

Proposed test cases include:
‒ that the MovePartitions can be successful when moving one partition from a deadline to another which proving period is in 24 hours after the partition's last proving period;
‒ that the MovePartitions would fail when moving one partition from a deadline to another which proving period is beyond 24 hours after the partition's last proving period;
‒ that the MovePartitions can be successful when moving multiple partitions from a deadline to another which proving period is in 24 hours after partitions' last proving period;
‒ that the MovePartitions would fail when moving multiple partitions from a deadline to another which proving period is beyond 24 hours after partitions' last proving period;
‒ that the WindowedPoSt works as normal in the DestDeadline for partitions moved from its OrigDeadline;
‒ that the CompactPartitions works as normal for the new moved-in partitions and existing partitions in the DestDeadline

Security Considerations

This proposal is to provide more flexibility for SPs to have full control of proving period of WindowPoSt for partitions. In the design, we still ask the SPs to follow the basic rule that one partition should have one proof every 24 hours, so there is no compromise in this regard.

There might be a concern that overall WindowPoSt messages in the network might imbalance over the 48 proving periods due to adjustments by SPs. Considering WindowPoSt only takes about 10% of the whole network bandwidth, and the network is decentralized, it should not be a problem, in addition, this mechanism actually provide a way for SPs to move proving period to avoid a expected congestion period.

Incentive Considerations

This FIP does not effect incentives in itself and is of only flexibility providing without any impact on economic factors.

Product Considerations

There is no any impact on Filecoin ecosystem application or platforms.

Implementation

An implementation can be provided after further discussion of the proposal.

Copyright

Copyright and related rights waived vis CC0.

[steven004]

PR #740 is submitted for this.

[zhiqiangxu]

Implementation wise, it seems that the move-able distance(in deadline unit) is quite limited:

For a specific deadline to be move-able, it also has to satisfy the deadline_available_for_compaction condition, this function basically says that a deadline can't be moved during these three periods:

1. when the target deadline is the current deadline. (duration is 1 ddl)
2. when the current deadline is in the dispute window of the target deadline just elapsed. (duration is 30 ddl)
3. when the current deadline is in the challenge window of the next occurrence for the target deadline. (duration is 1 ddl)

So for the 48 deadlines in a day, 32 deadlines are excluded by deadline_available_for_compaction, which leaves only 16 deadlines for operation.

When move partitions from deadline A to deadline B, it's necessary that deadline_available_for_compaction holds for both A and B. From this we can conclude that the ddl distance between A and B can not be greater than 16.

A workaround is that before moving partitions from a source deadline, do a window post verify against the source deadline forcefully, thus avoiding the requirement of deadline_available_for_compaction.

So there're two options:

1. carefully choose the deadlines so that deadline_available_for_compaction is met for both source/target deadline.
2. do window post verify against source deadline so that the source deadline doesn't have to meet the requirement of deadline_available_for_compaction.

The problem is: do we want both of these options or only prefer one of them?

[anorth]

I think we shouldn't allow moving to B if current deadline is exactly B or the challenge window of B for the same reason compact_partitions forbids modification to the partitions during that period.

Yes, I agree with that. Sorry my diagram was sloppy to not take that into account. I'll updated it.

I'm confident that we can allow moving the partition into the dispute window of B.

[zhiqiangxu]

@anorth Glad that we reach consensus on the other two periods.

Let's focus on the dispute window, what do you think is the reason that compact_partitions is not allowed during the dispute window?

[anorth]

what do you think is the reason that compact_partitions is not allowed during the dispute window?

Because if the dispute is successful, we need to mark the affected (old) partitions as faulty. But the old partitions aren't there any more, the sectors may have been spread into other partitions, some of which might have PoSted successfully. It's too expensive to load and process the faults on all the affected sectors individually, especially since we don't know which partition they're in now.

[zhiqiangxu]

@anorth So essentially, during dispute window, the partitions are append-only, that's why you say that compaction is disallowed but moving into is allowed, right?

[anorth]

Yes. During dispute window, each partition is append-only. We cannot remove a sector from a partition, because that would make it impossible to record a fault for it if PoSt is disputed. Compaction is disallowed because it can remove sectors from partitions. But adding new sectors is OK.

Also, the collection of partitions in the deadline is append-only. We cannot remove a partition from the deadline that it's currently in because then we couldn't fault any of its sectors. But adding a new one (moved from elsewhere) is fine.

[steven004]

Option (2) sounds attractive as a product experience, but difficult to implement. We don't have a means for the SP to voluntarily have the PoSt verified synchronously. I think we would have to change the parameter schema to add that. We also don't have an explicit record of which proofs were verified on chain in order to know if the dispute window is relevant. We might be able to infer it from the difference between Deadline.PartitionsPosted and Deadline.OptimisticPostSubmissions by reading it all from state though. Both of these are good things to consider next time we are changing anything about WindowPoST though.

I think a simple implementation might be possible, to just do verification of PoSt when MovePartitions, without other changes required, just like we do a dispute execution on the corresponding PoSt message. It would be better if we have the PoSt verification synchronously, or we have a record on chain to show that proofs were verified. But if there is not, we could just verify it (once again perhaps). This looks like a repeat work, but in the real life, it happens rarely since if the PoSt is correct, there will have no dispute message most likely, and while it is incorrect, it will fail anyway.

[anorth]

@Kubuxu and I discussed this today and reached the same idea that I think you are saying here. In the MovePartitions operation, we can read the previously submitted PoSt from state and verify it, and then be free to move the partition to any new deadline with 24hr of that original PoSt submission. We don't need to record in state that it was verified, because it's impossible to dispute it (we have just proven the proof is valid). And because disputes use snapshot state, we're free to remove the partition from the active deadline state.

[ZenGround0]

I've recently done some analysis on the load miner cron places on the system and have a few relevant results to share. These results get at the impact of a heightened concentration of partitions (i.e. majority within peak business hours in Asia) on network health.

On the positive side the per cron job processing time is quite low in the common case. For example it is not infeasible to process non faulting cron jobs for the whole system in a single deadline. Another encouraging result is that there is a quite a lot of overhead for individual cron jobs: ~65M gas for deadlines with live partitions and a small marginal cost to adding a partition to a deadline: ~0.7M gas. If SPs tend to cluster their partitions the system is unlikely to break and will be more efficient in the happy path.

On the more negative side faulting costs are expensive enough that high concentrations of partitions can cause significant problems at current network sizes. From the linked analysis

Today a 10% continued fault over the period of a deadline (60 epochs) is (1) ( 14k * [12M | 40M]) / 60 = 2.8B gas per epoch for continued faults, 9.3B gas per epoch for new faults. With 10 gas per ns that's .3 and .9 seconds per epoch for faulting cron.

I think this is actually an ok parameter region to operate in. I'm a bit more concerned about the mid term future should the network scale upwards significantly. I'm also concerned about the possibility of even tighter concentrations of partitions in fewer epochs.

@steven004 do you have predictions on the impact of this change on partition distribution across deadlines? Should we be concerned about large concentrations of partitions across ranges of fewer than, say, 60 epochs?

[steven004]

@ZenGround0 , you analysis and findings are wonderful.

@steven004 do you have predictions on the impact of this change on partition distribution across deadlines? Should we be concerned about large concentrations of partitions across ranges of fewer than, say, 60 epochs?

I do not have a quantitative analysis of the impact of this change on partition distribution across deadlines. But I do not think there will much impact since:

• the partitions moving usually happens when there is a need of a maintenance window, in this case, it actually avoid some expected failures (if there is no this change);
• some large SPs may want to balance the partition even distribution over 48 deadlines for cost saving for their WindowedPoSt machines being better utilized, while some small SPs may want to move partitions to their work hours;
• we are seeing more SPs located in EU or NA joining the network.

In addition, this change will actually help us to better distribute partitions if there is a mechanism to let the users pay the cron task gas in the future implemented. The problem right now is that cost of cron tasks is paid by the network, instead of real users. Once the users feel the pain of high gas usage because of too many partitions in a certain period of time, the partition moving will help to mitigate the congestion.

[jennijuju]

Thanks for the amazing work here @ZenGround0
I agree with most of the things, but

The problem right now is that cost of cron tasks is paid by the network, instead of real users. Once the users feel the pain of high gas usage because of too many partitions in a certain period of time, the partition moving will help to mitigate the congestion.

I think the goal should be avoid high cron tasks cost that might cause chain instability when designing the protocol. If we were going to accept this change, I think core devs to prioritize on safe cron sooner rather than later to mitigate the potential mid-term scalability risks.

[jennijuju]

It would be good to do some benchmarking while implementing this FIP and see if load_partitions_sectors_max is still valid, these values haven't be checked since the introduction of FVM and has a risk that a message that is under the protocol limit still can't fit in block gas limit.

[jennijuju]

Now that SPs can move partitions, it is easier to pack up as many as sectors in one deadline. We should confirm if there is a upper limit for the # of partitions for a deadline under the current gas usage, to make sure DisputeWindowPoSt message can be successfully submitted - as there exists network security concern if a PoSt dispute messages fails due to constant ErrOutOfGas.

[ZenGround0]

This is an interesting point. AFAIK dispute should be addressable at a single proof so number of partitions is not a strict problem. However there is the issue of many failed posts together requiring many dispute messages. It is also worth considering if there are particular actions (such as stacking many partitions in a deadline) that a miner actor could take on its state to make disputing only a single post very gas expensive.

[steven004]

AFAIK dispute is against one WindowedPoSt proof, possibly batch Proofs, and in the current implementation, there is a max-limit setting for a batch. That is, the dispute cost is not related to how many partitions in one deadline, but how many partitions allowed to be in one proof.

We may need to revisit the max-limit setting for batch proofs after the gas adjustments due to FEVM introduction, but anyway, this is another topic.

[Stebalien]

Creation of user-defined maintenance windows: SPs can create designated periods for maintenance activities without the risk of losing power.

I don't see this in the proposal. I believe this FIP needs to let storage providers configure their miner to only assign new sectors to specific deadlines. Otherwise, this could turn into a whack-a-mole problem.

[anorth]

@Stebalien I agree that facility would be useful, but not sure this FIP necessarily need to provide it. As soon as an SP stops onboarding constantly, they can make a maintenance window. Or if they stop onboarding during their maintenance window, etc.

We can roll such a change into direct onboarding (which has new onboarding methods) if you think it's really important to land together.

[Stebalien]

It really doesn't have to. And yeah, this does let the storage provider make a temporary maintenance window, I hadn't considered that.

[Stebalien]

Product Consideration

Storage services are expected to operate around the clock (e.g., to make data available). In theory, the current deadline assignment policy encourages storage providers to remain online at all times.

However, there are some reasons this may not be the case:

1. The data being proven still needs to be available for mining (winning post), providing an inceptive to remain online.
2. It's already possible to simply shard across multiple miner actors to try to align proving deadlines.

I'd like to see this addressed/discussed in the FIP.

[steven004]

@Stebalien I'm not sure if I fully understood what you mentioned. I guess you meant the current deadline assignment policy resulting in the partitions evenly distributed in deadlines so that the miner need to keep online at all times to avoid penalty or loss of rewards, however, this proposal makes it possible for SPs have flexible maintenance windows during which they might not be available for service.

That's fair. But, I think this impact is limited, since (1) as you mentioned, the winningPoSt mechanism, the SP could not get rewards if she is not online; (2) Actually, SPs providing services, especially for retrieval, not from the sealed data, but separated system, e.g. fast retrieval, IPFS, or other web services system; (3) the data service terms should be defined by clients and SPs, and data availability is better to be measured in the user space, so that the terms can take effect. That is the data service platform better to take care of this.

I could add this into Product Consideration section.

[Stebalien]

An empty partition is left behind in the source deadline after the move. This ensures that partition indices do not change.

This is a bit of a concern as it could be used to create a bunch of empty partitions. The user will pay very little for them (no proofs necessary) but the system will pay in cron (although it might be negligible, I'm not sure).

But I don't know of a great fix other than some kind of "forced compaction" mechanism (which, IMO, we should have anyways).

The partitions are moved as a whole, without any compaction.

Related to above, I'm a bit concerned this could be used to create a bunch of sparse partitions (whose sectors could be subsequently terminated to create empty partitions). What if we, instead:

1. Validate any optimistic proofs for the last partition in the destination deadline.
2. Move while compacting.

---

The alternative to both of these is to propose some mechanism where partition compaction would be required at some point. E.g., forbid moving partitions and/or sealing new sectors if any deadline has:

1. More than one partition.
2. Fewer live sectors than dead sectors.

That would address all my concerns about "empty sectors", but it would force compaction which isn't cheap.

[steven004]

While I think this is beyond the current proposal's scope, it definitely holds potential for further exploration.

This is a bit of a concern as it could be used to create a bunch of empty partitions. The user will pay very little for them (no proofs necessary) but the system will pay in cron (although it might be negligible, I'm not sure).

The problem here is that the network pays some fee for the SPs in cron, which results in different fairness issues.

The alternative to both of these is to propose some mechanism where partition compaction would be required at some point.

I do not know how much this will help. I recommend initiating a separated discussion to delve deeper into this matter and determine whether such a mechanism is truly necessary after the implementation of this proposal.

[anorth]

Empty partitions are skipped fairly early in cron processing. But the real answer to everything that rhymes with "system will pay in cron" is to change the miner construction so that this work happens under explicit calls from the SPs, and cron merely checks that they have done the work. I don't think we should make things unnecessarily difficult for ourselves in these various feature changes, but instead implement features simply and then fix use of cron.

1. Validating proofs is pretty complex, already the most complex part of this change (we initially didn't realise it was required at source)
2. The initial implementation sketch for this did compact at the destination, but I requested not doing so. Partly because it would require validating the destination window post, but also it's more work to compute and write that state vs moving a partition object (CID).

We could require the destination to be "already compact", but I don't really think it's worthwhile.

[Stebalien]

So, I missed something:

To adhere to the requirement of conducting a WindowPoSt every 24 hours, the MovePartitions function only permits the movement of partitions to a DestDeadline whose next proving period is scheduled to occur within 24 hours after the OrigDeadline's last proving period. If the DestDeadline falls outside of this time frame, it will fail. This restriction ensures that the sector's period aligns with the required WindowPoSt interval.

Every deadline is scheduled within 24 hours and, no matter what window we set here, there's nothing preventing someone from repeatedly moving partitions around the clock to never have to prove them.

As far as I can tell, the only solution here is to "mark" moved partitions and/or deadlines with recently moved partitions, preventing repeated moves.

[Stebalien]

Oh, I see what that I missed. You're forbidding moving partitions beyond 24 hours from when it was last proven.

Nevermind, that should work. Effectively, a partition can only be moved between it's current deadline and now "backwards" in time. I.e., going clockwise:

 x  x  x
  \ | /
o -   - t
  / | \
 n  o  o

Where t is the target deadline and n is the current deadline, t can be moved to any x. E.g., if moved "back" one deadline, we'd end up with the following picture:

 x  x  t
  \ | / ↰
o -   - o
  / | \
 n  o  o

The window within which it can move will shrink until the current deadline passes it.

[jennijuju]

Request as part of last call

@zhiqiangxu @steven004 - have you/your team tested/benchmark the miner cron impact/performance of under pessimistic situations? For example, assuming large amount of posts are scheduled in the same window; assuming aggressive amount of faults for the same window?
(While I believe zen’s analysis above is good enough for an estimate for the fip considered to be merged, however, for accepting an finalizing the fip in the network, I would love more concrete numbers and confident confirmation it won’t create a time bomb for the network.

In the fip - could you please also suggest what kind of monitoring should be setup for ensuring we catch any unexpected degradation post upgrade soon enough?

Could you please also briefly capture the potential future work (safe cron/user paid cron) mentioned in this discussion in the fip so they don’t get lost?

[anorth]

I do not think any practical benchmarking approach will generate more useful numbers than those given by @ZenGround0's analysis of actual mainnet state. Benchmarks would only be informative here if operated on a miner state of similar size and construction to mainnet. This is very difficult and expensive to construct, and we already have one! The numbers we have from @ZenGround0 are the concrete numbers you seek.

I am confident that the impacts implied by the analysis are fine for the network to absorb for quite a long period - certainly long enough for us to have improved underlying things like moving some of the cost into user-paid operations. At the current scale (which is still shrinking), even if the entire network moved all storage into a single shared 8-hour business day window (i.e. 960 epochs in same timezone), we'd still have a 16x headroom (960/60) before approaching a 1.0 sec cron execution time to fault 10% of network power, and 0.3s to maintain that fault. P90 cron execution today is just under 2.0 s and we survived much worse before FIP-0060.

[jennijuju]

I am confident that the impacts implied by the analysis are fine for the network to absorb for quite a long period

sounds good.

[zhiqiangxu]

We haven't done benchmark yet, just some functional test cases .

I was discussing one technical issue about find_sectors_by_expiration with @Stebalien and @anorth , and just pushed a fix today.

After the fix is confirmed, I'll add more test cases which @Stebalien proposed.

I think the most important use case for this FIP is that SP can move the window post to office hours, which is about 8 hours instead of 24 hours. So I expect the congestion to be about 3x for a specific SP, but each SP can have a different choice, network wise, the final distribution depends on the physical locations of SPs. But even if all SPs are from the same location, the expected congestion is about 3x(assuming the office hour is 8 hours).

There's no particular monitoring needed in my head, but we may want to test more edge cases.

In terms of potential future work, I'll leave it for @steven004 .

[jennijuju]

Well I think there are at least the following we should be monitoring

• usage of the newly introduced feature
• Miner cron performance (with potential alerts)
• #of partitions for each SP to verify if the assumptions are not too off (if we were building our confidence base of practical assumptions)

[steven004]

@jennijuju , the impact of newly introduced feature and # of partitions of each SP to verify could be evaluate by calculating gas assumptions.
Regarding the cron performance assessment, I believe we had done similar work before against previous FIPs. Is there an existing test code we could refer to, or any one we could work together on this, so that we could follow the best practice.

[steven004]

Thanks @anorth for the comment , which makes a lot of sense.

[jennijuju]

During nv21 testing, @arajasek discovered there is critical implementation design bug in the proposed solution. Moving the slack thread here:

@anorth

Re-stating the issue after talking it through with aayush. The partition state has a field expirations_epochs which maps epochs to sets of sector IDs and memoized power etc corresponding to those sectors. We don't re-quantize any of these entries. We can't add duplicate entries to this queue because the memoized values are used to update power etc cheaply when the sectors expire, without loading per-sector info.
The deadline state also has an expirations_epochs which maps epochs to partition numbers. This queue can have multiple entries for the same partition and by design it's ok to have entries naming a partition that doesn't actually have anything expiring at that epoch.
Consider a sector with declared expiration E , originally in deadline D1 and scheduled to have expiration processed at E1 according to D1's quantisation. Its partition is moved to new deadline D2 , and and that deadline's expiration queue is updated to have an entry for E1 (even though that's not quantised naturally for D2). If the sector faults and then recovered, the partition state's expiration_epochs is updated to have an entry for E2, a quantization of E for D2. But the deadline D2 expiration epoch's does not have an entry for E2 for that partition, it only has E1. If E1 < E2 then the expiration won't be processed (until the partition happens to be processed due to some other sector expiring later, but there might be none such).

Anorth also suggested the following options accordingly:

A few directions to resolve this while moving partitions:

• Add an entry to D2 expiration queue for E2 (we might have to actually add two values, quantized up and down, because we don't know E). The is a relatively simple change to the current impl
• Requantize the moved partition's expiration queue so that it only has E2 -type entries, naturally quantized for the new deadline. The deadline only needs E2. This requires loading the sector infos to get at the declared expirations, but the partition stays intact.
• Take the implementation approach of adding each moved sector to the destination deadline using the code paths for activating new sectors (
  @alanxu - venus
  originally proposed this path). This requires loading the sector infos and executing more logic to mutate/build the destination partition/s states (but is logic that is already exercised)
• Discard deadline-level partition queues and just process every partition in every proving period. This simplifies the code, but will add cost of loading the partition state each time. Today this is in cron, but we plan to move it to miner-initiated (and paid) code paths in time.
• Declare this area of code too complex to change safely and withdraw the FIP until some big overhaul can simplify it.

From governance perspective, it might worth consideration to withdraw the acceptance of FIP0070 and revert it to Draft first cc @kaitlin-beegle. FIP authors and core devs should continue iterating on the design of this FIP.

[luckyparadise]

Providing this information to document the rationale behind the status of this FIP.

As this FIP remains bugged, it has become unimplementable and therefore Core Devs have rejected this FIP for implementation. More information here - https://filecoinproject.slack.com/archives/C01EU76LPCJ/p1706041666706329