[Workflow API] Enable participants in FederatedRuntime to review and approve plan

[scngupta-dsp]

SUMMARY

This proposal introduces a structured mechanism for reviewing and agreeing on plans for executing FL experiments in FederatedRuntime within the Workflow API. This feature enables all participants, including the Director and Envoys, to review and approve an experiment plan before execution

The Director, acting as the orchestrator, manages this review process to ensure consistency, trust, and readiness across the federation. Additionally, the User/Data Scientist who submitted the experiment plan will be notified of the outcome (approval or rejection) on completion of the review process.

MOTIVATION

Currently in FederatedRuntime, users can initiate and execute Federated Machine Learning experiments without an explicit approval from all participants (including Directors or Envoys). While this approach allows for a swift action, the absence of formal agreement or review process may lead to following challenges:

• Trust Concerns: Directors and Envoy admins, as owners of sensitive data, may seek greater control and insights regarding the experiments being conducted. A formal review process is highly desirable for owners of sensitive data, allowing them to review the plan and enhance trust
• Risk of Execution Failures: If an Envoy is unprepared or disagrees with the proposed plan, it can result in execution failures. A structured agreement process can help ensure readiness and consensus among all parties involved

SCOPE

• This proposal is applicable only for the FederatedRuntime
• The scenario to dynamically add envoys to ongoing experiment is out of scope of current design
• Support for a configurable review policy that allows user to control the outcome of review process is not included in the current proposal. The current design is flexible and can be extended in the future if this feature is needed

KEY REQUIREMENTS

Feature Configuration:

• Each participant in Federation shall be able to configure this feature individually. Feature can be enabled or disabled by including an optional review_experiment flag in each participant’s configuration (YAML) file. The behavior of review_experiment flag in each participant’s configuration file shall be as follows

  • review_experiment: false - plan is auto-approved
  • review_experiment: true - node-admin review and approval is required

• Default: If the review_experiment setting is not present in configuration, the participant defaults to manually approving the experiment (i.e. as if review_experiment: True)

  • This ensures a stricter control over experiment execution in case this setting has been overlooked by node admin

Configuration File Interface

• To enable the plan review workflow, node-admin shall set the review_experiment flag to true in the configuration file as shown below:

settings:
    listen_host: localhost
    listen_port: 50050
    envoy_health_check_period: 5  # in seconds
    review_experiment: True

Persistence of Configuration:

• Configuration is read by the participant at startup and applies to all the experiments conducted after the node is started. Changes to the configuration require a participant node to be restarted

Director and Envoy Interface for reviewing and approving the plan

• If review_experiment flag is enabled, the plan for the received experiment shall be printed on console and following interface will be presented to the Director and Envoys for reviewing and responding to the experiment plan:

aggregator:
  defaults: plan/defaults/aggregator.yaml
  settings:
    rounds_to_train: 1
  template: openfl.experimental.workflow.component.Aggregator
collaborator:
  defaults: plan/defaults/collaborator.yaml
  settings: {}
  template: openfl.experimental.workflow.component.Collaborator
federated_flow:
  settings:
    checkpoint: true
    learning_rate: src.experiment.learning_rate
    model: null
    momentum: src.experiment.momentum
    optimizer: null
    rounds: 2
  template: src.experiment.FederatedFlow_TorchMNIST
network:
  settings:
    agg_addr: localhost
    agg_port: 49421
    client_reconnect_interval: 5
    disable_client_auth: false
    tls: false
  template: openfl.federation.Network
Do you want to accept the 📂 FederatedFlow_TorchMNIST❔ [y/N]:

Review Timeline

• If the review_experiment flag is set to true, Director and Envoy shall wait for manual approval (or rejection) of the plan

Notify plan review outcome to User/Data Scientist:

• On completion of review process, User shall be provided a notification including
  • Outcome of the review: Accepted / Rejected
  • Individual participants approval / rejection status including the timestamps (e.g. outcome as shown below)

Experiment review Status:

Director: approved Time: 2025-05-07 10:10:15 Portland: approved Time: 2025-05-07 10:11:10 Seattle: rejected Time: 2025-05-07 10:11:10

Experiment execution on Consensus:

• The experiment shall only be executed when all participants have reviewed and accepted the plan

PROPOSED APPROACH
The mechanism consists of two phases: the Review Phase and the Execution Phase.

Phase 1: Review Phase

Director Review: Director reviews the plan.

• If the Director rejects the plan: experiment is aborted immediately, and User is notified of the outcome (Rejected) and Director’s review status (Rejected) along with timestamp
• If the Director approves the plan: The experiment is forwarded to all authorized Envoys for further review

Envoy Review:

• Each Envoy receives the experiment plan for review and provide their response to Director

Director Consensus: Director waits for responses from all Envoys and

• If all Envoys approve: User is notified that the experiment was approved including timestamps of responses by the Director and all Envoys. The experiment moves to execution phase
• If any Envoy rejects: User is notified that the plan was rejected including timestamps of responses by the Director and all Envoys. The experiment is aborted, and cleanup is triggered on all participants

Phase 2: Execution Phase
If the Director and all Envoys approve the experiment:

• Director starts the Aggregator.
• All Envoys are notified to run the Collaborator.

SCENARIOS:

Scenario A: Director rejects the plan

Scenario B: Director approves the plan and (any) Envoy rejects the plan

Scenario C: ALL participants approve the plan

RISKS and MITITAGTION

Different combinations of participant approvals and rejections shall be validated thoroughly to ensure that an experiment proceeds only when all participants have approved the plan

VALIDATION

Manual

DEMONSTRATION

Feature shall be demonstrated by modifying the existing openfl-tutorials/experimental/workflow/FederatedRuntime/101_MNIST tutorial

[scngupta-dsp]

The proposal was discussed in an offline review with key stakeholders

SUMMARY

Default behavior when review_experiment is not configured by node admin:

• Received feedback that the default behavior should be modified to review_experiment: True. This would enforce stricter control over experiment execution in case node admin has overlooked this setting
• Agreement: Suggestion was accepted and the proposal shall be updated accordingly

Configurable Review policy:

• It was proposed to enhance the design by allowing provision for a configurable review policy. For example, user could specify a %ge of participants whose approval would be required to proceed with the experiment execution
• Feedback was discussed and following was agreed
  • To avoid premature complexity, this enhancement is deferred to future iterations. Based on user demand, the existing design could be extended in straightforward manner to support addition of review policies
  • The current approach, requiring full consensus, keeps the design simple and robust. Detailed feedback enables users to resolve disagreements offline. Users also have flexibility to remove non-consenting participants from the collaborator list if needed
• Agreement: Proposal will explicitly state that support for configurable review policies is not covered in current scope

Dynamic Addition of Envoys:

• A question was raised on support for dynamically adding envoys during an experiment.
• It was clarified that Director maintains a list of all envoys that are currently connected to Federation. While new envoys can connect at any time, they do not participate in ongoing experiments if they join after execution has started
• Agreement: The proposal will explicitly state that dynamic addition of envoys is not covered in current scope

NEXT STEPS:

• Update the proposal to:
  • Change the default behavior to review_experiment: True when the setting is not explicitly defined
  • Clearly state that dynamic addition of envoys during an ongoing experiment and support for configurable review policies is not supported in the current design
• Move forward with implementation based on the revised proposal

[teoparvanov]

Thanks for the summary, @scngupta-dsp! I believe the implementation can start based on the agreement points described above.