Any way to perform connectivity unit test?

[sarthakpati]

We are currently in the process of setting up a new federation for a grant between 4 different sites. Do we have a baseline unit test to establish connectivity with an aggregator? It doesn't have to be a compute-intensive process, we just need to check if the network protocols are fine between the compute instances and the aggregator.

[kminhta]

@noopurintel @payalcha - do you have any recommendations for Sarthak on this matter?

If not, maybe we take an AR to add a basic fx collaborator ping command @rahulga1

Thanks!

[teoparvanov]

Hey @kta-intel, would it be possible to just use a "dummy" FL plan with empty training/validation tasks, which just activates all components around the federation (incl. agg/coll connectivity)?

[kminhta]

Hey @kta-intel, would it be possible to just use a "dummy" FL plan with empty training/validation tasks, which just activates all components around the federation (incl. agg/coll connectivity)?

True - I don't think the task runner will allow a completely empty task, but you can set the federation to run just a validation task and then hard code the metric to 0:

# plan.yaml
assigner:
  settings:
    task_groups:
    - name: learning
      percentage: 1.0
      tasks:
      - aggregated_model_validation

# taskrunner.py
def validate_(self, validation_dataloader: Iterator[Tuple[np.ndarray, np.ndarray]]) -> Metric:
    accuracy = 0
    return Metric(name="accuracy", value=np.array(accuracy))

which can be a good sanity check for connectivity; however, I imagine @sarthakpati may be asking for a way to check connectivity after the actual plan/workspace has already been distributed (?)

[sarthakpati]

may be asking for a way to check connectivity after the actual plan/workspace has already been distributed (?)

Yup, that is correct. We are still in the process of finalizing the data science piece of the experiment, but I wanted to get a head start on trying to figure things out with the different IT folks of the 4 sites.

[teoparvanov]

@sarthakpati , do you plan to use OpenFL 1.8 for the new federation you're building?

[sarthakpati]

Sure, I can make that a requirement.

[payalcha]

My understanding of question -
Once we have plan circulated between aggregator and collaborators, is there a way to test the connections between them?

In our experience, if we have connection, firewall and ports properly set between aggregator and collaborators, we never faced issue in federation experiment.

Regarding tests - We don't have any connectivity specific test.

fx collaborator ping -- seems nice idea.

[sarthakpati]

Once we have plan circulated between aggregator and collaborators

Unfortunately, this is not the case here.

if we have connection

At this point, there is no connectivity between the collaborators and the aggregator. This is what I want to test. It is easy for us to set up OpenFL on the aggregator (which we control). However, setting this up on collaborators will be the main task, which involves getting into discussions with the IT folks to answer routine questions such as which ports to open, and so on.

[teoparvanov]

Thanks all for the great discussion! The consensus so far seems to settle around a new fx collaborator ping command. This, however, would require code changes and a release vehicle (either a 1.8.1 patch, or via the soon-to-be nightly build).

We'll discuss among the OpenFL team how this can be prioritized and delivered in an upcoming release.

[sarthakpati]

Running fx collaborator ping from the aggregator is a great idea! 👍🏽

A detailed instruction set for both sides of the federation (aggregator and collaborator) would be awesome). 🥳

[teoparvanov]

Here's a a draft code change for this - see #1516. Please review and discuss!

[teoparvanov]

Hi @sarthakpati, the above PR has been merged and it's part of the latest nightly build (incl. the fx collaborator ping command).
You can install it by running:

pip install openfl-nightly==1.9.0.dev202504091

To setup the federation E2E (incl. steps on both aggregator and collaborator side), please follow the Task Runner documentation.

PS: you can use this nightly build for the initial phases of setting up the federation, but ideally we recommend that you move to "production" with an official build.

Let us know how it goes!

[sarthakpati]

Brilliant, will do! Thank you so much!