For the same adaptor linear-probing, different metrics

[sujathakotte]

Hi Team,

I have set up the evaluation code locally.. For the same set of shots, cases and the same adaptor..giving different metrics and different loss. Want to check if I am doing anything wrong.
Also, how to pass the parameters to adaptor that we choose. Are we allowed to so.
@clemsgrs @marinadamato

Thanks

[clemsgrs]

Hi @sujathakotte,

Great to hear you’ve successfully set up the evaluation code locally!

Regarding the different metrics and loss values you're seeing: this is expected behavior in many cases. Most adaptors are inherently stochastic — they can converge to different local minima depending on initialization and the order of training examples. As a result, the final loss and evaluation metrics can vary slightly between runs, even with the same shots and cases.

We’ll look into seeding data loading, although that alone won’t eliminate all variability. Only fully deterministic adaptors like KNN will give you the same output every time.

Still, could you share a bit more details, e.g. how large is the difference in loss or metrics between runs?

As for passing parameters to the adaptor: currently, on the Grand Challenge platform, it's not possible to change adaptor parameters like learning rate or batch size directly. If you'd like to use a different configuration than what we provide, the best way is to submit a pull request with your own adaptor implementation!

Let me know if that helps or if you'd like assistance preparing such a PR.

[sujathakotte]

Hi @clemsgrs,

Thanks for the clarification! As seed is not set in the adaptors, I totally understand the stochastic nature of the adaptors except KNN, and that makes sense about the variability in metrics across runs.

Regarding the difference in metrics (Task04), the variations vary depend upon the tile size. At tile size of 512, linear-probing differs by 6% across different runs. At tile size of 256, mlp differs by 3%.

I appreciate the suggestion to explore setting a seed for data loading. But, not setting seed at global level in adaptors will not ensure fair comparison, right?

Regarding PR, if I come up with different adaptor, will connect with you. Thanks again for your help!