ROC produces an error with default args when dask is used

[nicholasloveday]

Currently, check_args needs to be set to False in the ROC metric if people are working with Dask data. This has happened to two people in the last week.

A quick fix would be, if the input data is an xarray-dask object, then automatically set check_args=False

This other issue is also related #908

[tennlee]

Thanks for raising the issue. That's interesting. Can you add a test showing this and post the stack trace (from CI/CD would be fine). I'd be interested to see the issue, and also determine whether or not there's an upstream issue with dask as well (as I would have thought they would aim for full compatibility).

[tennlee]

Or, if a test seems unnecessary, then just post a simple way to reproduce it, so I can take a look at what's happening.

[nicholasloveday]

@tennlee - see here brightbandtech/ExtremeWeatherBench#300 (comment)

Basically the docs say "if using dask, set check_args=False", but in practice most people don't read every detail of the docs and just expect it to work with default args, which I think is quite reasonable!

[tennlee]

Thanks, that was instructive. If we put a fix into scores, we should make it conditional on encountering the NotImplementedError. That way, if they implement it one day, then our code will go back to checking the args. We should also raise a warning along the lines of "Not checking arguments due to a gap in dask which may be implemented in future". Obviously better yet we should come up with an alternative way to check the args and do that instead. But I wanted to jump in so that we don't start relying on the existence of a bug...