Investigate lazy-data compliance in IMPROVER

[MoseleyS]

As an Improver developer I want to know which IMPROVER plugins are not lazy so that I can modify them and later take advantage of exciting new technologies such as Dask.

Related issues: #215.

Ideally, each plug-in should work in a lazy manner, i.e., no computation should be triggered and the dask graph should be constructed. The actual computation should be triggered at the point of saving the data to disk.

In order to get a rough estimate of how things look like with respect to being ready for Iris 2 and Dask, Tomek grepped through the code base for direct access to the cube .data attribute, which prevents the data from staying lazy. Currently, direct access to the cube .data attribute occurs in over 1/3 of IMPROVER files under lib (excluding unit tests) and 1/2 of unit tests (but that 1/3 of files accounts for almost 2/3 of lines of code).

Acceptance criteria:

• Document where cube.data is accessed directly.
• Open follow-on tickets to modify the non-lazy code.

[LaurenceBeard]

I think lazy computation was a realistic ambition pre-irisv2.0, but the implementation of dask is incomplete/inconsistent and this would require substantial rewriting of the plugins to try and achieve.
I don't think this would be possible in any reasonable timeframe

@TomekTrzeciak may have some thoughts

[TomekTrzeciak]

I think this ticket is still broadly relevant (despite spelling out my comment from Oct 2017!). I agree it might be too ambitious to make the entire code base lazy-compliant at this point, but we haven't really had a proper look at what it would take either.

It's possible we could get the majority of benefits by finding a better plugin structure and rewriting only the most expensive plugins, while retrofitting the others just enough to comply with the new structure. That is, we would not aim at eradicating cube.data use entirely, but instead focus on the top level computational interfaces and work downwards from there on case-by-case basis.

I think the critical step is to stop passing Iris cubes so much in the internals of plugins and instead implement the core of the scientific algorithms as pure numpy kernel functions that can be efficiently mapped over (suitably chunked) blocks of dask arrays. We could most likely leverage already existing dask and xarray functionality for the latter aspect.

I would like us to at least get a better understanding of the scope of the problem before closing this ticket off.