fix: apply fixes after doing all UKCP18 runs

[ciaransweet]

What this PR is

This PR applies several fixes after I ran the UKCP18 Indicator generation for all the inputs we require

Changes:

• Parameterised threshold_list and threshold_temperature so we can change them per run config
• Parameterised dask_cluster_kwargs so that we can make processing sequential (as needed for memory restricted environments)
• Changed src/hazard/sources/ukcp18.py's fsspec.filesystem to be a filecache ftp implementation. This will store downloaded files under /tmp/ukcp18cache/ so we don't have to download GB files every year of calculations
• Removed collection as a parameter to pass to the UKCP18 dataset, this can be derived from the desired resolution
• Fixed an issue caused by dynamic CRS handling for global 60km data

How you can test it

A 'nice' small dataset to test is the UK 60km Days Tas Above, here's an example input file:

source_dataset: UKCP18
source_dataset_kwargs: "{'resolution':'60km','domain':'uk'}"
scenario_list: "[rcp85]"
gcm_list: "[ukcp18]"
threshold_list: "[20,25,28,30,32,35,40,45,50,55]"
central_year_list: "[2030,2040,2050,2060,2070,2080,2090]"
central_year_historical: 2005
ceda_ftp_username: <your username>
ceda_ftp_url: "ftp.ceda.ac.uk"
ceda_ftp_password: "<your password>"
window_years: 20
inventory_format: "all"
write_xarray_compatible_zarr: true
store: "./indicator/uk_60km_dta"
indicator: "days_tas_above_indicator"

You should get:

uk_60km_dta_tree.txt

💡Useful note 💡

You can pass cwltool where you want /tmp in the container to be mounted to, I found this useful for keeping tabs on /tmp usage:

cwltool --tmpdir-prefix=<PATH_TO_LOCAL_DIR_OF_CHOICE> hazard_workflow.cwl#produce-hazard-indicator input_file.yml

[j08lue]

On first attempt to run the workflow as recommended above, I get

Exception: "FileNotFoundError('/badc/ukcp18/data/land-rcm/uk/60km')"

and later

ftplib.error_perm: 530 Sorry, the maximum number of clients (20) from your host are already connected.

This is the image version:

REPOSITORY                                        TAG       IMAGE ID       CREATED         SIZE
public.ecr.aws/c9k5s3u3/os-hazard-indicator       latest    14d61da778e7   5 days ago      2.29GB

[j08lue]

Hmm - I set --tmpdir-prefix=./ukcp-cache, but I got this:

When I add a trailing slash --tmpdir-prefix=./ukcp-cache/, the cache dirs go below. I guess this is expected and built-in behavior.

[j08lue]

Btw, unrelated to this PR - I also set

store: "./indicator/uk_60km_data"

in the inputs yaml file, but got the indicator results stored in ./uk_60km_data.

I think this might be an issue to look into, because output dirs really need to be predictable. 🤷

[j08lue]

I can confirm that

• ✅ the --tmpdir-prefix option affects where the UKCP data is cached and
• ✅ the indicator output files get produced as expected. 👌
• ✅ read and plotted _map and _xarray with xarray - see notebook

[j08lue]

Had another look, this time reviewing the changes to the STAC item metadata that gets produced.

• ✅ new indicator STAC metadata includes product-specific paths, rather than templates, and shortened descriptions (nice way to diff, from @ciaransweet: https://www.diffchecker.com/yitCSG42/)