-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
bugSomething isn't workingSomething isn't workingforcingIssue related to forcing (generation, loading, etc.)Issue related to forcing (generation, loading, etc.)
Description
When running the Makkink forcing generation for ERA5 for a range of a couple of years, it can use excessive memory causing the kernel to crash.
However, when defining ~/.esmvaltool/dask.yml
as the following:
cluster:
type: distributed.LocalCluster
n_workers: 2
threads_per_worker: 1
memory_limit: 5 GiB
It runs fine with a memory use <2GB.
The issue probably occurs in the post-processing function where the potential evaporation is computed. However, we do run into conflicting advice:
- Dask recommends using the distributed scheduler.
- ESMValTool recommends the local sheduler as not all iris (and ESMF...) functionality works well with Dask Distributed.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingforcingIssue related to forcing (generation, loading, etc.)Issue related to forcing (generation, loading, etc.)