Skip to content
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
e019ab5
added custom CMOR tables for ESACCI-PERMAFROST
axel-lauer Feb 28, 2024
fbce425
added new coordinate to custom coordinate table
axel-lauer Feb 28, 2024
cc102fa
changed sdepth to depth
axel-lauer Mar 4, 2024
c38bda3
Merge branch 'main' into esacci-permafrost
axel-lauer Jun 12, 2024
659494d
added derivation script for permafrost extent (pfr)
axel-lauer Jun 14, 2024
b4f46b9
permafrost extent: switched from sftgif to mrsos
axel-lauer Jun 17, 2024
0c86835
Merge branch 'main' into esacci-permafrost
axel-lauer Jul 23, 2025
4db4a42
updated pfr.py
axel-lauer Jul 23, 2025
9ee7825
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Dec 19, 2025
480d33d
fix ruff issues
axel-lauer Dec 19, 2025
26c19da
update pfr.py
axel-lauer Dec 19, 2025
e562830
removed comments
axel-lauer Dec 19, 2025
f4fb071
Merge branch 'main' into esacci-permafrost
axel-lauer Dec 19, 2025
b616d92
non-working draft of pfr derivation unit test
axel-lauer Dec 19, 2025
8a92845
added test for derived variable pfr
axel-lauer Jan 9, 2026
1db80de
pre-commit results
axel-lauer Jan 9, 2026
5d0f518
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Jan 13, 2026
66eb70a
Merge branch 'main' into esacci-permafrost
axel-lauer Jan 13, 2026
b3d0fca
extended unit test for derivation of pfr
axel-lauer Jan 13, 2026
3177cd8
Merge branch 'esacci-permafrost' of github.com:ESMValGroup/ESMValCore…
axel-lauer Jan 13, 2026
d370a3b
update test_pfr.py
axel-lauer Jan 13, 2026
01c8227
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Jan 15, 2026
cce9088
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Jan 15, 2026
59bc9c0
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Jan 15, 2026
ee34de9
split unit tests into separate functions
axel-lauer Jan 15, 2026
f20e3d7
updated pfr
axel-lauer Jan 15, 2026
c5453bb
Merge branch 'main' into esacci-permafrost
axel-lauer Jan 15, 2026
6913302
Update esmvalcore/preprocessor/_derive/pfr.py
axel-lauer Jan 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions esmvalcore/cmor/tables/custom/CMOR_alt.dat
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
SOURCE: CMIP6
!============
variable_entry: alt
!============
modeling_realm: land
frequency: yr
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:
units: m
cell_methods: time: mean
cell_measures: area: areacella
long_name: Active Layer Thickness
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions: longitude latitude time
type: real
!----------------------------------
!
22 changes: 22 additions & 0 deletions esmvalcore/cmor/tables/custom/CMOR_coordinates.dat
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,28 @@ must_have_bounds: no
!----------------------------------
!

!============
axis_entry: sdepth
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name: depth
units: m
axis: Z ! X, Y, Z, T (default: undeclared)
positive: down ! up or down (default: undeclared)
long_name: depth
!----------------------------------
! Additional axis information:
!----------------------------------
out_name: depth
valid_min: 0.0
valid_max: 200.0
stored_direction: increasing
type: double
must_have_bounds: yes
!----------------------------------
!

!============
axis_entry: time
Expand Down
22 changes: 22 additions & 0 deletions esmvalcore/cmor/tables/custom/CMOR_gtd.dat
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
SOURCE: CMIP6
!============
variable_entry: gtd
!============
modeling_realm: land
frequency: yr
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:
units: K
cell_methods: time: mean
cell_measures: area: areacella
long_name: Permafrost Ground Temperature
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions: longitude latitude sdepth time
out_name: gtd
type: real
!----------------------------------
!
25 changes: 25 additions & 0 deletions esmvalcore/cmor/tables/custom/CMOR_pfr.dat
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
SOURCE: CMIP6
!============
variable_entry: pfr
!============
modeling_realm: land
frequency: yr
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:
units: %
cell_methods: time: mean
cell_measures: area: areacella
long_name: Permafrost Extent
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions: longitude latitude time
type: real
valid_min: 0.0
valid_max: 100.0
ok_min_mean_abs: 0.0
ok_max_mean_abs: 100.0
!----------------------------------
!
154 changes: 154 additions & 0 deletions esmvalcore/preprocessor/_derive/pfr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
"""Derivation of variable `pfr`."""

import logging

import dask.array as da
import iris
import numpy as np
from iris import NameConstraint
from iris.time import PartialDateTime

from ._baseclass import DerivedVariableBase

logger = logging.getLogger(__name__)

# Constants
THRESH_TEMPERATURE = 273.15
FROZEN_MONTHS = 24 # valid range: 12-36


class DerivedVariable(DerivedVariableBase):
"""Derivation of variable `pfr` (permafrost extent)."""

@staticmethod
def required(project):

Check warning on line 24 in esmvalcore/preprocessor/_derive/pfr.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

esmvalcore/preprocessor/_derive/pfr.py#L24

Function is missing a type annotation. (no-untyped-def)
"""Declare the variables needed for derivation."""
return [
{"short_name": "tsl", "mip": "Lmon"},
{"short_name": "sftlf", "mip": "fx"},
# {'short_name': 'sftgif', 'mip': 'fx'}]
{"short_name": "mrsos", "mip": "Lmon"},
]

@staticmethod
def calculate(cubes):

Check warning on line 34 in esmvalcore/preprocessor/_derive/pfr.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

esmvalcore/preprocessor/_derive/pfr.py#L34

Function is missing a type annotation. (no-untyped-def)

Check notice on line 34 in esmvalcore/preprocessor/_derive/pfr.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

esmvalcore/preprocessor/_derive/pfr.py#L34

Too many local variables (24/15) (too-many-locals)
"""Compute permafrost extent.

Permafrost is assumed if
- soil temperature in the deepest level is < 0°C
- for at least 24 consecutive months
- ice covered part of grid cell is excluded
Reference: Burke, E. J., Y. Zhang, and G. Krinner:
Evaluating permafrost physics in the Coupled Model
Intercomparison Project 6 (CMIP6) models and their
sensitivity to climate change, The Cryosphere, 14,
3155-3174, doi: 10.5194/tc-14-3155-2020, 2020.
"""
# create a mask of land fraction (%) over ice-free grid cells

# # use ice fraction (sftgif) --- only available from very few models
# # 1) annual mean of fraction of grid cell covered with ice (%)
# icefrac = cubes.extract_cube(NameConstraint(var_name='sftgif'))
# iris.coord_categorisation.add_year(icefrac, 'time')
# icefrac_yr = icefrac.aggregated_by(['year'], iris.analysis.MEAN)
# # 2) fraction of land cover of grid cell (%) (constant)
# landfrac = cubes.extract_cube(NameConstraint(var_name='sftlf'))
# # 3) create mask with fraction of ice-free land (%)
# mask = iris.analysis.maths.subtract(landfrac, icefrac_yr)
# # remove slightly negative values that might occur because of
# # rounding errors between ice and land fractions
# mask.data = da.where(mask.data < 0.0, 0.0, mask.data)

# use soil moisture as proxy for ice / ice-free grid cells
# 1) annual mean of fraction of grid cell covered with ice (%)
# assumption: top soil moisture = 0 --> ice covered
mrsos = cubes.extract_cube(NameConstraint(var_name="mrsos"))
iris.coord_categorisation.add_year(mrsos, "time")
mrsos_yr = mrsos.aggregated_by(["year"], iris.analysis.MEAN)
mrsos_yr.data = da.where(mrsos_yr.data < 0.001, 0.0, 1.0)
# 2) fraction of land cover of grid cell (%) (constant)
landfrac = cubes.extract_cube(NameConstraint(var_name="sftlf"))
# 3) create mask with fraction of ice-free land (%)

# latitude/longitude coordinates of mrsos and sftlf sometimes
# differ by a very small amount for some models (probably because
# of rounding errors) preventing iris to do the math
# --> overwrite latitudes/longitudes in sftlf

# fix longitudes if maximum differences are smaller than 1.0e-4
x_coord1 = mrsos.coord(axis="X")
x_coord2 = landfrac.coord(axis="X")
delta_x_max = np.amax(x_coord1.core_points() - x_coord2.core_points())
if delta_x_max != 0.0:
if abs(delta_x_max) < 1.0e-4:
x_coord2.points = x_coord1.points
x_coord2.bounds = x_coord1.bounds
else:
logger.error(
"Longitudes of mrsos and stflf fields differ (max = %f).",
delta_x_max,
)

# fix latitudes if maximum differences are smaller than 1.0e-4
y_coord1 = mrsos.coord(axis="Y")
y_coord2 = landfrac.coord(axis="Y")
delta_y_max = np.amax(y_coord1.core_points() - y_coord2.core_points())
if delta_y_max != 0.0:
if abs(delta_y_max) < 1.0e-4:
y_coord2.points = y_coord1.points
y_coord2.bounds = y_coord1.bounds
else:
logger.error(
"Latitudes of mrsos and stflf fields differ (max = %f).",
delta_y_max,
)

mask = iris.analysis.maths.multiply(mrsos_yr, landfrac)

# extract deepest soil level
soiltemp = cubes.extract_cube(NameConstraint(var_name="tsl"))
z_coord = soiltemp.coord(axis="Z")
zmax = np.amax(z_coord.core_points())
soiltemp = soiltemp.extract(iris.Constraint(depth=zmax))
soiltemp.data = da.where(soiltemp.data < THRESH_TEMPERATURE, 1, 0)
iris.coord_categorisation.add_year(soiltemp, "time")

# prepare cube for permafrost extent with yearly time steps
pfr_yr = soiltemp.aggregated_by(["year"], iris.analysis.MEAN)
# get years to process
year_coord = pfr_yr.coord("year")
# calculate time period before and after current year to include
# in test for permafrost
test_period = (FROZEN_MONTHS - 12) / 2
# loop over all years and test if frost is present throughout
# the whole test period, i.e. [year-test_period, year+test_period]

for tidx, year in enumerate(year_coord.points):
# extract test period
pdt1 = PartialDateTime(
year=year - 1,
month=13 - test_period,
day=1,
)
pdt2 = PartialDateTime(year=year + 1, month=test_period + 1, day=1)
daterange = iris.Constraint(
time=lambda cell, pdt1=pdt1, pdt2=pdt2: pdt1
<= cell.point
< pdt2,
)
soiltemp_window = soiltemp.extract(daterange)
# remove auxiliary coordinate 'year' to avoid lots of warnings
# from iris
soiltemp_window.remove_coord("year")
# calculate mean over test period
test_cube = soiltemp_window.collapsed("time", iris.analysis.MEAN)
# if all months in test period show soil tempeatures below zero
# then mark grid cell with "1" as permafrost and "0" otherwise
pfr_yr.data[tidx, :, :] = da.where(test_cube.data > 0.99, 1, 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be replaced by the rolling_window_statistics preprocessor function? Something like:

pfr_yr = esmvalcore.preprocessor.rolling_window_statistics(
    soiltemp,
    coordinate="time",
    operator="mean",
    window_length=FROZEN_MONTHS,
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. I tried a few options but could not reproduce the original output. Not sure it is worth spending much more time on this possible replacement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rolling_window_statistics could be used if you extend the input cube by test_period time steps to both sides (the rolling window with a window > 1 will produce a shorter output cube; this accounts for that). Not sure if that's makes the code simpler, probably not...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the current implementation creates fairly small dask chunks, which may lead to slow/inefficient computations. Every time you create a dask array and use it to build another dask array, you insert all the dask chunks needed to build that subarray into the final array. Looping over the time-axis and computing things separately for each time step is a typical example of this. This may be aggravated by the use of aggregated_by SciTools/iris#5455. Therefore, I think it would be worth investigating if rolling_window_statistics can be used to reduce the size of the Dask task graph and increase the chunk sizes.

However, maybe this works just fine, it's hard to tell from just looking at the code. Did you test the runtime? Can you run the computation using the distributed scheduler? And did you test how long it takes to compute a multi-model mean over e.g. 100 datasets?

It would be nice to have a good unit test, so we can look at improving computational performance later https://github.com/ESMValGroup/ESMValCore/pull/2358/files#r2690416067 if needed.


pfr_yr = pfr_yr * mask
pfr_yr.units = "%"
pfr_yr.rename("Permafrost extent")
pfr_yr.var_name = "pfr"

return pfr_yr