Skip to content

"The behaviour of Join.collect_supporting_arrays() is not well defined" warning before lam-training #825

New issue
New issue
Open
Open
"The behaviour of Join.collect_supporting_arrays() is not well defined" warning before lam-training#825
Labels
bugSomething isn't working
@inakbk

Description

@inakbk
inakbk
opened on Jan 23, 2026

What happened?

Before the training starts we get this error

/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")

Which raises the question, are there any unintended consequences in the loading of our dataset?

NOTE:

  • we have gotten this warning also for much older anemoi versions
  • tag my colleague @mateuszmatu (profile) in any discussion of this issue as I will not be able to respond for a few months
  • we use cutout & concat in the dataloader when creating the graph, but not in the below training-yaml. see this issue

What are the steps to reproduce the bug?

See this config for our separate graph-creation. It uses cutout & concat, while the below training-yaml does not.

yaml-file:

defaults:
- data: zarr
- dataloader: native_grid
- datamodule: single
- diagnostics: evaluation
- hardware: slurm
#- graph: limited_area 
- model: graphtransformer
#- training: lam 
- override hydra/hydra_logging: disabled
- override hydra/job_logging: disabled
- _self_

config_validation: True

hydra:  
  output_subdir: null  
  run:  
    dir: .

data:
  frequency: 3h
  timestep: 3h
  forcing:
  - h
  - Pair             
  - Qair             
  - Tair   
  - Uwind
  - Vwind                 
  - cos_julian_day   
  - cos_latitude     
  - cos_local_time   
  - cos_longitude    
  - f                
  - insolation       
  - rain             
  - river_binary_mask
  - sin_julian_day   
  - sin_latitude   
  - sin_local_time
  - sin_longitude
  - sea_mask

  diagnostic: []
    
  processors:
    imputer:
      _target_: anemoi.models.preprocessing.imputer.InputImputer
      config: 
        default: "none"
        mean:  
        - salinity_0
        - temperature_0
        - u_eastward_0
        - v_northward_0
        - zeta
  
    normalizer:
      _target_: anemoi.models.preprocessing.normalizer.InputNormalizer

      config:
        default: "mean-std"
        std: []
        min-max:
        - salinity_0
        max: 
        - h
        none:
        - cos_julian_day   
        - cos_latitude     
        - cos_local_time   
        - cos_longitude  
        - insolation           
        - sin_julian_day   
        - sin_latitude   
        - sin_local_time
        - sin_longitude
        - river_binary_mask
        - sea_mask
      
dataloader:
  num_workers:
    training: 2
    validation: 2
    test: 2
  batch_size: 
    training: 1 # has to be 1 for model-paralell
    validation: 1
    test: 1

  limit_batches:
    training: null
    validation: null
    test: 20 
  
  dataset:
    join:
      - dataset: ${hardware.paths.data}/${hardware.files.dataset.dataset_main}
        frequency: ${data.frequency}
      - dataset: ${hardware.paths.data}/${hardware.files.dataset.dataset_force}
        frequency: ${data.frequency}
    adjust: ["start", "end"]
  grid_indices:
    _target_: anemoi.training.data.grid_indices.MaskedGrid
    nodes_name: data
    node_attribute_name: indices_connected_nodes

  training:
    dataset: ${dataloader.dataset} 
    start: 2012-01-01
    end: 2022-06-30
    drop: ['vbar_northward', 'ubar_eastward', 'Uwind_eastward', 'Vwind_northward']

  validation_rollout: 1 

  validation:
    dataset: ${dataloader.dataset}
    start: 2022-07-01
    end: 2023-09-30
    drop: ['vbar_northward', 'ubar_eastward', 'Uwind_eastward', 'Vwind_northward']
  test:
    dataset: ${dataloader.dataset}
    start: 2023-10-01
    end: 2024-12-31
    drop: ['vbar_northward', 'ubar_eastward', 'Uwind_eastward', 'Vwind_northward']

diagnostics:
  plot: 
    callbacks: []
  log:
    mlflow:
      enabled: True
      offline: True
      authentication: True
      tracking_uri: https://mlflow.ecmwf.int
      experiment_name: 'metno-fou'
      run_name: '<replace-me>' 
    wandb: 
      entity: null
  print_memory_summary: True
  checkpoint:
    every_n_minutes:
      save_frequency: 30 # Approximate, as this is checked at the end of training steps
      num_models_saved: 3 
  
    every_n_epochs:
      save_frequency: 1
      num_models_saved: -1 
  
    every_n_train_steps:
      save_frequency: 1000 # Does not scale with rollout
      num_models_saved: 5

hardware:
  paths:
    data: /cluster/work/projects/nn12017k/datasets2/
    output: /cluster/projects/nn12017k/experiments/<replace-me>/
    graph: /cluster/projects/nn12017k/graphs/
  files:
    dataset:
      dataset_main: norkystv3_hindcast_2012-2024.zarr
      dataset_force: forcing_norkystv3_hindcast_2012-2024.zarr 
    graph: LAM-trimedge-10-res-12-scale12.pt 
    warm_start: null 

  num_gpus_per_node: 4
  num_nodes: 8
  num_gpus_per_model: 4

graph:
  overwrite: False # only create/re-generate the graph with True

model: 
  num_channels: 256 #1024 
  # ------
  # go back to default?
  keep_batch_sharded: False
  encoder:
    num_chunks: 1
    shard_strategy: "heads"
  decoder:
    num_chunks: 1
    shard_strategy: "heads"
  #-------
  trainable_parameters:
    data: 0
    hidden: 0
    data2hidden: 0
    hidden2data: 0
    hidden2hidden: 0 # GNN and GraphTransformer Processor only
  bounding: #These are applied in order
    - _target_: anemoi.models.layers.bounding.ReluBounding
      variables: [] 
  output_mask: # suggested by lam.yaml
    _target_: anemoi.training.utils.masks.Boolean1DMask
    nodes_name: ${graph.data}
    attribute_name: cutout_mask

# ============================================================================
# TRAINING - adding all from lam.yaml "manually" to turn off tendency scalers
# ============================================================================
training:
  run_id: null
  fork_run_id: null
  transfer_learning: False 
  load_weights_only: False

  deterministic: False

  precision: bf16-mixed # todo: test go back to default: 16-mixed

  #----------------------------
  # Using defaults from lam.yaml
  multistep_input: 2
  accum_grad_batches: 1
  num_sanity_val_steps: 6
  gradient_clip:
    val: 32.
    algorithm: value
  swa:
    enabled: False
    lr: 1.e-4
  optimizer:
    _target_: torch.optim.AdamW
    betas: [0.9, 0.95]
  model_task: anemoi.training.train.tasks.GraphForecaster
  strategy:
    _target_: anemoi.training.distributed.strategy.DDPGroupStrategy
    num_gpus_per_model: ${system.hardware.num_gpus_per_model}
    read_group_size: ${dataloader.read_group_size}
  loss_gradient_scaling: False
  #----------------------------

  training_loss:
    _target_: anemoi.training.losses.MSELoss
    scalers: ['general_variable', 'nan_mask_weights', 'node_weights']
    ignore_nans: False

  scalers:
    # Same as scalers/lam.yaml, except not adding stdev_tendency and var_tendency
    general_variable:
      _target_: anemoi.training.losses.scalers.GeneralVariableLossScaler
      weights:
        default: 1
    
    nan_mask_weights:
      _target_: anemoi.training.losses.scalers.NaNMaskScaler
    
    node_weights:
      _target_: anemoi.training.losses.scalers.GraphNodeAttributeScaler
      nodes_name: ${graph.data}
      nodes_attribute_name: area_weight
      norm: "unit-sum"
    
    limited_area_mask:
      _target_: anemoi.training.losses.scalers.GraphNodeAttributeScaler
      nodes_name: ${graph.data}
      nodes_attribute_name: cutout_mask
      norm: null

  validation_metrics:
    mse_inside_lam:
      _target_: anemoi.training.losses.MSELoss
      scalers: ["node_weights"]
      ignore_nans: True
  variable_groups:
    default: sfc
    # pressure levels removed

  metrics:
  - 'all' 

  #----------------------------
  # Using defaults from lam.yaml
  rollout:
    start: 1
    epoch_increment: 0
    max: 1
  submodules_to_freeze: []
  recompile_limit: 32
  #----------------------------

  max_epochs: null
  max_steps: 100000
  lr:
    warmup: 1000
    rate: 6.25e-4
    iterations: ${training.max_steps}
    min: 3e-7

Version

, anemoi-training 0.8.3, anemoi-models 0.11.2, anemoi-graphs 0.8.2, anemoi-datasets 0.5.29.dev13+gef26767bc (and older), torch 2.6.0a0+ecf3bae40a.nv25.01

Platform (OS and architecture)

NVIDIA GraceHopper Superchip (NVIDIA GH200 120GiB)

Relevant log output

/usr/local/lib/python3.12/dist-packages/mlflow/tracking/_tracking_service/utils.py:178: FutureWarning: The filesystem tracking backend (e.g., './mlruns') will be deprecated in February 2026. Consider transitioning to a database backend (e.g., 'sqlite:///mlflow.db') to take advantage of the latest MLflow features. See https://github.com/mlflow/mlflow/issues/18534 for more details and migration guidance. For migrating existing data, https://github.com/mlflow/mlflow-export-import can be used.
  return FileStore(store_uri, store_uri)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/utils/provenance.py:143: UserWarning: The '__version__' attribute is deprecated and will be removed in MarkupSafe 3.1. Use feature detection, or `importlib.metadata.version("markupsafe")`, instead.
  versions[name] = str(module.__version__)
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/training/losses/scalers/variable_tendency.py:57: UserWarning: Dataset has no tendency statistics! Are you sure you want to use a tendency scaler?
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/anemoi/datasets/data/forwards.py:517: UserWarning: The behaviour of Join.collect_supporting_arrays() is not well defined
  warnings.warn(f"The behaviour of {self.__class__.__name__}.collect_supporting_arrays() is not well defined")
/usr/local/lib/python3.12/dist-packages/pytorch_lightning/utilities/model_summary/model_summary.py:242: Precision bf16-mixed is not supported by the model summary.  Estimated model size in MB will not be accurate. Using 32 bits instead.
/usr/local/lib/python3.12/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:378: You have overridden `on_after_batch_transfer` in `LightningModule` but have passed in a `LightningDataModule`. It will use the implementation from `LightningModule` instance.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2026-01-23T10:41:54.228] error: *** STEP 97342.1 ON gpu-1-3 CANCELLED AT 2026-01-23T10:41:54 DUE to SIGNAL Terminated ***
[2026-01-23T10:41:54.229] error: *** JOB 97342 ON gpu-1-3 CANCELLED AT 2026-01-23T10:41:54 DUE to SIGNAL Terminated ***

Accompanying data

No response

Organisation

MET Norway, Division for Ocean and Ice

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Anemoi-dev

    Status

    To be triaged

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions