Skip to content

Pushforward feature from fe param search #1745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
ankitpatnala wants to merge 52 commits into
base: develop
Choose a base branch
from
Draft

Pushforward feature from fe param search #1745

ankitpatnala wants to merge 52 commits into from

Conversation

@ankitpatnala
Copy link
Contributor

@ankitpatnala ankitpatnala commented Jan 29, 2026

Description

It takes around push-forward experiments where the loss is only calculated on 1 step rather than applying to all steps.
Only meant for experimentation. Built on top of fe-param-search branch of Matthias.

closes #1740

Checklist before asking for review

  • I have performed a self-review of my code
  • My changes comply with basic sanity checks:
    • I have fixed formatting issues with ./scripts/actions.sh lint
    • I have run unit tests with ./scripts/actions.sh unit-test
    • I have documented my code and I have updated the docstrings.
    • I have added unit tests, if relevant
  • I have tried my changes with data and code:
    • I have run the integration tests with ./scripts/actions.sh integration-test
    • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
    • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
  • I have informed and aligned with people impacted by my change:
    • for config changes: the MatterMost channels and/or a design doc
    • for changes of dependencies: the MatterMost software development channel

sophie-xhonneux and others added 30 commits August 6, 2025 12:24
@sophie-xhonneux
Log gradient norms
a0039ec
@sophie-xhonneux
Prototype for recording grad norms
e83903b
@sophie-xhonneux
Address review changes + hide behind feature flag
d2995b4
@sophie-xhonneux
Final fixes including backward compatibility
26c6869
@sophie-xhonneux
Merge branch 'develop' into sophiex/dev/log-grad-norms
66da0d7
@sophie-xhonneux
Ruff
9a66f72
@sophie-xhonneux
More ruff stuff
22a6fd7
@MatKbauer
Update to develop, prepare for new experiment series
a1d7a27
@MatKbauer
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into mk/d…
128aeb1 
…evelop/fe_experiments
@MatKbauer
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into mk/d…
d4be568 
…evelop/fe_experiments
@MatKbauer
Rebase to latest develop
6504fc7
@MatKbauer
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into mk/d…
4f62e1a 
…evelop/fe_experiments
@Jubeku
Merge branch 'develop' into jk/log-grad-norms/log-grad-norms
87e7d3b
@Jubeku
forecast config with small decoder
754d31c
@Jubeku
Merge branch 'develop' into jk/log-grad-norms/log-grad-norms
cd7948f
@Jubeku
fixed uv.lock
7c756a3
@Jubeku
test gradient logging on mutli gpus
41716a6
Update branch to latest develop with configured o48 settings
b5ce171
Setting o48 as default in era5 config
c12e190 
Committer: Matthias Karlbauer <[email protected]>

On branch mk/develop/fe_experiments
Your branch is ahead of 'origin/mk/develop/fe_experiments' by 57 commits.
  (use "git push" to publish your local commits)

Changes to be committed:
  modified:   config/streams/era5_1deg/era5.yml
@MatKbauer
Updated default config to 256 dim latent size
d95277e 
On branch mk/develop/fe_experiments
Your branch is ahead of 'origin/mk/develop/fe_experiments' by 58 commits.
  (use "git push" to publish your local commits)

Changes to be committed:
	modified:   config/default_config.yml
@MatKbauer
Update branch to latest develop
a734471
@MatKbauer
Merge branch 'develop' of github.com:ecmwf/WeatherGenerator into mk/d…
3ae99dd 
…evelop/fe_experiments
@MatKbauer
Change epochs from 64 to 32
eba89a6
@MatKbauer
LayerNorm replication and analysis tools
5615634
@MatKbauer
Rename fe_layer_norm_at_layers to fe_layer_norm_after_blocks
9ccc95e
@MatKbauer
Increase epochs from 32 to 64 and resolve minor bug
240031d
@MatKbauer
Merged gradient logging
75b81fe
@MatKbauer
Update to develop
f65ac37
@MatKbauer
Update default_config back to d2048 on the O96 grid
20ae505
@MatKbauer
Update ERA5 stream to O96 grid
2731d29
MatKbauer and others added 22 commits November 17, 2025 13:48
@MatKbauer
Update to latest develop having mini-epoch notation
912c406
@MatKbauer
Resolving bug after merging with develop and updating default_config
028bb98
@MatKbauer
Enable loading old model checkpoints after recent merges
ba84066
@MatKbauer
Update WeatherGenReader with mini-epoch notation
4f00cc6
@MatKbauer
Minor modifications to latent histogram plotting
e44e139
@MatKbauer
Resolve bug in histogram plotting
c979ab4
@MatKbauer
Replace getattr by cf.get
d24c4b6
@MatKbauer
Change target read-out engine from 1 to 2 layers
89670bf
@MatKbauer
Set aux-info for fe-blocks to none
58474b2
@SavvasMel
fix a plotting bug (ecmwf#1453)
184dcd9
@MatKbauer
Update train/val dates, HL=5, fsteps=2, lat-weighting
d3b63d2
@MatKbauer
Defined base config for parameter search
a584e41
@MatKbauer
Increase encoder/decoder size and add mlflow tags
d7e75eb
@MatKbauer
Added plot_train content
9bc45c5
@MatKbauer
Adam betas per cli and aifs channel weighting option
40b070e
@MatKbauer
Updated launch script
8c6dafb
@MatKbauer
Updated multiple job launch script with pre-training dropout and nois…
b5c98cb 
…e ablation
@MatKbauer
Updated experiment launch script
76cb62c
@ankitpatnala
added config and scripts for launching multiple runs
d14291c
@ankitpatnala
yml and multi scripts
42b4318
@ankitpatnala
changed eval_config and inference script
892f509
@ankitpatnala
added spike function to the fstep weighting
a44d5a4
@ankitpatnala
Copy link
Contributor Author

ankitpatnala commented Jan 29, 2026

cc @SavvasMel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

WeatherGen-dev
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Run pushforward experiments (4,6,8,12) steps

5 participants

@ankitpatnala @sophie-xhonneux @MatKbauer @Jubeku @SavvasMel