Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean nodes after building the graph #23

Merged
merged 64 commits into from
Jul 26, 2024
Merged

Clean nodes after building the graph #23

merged 64 commits into from
Jul 26, 2024

Conversation

JPXKQX
Copy link
Member

@JPXKQX JPXKQX commented Jul 25, 2024

This PR implements a post-process function to remove node attributes that are no longer needed, such as the networkx.Graph, the node_ordering or the refinement levels.

These attributes are stored in the HeteroData object during the node construction and used later by the edge builders.

JPXKQX and others added 30 commits July 5, 2024 16:36
@JPXKQX
Global Encoder-Processor-Decoder graph (#9)
3609681 
* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
@JPXKQX
fix: attributes as torch.float32
a973c2d
@JPXKQX
new test: attributes must be float32
af111a6
@JPXKQX
fix typo
a8a1620
@JPXKQX
Homogeneize base builders
926c75b
@JPXKQX
improve test docstrings
f342073
@JPXKQX
homogeneize (name as class attribute)
9d9fea8
@JPXKQX
new input config
1b20845
@JPXKQX
new default
4e62431
@JPXKQX
Merge branch 'develop' into fix/global-version
6825850
@theissenhelen
feat: Initial implementation of global graphs
d25c47b 
Co-authored by: Mario Santa Cruz <[email protected]>
@JPXKQX @theissenhelen
add cli command
8f0415e
@JPXKQX @theissenhelen
Ignore .pt files
22fba0e
@JPXKQX @theissenhelen
run pre-commit
2879c7c
@JPXKQX @theissenhelen
docstring + log erros
1ca0633
@JPXKQX @theissenhelen
initial tests
277c231
@theissenhelen
feat: initial version of AttributeBuilder
7a4e9cb
@theissenhelen
refactor: separate into node edge attribute builders
96a6ed5
@JPXKQX @theissenhelen
feat: edge_length moved to edges/attributes.py
1c88ee8
@JPXKQX @theissenhelen
remove __init__
92988d7
@JPXKQX @theissenhelen
bugfix (encoder edge lengths) + refector
de07488
@JPXKQX @theissenhelen
feat: support path and dict for config argument
c900bfd
@JPXKQX @theissenhelen
fix: error
9768204
@theissenhelen
refactor: naming
74ef4cc
@theissenhelen
fix: pre-commit
91e15b3
@theissenhelen
feat: builders icosahedral
66d3aa1
@theissenhelen @JPXKQX
feat: Add icosahedral graph generation
23b3110 
Co-authored-by: Mario Santa Cruz <[email protected]>
@theissenhelen
refactor: remove create_shere
469b56b
@theissenhelen
feat: Icosahedral edge builder
4e864c3
@theissenhelen @JPXKQX
feat: hexagonal graph generation
d573905 
Co-authored-by: Mario Santa Cruz <[email protected]>
theissenhelen and others added 20 commits July 8, 2024 13:27
@theissenhelen
test: add tests for TriIcosahedralEdges
02496ec
@theissenhelen
fix: assert missing for Hexagonal edges
59fcad3
@theissenhelen
test: hexagonal edges
b41f88e
@theissenhelen
fix: avoid same name
5a43185
@theissenhelen
fix: imports
a0259f8
@theissenhelen
fix: conflicts
7289e32
@theissenhelen
Merge branch 'develop' into 6-generate-graphs-from-icosahedral-meshes
9a6decc
@JPXKQX
update tests
4ca717b
@theissenhelen
Merge branch 'develop' into 6-generate-graphs-from-icosahedral-meshes
fb93b8b
@JPXKQX
Include xhops to hexagonal edges
fe8a8e5
@JPXKQX
Merge pull request #14 from ecmwf/hotfix/config_node_name
5865b35 
Hotfix: nodes config name

Co-authored-by: <[email protected]>
Co-authored-by: <[email protected]>
@theissenhelen
Merge remote-tracking branch 'origin/develop' into 6-generate-graphs-…
633091f 
…from-icosahedral-meshes
@theissenhelen
Merge remote-tracking branch 'origin/develop' into 6-generate-graphs-…
b2820e2 
…from-icosahedral-meshes
@theissenhelen
docs: update docstrings
463911c
@JPXKQX
fix: update attribute name
b0a35b8
@theissenhelen
refactor: rename multiscale nodes
4f445a9
@theissenhelen
refactor: rename icosahedral nodes
fa812eb
@JPXKQX
Clean output graph object
1bd2c0c
@JPXKQX
tests: update
2bc8fb2
@JPXKQX
Merge remote-tracking branch 'origin/develop' into 19-feature-clean-n…
79fa4a6 
…odes-after-building-the-graph
@JPXKQX JPXKQX added the enhancement New feature or request label Jul 25, 2024
@JPXKQX JPXKQX added this to the Multi-Scale Grids milestone Jul 25, 2024
@JPXKQX JPXKQX self-assigned this Jul 25, 2024
@JPXKQX JPXKQX linked an issue Jul 25, 2024 that may be closed by this pull request
[feature] Clean nodes after building the graph #19
Closed
theissenhelen
theissenhelen approved these changes Jul 26, 2024
View reviewed changes
Copy link
Collaborator

@theissenhelen theissenhelen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to go.

@JPXKQX JPXKQX merged commit 7fd9c74 into develop Jul 26, 2024
4 checks passed
@JPXKQX JPXKQX deleted the 19-feature-clean-nodes-after-building-the-graph branch July 26, 2024 13:31
theissenhelen added a commit that referenced this pull request Sep 11, 2024
@theissenhelen @JPXKQX
@JesperDramsch @gmertes @iainrussell @Rilwan-Adewoyin @anaprietonem @pre-commit-ci @github-actions
Latest release 0.3.0 (#47)
17c2847 
* Fix (global version) (#12)

* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs + fixes

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* remove dataclass from attribute classes

* fix: config nodes name

* 6 generate graphs from icosahedral meshes (#11)

* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>

* add cli command

* Ignore .pt files

* run pre-commit

* docstring + log erros

* initial tests

* feat: initial version of AttributeBuilder

* refactor: separate into node edge attribute builders

* feat: edge_length moved to edges/attributes.py

* remove __init__

* bugfix (encoder edge lengths) + refector

* feat: support path and dict for `config` argument

* fix: error

* refactor: naming

* fix: pre-commit

* feat: builders icosahedral

* feat: Add icosahedral graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* refactor: remove create_shere

* feat: Icosahedral edge builder

* feat: hexagonal graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* feat: hexagonal builders

* fix: AOI not implemented yet

* fix: abstractmethod and renaming

* chore: add dependencies

* test: add tests for trimesh

* test: add tests for hex (h3)

* fix: imports

* fix: output type

* refactor: delete unused file

* refactor: renaming and positioning

* feat: ensure src and dst always the same

* fix: imports

* fix: edge_name not supported

* test: add tests for TriIcosahedralEdges

* fix: assert missing for Hexagonal edges

* test: hexagonal edges

* fix: avoid same name

* fix: imports

* fix: conflicts

* update tests

* Include xhops to hexagonal edges

* docs: update docstrings

* fix: update attribute name

* refactor: rename multiscale nodes

* refactor: rename icosahedral nodes

* improve: clarity of function

* improve: function syntax

* refactor: simplify resolution assignment

* refactor: improve variable naming in icosahedral graph generation

* more comments

* refactor: naming

* refactor: separate into functions

* refactor: remove unused code

* refactor: remove unused options and rename

* doc: clarify cells and nodes

* fix: add return statements

* homogeneize: tri & hex edges

* naming: xhops to x_hops

* naming: x_hops in docstring

* docstring

* blank lines

* fix: add return statement

* simplify icoshaedral edges

* feat: select edge builder method based on ico node type

* test: adjust tests to new MultiScaleEdges

* docs: improve docstrings

* fix: remove LAM filtering for icosahedral, leave for next PR

* h3 (v4) not supported

---------

* [feature] add changelog (#21)

* ci: changelog check

* feat: add changelog

* fix: on develop

* Skip no-commit-to-branch in code QA (#17)

* Clean nodes after building the graph (#23)

* feat: clean graph of unneeded attributes after creation

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* Make graph filename argument optional (#24)

* hotfix: make graph saving optional

Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>

* doc: update CHANGELOG

Co-authored-by: Jesper Dramsch <[email protected]>

* chore: add rtd pr preview

* HEALPix nodes (#16)

[feature] HEALPix node builder

Co-authored-by: theissenhelen <[email protected]>

* 25 integrate reusable workflows (#26)

* ci: add public pr label

* ci: add downstream-ci workflow

* ci: add ci-config

* fix: pre-commit

* docs: update changelog

* ci: switch main on downstream-ci

* ci: run ci on push on main

* Hotfix/clean graph (#28)

* Add binary dependencies to ci-config.yml (#29)

* Add binary dependencies to ci-config.yml

* docs: update changelog

---------

Co-authored-by: theissenhelen <[email protected]>

* ci: inherit pypi publish flow (#27)

* ci: inherit pypi publish flow

Co-authored-by: Jesper Dramsch <[email protected]>

* docs: update changelog

* ci: only downstream-ci for changes in src and tests

---------

Co-authored-by: Jesper Dramsch <[email protected]>

* Bugfix/31 ensure latlon input numpy array (#32)

* #31 Ensure lat lon values passed as numpy arrays

* fix formatting

* Updated CHANGELOG.md

* Remove bug in the graph cleaning function (#33)

* fix: remove the bug in the graph cleaning function

Co-authored-by: Jesper Dramsch <[email protected]>

* docs: update changelog

---------

Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Gert Mertes <[email protected]>

* Move to reusable workflows for QA and Testing (#34)

* ci: move to reusable workflows for QA and Testing

* docs: changelog add ci reusable workflow

* ci: downstream ci ignore docs

* ci: remove anchor, unsupported by github

* ci: add changelog release updater (#35)

* ci: changelog updater PR (#36)

* ci: correct inputs typo

* ci: fix changelog updater

* docs: ci fixes changelog

* ci: premissions

* ci: typo

* Ci/fix pr permission (#37)

* ci: add pull-requests permissions

* ci: add on workflow dispatch

* ci: remove 3.9 tests (#39)

* ci: remove 3.9 tests

* ci: remove testing of python 3.9

* dics: update changelog

* Fix support for Python 3.9 (#38)

* fix: support py3.9

Co-authored-by: Jesper Dramsch <[email protected]>

* fix: update & homogeneize changelog

* fix: update tests to support py39

* fix: style

* ci: rollback 3.9 tests

* refactor: annotations not used in tests

---------

Co-authored-by: Jesper Dramsch <[email protected]>

* Ci/fix_double_notes (#40)

* ci: avoid duplication of release notes

* ci: try write-all

* ci: fix permissions

* docs: update changelog

* ci: fix indent

* Inspection tools (#22)


Co-authored-by: Ana Prieto Nemesio <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>

* [pre-commit.ci] pre-commit autoupdate (#41)

updates:
- [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](psf/black-pre-commit-mirror@24.4.2...24.8.0)
- [github.com/astral-sh/ruff-pre-commit: v0.4.6 → v0.6.2](astral-sh/ruff-pre-commit@v0.4.6...v0.6.2)
- [github.com/tox-dev/pyproject-fmt: 2.1.3 → 2.2.1](tox-dev/pyproject-fmt@2.1.3...2.2.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#43)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.2 → v0.6.3](astral-sh/ruff-pre-commit@v0.6.2...v0.6.3)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* First version of documentation (#18)

Documentation web page
---------
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* [Changelog] Update to 0.3.0 (#44)

* [create-pull-request] automated change

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: theissenhelen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Gert Mertes <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Iain Russell <[email protected]>
Co-authored-by: Rilwan (Akanni) Adewoyin <[email protected]>
Co-authored-by: Ana Prieto Nemesio <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: theissenhelen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@theissenhelen theissenhelen theissenhelen approved these changes

Assignees

@JPXKQX JPXKQX

Labels
enhancement New feature or request
Projects
None yet
Milestone
Multi-Scale Grids
Development

Successfully merging this pull request may close these issues.

[feature] Clean nodes after building the graph
2 participants
@JPXKQX @theissenhelen