Skip to content

Repeated reading of attributes is harming performance #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davidhassell opened this issue Apr 11, 2025 · 1 comment
Open

Repeated reading of attributes is harming performance #8

davidhassell opened this issue Apr 11, 2025 · 1 comment

Comments

@davidhassell
Copy link
Collaborator

Repeated reading of attributes is harming performance - this is because every time h5netcdf is going back to self._h5ds to get the attributes, and this creates a new Variable every time. So even if a hd55/pyfive variable itself can cache its attributes, this is lost to h5netcdf.

This is also the case for dimensions, which are (often) associated with a variable (i.e. dimension coordinate variable), and so get their dimensions read.

The answer has to be (?) to apply store caching on the objects which are persistent in h5netcdf, such as Dimension and File.

@davidhassell
Copy link
Collaborator Author

I have a hacky branch (https://github.com/NCAS-CMS/h5netcdf/tree/dimension-access) where I'm playing with this

This was referenced Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant