Skip to content
This repository was archived by the owner on Sep 1, 2022. It is now read-only.
This repository was archived by the owner on Sep 1, 2022. It is now read-only.

Performance Issue with high variable and dimension count #1295

@GTOL

Description

@GTOL

I try to use CDF format to store my data so that each piece of data is using one dimension and two variables. I noticed that the loading speed of the CDF file is increasing significantly with the dimension and variable count.
I also run a simple test on this issue. The dimension length is randomized from 1 to 10000 for each data, and the data size is from 1000 to 10000 with a step of 1000 (for example, a data size of 1000 will give dimension count of 1000 and variable count of 2000).
Here is the result:
Size: 1000 Time: 48 ms
Size: 2000 Time: 133 ms
Size: 3000 Time: 150 ms
Size: 4000 Time: 404 ms
Size: 5000 Time: 622 ms
Size: 6000 Time: 1426 ms
Size: 7000 Time: 2374 ms
Size: 8000 Time: 2773 ms
Size: 9000 Time: 4044 ms
Size: 10000 Time: 5159 ms
Maybe you can find a way to fix this? Thank you.

The file format is netcdf3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions