Skip to content

parallel gets for list slices #111

@MRossol

Description

@MRossol

Why this feature is necessary:
Even with the slicing tricks implemented here: https://github.com/NREL/rex/blob/ae280dbfc149db95135b25b244c585f09d2d7cb4/rex/resource.py#L133-L445
extracting data from h5py and h5pyd using a list is slow.

A possible solution is:
The most performant solution would be to use concurrent futures to extract each item associated with the list in parallel and then recombine the requests. This would greatly simplify the logic noted above. The only issue is that it wouldn't be compatible with the already parallel nature of reV Generation as you can't run parallel workers on parellel workers.

Additional Context
Relevant for: NatLabRockies/reV#335

Charge code
reV

Urgency / Timeframe
TBD

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementUpdate to logic or general code improvementstopic-resource-handlerIssues/pull requests related to the main rex Resource handlerwishlistFeatures we would ideally want to support, but not right now

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions