-
Notifications
You must be signed in to change notification settings - Fork 25
Open
Labels
enhancementUpdate to logic or general code improvementsUpdate to logic or general code improvementstopic-resource-handlerIssues/pull requests related to the main rex Resource handlerIssues/pull requests related to the main rex Resource handlerwishlistFeatures we would ideally want to support, but not right nowFeatures we would ideally want to support, but not right now
Description
Why this feature is necessary:
Even with the slicing tricks implemented here: https://github.com/NREL/rex/blob/ae280dbfc149db95135b25b244c585f09d2d7cb4/rex/resource.py#L133-L445
extracting data from h5py and h5pyd using a list is slow.
A possible solution is:
The most performant solution would be to use concurrent futures to extract each item associated with the list in parallel and then recombine the requests. This would greatly simplify the logic noted above. The only issue is that it wouldn't be compatible with the already parallel nature of reV Generation as you can't run parallel workers on parellel workers.
Additional Context
Relevant for: NatLabRockies/reV#335
Charge code
reV
Urgency / Timeframe
TBD
Metadata
Metadata
Assignees
Labels
enhancementUpdate to logic or general code improvementsUpdate to logic or general code improvementstopic-resource-handlerIssues/pull requests related to the main rex Resource handlerIssues/pull requests related to the main rex Resource handlerwishlistFeatures we would ideally want to support, but not right nowFeatures we would ideally want to support, but not right now