Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This partially addresses the issue I had in #3915. The main missing piece is documentation updates that shows the user how they can define a new run spec independent of modifying HELM source.
The main idea is add an argument
--pluginwhich can be a list of module names or paths to python modules. The new behavior is that these specified modules are imported at the start of therun_benchmarkingfunction, so custom run specs will be found.Importing modules based on their importable python name is fairly straight forward, and sometimes useful, however, I would almost always prefer to specify a path to a plugin file so it is explicit what custom logic I'm using. Unfortunately the stdlib doesn't have a mechanism to import a module from its path, but I've been maintaining one in my ubelt library for years, and I simply ported the logic of that into
helm/common/import_utils.py. (The way it works is by temporarily adding the module's directory to the PYTHONPATH and then using normal import mechanisms).If maintainers like the basic idea then I'll go ahead and update the documentation and tests in this PR as well.