Skip to content

Conversation

@Erotemic
Copy link
Contributor

@Erotemic Erotemic commented Nov 7, 2025

This partially addresses the issue I had in #3915. The main missing piece is documentation updates that shows the user how they can define a new run spec independent of modifying HELM source.

The main idea is add an argument --plugin which can be a list of module names or paths to python modules. The new behavior is that these specified modules are imported at the start of the run_benchmarking function, so custom run specs will be found.

Importing modules based on their importable python name is fairly straight forward, and sometimes useful, however, I would almost always prefer to specify a path to a plugin file so it is explicit what custom logic I'm using. Unfortunately the stdlib doesn't have a mechanism to import a module from its path, but I've been maintaining one in my ubelt library for years, and I simply ported the logic of that into helm/common/import_utils.py. (The way it works is by temporarily adding the module's directory to the PYTHONPATH and then using normal import mechanisms).

If maintainers like the basic idea then I'll go ahead and update the documentation and tests in this PR as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant