-
Notifications
You must be signed in to change notification settings - Fork 7
Description
Issue:
The 4_get_evaluations.ipynb step triggers an error when the experiment uses EC2 instances as the model serving stack.
Details:
The error shows that 'verdict' column was not created.
Step to reproduce:
Create a fmbench-orchestrator config file specifying custom fmbench_config. In the fmbench_config file, specify run_steps: 4_get_evaluations.ipynb: yes
and model_evaluations: model_eval_all_info.yml
Possible solution:
fmbench is trying to write model evaluation results into a temp folder, e.g.:
'/tmp/fmbench-write/modelname/data/metrics/yyyy=/mm=/dd=/hh=/mm=/per_poll_eval/http:/127.0.0.1:8080/invocations_1736315403.2303681_According_to_the_provided_context_.json'
The part 'http:/127.0.0.1:8080/invocations_1736315403.2303681_According_to_the_provided_context_.json' was the file name generated in previous steps, but linux reads each '/' as a sub folder. This file name needs to to be re-formated, removing '/' before saving.