Skip to content

Fix the error when running model evaluation on EC2 #40

@Yiwen-Zhang

Description

@Yiwen-Zhang

Issue:
The 4_get_evaluations.ipynb step triggers an error when the experiment uses EC2 instances as the model serving stack.

Details:
The error shows that 'verdict' column was not created.

Step to reproduce:
Create a fmbench-orchestrator config file specifying custom fmbench_config. In the fmbench_config file, specify run_steps: 4_get_evaluations.ipynb: yes and model_evaluations: model_eval_all_info.yml

Possible solution:
fmbench is trying to write model evaluation results into a temp folder, e.g.:
'/tmp/fmbench-write/modelname/data/metrics/yyyy=/mm=/dd=/hh=/mm=/per_poll_eval/http:/127.0.0.1:8080/invocations_1736315403.2303681_According_to_the_provided_context_.json'

The part 'http:/127.0.0.1:8080/invocations_1736315403.2303681_According_to_the_provided_context_.json' was the file name generated in previous steps, but linux reads each '/' as a sub folder. This file name needs to to be re-formated, removing '/' before saving.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions