[BUG] When using accelerator, current model_args parser won't correctly parse dict-like arguments like generation_parameters

[Ignoramus0817]

Describe the bug

Current parser split the arguments with ",", which will cut the dict-like arguments into pieces and causing TypeError.
For example, here is the model_args:

MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"

When I print the parsed arguments, I got something like this:

{
    ...other args...

    "generation_parameters": "{do_sample:True",
    "max_new_tokens:32768": True,
    "temperature:0.6": True,
    "top_p:0.95}": True
}

Clearly the later parts of the generation parameters are parsed as flags.

To Reproduce

NUM_GPUS=1
MODEL=/path/to/model
OUTPUT_DIR=/path/to/output
MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"

lighteval accelerate $MODEL_ARGS "any_task" \
    --use-chat-template \
    --override-batch-size 1 \
    --output-dir $OUTPUT_DIR \
    --save-details

Expected behavior

The parsed args should look like this:

{
    ...other args...

    "generation_parameters":  {
        "do_sample": True,
        "max_new_tokens": 32768,
        "temperature": 0.6,
        "top_p": 0.95
    }
}

Version info

lighteval==0.8.0

[ashim-mahara]

+1

[NathanHB]

Hi ! Thanks for the issue, this should be fixed now :)
Don't hesitate to reopen if you see anything other issue

[NathanHB]

small nit in your command, generation_parameters={do_sample:True,max_new_tokens:32768,temperature:0.6,top_p:0.95}"

True should be true because it's parsed as json