Possible broken logic in llama_agent/function_calling_agent.py

[ebarrragn]

Hi,
First, I have to say a big thank you for your work!.

Second, the issue:
I think there is a problem in the logic dealing with the generation settings in the __init__ of the class FunctionCallingAgent.
In this snippet of code:

settings = LlamaLLMSettings(
        # model_path="/mnt/WorkStuff/AI/Models/yarn-mistral-7b-64k.Q5_K_M.gguf",
        model_path="/mnt/WorkStuff/AI/Models/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
        # model_path="/mnt/WorkStuff/AI/Models/TheBloke/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
        # model_path="/mnt/WorkStuff/AI/Models/Nous-Hermes-2-Mixtral-8x7B-DPO.Q2_K.gguf",
        # model_path="/mnt/WorkStuff/AI/Models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf",
        # model_path="/mnt/WorkStuff/AI/Models/wizardcoder-python-13b-v1.0.Q4_K_S.gguf",
        n_gpu_layers=25,
        f16_kv=False,
        offload_kqv=True,
        use_mlock=False,
        embedding=False,
        n_threads=8,
        n_batch=1024,
        n_ctx=8192,
        last_n_tokens_size=1024,
        verbose=True,
        seed=42,
    )

    # settings.save("deepseek-coder-6.7b-instruct.Q4_K_S.gguf")
    from llama_cpp import Llama
    main_model = Llama(
        **settings.as_dict()
    )

function_call_agent = FunctionCallingAgent(
    # Can be lama-cpp-python Llama class, llama_cpp_agent.llm_settings.LlamaLLMSettings class or llama_cpp_agent.providers.llama_cpp_server_provider.LlamaCppServerLLMSettings.
    main_model,
    # llama_cpp_agent.llm_settings.LlamaLLMGenerationSettings  class or llama_cpp_agent.providers.llama_cpp_server_provider.LlamaCppServerGenerationSettings.
    llama_generation_settings=generation_settings,
    # A tuple of the OpenAI style function definitions and the actual functions
    open_ai_functions=(tools, tool_functions),
    # Just a list of type hinted functions for normal Python functions
    python_functions=[get_current_datetime],
    # Just a list of pydantic types
    pydantic_functions=[calculator],
    # Callback for receiving messages for the user.
    send_message_to_user_callback=send_message_to_user_callback,
    # Set to true to allow parallel function calling
    allow_parallel_function_calling=False,
    messages_formatter_type=MessagesFormatterType.CHATML,
    debug_output=True)

It doesn't matter if I provide the generation settings as:

generation_settings = LlamaLLMGenerationSettings(max_tokens=4096,
                                                    temperature=0.4, top_k=0, top_p=1.0, repeat_penalty=1.1,
                                                    min_p=0.1, tfs_z=0.95, stream=True, print_output=True )

or

generation_settings = LlamaCppGenerationSettings(n_predict=4096,
                                                 temperature=0.4, top_k=0, top_p=1.0, repeat_penalty=1.1,
                                                 repeat_last_n=512,
                                                 min_p=0.1, tfs_z=0.95, penalize_nl=False, stream=True)

It always generates an exception, even if the llama_generation_settings parameter is None.

I brute forced it commenting the lines 213 to 229 in function_calling_agent.py, so now it works passing a LlamaCppGenerationSettings (I am not sure if this class is the intended one)

Thanks.

[Maximilian-Winter]

I finally fixed this in the latest release.

[ebarrragn]

Gosh, that was quick.