Skip to content

Finetuning text dataset with function calls fails for gemini 2.5 (2.0 works) #6127

@danielLublinsky

Description

@danielLublinsky

Hey again,

I am tuning a model with Vertex supervised text tuning. The model has an SQL tool that
will be used in the tuning.

My dataset has a system prompt and tools at the first lines, just like the docs say…

But the issue I am having is with the tuning process for models above Gemini 2.0 flash.
So my dataset works for Gemini 2.0 flash / 2.0 flash lite,
But for 2.5 flash and up, I get this error (same dataset):

Dataset example 1 of 14 contains an unsupported modality [function call] in the “parts” field, which is not supported by this model. The supported modalities are: [image, PDF document, audio, video]. JSON: {“systemInstruction”:{“role”:“system”,“parts”:[{“text”:"You are an assistant who helps the user break down large business/company financial and Cost Accounting data.\nAlways think before answering and then run every request end-to-end using:\n\n1. Read the OBS List from the cached context file.\n2. …

This is a breakdown of the JSONL contents line format I use:

{
  "contents": [
    {
      "role": "user",
      "parts": [
        { "text": "why does my tuning fail???" }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "run_sql",
            "args": { "sql": "SELECT...." }
          }
        }
      ]
    },
    {
      "role": "function",
      "parts": [
        {
          "functionResponse": {
            "name": "run_sql",
            "response": {"blabla"}
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        { "text": "bla bla" }
      ]
    }
  ]
}

and from the docs models 2.5 do support text tuning…
What am I missing?

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: vertex-aiIssues related to the googleapis/python-aiplatform API.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions