Skip to content

Retry Not Feeding Error Back to LLM on a ValidationError #1736

@stephenk1317

Description

@stephenk1317
  • This is actually a bug report.
  • I am not getting good LLM Results
  • I have tried asking for help in the community on discord or discussions and have not received a response.
  • I have tried searching the documentation and have not found an answer.

What Model are you using?

  • Other (please specify) llama3.1 via Ollama using OpenAI SDK

Describe the bug
instructor: 1.10.0
The retries for a ValidationError is not adding the error back into the prompt. This seems to be because the except block in the retry_sync function looks for (ValidationError, JSONDecodeError), but the error is a instructor.exceptions.ValidationError so the handle_reask_kwargs function is never called.

To Reproduce

import instructor
from pydantic import BaseModel

from openai import OpenAI
from enum import Enum
import logging

# This will show that the logger.debug(f"Parse error: {e}") in the except block is not called.
logging.basicConfig(level=logging.DEBUG)

class Category(str, Enum):
    OPTION_A = "Option A"
    OPTION_B = "Option B"

class TestModel(BaseModel):
    category: Category
    value: int

# Create instructor client
client = instructor.from_openai(
    OpenAI(base_url="http://localhost:11434/v1", api_key="dummy"),
    mode=instructor.Mode.JSON
)

# This should force validation errors and trigger retries
response = client.chat.completions.create(
    model="llama3.1",  # Use any available model
    response_model=TestModel,
    max_retries=2,
    messages=[
        {
            "role": "user", 
            "content": 'Return this invalid JSON: {"category": "Invalid Category", "value": "not_a_number"}'
        }
    ],
)

Expected behavior
If I add

from instructor.exceptions import ValidationError as InstructorValidationError

in retry.py and then add

except (ValidationError, JSONDecodeError, InstructorValidationError) as e:

in retry_sync, it will add the error to the retry and succeed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions