-
-
Notifications
You must be signed in to change notification settings - Fork 827
Open
Labels
bugSomething isn't workingSomething isn't working
Description
- This is actually a bug report.
- I am not getting good LLM Results
- I have tried asking for help in the community on discord or discussions and have not received a response.
- I have tried searching the documentation and have not found an answer.
What Model are you using?
- Other (please specify) llama3.1 via Ollama using OpenAI SDK
Describe the bug
instructor: 1.10.0
The retries for a ValidationError
is not adding the error back into the prompt. This seems to be because the except
block in the retry_sync
function looks for (ValidationError, JSONDecodeError)
, but the error is a instructor.exceptions.ValidationError
so the handle_reask_kwargs
function is never called.
To Reproduce
import instructor
from pydantic import BaseModel
from openai import OpenAI
from enum import Enum
import logging
# This will show that the logger.debug(f"Parse error: {e}") in the except block is not called.
logging.basicConfig(level=logging.DEBUG)
class Category(str, Enum):
OPTION_A = "Option A"
OPTION_B = "Option B"
class TestModel(BaseModel):
category: Category
value: int
# Create instructor client
client = instructor.from_openai(
OpenAI(base_url="http://localhost:11434/v1", api_key="dummy"),
mode=instructor.Mode.JSON
)
# This should force validation errors and trigger retries
response = client.chat.completions.create(
model="llama3.1", # Use any available model
response_model=TestModel,
max_retries=2,
messages=[
{
"role": "user",
"content": 'Return this invalid JSON: {"category": "Invalid Category", "value": "not_a_number"}'
}
],
)
Expected behavior
If I add
from instructor.exceptions import ValidationError as InstructorValidationError
in retry.py
and then add
except (ValidationError, JSONDecodeError, InstructorValidationError) as e:
in retry_sync
, it will add the error to the retry and succeed.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working