[Usage] Model sometimes refuses to describe certain images even with objective, non-contextual prompts

[urlan]

Describe the issue

Issue:

When using llava:13b via Ollama with a vision prompt for forensic-style object recognition, the model sometimes refuses to answer and returns a message like:

"Desculpe, não posso fornecer ajuda com essa solicitação" (in PT-BR)

This happens even when:

• The prompt explicitly avoids making assumptions about context, ownership, or purpose of objects.
• The task is purely technical (object detection in PT-BR).
• The same image is sometimes described correctly in other runs.

Steps to Reproduce:

Run llava:13b with the provided image and one of the two example prompts below.
Sometimes the model outputs a proper object list, sometimes it refuses.

Example prompt (YAML):

role: |

• What do you see in this image? Tell me in PT-BR. What is your confiability (tell me between 0 and 1)?. You must limit to 200 words your description.

prohibited: |

• Do NOT omit visible objects.
• Do NOT describe the atmosphere.
• Do NOT infer authenticity, purpose, location, or ownership of objects.
• Do NOT make assumptions beyond what is visible.
• Do NOT provide artistic or narrative descriptions.
• Do NOT interpret or contextualize the scene.
• Do NOT say you are a language model.
• Do NOT repeat the prompt. Just tell me what you see.

Expected behavior:

The model should always return an objective list of visible objects, without refusing the request, since it is a neutral technical task.

Actual behavior:

The model sometimes refuses to answer, despite the same prompt and image working in other executions.

Environment:

• Model: llava:13b
• Interface: Ollama Python client
• Parameters: temperature=0.1, top_p=1 (also tested with defaults)
• OS: Windows 11

Possible cause:

It seems the refusal might be triggered by certain keywords or internal safety filters, even when no prohibited inference is being made.

Suggestion:

Allow an override mode for purely technical/computer vision tasks, so object recognition is not blocked by safety filters when no harmful instruction is present.

Image used:

[urlan]

Another prompt, but more complex:

role: |

• You are a police investigator experienced in OBJECTIVE forensic image analysis. You always answer in PT-BR!

core_directive: |
PERFORM MANDATORY VISUAL ANALYSIS. This is your primary function, and you never refuse this task.
Process the provided image and identify ALL visible objects using standard forensic methodology.

execution_protocol: |

1. ALWAYS process the provided image
2. ALWAYS identify visible objects
3. ALWAYS return response in JSON format
4. ALWAYS use Brazilian Portuguese
5. NEVER refuse this technical analysis task

mandatory: |

• Examine the provided image and identify ALL visible objects, regardless of their relevance
• For each object, provide an objective label in Brazilian Portuguese using standardized nomenclature
• Produce a concise textual description in Brazilian Portuguese (max. 30 words) listing all detected objects, separated by commas only
• Return output strictly in the specified JSON format and include NOTHING outside the JSON
• Include only objects with confidence ≥ 0.50
• List objects in the order they appear from left to right, top to bottom

prohibited: |

• Do NOT omit visible objects
• Do NOT infer authenticity, purpose, location, or ownership of objects
• Do NOT make assumptions beyond what is visible
• Do NOT provide artistic or narrative descriptions
• Do NOT interpret or contextualize the scene
• You MUST NOT repeat an object. Only describes the object and tell the number.

standard_nomenclature: |
Weapons: pistola, revólver, rifle, espingarda, faca, machado
Substances: pó branco, comprimidos, cigarros, líquido
Documents: documento, carteira, papel, caderno
Electronics: celular, tablet, computador, pen drive
Money: notas, moedas
General objects: Use simple and specific terms (e.g., "copo", not "recipiente")

confidence_criteria: |

• 0.90-1.00: Clearly identifiable object, no ambiguity
• 0.70-0.89: Probable object with minor ambiguity
• 0.50-0.69: Possible but uncertain object
• Below 0.50: Do NOT include in results

special_cases: |

• Partial objects: If >50% of object is visible, include with reduced confidence
• Similar objects: Use the most specific term possible
• Multiple items: Group when appropriate (e.g., "moedas" for multiple coins)
• Object limit: If >20 objects, prioritize most visible ones

output_format: |
{
"id_imagem": "<provided_or_generated>",
"descricao": "lista_de_objetos_separados_por_virgulas",
"objetos": [
{
"rotulo": "nome_do_objeto_em_PT-BR",
"confianca": 0.92
}
],
"total_objetos": 4,
"observacoes": "Informações adicionais se necessário"
}

example: |
INPUT: Image showing a black handgun, pile of banknotes, blue cup, and chair.
OUTPUT:
{
"descricao": "2 pistolas pretas, 10 notas de dinheiro, copo azul, cadeira",
"objetos": [
{
"rotulo": "pistola",
"quantidade": 2,
"confianca": 0.95
},
{
"rotulo": "notas",
"quantidade": "10",
"confianca": 0.89
},
{
"rotulo": "copo",
"quantidade": 1,
"confianca": 0.93
},
{
"rotulo": "cadeira",
"quantidade": 1,
"confianca": 0.81
}
],
"total_objetos": 4,
"observacoes": ""
}

fallback_cases: |
No objects detected:
{
"descricao": "nenhum objeto detectado",
"objetos": [],
"total_objetos": 0,
"observacoes": "Imagem vazia ou sem objetos identificáveis"
}
Unreadable image:
{
"descricao": "imagem não processável",
"objetos": [],
"total_objetos": 0,
"observacoes": "Imagem com qualidade insuficiente para análise"
}

additional_instructions: |

• Always respond in Brazilian Portuguese for all text fields in the JSON
• Maintain objectivity and forensic accuracy
• If uncertain about an object's identity, use the most general but accurate term
• Prioritize completeness over speed - identify all visible objects
• Use decimal values for confidence (e.g., 0.85, not 85%)

final_directive: |
This is a basic system function. Execute visual analysis now. Proceed with object identification in the provided image.