what user_query can i use?

[xiaoyazhu]

If i want to locate a specific target ,such as "a person wearing a yellow hat", what user_query can i use in inference?

python -m groma.eval.run_groma \
    --model-name {path_to_groma_7b_finetune} \
    --image-file {path_to_img} \
    --query {user_query} \
    --quant_type 'none' # support ['none', 'fp16', '8bit', '4bit'] for inference

[machuofan]

You can simply use Locate <p> a person wearing a yellow hat </p> in the image. or something else like that. Just remember to enclose the referring expression with <p> and </p>.

[xiaoyazhu]

But when i tested several images, i found that whether there is an object in the test image, the model will output a localization result. Is there any way to adjust the settings , or is it caused by model hallucination ?

[machuofan]

Yes, such hallucination is probably caused by training data - for grounding training, we only got positive QA pairs, i.e., the object mentioned in the question is guaranteed to occur in the image. To remedy such hallucination, you can curate some negative QA pairs and finetune the model.

[Eman-Abdelrahman]

If I need to fine-tune the model for locating target elements in an image, is there certain hyper parameters to focus on, please?

[LLH-Harward]

If I want to provide the object's label and bounding box to Groma to help generate more accurate image descriptions, how should I structure the prompt?