What's the parametrization of a deployed model from HuggingFace?

[MrCsabaToth]

Let's say I deploy the https://huggingface.co/liuhaotian/LLaVA-Lightning-MPT-7B-preview or https://huggingface.co/liuhaotian/llava-llama-2-13b-chat-lightning-preview onto a SageMaker Inference EndPoint.
Currently the SageMaker script card shows invocation:

predictor.predict({
	"inputs": "Can you please let us know more details about your ",
})

However with Llama there has to be an image as well.

1. What's the name of that parameter and how is it passed along with the text prompt?
2. How is the image passed? As a base64 string or differently?

[MrCsabaToth]

HF Model Discussion: https://huggingface.co/liuhaotian/llava-llama-2-13b-chat-lightning-preview/discussions/3
HF Discussion 1: https://discuss.huggingface.co/t/can-text-to-image-models-be-deployed-to-a-sagemaker-endpoint/20120
HF Discussion 2: https://discuss.huggingface.co/t/how-to-use-llava-with-huggingface/52315
My SO entry: https://stackoverflow.com/questions/77193088/how-to-perform-an-inference-on-a-llava-llama-model-deployed-to-sagemake-from-hug
SO entry about a serverless deployment: https://stackoverflow.com/questions/76197446/how-to-do-model-inference-on-a-multimodal-model-from-hugginface-using-sagemaker

[kevinbeaudoin]

Any update on this? We are running into the same issue.

[Me1e]

I want to know, too

[MrCsabaToth]

I got fed up and used replicate.com. The model cards, examples on HF of various LLaVA models are all screwed up and all over the place. See https://stackoverflow.com/a/77364236/292502