Running LLaVA Without Images:

[chancharikmitra]

Hello! I am curious to run LLaVA without images. I know this is certainly possible and was about to create my own mask for model.generate, but given that LLaVA handles inputs in its own way, I thought I would check if there was a more integrated solution. So for example, in the the following line from model_vqa_loader.py, I would like to generate a response without the image:

output_ids = model.generate(
                input_ids,
                images=image_tensor.to(dtype=torch.float16, device='cuda', non_blocking=True),
                do_sample=True if args.temperature > 0 else False,
                temperature=args.temperature,
                top_p=args.top_p,
                num_beams=args.num_beams,
                max_new_tokens=args.max_new_tokens,
                use_cache=True)

It seems based on the architecture in llava_arch.py: simply passing None for the image is enough. Just wanted to check in case anyone has tried this themselves:

def prepare_inputs_labels_for_multimodal(
        self, input_ids, position_ids, attention_mask, past_key_values, labels, images
    ):
        vision_tower = self.get_vision_tower()
        if vision_tower is None or images is None or input_ids.shape[1] == 1:
            if past_key_values is not None and vision_tower is not None and images is not None and input_ids.shape[1] == 1:
                target_shape = past_key_values[-1][-1].shape[-2] + 1
                attention_mask = torch.cat((attention_mask, torch.ones(
                    (attention_mask.shape[0], target_shape - attention_mask.shape[1]),
                    dtype=attention_mask.dtype,
                    device=attention_mask.device
                )), dim=1)
                position_ids = torch.sum(attention_mask, dim=1).unsqueeze(-1) - 1
            return input_ids, position_ids, attention_mask, past_key_values, None, labels

[chancharikmitra]

It seems making images=None yields the following CUDA error:

../aten/src/ATen/native/cuda/Indexing.cu:1146: indexSelectLargeIndex: block: [28
6,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.