Fix Llava tests for torch too!#44476

Rocketknight1 merged 3 commits intomainhuggingface/transformers:mainfrom

fix_llava_test_round_2huggingface/transformers:fix_llava_test_round_2Copy head branch name to clipboard