[CI][TorchModels] Update flags for CLIP test.#22413

MaheshRavishankar wants to merge 2 commits intoiree-org:mainiree-org/iree:mainfrom

MaheshRavishankar:users/MaheshRavishankar/updateClipTestMaheshRavishankar/iree:users/MaheshRavishankar/updateClipTestCopy head branch name to clipboard