Make tests run in less time by reducing batch_size#42213

ydshieh merged 15 commits intomainhuggingface/transformers:mainfrom

fix_tests_being_slow_002huggingface/transformers:fix_tests_being_slow_002Copy head branch name to clipboard