Make PegasusXModelTest run in less time

[ydshieh]

What does this PR do?

Saving 40 seconds for

python -m pytest -v -m 'not generate' tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest --make-reports=test_dummy

The most significant one is

test_model_outputs_equivalence

20 sec --> 4 sec.

Have no clear idea why this model use more time, but it's encoder-decoder, each of 2 layers ... and this test test_model_outputs_equivalence have many test cases, and being even slower if we have output_attentions=True.

Let's simply use smaller batch_size for this model

Before

slowest durations
20.34s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_outputs_equivalence
3.74s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant
3.71s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant_false
3.71s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing
3.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_tokens_embeddings
2.75s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_attention_outputs
2.40s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_bc_torch_dtype
2.01s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_feed_forward_chunking
2.01s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_determinism
1.74s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load
1.66s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_generate_fp16
1.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_hidden_states_output
1.45s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_retain_grad_hidden_states_attentions
1.44s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training
1.08s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_encoder_decoder_model_standalone
1.07s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_batching_equivalence
0.79s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds
0.79s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds_matches_input_ids
0.78s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_embeddings_untied
0.24s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_use_safetensors
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_load_save_without_tied_weights
0.18s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_from_pretrained_no_checkpoint
0.14s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load_strict
0.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_be_initialized_on_meta
0.07s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_weights_reload_no_missing_tied_weights
509 durations < 0.05 secs were omitted

After

slowest durations
3.96s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_outputs_equivalence
2.46s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_bc_torch_dtype
1.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_tokens_embeddings
0.83s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing
0.81s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant
0.78s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant_false
0.61s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_generate_fp16
0.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load
0.50s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_attention_outputs
0.47s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_determinism
0.34s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_feed_forward_chunking
0.33s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_retain_grad_hidden_states_attentions
0.33s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_batching_equivalence
0.32s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training
0.27s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_encoder_decoder_model_standalone
0.22s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_load_save_without_tied_weights
0.22s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_use_safetensors
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_hidden_states_output
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load_strict
0.17s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_embeddings_untied
0.17s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_from_pretrained_no_checkpoint
0.11s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds
0.11s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_weights_reload_no_missing_tied_weights
0.10s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds_matches_input_ids
0.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_be_initialized_on_meta
0.06s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_torch_save_load
508 durations < 0.05 secs were omitted

[vasqu]

Why was this so high in the first place 😭 take my approval

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

This test test_model_outputs_equivalence appears 447 times in the report of long run tests.

It test many cases (which is not a good approach for testing), and output_attentions=True case is even much slower
(large tensor, I guess).

For this model , it's encoder-decoder, each of 2 layers. (But not sure if this play a significant role for slowness here.)

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: beit, big_bird, bigbird_pegasus, deformable_detr, pegasus_x, x_clip