Mark test_prompt_lookup_decoding as flaky

[Rocketknight1]

The CI seems a bit unstable and this test is the most common culprit! cc @ydshieh

[Rocketknight1]

cc @ydshieh

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Hi, I think there have been some efforts putting to make it less flaky. Where did you see this being fail, could you share a job run link?

[ydshieh]

Hi @Rocketknight1 I am running inside CircleCI runners. Run each test 20 times and all pass

root@4c11ad497105:~/project# python -m pytest -v --flake-finder --flake-runs=20 tests/models/ -k "test_prompt_lookup_decoding_matches_greedy_search"

============================================= 2565 passed, 600 skipped, 1484595 deselected, 5 warnings in 245.93s (0:04:05) ==============================================

I am not pro for putting @is_flaky to a test defined in GenerationTesterMixin, (putting it in a specific model test file is fine), especially there was already some work done before.

I don't mean non of these tests are flaky, it may happens if I specify --flake-runs=1000 for example. But it's time consuming if I do so for all these tests test_prompt_lookup_decoding_matches_greedy_search.

If you see in the future which specific model having its test_prompt_lookup_decoding_matches_greedy_search flaky, please ping me (or you can dive into it if you are motivated).

For this PR, I think we can close, or if you have any other idea to propose.

[Rocketknight1]

That's interesting, it was blocking me yesterday here: https://app.circleci.com/pipelines/github/huggingface/transformers/153284/workflows/feadf2bc-dc65-42f1-a693-8001f074db44/jobs/2015365

[ydshieh]

Great, I will check this one!

BTW, if you need to merge the mentioned PR, don't hesitate to ping me or core maintainers 🙏 no need to be blocked