FIX: Generation nightly CI failing due to gemma (huggingface#2580)

For a month now, nightly CI has failed with dozens of tests causing this
error:

> RuntimeError: Offset increment outside graph capture encountered
unexpectedly.

(link: https://github.com/huggingface/peft/actions/runs/14850392078/job/41692748031)

It turns out that huggingface#2458, which
added a gemma model to the test suite, is most likely the culprit. Since
that commit, on nightly CI (with GPU), when transformers generates with
gemma, which uses torch.compile, an error can be triggered. For some
reason, this has a side effect on other tests that then results in the
error quoted above.

As is, there is no solution for the gemma issue. To still allow the
tests to run and help discover potential issues, this PR skips the
corresponding gemma tests, which should allow the other tests to pass
again.

I could confirm locally that these tests only fail when the gemma tests
are run in the same session. Hopefully, this generalizes to the CI
environment.

---------

Co-authored-by: githubnemo <[email protected]>

Author: 2 people

Makefile

@@ -31,8 +31,12 @@ tests_core_multi_gpu:
 tests_core_single_gpu:
 	python -m pytest -m single_gpu_tests tests/test_common_gpu.py $(if $(IS_GITHUB_CI),--report-log "core_single_gpu.log",)
 
+# exclude gemma tests, as generation fails with torch.compile, these failures
+# trigger side effects that make other tests fail with 'RuntimeError: Offset
+# increment outside graph capture encountered unexpectedly.' 
+# TODO re-enable gemma once/if it is fixed
 tests_common_gpu:
-	python -m pytest tests/test_decoder_models.py $(if $(IS_GITHUB_CI),--report-log "common_decoder.log",)
+	python -m pytest tests/test_decoder_models.py -k "not gemma" $(if $(IS_GITHUB_CI),--report-log "common_decoder.log",)
 	python -m pytest tests/test_encoder_decoder_models.py $(if $(IS_GITHUB_CI),--report-log "common_encoder_decoder.log",)
 	python -m pytest tests/test_gptqmodel.py $(if $(IS_GITHUB_CI),--report-log "gptqmodel_gpu.log",)