Models: Add Gemma-2-9b-it-GGUF

[ThiloteE]

Describe your changes

Adds model support for Gemma-2-9b-it

Description of Model

At the date of writing, the model has strong results in benchmarks (for its parameter size). It claims to support a context of up to 8k.

• The model was apparently trained and finetuned on mostly English datasets
• License: Gemma

Personal Impression:

For 9 billion parameters, the model has reasonable output. I tested the model with a 14k character conversation and there were no tokenizer issues and no severe repetition problems as far as I could discern. I have seen refusals when it was tasked with certain things and it seems to be finetuned with a particular alignment. Its quality of responses makes it a good model, if you can bear its alignment or your use case happens to fall within the originally intended use cases of the model. It mainly will appeal to English speaking users.

Clayton reported, the model has a tendency to keep asking questions, even if instructed not to.

Critique:

• The license is very restrictive.
• Its context window of 8192 is a little short compared to other state of the art models with roughly similar architecture and within its parameter size range.
• only works on CPU and Cuda backend.

Motivation for this pull-request

• Other quants uploaded to huggingface and that are accessible via the search feature of GPT4All have tokenizer eos issues.
• To date, the model is rumoured to be one of the better models out there.
• For it's size it is high on the huggingface open leaderboard benchmark
• Made by Google, the model has a certain reputation

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• I have added thorough documentation for my code.
• I have tagged PR with relevant project labels. I acknowledge that a PR without labels may be dismissed.
• If this PR addresses a bug, I have provided both a screenshot/video of the original bug and the working solution.

[ThiloteE]

I am a little unsure, if the \n at the end of the chat template is really necessary or not. It is in the tokenizer_config.json, so it should be in there by default though. If anybody wants to do extensive tests, go ahead, but my 14k characters test was done without a new line and it still worked.

Ready for review.

[ThiloteE]

[ThiloteE]

This model is not supported on the Nomic Vulkan backend.

[llorensr]

First of all, thank you for the time and effort you've put into this Pull Request. Your work is greatly appreciated! 🙌

I was wondering if there’s an expected timeline for merging this PR? If there’s anything I can do to help move it forward, please let me know.

Thanks again for your support and for maintaining this awesome project!

[manyoso]

We need configurable stop sequences first