Vicuna v1.5 released with 4K and 16K context lengths

[ianscrivener]

"Vicuna v1.5 series, featuring 4K and 16K context lengths with improved performance on almost all benchmarks.. based on the commercial-friendly Llama2"

Excited to see llama.cpp support soon?!

[klosax]

The 16k model should already work using the parameter --rope-freq-scale 4.0

[ianscrivener]

hmm.. no luck for me on Mac with Metal.
Just starts outputting unpromtped... mixed languages and a lots of new lines

./main -t 4 -ngl 1 --model vicuna-7b-v1.5-16k.ggmlv3.q4_0.bin --ctx-size 2048 --rope-freq-scale 4.0 -i

./main -t 4 -ngl 1 --model vicuna-7b-v1.5-16k.ggmlv3.q4_0.bin --ctx-size 16384 --rope-freq-scale 4.0 -i

[ianscrivener]

am I missing something?

[klosax]

I got it wrong. The --rope-freq-scale should be the inverse of the context scaling factor.

In PR ggml-org/llama.cpp#2544 I added a context scaling parameter --rope-scale which makes more sense:
--rope-scale 4.0