Introducing my 'VowelReconstruct' Method - A Tangible Test for Comparing LLM's General Intelligence

[mounta11n]

TL;DR I have created a test method I call it "VowelReconstruct", where texts with almost all vowels were removed are presented to language models, and its job is to reconstruct the original text. I am very excited to introduce it to you. My approach is interesting because the model needs various cognitive capabilities at same time to be able to achieve this task. The result is evaluated by comparing the reconstructed text to the original text using two metrics, the Levenshtein-distance and a simple characters similarity score. After that I calculate a new Score (YASimScore), which provide insights into the performance of different language models and helping assess their intelligence.. This method aims to provide a practical way of assessing and comparing language models intelligence.

I've also decided to start my own blog and there you can read more about the method, if you are interested:

https://publish.obsidian.md/mountaiin/VowelReconstruct

---

Here you'll find the files, if you want to use this method too:

https://github.com/mounta11n/VowelReconstruct

---

And here you can see how some of my results look like:

Name	Size	Specifications	Similarity	Levenshtein	YASimScore
Guanaco	7B	/	39.81%	151	379.29
WizardLM	7B	q40	42.36%	194	457.90
--------	---	------	------	---	---
Vicuna	13B	q41_v3	44.90%	109	242.76
Vicuna	13B	q41_v3	57.64%	29	50.31
Vicuna	13B	q6k	51.27%	190	370.82
WizardLM	13B	q40_v3	51.27%	41	79.95
WizardLM	13B	q40_v3	51.59%	31	60.09
WizardLM	13B	q40_v3	51.27%	42	81.92
WizardLM	13B	q40_v3	50.00%	29	58.00
WizardLM	13B	q4km	57.96%	34	58.64
WizardLM	13B	q6k	55.73%	41	73.52
--------	---	------	------	---	---
based	30B	q40_v3	53.50%	108	201.87
LLaMA	30B	s-hotcot	67.20%	83	123.45
--------	---	------	------	---	---
Guanaco	65B	q40_v3	99%	2	2.01
--------	---	------	------	---	---
Claude+	/	100k	93%	12	12.90
GPT-3.5	/	/	96.18%	12	12.48
GPT-4	/	/	97.77%	2	2.04