Skip to content

Conversation

@jjbuschhoff
Copy link

Cherrypicked some German task implementations from https://github.com/bjoernpl/lm-evaluation-harness-de/tree/mmlu_de.
Contains Hellaswag_de, TruthfulQA_de, ARC-Challenge_de and HendrycksTest_de.

@KlaudiaTH KlaudiaTH marked this pull request as ready for review November 5, 2023 17:59
DATASET_NAME = None

def __init__(self, subject):
self.DATASET_NAME = subject
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seems to be a problem with the dataset: Even when a dataset config is specified, documents from many different subjects are returned, e.g. for ogx_hendrycksTest_de-abstract_algebra:

Es folgen multiple-choice Fragen (mit Antworten) über das Thema Abstrakte Algebra.

Frage: Welche der folgenden listet die elektromagnetischen Spektralbereiche in absteigender Reihenfolge der Wellenlänge auf?
Optionen:
A. Ultraviolett, sichtbar, Infrarot, Röntgen
B. Röntgen, sichtbar, Ultraviolett, Infrarot
C. Röntgen, Ultraviolett, sichtbar, Infrarot
D. Infrarot, sichtbar, Ultraviolett, Röntgen
Antwort:

This question is actually a physics question, and not from abstract algebra.

If this cannot be worked around, it probably doesn't make sense to include this benchmark before the bug is fixed in the HF dataset.

Copy link
Author

@jjbuschhoff jjbuschhoff Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It indeed appears that the HF dataset builder script is faulty and suppllies the entire test-set over all subjects regardless of which subject is supplied in datasets.load_dataset().

@jjbuschhoff
Copy link
Author

I'd advocate in favour of closing this since #99 is a superset of the here implemented tasks.

@malteos
Copy link

malteos commented Nov 21, 2023

I'd advocate in favour of closing this since #99 is a superset of the here implemented tasks.

Sure about this? Even though they are technically the same tasks but I think the translations were created differently. I'd recommend having both variations implemented to be comparable to the literature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants