diff --git a/lm_eval/tasks/openai_mmmlu/README.md b/lm_eval/tasks/openai_mmmlu/README.md new file mode 100644 index 0000000000..6ad89e2a59 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/README.md @@ -0,0 +1,75 @@ +# OpenAI MMMLU + +### Technical Report + +The task/dataset contains a professional, human-translation of the common MMLU task (originally in the English language) into 14 different languages. + +Title: OpenAI o1 System Card + +Homepage: https://openai.com/index/openai-o1-system-card/ + +Technical Report: https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iOfeOP/71551c3d223cd97e591aa89567306912/o1_system_card.pdf + +[Measuring Massive Multitask Language Understanding](https://arxiv.org/pdf/2009.03300) by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (ICLR 2021). + +### Groups and Tasks + +The `default` variant to the common MMLU-style prompting, output from a `--write-out`: + +```bash +[...] + +document 0; context prompt (starting on next line): +The following are multiple choice questions (with answers) about anatomy. + +Ermitteln Sie den Grad für die gegebene Felderweiterung Q(sqrt(2), sqrt(3), sqrt(18)) über Q. +A. 0 +B. 4 +C. 2 +D. 6 +Antwort: +(end of prompt on previous line) +target string or answer choice index (starting on next line): +B +(end of target on previous line) +``` + +Note that + * the `description` is in English, while the question itself is in the target language, and the "Answer: " prefix is in the target language [this last bit was my choice]. + * in the paper, the prompt is [significantly different](https://github.com/openai/simple-evals/blob/2df1a92bbddb8c89fbeb3670e2dd125b10632bca/common.py#L12) and includes COT plus [generous regexps](https://github.com/openai/simple-evals/blob/2df1a92bbddb8c89fbeb3670e2dd125b10632bca/common.py#L29) (filters) to extract the answer. I am of the opinion one should implement a different variant to reproduce those results. + * split information is not present in the [dataset on hf](https://huggingface.co/datasets/openai/MMMLU), so currently this dataset doesn't support fewshot or decontamination. + +#### Groups + + * `openai_mmmlu_default` # supergroup of the following groups + * `openai_mmmlu_default_ar_xy` + * `openai_mmmlu_default_bn_bd` + * `openai_mmmlu_default_de_de` + * `openai_mmmlu_default_es_la` + * `openai_mmmlu_default_fr_fr` + * `openai_mmmlu_default_hi_in` + * `openai_mmmlu_default_id_id` + * `openai_mmmlu_default_it_it` + * `openai_mmmlu_default_ja_jp` + * `openai_mmmlu_default_ko_kr` + * `openai_mmmlu_default_pt_br` + * `openai_mmmlu_default_sw_ke` + * `openai_mmmlu_default_yo_ng` + * `openai_mmmlu_default_zh_ch` + +#### Tasks + +* `openai_mmmlu_default__`: The mmlu translation combined with prompt, language and subject from mmlu. + +### Checklist + +For adding novel benchmarks/datasets to the library: +* [x] Is the task an existing benchmark in the literature? + * [x] Have you referenced the original paper that introduced the task? + * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test? + + +If other tasks on this dataset are already supported: +* [x] Is the "Main" variant of this task clearly denoted? Yes, it would be the `default` folder. +* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates? +* [ ] Have you noted which, if any, published evaluation setups are matched by this variant? diff --git a/lm_eval/tasks/openai_mmmlu/_default_template_yaml b/lm_eval/tasks/openai_mmmlu/_default_template_yaml new file mode 100644 index 0000000000..2250681e48 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/_default_template_yaml @@ -0,0 +1,12 @@ +dataset_path: giuliolovisotto/openai_multilingual_mmlu # a copy of `cais/mmlu` with no auxiliary_train split +fewshot_split: null +fewshot_config: null +output_type: multiple_choice +metric_list: + - metric: acc + aggregation: mean + higher_is_better: true +metadata: + version: 1.0.0 +dataset_kwargs: + trust_remote_code: true diff --git a/lm_eval/tasks/openai_mmmlu/_generate_configs.py b/lm_eval/tasks/openai_mmmlu/_generate_configs.py new file mode 100644 index 0000000000..013ffe1bf6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/_generate_configs.py @@ -0,0 +1,176 @@ +""" +Caveats: + * I asked an LLM to translate "Answer:" into the 14 languages following, output of its conversation below. + * OpenAI uses slightly different formatting for Q/A -> https://github.com/openai/simple-evals/blob/main/common.py + a. They use a list of regexps in various langauages to extract a line that contains "Answer: .." to get the response, + b. They have an actual prompt `QUERY_TEMPLATE_MULTICHOICE` with cot priming - I added this. + +====== +Can you make me a dictionary that maps these languages to the way one would say "Answer:" in those languages? Let me know of when you have doubts and why + +LANGUAGES = { + "AR_XY": "Arabic (Generic)", + "BN_BD": "Bengali (Bangladesh)", + "DE_DE": "German (Germany)", + "ES_LA": "Spanish (Latin America)", + "FR_FR": "French (France)", + "HI_IN": "Hindi (India)", + "ID_ID": "Indonesian (Indonesia)", + "IT_IT": "Italian (Italy)", + "JA_JP": "Japanese (Japan)", + "KO_KR": "Korean (South Korea)", + "PT_BR": "Portuguese (Brazil)", + "ZH_CN": "Chinese (China)", + "SW_KE": "Swahili (Kenya)", + "YO_NG": "Yoruba (Nigeria)" +} + +I'll create a dictionary that maps these languages to their respective translations of "Answer:". I'll think through this step-by-step and highlight any uncertainties. + First, I'll create a new dictionary with the same keys as the LANGUAGES dictionary. + For each language, I'll provide the translation of "Answer:". + I'll note any doubts or uncertainties as I go. + +Here's the process: + AR_XY (Arabic): "الإجابة:" No doubts here. + BN_BD (Bengali): "উত্তর:" No doubts. + DE_DE (German): "Antwort:" Confident about this one. + ES_LA (Spanish): "Respuesta:" Note: This is standard Spanish, but there might be slight variations in Latin American countries. + FR_FR (French): "Réponse :" Note: In French, there's usually a space before the colon. + HI_IN (Hindi): "उत्तर:" No doubts. + ID_ID (Indonesian): "Jawaban:" Fairly confident, but there might be variations. + IT_IT (Italian): "Risposta:" No doubts. + JA_JP (Japanese): "答え：" Note: Japanese uses a full-width colon (：) instead of a half-width one (:). + KO_KR (Korean): "답변:" Fairly confident, but there might be other ways to express this. + PT_BR (Portuguese): "Resposta:" No doubts for Brazilian Portuguese. + ZH_CN (Chinese): "答案：" Note: Using Simplified Chinese characters. Also uses full-width colon. + SW_KE (Swahili): "Jibu:" Some uncertainty here, as my knowledge of Swahili is not as extensive. + YO_NG (Yoruba): "Ìdáhùn:" Significant uncertainty. My knowledge of Yoruba is limited, and this might not be the most common or accurate translation. + +Main points of uncertainty: + For ES_LA (Spanish), there might be slight regional variations in Latin America. + For SW_KE (Swahili) and YO_NG (Yoruba), my confidence is lower due to limited knowledge of these languages. + For some languages, there might be multiple ways to express "Answer:" depending on context or formality level. + +It would be advisable to have these translations verified by native speakers or language experts, especially for Swahili and Yoruba. +====== + + +""" + +import argparse +import logging +from pathlib import Path + +import yaml +from tqdm import tqdm +from utils import ANSWERS, LANGUAGES, SUBJECTS + + +eval_logger = logging.getLogger("lm-eval") + + +def parse_args(): + parser = argparse.ArgumentParser() + return parser.parse_args() + + +if __name__ == "__main__": + args = parse_args() + + PROMPT_FLAVOURS = { + # default in the version of standard MMLU + # honestly I think we should translate these into the target language. + "default": { + "description": "The following are multiple choice questions (with answers) about {}.\n\n", + "prompt": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\nD. {{D.strip()}}\n", + "add_answer": True, + }, + # this one in the version found on simple-evals from openai + # "cot": { + # "description": "Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.\n\n", + # "prompt": "{{Question.strip()}}\n\nA) {{A.strip()}}\nB) {{B.strip()}}\nC) {{C.strip()}}\nD) {{D.strip()}}\n", + # "add_answer": False + # } + } + + ALL_CATEGORIES = [] + ALL_TASKS = [] + for prompt_key, prompt_info in PROMPT_FLAVOURS.items(): + for langgode, language_full_name in tqdm(LANGUAGES.items()): + _langgode = langgode.lower() + out_folder = Path(prompt_key) / _langgode + out_folder.mkdir(exist_ok=True, parents=True) + for subject, category in SUBJECTS.items(): + if category not in ALL_CATEGORIES: + ALL_CATEGORIES.append(category) + + yaml_dict = { + "include": "../../_default_template_yaml", + "tag": f"openai_mmmlu_{prompt_key}_{_langgode}_{category}", + "task": f"openai_mmmlu_{prompt_key}_{_langgode}_{subject}", + "task_alias": f'{_langgode} {subject.replace("_", " ")}', + "dataset_name": subject, + "test_split": langgode, + "description": prompt_info["description"].format(subject), + "doc_to_text": prompt_info["prompt"] + + (ANSWERS[langgode] if prompt_info["add_answer"] else ""), + "doc_to_choice": ["A", "B", "C", "D"], + "doc_to_target": "{{Answer.strip()}}", + } + + file_save_path = ( + out_folder / f"openai_mmmlu_{prompt_key}_{subject}.yaml" + ) + eval_logger.info( + f"Saving yaml for subset {_langgode},{subject} to {file_save_path}" + ) + with open(file_save_path, "w", encoding="utf-8") as yaml_file: + yaml.dump( + yaml_dict, + yaml_file, + allow_unicode=True, + default_style='"', + ) + + # (sub)group for prompt/language pair + subgroup_info_path = ( + out_folder / f"_{prompt_key}_{_langgode}_group_info.yaml" + ) + with open(subgroup_info_path, "w", encoding="utf-8") as yaml_file: + # list of task for this pair of prompt/language + _tasks = [ + f"openai_mmmlu_{prompt_key}_{_langgode}_{_subject}" + for _subject in SUBJECTS.keys() + ] + dct = { + "group": f"openai_mmmlu_{prompt_key}_{_langgode}", + "task": _tasks, + "aggregate_metric_list": [ + {"metric": "acc", "weight_by_size": True} + ], + "metadata": {"version": "1.0.0"}, + } + ALL_TASKS.extend(_tasks) + yaml.dump( + dct, + yaml_file, + indent=4, + default_flow_style=False, + ) + # (super)group for promptkey + out_folder = Path(prompt_key) + supergroup_info_path = out_folder / f"_{prompt_key}_group_info.yaml" + with open(supergroup_info_path, "w", encoding="utf-8") as yaml_file: + dct = { + "group": f"openai_mmmlu_{prompt_key}", + "task": ALL_TASKS, + "aggregate_metric_list": [{"metric": "acc", "weight_by_size": True}], + "metadata": {"version": "1.0.0"}, + } + + yaml.dump( + dct, + yaml_file, + indent=4, + default_flow_style=False, + ) diff --git a/lm_eval/tasks/openai_mmmlu/default/_default_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/_default_group_info.yaml new file mode 100644 index 0000000000..cf23bee3c6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/_default_group_info.yaml @@ -0,0 +1,862 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_ar_xy_abstract_algebra +- openai_mmmlu_default_ar_xy_anatomy +- openai_mmmlu_default_ar_xy_astronomy +- openai_mmmlu_default_ar_xy_business_ethics +- openai_mmmlu_default_ar_xy_clinical_knowledge +- openai_mmmlu_default_ar_xy_college_biology +- openai_mmmlu_default_ar_xy_college_chemistry +- openai_mmmlu_default_ar_xy_college_computer_science +- openai_mmmlu_default_ar_xy_college_mathematics +- openai_mmmlu_default_ar_xy_college_medicine +- openai_mmmlu_default_ar_xy_college_physics +- openai_mmmlu_default_ar_xy_computer_security +- openai_mmmlu_default_ar_xy_conceptual_physics +- openai_mmmlu_default_ar_xy_econometrics +- openai_mmmlu_default_ar_xy_electrical_engineering +- openai_mmmlu_default_ar_xy_elementary_mathematics +- openai_mmmlu_default_ar_xy_formal_logic +- openai_mmmlu_default_ar_xy_global_facts +- openai_mmmlu_default_ar_xy_high_school_biology +- openai_mmmlu_default_ar_xy_high_school_chemistry +- openai_mmmlu_default_ar_xy_high_school_computer_science +- openai_mmmlu_default_ar_xy_high_school_european_history +- openai_mmmlu_default_ar_xy_high_school_geography +- openai_mmmlu_default_ar_xy_high_school_government_and_politics +- openai_mmmlu_default_ar_xy_high_school_macroeconomics +- openai_mmmlu_default_ar_xy_high_school_mathematics +- openai_mmmlu_default_ar_xy_high_school_microeconomics +- openai_mmmlu_default_ar_xy_high_school_physics +- openai_mmmlu_default_ar_xy_high_school_psychology +- openai_mmmlu_default_ar_xy_high_school_statistics +- openai_mmmlu_default_ar_xy_high_school_us_history +- openai_mmmlu_default_ar_xy_high_school_world_history +- openai_mmmlu_default_ar_xy_human_aging +- openai_mmmlu_default_ar_xy_human_sexuality +- openai_mmmlu_default_ar_xy_international_law +- openai_mmmlu_default_ar_xy_jurisprudence +- openai_mmmlu_default_ar_xy_logical_fallacies +- openai_mmmlu_default_ar_xy_machine_learning +- openai_mmmlu_default_ar_xy_management +- openai_mmmlu_default_ar_xy_marketing +- openai_mmmlu_default_ar_xy_medical_genetics +- openai_mmmlu_default_ar_xy_miscellaneous +- openai_mmmlu_default_ar_xy_moral_disputes +- openai_mmmlu_default_ar_xy_moral_scenarios +- openai_mmmlu_default_ar_xy_nutrition +- openai_mmmlu_default_ar_xy_philosophy +- openai_mmmlu_default_ar_xy_prehistory +- openai_mmmlu_default_ar_xy_professional_accounting +- openai_mmmlu_default_ar_xy_professional_law +- openai_mmmlu_default_ar_xy_professional_medicine +- openai_mmmlu_default_ar_xy_professional_psychology +- openai_mmmlu_default_ar_xy_public_relations +- openai_mmmlu_default_ar_xy_security_studies +- openai_mmmlu_default_ar_xy_sociology +- openai_mmmlu_default_ar_xy_us_foreign_policy +- openai_mmmlu_default_ar_xy_virology +- openai_mmmlu_default_ar_xy_world_religions +- openai_mmmlu_default_bn_bd_abstract_algebra +- openai_mmmlu_default_bn_bd_anatomy +- openai_mmmlu_default_bn_bd_astronomy +- openai_mmmlu_default_bn_bd_business_ethics +- openai_mmmlu_default_bn_bd_clinical_knowledge +- openai_mmmlu_default_bn_bd_college_biology +- openai_mmmlu_default_bn_bd_college_chemistry +- openai_mmmlu_default_bn_bd_college_computer_science +- openai_mmmlu_default_bn_bd_college_mathematics +- openai_mmmlu_default_bn_bd_college_medicine +- openai_mmmlu_default_bn_bd_college_physics +- openai_mmmlu_default_bn_bd_computer_security +- openai_mmmlu_default_bn_bd_conceptual_physics +- openai_mmmlu_default_bn_bd_econometrics +- openai_mmmlu_default_bn_bd_electrical_engineering +- openai_mmmlu_default_bn_bd_elementary_mathematics +- openai_mmmlu_default_bn_bd_formal_logic +- openai_mmmlu_default_bn_bd_global_facts +- openai_mmmlu_default_bn_bd_high_school_biology +- openai_mmmlu_default_bn_bd_high_school_chemistry +- openai_mmmlu_default_bn_bd_high_school_computer_science +- openai_mmmlu_default_bn_bd_high_school_european_history +- openai_mmmlu_default_bn_bd_high_school_geography +- openai_mmmlu_default_bn_bd_high_school_government_and_politics +- openai_mmmlu_default_bn_bd_high_school_macroeconomics +- openai_mmmlu_default_bn_bd_high_school_mathematics +- openai_mmmlu_default_bn_bd_high_school_microeconomics +- openai_mmmlu_default_bn_bd_high_school_physics +- openai_mmmlu_default_bn_bd_high_school_psychology +- openai_mmmlu_default_bn_bd_high_school_statistics +- openai_mmmlu_default_bn_bd_high_school_us_history +- openai_mmmlu_default_bn_bd_high_school_world_history +- openai_mmmlu_default_bn_bd_human_aging +- openai_mmmlu_default_bn_bd_human_sexuality +- openai_mmmlu_default_bn_bd_international_law +- openai_mmmlu_default_bn_bd_jurisprudence +- openai_mmmlu_default_bn_bd_logical_fallacies +- openai_mmmlu_default_bn_bd_machine_learning +- openai_mmmlu_default_bn_bd_management +- openai_mmmlu_default_bn_bd_marketing +- openai_mmmlu_default_bn_bd_medical_genetics +- openai_mmmlu_default_bn_bd_miscellaneous +- openai_mmmlu_default_bn_bd_moral_disputes +- openai_mmmlu_default_bn_bd_moral_scenarios +- openai_mmmlu_default_bn_bd_nutrition +- openai_mmmlu_default_bn_bd_philosophy +- openai_mmmlu_default_bn_bd_prehistory +- openai_mmmlu_default_bn_bd_professional_accounting +- openai_mmmlu_default_bn_bd_professional_law +- openai_mmmlu_default_bn_bd_professional_medicine +- openai_mmmlu_default_bn_bd_professional_psychology +- openai_mmmlu_default_bn_bd_public_relations +- openai_mmmlu_default_bn_bd_security_studies +- openai_mmmlu_default_bn_bd_sociology +- openai_mmmlu_default_bn_bd_us_foreign_policy +- openai_mmmlu_default_bn_bd_virology +- openai_mmmlu_default_bn_bd_world_religions +- openai_mmmlu_default_de_de_abstract_algebra +- openai_mmmlu_default_de_de_anatomy +- openai_mmmlu_default_de_de_astronomy +- openai_mmmlu_default_de_de_business_ethics +- openai_mmmlu_default_de_de_clinical_knowledge +- openai_mmmlu_default_de_de_college_biology +- openai_mmmlu_default_de_de_college_chemistry +- openai_mmmlu_default_de_de_college_computer_science +- openai_mmmlu_default_de_de_college_mathematics +- openai_mmmlu_default_de_de_college_medicine +- openai_mmmlu_default_de_de_college_physics +- openai_mmmlu_default_de_de_computer_security +- openai_mmmlu_default_de_de_conceptual_physics +- openai_mmmlu_default_de_de_econometrics +- openai_mmmlu_default_de_de_electrical_engineering +- openai_mmmlu_default_de_de_elementary_mathematics +- openai_mmmlu_default_de_de_formal_logic +- openai_mmmlu_default_de_de_global_facts +- openai_mmmlu_default_de_de_high_school_biology +- openai_mmmlu_default_de_de_high_school_chemistry +- openai_mmmlu_default_de_de_high_school_computer_science +- openai_mmmlu_default_de_de_high_school_european_history +- openai_mmmlu_default_de_de_high_school_geography +- openai_mmmlu_default_de_de_high_school_government_and_politics +- openai_mmmlu_default_de_de_high_school_macroeconomics +- openai_mmmlu_default_de_de_high_school_mathematics +- openai_mmmlu_default_de_de_high_school_microeconomics +- openai_mmmlu_default_de_de_high_school_physics +- openai_mmmlu_default_de_de_high_school_psychology +- openai_mmmlu_default_de_de_high_school_statistics +- openai_mmmlu_default_de_de_high_school_us_history +- openai_mmmlu_default_de_de_high_school_world_history +- openai_mmmlu_default_de_de_human_aging +- openai_mmmlu_default_de_de_human_sexuality +- openai_mmmlu_default_de_de_international_law +- openai_mmmlu_default_de_de_jurisprudence +- openai_mmmlu_default_de_de_logical_fallacies +- openai_mmmlu_default_de_de_machine_learning +- openai_mmmlu_default_de_de_management +- openai_mmmlu_default_de_de_marketing +- openai_mmmlu_default_de_de_medical_genetics +- openai_mmmlu_default_de_de_miscellaneous +- openai_mmmlu_default_de_de_moral_disputes +- openai_mmmlu_default_de_de_moral_scenarios +- openai_mmmlu_default_de_de_nutrition +- openai_mmmlu_default_de_de_philosophy +- openai_mmmlu_default_de_de_prehistory +- openai_mmmlu_default_de_de_professional_accounting +- openai_mmmlu_default_de_de_professional_law +- openai_mmmlu_default_de_de_professional_medicine +- openai_mmmlu_default_de_de_professional_psychology +- openai_mmmlu_default_de_de_public_relations +- openai_mmmlu_default_de_de_security_studies +- openai_mmmlu_default_de_de_sociology +- openai_mmmlu_default_de_de_us_foreign_policy +- openai_mmmlu_default_de_de_virology +- openai_mmmlu_default_de_de_world_religions +- openai_mmmlu_default_es_la_abstract_algebra +- openai_mmmlu_default_es_la_anatomy +- openai_mmmlu_default_es_la_astronomy +- openai_mmmlu_default_es_la_business_ethics +- openai_mmmlu_default_es_la_clinical_knowledge +- openai_mmmlu_default_es_la_college_biology +- openai_mmmlu_default_es_la_college_chemistry +- openai_mmmlu_default_es_la_college_computer_science +- openai_mmmlu_default_es_la_college_mathematics +- openai_mmmlu_default_es_la_college_medicine +- openai_mmmlu_default_es_la_college_physics +- openai_mmmlu_default_es_la_computer_security +- openai_mmmlu_default_es_la_conceptual_physics +- openai_mmmlu_default_es_la_econometrics +- openai_mmmlu_default_es_la_electrical_engineering +- openai_mmmlu_default_es_la_elementary_mathematics +- openai_mmmlu_default_es_la_formal_logic +- openai_mmmlu_default_es_la_global_facts +- openai_mmmlu_default_es_la_high_school_biology +- openai_mmmlu_default_es_la_high_school_chemistry +- openai_mmmlu_default_es_la_high_school_computer_science +- openai_mmmlu_default_es_la_high_school_european_history +- openai_mmmlu_default_es_la_high_school_geography +- openai_mmmlu_default_es_la_high_school_government_and_politics +- openai_mmmlu_default_es_la_high_school_macroeconomics +- openai_mmmlu_default_es_la_high_school_mathematics +- openai_mmmlu_default_es_la_high_school_microeconomics +- openai_mmmlu_default_es_la_high_school_physics +- openai_mmmlu_default_es_la_high_school_psychology +- openai_mmmlu_default_es_la_high_school_statistics +- openai_mmmlu_default_es_la_high_school_us_history +- openai_mmmlu_default_es_la_high_school_world_history +- openai_mmmlu_default_es_la_human_aging +- openai_mmmlu_default_es_la_human_sexuality +- openai_mmmlu_default_es_la_international_law +- openai_mmmlu_default_es_la_jurisprudence +- openai_mmmlu_default_es_la_logical_fallacies +- openai_mmmlu_default_es_la_machine_learning +- openai_mmmlu_default_es_la_management +- openai_mmmlu_default_es_la_marketing +- openai_mmmlu_default_es_la_medical_genetics +- openai_mmmlu_default_es_la_miscellaneous +- openai_mmmlu_default_es_la_moral_disputes +- openai_mmmlu_default_es_la_moral_scenarios +- openai_mmmlu_default_es_la_nutrition +- openai_mmmlu_default_es_la_philosophy +- openai_mmmlu_default_es_la_prehistory +- openai_mmmlu_default_es_la_professional_accounting +- openai_mmmlu_default_es_la_professional_law +- openai_mmmlu_default_es_la_professional_medicine +- openai_mmmlu_default_es_la_professional_psychology +- openai_mmmlu_default_es_la_public_relations +- openai_mmmlu_default_es_la_security_studies +- openai_mmmlu_default_es_la_sociology +- openai_mmmlu_default_es_la_us_foreign_policy +- openai_mmmlu_default_es_la_virology +- openai_mmmlu_default_es_la_world_religions +- openai_mmmlu_default_fr_fr_abstract_algebra +- openai_mmmlu_default_fr_fr_anatomy +- openai_mmmlu_default_fr_fr_astronomy +- openai_mmmlu_default_fr_fr_business_ethics +- openai_mmmlu_default_fr_fr_clinical_knowledge +- openai_mmmlu_default_fr_fr_college_biology +- openai_mmmlu_default_fr_fr_college_chemistry +- openai_mmmlu_default_fr_fr_college_computer_science +- openai_mmmlu_default_fr_fr_college_mathematics +- openai_mmmlu_default_fr_fr_college_medicine +- openai_mmmlu_default_fr_fr_college_physics +- openai_mmmlu_default_fr_fr_computer_security +- openai_mmmlu_default_fr_fr_conceptual_physics +- openai_mmmlu_default_fr_fr_econometrics +- openai_mmmlu_default_fr_fr_electrical_engineering +- openai_mmmlu_default_fr_fr_elementary_mathematics +- openai_mmmlu_default_fr_fr_formal_logic +- openai_mmmlu_default_fr_fr_global_facts +- openai_mmmlu_default_fr_fr_high_school_biology +- openai_mmmlu_default_fr_fr_high_school_chemistry +- openai_mmmlu_default_fr_fr_high_school_computer_science +- openai_mmmlu_default_fr_fr_high_school_european_history +- openai_mmmlu_default_fr_fr_high_school_geography +- openai_mmmlu_default_fr_fr_high_school_government_and_politics +- openai_mmmlu_default_fr_fr_high_school_macroeconomics +- openai_mmmlu_default_fr_fr_high_school_mathematics +- openai_mmmlu_default_fr_fr_high_school_microeconomics +- openai_mmmlu_default_fr_fr_high_school_physics +- openai_mmmlu_default_fr_fr_high_school_psychology +- openai_mmmlu_default_fr_fr_high_school_statistics +- openai_mmmlu_default_fr_fr_high_school_us_history +- openai_mmmlu_default_fr_fr_high_school_world_history +- openai_mmmlu_default_fr_fr_human_aging +- openai_mmmlu_default_fr_fr_human_sexuality +- openai_mmmlu_default_fr_fr_international_law +- openai_mmmlu_default_fr_fr_jurisprudence +- openai_mmmlu_default_fr_fr_logical_fallacies +- openai_mmmlu_default_fr_fr_machine_learning +- openai_mmmlu_default_fr_fr_management +- openai_mmmlu_default_fr_fr_marketing +- openai_mmmlu_default_fr_fr_medical_genetics +- openai_mmmlu_default_fr_fr_miscellaneous +- openai_mmmlu_default_fr_fr_moral_disputes +- openai_mmmlu_default_fr_fr_moral_scenarios +- openai_mmmlu_default_fr_fr_nutrition +- openai_mmmlu_default_fr_fr_philosophy +- openai_mmmlu_default_fr_fr_prehistory +- openai_mmmlu_default_fr_fr_professional_accounting +- openai_mmmlu_default_fr_fr_professional_law +- openai_mmmlu_default_fr_fr_professional_medicine +- openai_mmmlu_default_fr_fr_professional_psychology +- openai_mmmlu_default_fr_fr_public_relations +- openai_mmmlu_default_fr_fr_security_studies +- openai_mmmlu_default_fr_fr_sociology +- openai_mmmlu_default_fr_fr_us_foreign_policy +- openai_mmmlu_default_fr_fr_virology +- openai_mmmlu_default_fr_fr_world_religions +- openai_mmmlu_default_hi_in_abstract_algebra +- openai_mmmlu_default_hi_in_anatomy +- openai_mmmlu_default_hi_in_astronomy +- openai_mmmlu_default_hi_in_business_ethics +- openai_mmmlu_default_hi_in_clinical_knowledge +- openai_mmmlu_default_hi_in_college_biology +- openai_mmmlu_default_hi_in_college_chemistry +- openai_mmmlu_default_hi_in_college_computer_science +- openai_mmmlu_default_hi_in_college_mathematics +- openai_mmmlu_default_hi_in_college_medicine +- openai_mmmlu_default_hi_in_college_physics +- openai_mmmlu_default_hi_in_computer_security +- openai_mmmlu_default_hi_in_conceptual_physics +- openai_mmmlu_default_hi_in_econometrics +- openai_mmmlu_default_hi_in_electrical_engineering +- openai_mmmlu_default_hi_in_elementary_mathematics +- openai_mmmlu_default_hi_in_formal_logic +- openai_mmmlu_default_hi_in_global_facts +- openai_mmmlu_default_hi_in_high_school_biology +- openai_mmmlu_default_hi_in_high_school_chemistry +- openai_mmmlu_default_hi_in_high_school_computer_science +- openai_mmmlu_default_hi_in_high_school_european_history +- openai_mmmlu_default_hi_in_high_school_geography +- openai_mmmlu_default_hi_in_high_school_government_and_politics +- openai_mmmlu_default_hi_in_high_school_macroeconomics +- openai_mmmlu_default_hi_in_high_school_mathematics +- openai_mmmlu_default_hi_in_high_school_microeconomics +- openai_mmmlu_default_hi_in_high_school_physics +- openai_mmmlu_default_hi_in_high_school_psychology +- openai_mmmlu_default_hi_in_high_school_statistics +- openai_mmmlu_default_hi_in_high_school_us_history +- openai_mmmlu_default_hi_in_high_school_world_history +- openai_mmmlu_default_hi_in_human_aging +- openai_mmmlu_default_hi_in_human_sexuality +- openai_mmmlu_default_hi_in_international_law +- openai_mmmlu_default_hi_in_jurisprudence +- openai_mmmlu_default_hi_in_logical_fallacies +- openai_mmmlu_default_hi_in_machine_learning +- openai_mmmlu_default_hi_in_management +- openai_mmmlu_default_hi_in_marketing +- openai_mmmlu_default_hi_in_medical_genetics +- openai_mmmlu_default_hi_in_miscellaneous +- openai_mmmlu_default_hi_in_moral_disputes +- openai_mmmlu_default_hi_in_moral_scenarios +- openai_mmmlu_default_hi_in_nutrition +- openai_mmmlu_default_hi_in_philosophy +- openai_mmmlu_default_hi_in_prehistory +- openai_mmmlu_default_hi_in_professional_accounting +- openai_mmmlu_default_hi_in_professional_law +- openai_mmmlu_default_hi_in_professional_medicine +- openai_mmmlu_default_hi_in_professional_psychology +- openai_mmmlu_default_hi_in_public_relations +- openai_mmmlu_default_hi_in_security_studies +- openai_mmmlu_default_hi_in_sociology +- openai_mmmlu_default_hi_in_us_foreign_policy +- openai_mmmlu_default_hi_in_virology +- openai_mmmlu_default_hi_in_world_religions +- openai_mmmlu_default_id_id_abstract_algebra +- openai_mmmlu_default_id_id_anatomy +- openai_mmmlu_default_id_id_astronomy +- openai_mmmlu_default_id_id_business_ethics +- openai_mmmlu_default_id_id_clinical_knowledge +- openai_mmmlu_default_id_id_college_biology +- openai_mmmlu_default_id_id_college_chemistry +- openai_mmmlu_default_id_id_college_computer_science +- openai_mmmlu_default_id_id_college_mathematics +- openai_mmmlu_default_id_id_college_medicine +- openai_mmmlu_default_id_id_college_physics +- openai_mmmlu_default_id_id_computer_security +- openai_mmmlu_default_id_id_conceptual_physics +- openai_mmmlu_default_id_id_econometrics +- openai_mmmlu_default_id_id_electrical_engineering +- openai_mmmlu_default_id_id_elementary_mathematics +- openai_mmmlu_default_id_id_formal_logic +- openai_mmmlu_default_id_id_global_facts +- openai_mmmlu_default_id_id_high_school_biology +- openai_mmmlu_default_id_id_high_school_chemistry +- openai_mmmlu_default_id_id_high_school_computer_science +- openai_mmmlu_default_id_id_high_school_european_history +- openai_mmmlu_default_id_id_high_school_geography +- openai_mmmlu_default_id_id_high_school_government_and_politics +- openai_mmmlu_default_id_id_high_school_macroeconomics +- openai_mmmlu_default_id_id_high_school_mathematics +- openai_mmmlu_default_id_id_high_school_microeconomics +- openai_mmmlu_default_id_id_high_school_physics +- openai_mmmlu_default_id_id_high_school_psychology +- openai_mmmlu_default_id_id_high_school_statistics +- openai_mmmlu_default_id_id_high_school_us_history +- openai_mmmlu_default_id_id_high_school_world_history +- openai_mmmlu_default_id_id_human_aging +- openai_mmmlu_default_id_id_human_sexuality +- openai_mmmlu_default_id_id_international_law +- openai_mmmlu_default_id_id_jurisprudence +- openai_mmmlu_default_id_id_logical_fallacies +- openai_mmmlu_default_id_id_machine_learning +- openai_mmmlu_default_id_id_management +- openai_mmmlu_default_id_id_marketing +- openai_mmmlu_default_id_id_medical_genetics +- openai_mmmlu_default_id_id_miscellaneous +- openai_mmmlu_default_id_id_moral_disputes +- openai_mmmlu_default_id_id_moral_scenarios +- openai_mmmlu_default_id_id_nutrition +- openai_mmmlu_default_id_id_philosophy +- openai_mmmlu_default_id_id_prehistory +- openai_mmmlu_default_id_id_professional_accounting +- openai_mmmlu_default_id_id_professional_law +- openai_mmmlu_default_id_id_professional_medicine +- openai_mmmlu_default_id_id_professional_psychology +- openai_mmmlu_default_id_id_public_relations +- openai_mmmlu_default_id_id_security_studies +- openai_mmmlu_default_id_id_sociology +- openai_mmmlu_default_id_id_us_foreign_policy +- openai_mmmlu_default_id_id_virology +- openai_mmmlu_default_id_id_world_religions +- openai_mmmlu_default_it_it_abstract_algebra +- openai_mmmlu_default_it_it_anatomy +- openai_mmmlu_default_it_it_astronomy +- openai_mmmlu_default_it_it_business_ethics +- openai_mmmlu_default_it_it_clinical_knowledge +- openai_mmmlu_default_it_it_college_biology +- openai_mmmlu_default_it_it_college_chemistry +- openai_mmmlu_default_it_it_college_computer_science +- openai_mmmlu_default_it_it_college_mathematics +- openai_mmmlu_default_it_it_college_medicine +- openai_mmmlu_default_it_it_college_physics +- openai_mmmlu_default_it_it_computer_security +- openai_mmmlu_default_it_it_conceptual_physics +- openai_mmmlu_default_it_it_econometrics +- openai_mmmlu_default_it_it_electrical_engineering +- openai_mmmlu_default_it_it_elementary_mathematics +- openai_mmmlu_default_it_it_formal_logic +- openai_mmmlu_default_it_it_global_facts +- openai_mmmlu_default_it_it_high_school_biology +- openai_mmmlu_default_it_it_high_school_chemistry +- openai_mmmlu_default_it_it_high_school_computer_science +- openai_mmmlu_default_it_it_high_school_european_history +- openai_mmmlu_default_it_it_high_school_geography +- openai_mmmlu_default_it_it_high_school_government_and_politics +- openai_mmmlu_default_it_it_high_school_macroeconomics +- openai_mmmlu_default_it_it_high_school_mathematics +- openai_mmmlu_default_it_it_high_school_microeconomics +- openai_mmmlu_default_it_it_high_school_physics +- openai_mmmlu_default_it_it_high_school_psychology +- openai_mmmlu_default_it_it_high_school_statistics +- openai_mmmlu_default_it_it_high_school_us_history +- openai_mmmlu_default_it_it_high_school_world_history +- openai_mmmlu_default_it_it_human_aging +- openai_mmmlu_default_it_it_human_sexuality +- openai_mmmlu_default_it_it_international_law +- openai_mmmlu_default_it_it_jurisprudence +- openai_mmmlu_default_it_it_logical_fallacies +- openai_mmmlu_default_it_it_machine_learning +- openai_mmmlu_default_it_it_management +- openai_mmmlu_default_it_it_marketing +- openai_mmmlu_default_it_it_medical_genetics +- openai_mmmlu_default_it_it_miscellaneous +- openai_mmmlu_default_it_it_moral_disputes +- openai_mmmlu_default_it_it_moral_scenarios +- openai_mmmlu_default_it_it_nutrition +- openai_mmmlu_default_it_it_philosophy +- openai_mmmlu_default_it_it_prehistory +- openai_mmmlu_default_it_it_professional_accounting +- openai_mmmlu_default_it_it_professional_law +- openai_mmmlu_default_it_it_professional_medicine +- openai_mmmlu_default_it_it_professional_psychology +- openai_mmmlu_default_it_it_public_relations +- openai_mmmlu_default_it_it_security_studies +- openai_mmmlu_default_it_it_sociology +- openai_mmmlu_default_it_it_us_foreign_policy +- openai_mmmlu_default_it_it_virology +- openai_mmmlu_default_it_it_world_religions +- openai_mmmlu_default_ja_jp_abstract_algebra +- openai_mmmlu_default_ja_jp_anatomy +- openai_mmmlu_default_ja_jp_astronomy +- openai_mmmlu_default_ja_jp_business_ethics +- openai_mmmlu_default_ja_jp_clinical_knowledge +- openai_mmmlu_default_ja_jp_college_biology +- openai_mmmlu_default_ja_jp_college_chemistry +- openai_mmmlu_default_ja_jp_college_computer_science +- openai_mmmlu_default_ja_jp_college_mathematics +- openai_mmmlu_default_ja_jp_college_medicine +- openai_mmmlu_default_ja_jp_college_physics +- openai_mmmlu_default_ja_jp_computer_security +- openai_mmmlu_default_ja_jp_conceptual_physics +- openai_mmmlu_default_ja_jp_econometrics +- openai_mmmlu_default_ja_jp_electrical_engineering +- openai_mmmlu_default_ja_jp_elementary_mathematics +- openai_mmmlu_default_ja_jp_formal_logic +- openai_mmmlu_default_ja_jp_global_facts +- openai_mmmlu_default_ja_jp_high_school_biology +- openai_mmmlu_default_ja_jp_high_school_chemistry +- openai_mmmlu_default_ja_jp_high_school_computer_science +- openai_mmmlu_default_ja_jp_high_school_european_history +- openai_mmmlu_default_ja_jp_high_school_geography +- openai_mmmlu_default_ja_jp_high_school_government_and_politics +- openai_mmmlu_default_ja_jp_high_school_macroeconomics +- openai_mmmlu_default_ja_jp_high_school_mathematics +- openai_mmmlu_default_ja_jp_high_school_microeconomics +- openai_mmmlu_default_ja_jp_high_school_physics +- openai_mmmlu_default_ja_jp_high_school_psychology +- openai_mmmlu_default_ja_jp_high_school_statistics +- openai_mmmlu_default_ja_jp_high_school_us_history +- openai_mmmlu_default_ja_jp_high_school_world_history +- openai_mmmlu_default_ja_jp_human_aging +- openai_mmmlu_default_ja_jp_human_sexuality +- openai_mmmlu_default_ja_jp_international_law +- openai_mmmlu_default_ja_jp_jurisprudence +- openai_mmmlu_default_ja_jp_logical_fallacies +- openai_mmmlu_default_ja_jp_machine_learning +- openai_mmmlu_default_ja_jp_management +- openai_mmmlu_default_ja_jp_marketing +- openai_mmmlu_default_ja_jp_medical_genetics +- openai_mmmlu_default_ja_jp_miscellaneous +- openai_mmmlu_default_ja_jp_moral_disputes +- openai_mmmlu_default_ja_jp_moral_scenarios +- openai_mmmlu_default_ja_jp_nutrition +- openai_mmmlu_default_ja_jp_philosophy +- openai_mmmlu_default_ja_jp_prehistory +- openai_mmmlu_default_ja_jp_professional_accounting +- openai_mmmlu_default_ja_jp_professional_law +- openai_mmmlu_default_ja_jp_professional_medicine +- openai_mmmlu_default_ja_jp_professional_psychology +- openai_mmmlu_default_ja_jp_public_relations +- openai_mmmlu_default_ja_jp_security_studies +- openai_mmmlu_default_ja_jp_sociology +- openai_mmmlu_default_ja_jp_us_foreign_policy +- openai_mmmlu_default_ja_jp_virology +- openai_mmmlu_default_ja_jp_world_religions +- openai_mmmlu_default_ko_kr_abstract_algebra +- openai_mmmlu_default_ko_kr_anatomy +- openai_mmmlu_default_ko_kr_astronomy +- openai_mmmlu_default_ko_kr_business_ethics +- openai_mmmlu_default_ko_kr_clinical_knowledge +- openai_mmmlu_default_ko_kr_college_biology +- openai_mmmlu_default_ko_kr_college_chemistry +- openai_mmmlu_default_ko_kr_college_computer_science +- openai_mmmlu_default_ko_kr_college_mathematics +- openai_mmmlu_default_ko_kr_college_medicine +- openai_mmmlu_default_ko_kr_college_physics +- openai_mmmlu_default_ko_kr_computer_security +- openai_mmmlu_default_ko_kr_conceptual_physics +- openai_mmmlu_default_ko_kr_econometrics +- openai_mmmlu_default_ko_kr_electrical_engineering +- openai_mmmlu_default_ko_kr_elementary_mathematics +- openai_mmmlu_default_ko_kr_formal_logic +- openai_mmmlu_default_ko_kr_global_facts +- openai_mmmlu_default_ko_kr_high_school_biology +- openai_mmmlu_default_ko_kr_high_school_chemistry +- openai_mmmlu_default_ko_kr_high_school_computer_science +- openai_mmmlu_default_ko_kr_high_school_european_history +- openai_mmmlu_default_ko_kr_high_school_geography +- openai_mmmlu_default_ko_kr_high_school_government_and_politics +- openai_mmmlu_default_ko_kr_high_school_macroeconomics +- openai_mmmlu_default_ko_kr_high_school_mathematics +- openai_mmmlu_default_ko_kr_high_school_microeconomics +- openai_mmmlu_default_ko_kr_high_school_physics +- openai_mmmlu_default_ko_kr_high_school_psychology +- openai_mmmlu_default_ko_kr_high_school_statistics +- openai_mmmlu_default_ko_kr_high_school_us_history +- openai_mmmlu_default_ko_kr_high_school_world_history +- openai_mmmlu_default_ko_kr_human_aging +- openai_mmmlu_default_ko_kr_human_sexuality +- openai_mmmlu_default_ko_kr_international_law +- openai_mmmlu_default_ko_kr_jurisprudence +- openai_mmmlu_default_ko_kr_logical_fallacies +- openai_mmmlu_default_ko_kr_machine_learning +- openai_mmmlu_default_ko_kr_management +- openai_mmmlu_default_ko_kr_marketing +- openai_mmmlu_default_ko_kr_medical_genetics +- openai_mmmlu_default_ko_kr_miscellaneous +- openai_mmmlu_default_ko_kr_moral_disputes +- openai_mmmlu_default_ko_kr_moral_scenarios +- openai_mmmlu_default_ko_kr_nutrition +- openai_mmmlu_default_ko_kr_philosophy +- openai_mmmlu_default_ko_kr_prehistory +- openai_mmmlu_default_ko_kr_professional_accounting +- openai_mmmlu_default_ko_kr_professional_law +- openai_mmmlu_default_ko_kr_professional_medicine +- openai_mmmlu_default_ko_kr_professional_psychology +- openai_mmmlu_default_ko_kr_public_relations +- openai_mmmlu_default_ko_kr_security_studies +- openai_mmmlu_default_ko_kr_sociology +- openai_mmmlu_default_ko_kr_us_foreign_policy +- openai_mmmlu_default_ko_kr_virology +- openai_mmmlu_default_ko_kr_world_religions +- openai_mmmlu_default_pt_br_abstract_algebra +- openai_mmmlu_default_pt_br_anatomy +- openai_mmmlu_default_pt_br_astronomy +- openai_mmmlu_default_pt_br_business_ethics +- openai_mmmlu_default_pt_br_clinical_knowledge +- openai_mmmlu_default_pt_br_college_biology +- openai_mmmlu_default_pt_br_college_chemistry +- openai_mmmlu_default_pt_br_college_computer_science +- openai_mmmlu_default_pt_br_college_mathematics +- openai_mmmlu_default_pt_br_college_medicine +- openai_mmmlu_default_pt_br_college_physics +- openai_mmmlu_default_pt_br_computer_security +- openai_mmmlu_default_pt_br_conceptual_physics +- openai_mmmlu_default_pt_br_econometrics +- openai_mmmlu_default_pt_br_electrical_engineering +- openai_mmmlu_default_pt_br_elementary_mathematics +- openai_mmmlu_default_pt_br_formal_logic +- openai_mmmlu_default_pt_br_global_facts +- openai_mmmlu_default_pt_br_high_school_biology +- openai_mmmlu_default_pt_br_high_school_chemistry +- openai_mmmlu_default_pt_br_high_school_computer_science +- openai_mmmlu_default_pt_br_high_school_european_history +- openai_mmmlu_default_pt_br_high_school_geography +- openai_mmmlu_default_pt_br_high_school_government_and_politics +- openai_mmmlu_default_pt_br_high_school_macroeconomics +- openai_mmmlu_default_pt_br_high_school_mathematics +- openai_mmmlu_default_pt_br_high_school_microeconomics +- openai_mmmlu_default_pt_br_high_school_physics +- openai_mmmlu_default_pt_br_high_school_psychology +- openai_mmmlu_default_pt_br_high_school_statistics +- openai_mmmlu_default_pt_br_high_school_us_history +- openai_mmmlu_default_pt_br_high_school_world_history +- openai_mmmlu_default_pt_br_human_aging +- openai_mmmlu_default_pt_br_human_sexuality +- openai_mmmlu_default_pt_br_international_law +- openai_mmmlu_default_pt_br_jurisprudence +- openai_mmmlu_default_pt_br_logical_fallacies +- openai_mmmlu_default_pt_br_machine_learning +- openai_mmmlu_default_pt_br_management +- openai_mmmlu_default_pt_br_marketing +- openai_mmmlu_default_pt_br_medical_genetics +- openai_mmmlu_default_pt_br_miscellaneous +- openai_mmmlu_default_pt_br_moral_disputes +- openai_mmmlu_default_pt_br_moral_scenarios +- openai_mmmlu_default_pt_br_nutrition +- openai_mmmlu_default_pt_br_philosophy +- openai_mmmlu_default_pt_br_prehistory +- openai_mmmlu_default_pt_br_professional_accounting +- openai_mmmlu_default_pt_br_professional_law +- openai_mmmlu_default_pt_br_professional_medicine +- openai_mmmlu_default_pt_br_professional_psychology +- openai_mmmlu_default_pt_br_public_relations +- openai_mmmlu_default_pt_br_security_studies +- openai_mmmlu_default_pt_br_sociology +- openai_mmmlu_default_pt_br_us_foreign_policy +- openai_mmmlu_default_pt_br_virology +- openai_mmmlu_default_pt_br_world_religions +- openai_mmmlu_default_zh_cn_abstract_algebra +- openai_mmmlu_default_zh_cn_anatomy +- openai_mmmlu_default_zh_cn_astronomy +- openai_mmmlu_default_zh_cn_business_ethics +- openai_mmmlu_default_zh_cn_clinical_knowledge +- openai_mmmlu_default_zh_cn_college_biology +- openai_mmmlu_default_zh_cn_college_chemistry +- openai_mmmlu_default_zh_cn_college_computer_science +- openai_mmmlu_default_zh_cn_college_mathematics +- openai_mmmlu_default_zh_cn_college_medicine +- openai_mmmlu_default_zh_cn_college_physics +- openai_mmmlu_default_zh_cn_computer_security +- openai_mmmlu_default_zh_cn_conceptual_physics +- openai_mmmlu_default_zh_cn_econometrics +- openai_mmmlu_default_zh_cn_electrical_engineering +- openai_mmmlu_default_zh_cn_elementary_mathematics +- openai_mmmlu_default_zh_cn_formal_logic +- openai_mmmlu_default_zh_cn_global_facts +- openai_mmmlu_default_zh_cn_high_school_biology +- openai_mmmlu_default_zh_cn_high_school_chemistry +- openai_mmmlu_default_zh_cn_high_school_computer_science +- openai_mmmlu_default_zh_cn_high_school_european_history +- openai_mmmlu_default_zh_cn_high_school_geography +- openai_mmmlu_default_zh_cn_high_school_government_and_politics +- openai_mmmlu_default_zh_cn_high_school_macroeconomics +- openai_mmmlu_default_zh_cn_high_school_mathematics +- openai_mmmlu_default_zh_cn_high_school_microeconomics +- openai_mmmlu_default_zh_cn_high_school_physics +- openai_mmmlu_default_zh_cn_high_school_psychology +- openai_mmmlu_default_zh_cn_high_school_statistics +- openai_mmmlu_default_zh_cn_high_school_us_history +- openai_mmmlu_default_zh_cn_high_school_world_history +- openai_mmmlu_default_zh_cn_human_aging +- openai_mmmlu_default_zh_cn_human_sexuality +- openai_mmmlu_default_zh_cn_international_law +- openai_mmmlu_default_zh_cn_jurisprudence +- openai_mmmlu_default_zh_cn_logical_fallacies +- openai_mmmlu_default_zh_cn_machine_learning +- openai_mmmlu_default_zh_cn_management +- openai_mmmlu_default_zh_cn_marketing +- openai_mmmlu_default_zh_cn_medical_genetics +- openai_mmmlu_default_zh_cn_miscellaneous +- openai_mmmlu_default_zh_cn_moral_disputes +- openai_mmmlu_default_zh_cn_moral_scenarios +- openai_mmmlu_default_zh_cn_nutrition +- openai_mmmlu_default_zh_cn_philosophy +- openai_mmmlu_default_zh_cn_prehistory +- openai_mmmlu_default_zh_cn_professional_accounting +- openai_mmmlu_default_zh_cn_professional_law +- openai_mmmlu_default_zh_cn_professional_medicine +- openai_mmmlu_default_zh_cn_professional_psychology +- openai_mmmlu_default_zh_cn_public_relations +- openai_mmmlu_default_zh_cn_security_studies +- openai_mmmlu_default_zh_cn_sociology +- openai_mmmlu_default_zh_cn_us_foreign_policy +- openai_mmmlu_default_zh_cn_virology +- openai_mmmlu_default_zh_cn_world_religions +- openai_mmmlu_default_sw_ke_abstract_algebra +- openai_mmmlu_default_sw_ke_anatomy +- openai_mmmlu_default_sw_ke_astronomy +- openai_mmmlu_default_sw_ke_business_ethics +- openai_mmmlu_default_sw_ke_clinical_knowledge +- openai_mmmlu_default_sw_ke_college_biology +- openai_mmmlu_default_sw_ke_college_chemistry +- openai_mmmlu_default_sw_ke_college_computer_science +- openai_mmmlu_default_sw_ke_college_mathematics +- openai_mmmlu_default_sw_ke_college_medicine +- openai_mmmlu_default_sw_ke_college_physics +- openai_mmmlu_default_sw_ke_computer_security +- openai_mmmlu_default_sw_ke_conceptual_physics +- openai_mmmlu_default_sw_ke_econometrics +- openai_mmmlu_default_sw_ke_electrical_engineering +- openai_mmmlu_default_sw_ke_elementary_mathematics +- openai_mmmlu_default_sw_ke_formal_logic +- openai_mmmlu_default_sw_ke_global_facts +- openai_mmmlu_default_sw_ke_high_school_biology +- openai_mmmlu_default_sw_ke_high_school_chemistry +- openai_mmmlu_default_sw_ke_high_school_computer_science +- openai_mmmlu_default_sw_ke_high_school_european_history +- openai_mmmlu_default_sw_ke_high_school_geography +- openai_mmmlu_default_sw_ke_high_school_government_and_politics +- openai_mmmlu_default_sw_ke_high_school_macroeconomics +- openai_mmmlu_default_sw_ke_high_school_mathematics +- openai_mmmlu_default_sw_ke_high_school_microeconomics +- openai_mmmlu_default_sw_ke_high_school_physics +- openai_mmmlu_default_sw_ke_high_school_psychology +- openai_mmmlu_default_sw_ke_high_school_statistics +- openai_mmmlu_default_sw_ke_high_school_us_history +- openai_mmmlu_default_sw_ke_high_school_world_history +- openai_mmmlu_default_sw_ke_human_aging +- openai_mmmlu_default_sw_ke_human_sexuality +- openai_mmmlu_default_sw_ke_international_law +- openai_mmmlu_default_sw_ke_jurisprudence +- openai_mmmlu_default_sw_ke_logical_fallacies +- openai_mmmlu_default_sw_ke_machine_learning +- openai_mmmlu_default_sw_ke_management +- openai_mmmlu_default_sw_ke_marketing +- openai_mmmlu_default_sw_ke_medical_genetics +- openai_mmmlu_default_sw_ke_miscellaneous +- openai_mmmlu_default_sw_ke_moral_disputes +- openai_mmmlu_default_sw_ke_moral_scenarios +- openai_mmmlu_default_sw_ke_nutrition +- openai_mmmlu_default_sw_ke_philosophy +- openai_mmmlu_default_sw_ke_prehistory +- openai_mmmlu_default_sw_ke_professional_accounting +- openai_mmmlu_default_sw_ke_professional_law +- openai_mmmlu_default_sw_ke_professional_medicine +- openai_mmmlu_default_sw_ke_professional_psychology +- openai_mmmlu_default_sw_ke_public_relations +- openai_mmmlu_default_sw_ke_security_studies +- openai_mmmlu_default_sw_ke_sociology +- openai_mmmlu_default_sw_ke_us_foreign_policy +- openai_mmmlu_default_sw_ke_virology +- openai_mmmlu_default_sw_ke_world_religions +- openai_mmmlu_default_yo_ng_abstract_algebra +- openai_mmmlu_default_yo_ng_anatomy +- openai_mmmlu_default_yo_ng_astronomy +- openai_mmmlu_default_yo_ng_business_ethics +- openai_mmmlu_default_yo_ng_clinical_knowledge +- openai_mmmlu_default_yo_ng_college_biology +- openai_mmmlu_default_yo_ng_college_chemistry +- openai_mmmlu_default_yo_ng_college_computer_science +- openai_mmmlu_default_yo_ng_college_mathematics +- openai_mmmlu_default_yo_ng_college_medicine +- openai_mmmlu_default_yo_ng_college_physics +- openai_mmmlu_default_yo_ng_computer_security +- openai_mmmlu_default_yo_ng_conceptual_physics +- openai_mmmlu_default_yo_ng_econometrics +- openai_mmmlu_default_yo_ng_electrical_engineering +- openai_mmmlu_default_yo_ng_elementary_mathematics +- openai_mmmlu_default_yo_ng_formal_logic +- openai_mmmlu_default_yo_ng_global_facts +- openai_mmmlu_default_yo_ng_high_school_biology +- openai_mmmlu_default_yo_ng_high_school_chemistry +- openai_mmmlu_default_yo_ng_high_school_computer_science +- openai_mmmlu_default_yo_ng_high_school_european_history +- openai_mmmlu_default_yo_ng_high_school_geography +- openai_mmmlu_default_yo_ng_high_school_government_and_politics +- openai_mmmlu_default_yo_ng_high_school_macroeconomics +- openai_mmmlu_default_yo_ng_high_school_mathematics +- openai_mmmlu_default_yo_ng_high_school_microeconomics +- openai_mmmlu_default_yo_ng_high_school_physics +- openai_mmmlu_default_yo_ng_high_school_psychology +- openai_mmmlu_default_yo_ng_high_school_statistics +- openai_mmmlu_default_yo_ng_high_school_us_history +- openai_mmmlu_default_yo_ng_high_school_world_history +- openai_mmmlu_default_yo_ng_human_aging +- openai_mmmlu_default_yo_ng_human_sexuality +- openai_mmmlu_default_yo_ng_international_law +- openai_mmmlu_default_yo_ng_jurisprudence +- openai_mmmlu_default_yo_ng_logical_fallacies +- openai_mmmlu_default_yo_ng_machine_learning +- openai_mmmlu_default_yo_ng_management +- openai_mmmlu_default_yo_ng_marketing +- openai_mmmlu_default_yo_ng_medical_genetics +- openai_mmmlu_default_yo_ng_miscellaneous +- openai_mmmlu_default_yo_ng_moral_disputes +- openai_mmmlu_default_yo_ng_moral_scenarios +- openai_mmmlu_default_yo_ng_nutrition +- openai_mmmlu_default_yo_ng_philosophy +- openai_mmmlu_default_yo_ng_prehistory +- openai_mmmlu_default_yo_ng_professional_accounting +- openai_mmmlu_default_yo_ng_professional_law +- openai_mmmlu_default_yo_ng_professional_medicine +- openai_mmmlu_default_yo_ng_professional_psychology +- openai_mmmlu_default_yo_ng_public_relations +- openai_mmmlu_default_yo_ng_security_studies +- openai_mmmlu_default_yo_ng_sociology +- openai_mmmlu_default_yo_ng_us_foreign_policy +- openai_mmmlu_default_yo_ng_virology +- openai_mmmlu_default_yo_ng_world_religions +- openai_mmmlu_default_en_us_abstract_algebra +- openai_mmmlu_default_en_us_anatomy +- openai_mmmlu_default_en_us_astronomy +- openai_mmmlu_default_en_us_business_ethics +- openai_mmmlu_default_en_us_clinical_knowledge +- openai_mmmlu_default_en_us_college_biology +- openai_mmmlu_default_en_us_college_chemistry +- openai_mmmlu_default_en_us_college_computer_science +- openai_mmmlu_default_en_us_college_mathematics +- openai_mmmlu_default_en_us_college_medicine +- openai_mmmlu_default_en_us_college_physics +- openai_mmmlu_default_en_us_computer_security +- openai_mmmlu_default_en_us_conceptual_physics +- openai_mmmlu_default_en_us_econometrics +- openai_mmmlu_default_en_us_electrical_engineering +- openai_mmmlu_default_en_us_elementary_mathematics +- openai_mmmlu_default_en_us_formal_logic +- openai_mmmlu_default_en_us_global_facts +- openai_mmmlu_default_en_us_high_school_biology +- openai_mmmlu_default_en_us_high_school_chemistry +- openai_mmmlu_default_en_us_high_school_computer_science +- openai_mmmlu_default_en_us_high_school_european_history +- openai_mmmlu_default_en_us_high_school_geography +- openai_mmmlu_default_en_us_high_school_government_and_politics +- openai_mmmlu_default_en_us_high_school_macroeconomics +- openai_mmmlu_default_en_us_high_school_mathematics +- openai_mmmlu_default_en_us_high_school_microeconomics +- openai_mmmlu_default_en_us_high_school_physics +- openai_mmmlu_default_en_us_high_school_psychology +- openai_mmmlu_default_en_us_high_school_statistics +- openai_mmmlu_default_en_us_high_school_us_history +- openai_mmmlu_default_en_us_high_school_world_history +- openai_mmmlu_default_en_us_human_aging +- openai_mmmlu_default_en_us_human_sexuality +- openai_mmmlu_default_en_us_international_law +- openai_mmmlu_default_en_us_jurisprudence +- openai_mmmlu_default_en_us_logical_fallacies +- openai_mmmlu_default_en_us_machine_learning +- openai_mmmlu_default_en_us_management +- openai_mmmlu_default_en_us_marketing +- openai_mmmlu_default_en_us_medical_genetics +- openai_mmmlu_default_en_us_miscellaneous +- openai_mmmlu_default_en_us_moral_disputes +- openai_mmmlu_default_en_us_moral_scenarios +- openai_mmmlu_default_en_us_nutrition +- openai_mmmlu_default_en_us_philosophy +- openai_mmmlu_default_en_us_prehistory +- openai_mmmlu_default_en_us_professional_accounting +- openai_mmmlu_default_en_us_professional_law +- openai_mmmlu_default_en_us_professional_medicine +- openai_mmmlu_default_en_us_professional_psychology +- openai_mmmlu_default_en_us_public_relations +- openai_mmmlu_default_en_us_security_studies +- openai_mmmlu_default_en_us_sociology +- openai_mmmlu_default_en_us_us_foreign_policy +- openai_mmmlu_default_en_us_virology +- openai_mmmlu_default_en_us_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/_default_ar_xy_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/_default_ar_xy_group_info.yaml new file mode 100644 index 0000000000..6f05e303ab --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/_default_ar_xy_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_ar_xy +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_ar_xy_abstract_algebra +- openai_mmmlu_default_ar_xy_anatomy +- openai_mmmlu_default_ar_xy_astronomy +- openai_mmmlu_default_ar_xy_business_ethics +- openai_mmmlu_default_ar_xy_clinical_knowledge +- openai_mmmlu_default_ar_xy_college_biology +- openai_mmmlu_default_ar_xy_college_chemistry +- openai_mmmlu_default_ar_xy_college_computer_science +- openai_mmmlu_default_ar_xy_college_mathematics +- openai_mmmlu_default_ar_xy_college_medicine +- openai_mmmlu_default_ar_xy_college_physics +- openai_mmmlu_default_ar_xy_computer_security +- openai_mmmlu_default_ar_xy_conceptual_physics +- openai_mmmlu_default_ar_xy_econometrics +- openai_mmmlu_default_ar_xy_electrical_engineering +- openai_mmmlu_default_ar_xy_elementary_mathematics +- openai_mmmlu_default_ar_xy_formal_logic +- openai_mmmlu_default_ar_xy_global_facts +- openai_mmmlu_default_ar_xy_high_school_biology +- openai_mmmlu_default_ar_xy_high_school_chemistry +- openai_mmmlu_default_ar_xy_high_school_computer_science +- openai_mmmlu_default_ar_xy_high_school_european_history +- openai_mmmlu_default_ar_xy_high_school_geography +- openai_mmmlu_default_ar_xy_high_school_government_and_politics +- openai_mmmlu_default_ar_xy_high_school_macroeconomics +- openai_mmmlu_default_ar_xy_high_school_mathematics +- openai_mmmlu_default_ar_xy_high_school_microeconomics +- openai_mmmlu_default_ar_xy_high_school_physics +- openai_mmmlu_default_ar_xy_high_school_psychology +- openai_mmmlu_default_ar_xy_high_school_statistics +- openai_mmmlu_default_ar_xy_high_school_us_history +- openai_mmmlu_default_ar_xy_high_school_world_history +- openai_mmmlu_default_ar_xy_human_aging +- openai_mmmlu_default_ar_xy_human_sexuality +- openai_mmmlu_default_ar_xy_international_law +- openai_mmmlu_default_ar_xy_jurisprudence +- openai_mmmlu_default_ar_xy_logical_fallacies +- openai_mmmlu_default_ar_xy_machine_learning +- openai_mmmlu_default_ar_xy_management +- openai_mmmlu_default_ar_xy_marketing +- openai_mmmlu_default_ar_xy_medical_genetics +- openai_mmmlu_default_ar_xy_miscellaneous +- openai_mmmlu_default_ar_xy_moral_disputes +- openai_mmmlu_default_ar_xy_moral_scenarios +- openai_mmmlu_default_ar_xy_nutrition +- openai_mmmlu_default_ar_xy_philosophy +- openai_mmmlu_default_ar_xy_prehistory +- openai_mmmlu_default_ar_xy_professional_accounting +- openai_mmmlu_default_ar_xy_professional_law +- openai_mmmlu_default_ar_xy_professional_medicine +- openai_mmmlu_default_ar_xy_professional_psychology +- openai_mmmlu_default_ar_xy_public_relations +- openai_mmmlu_default_ar_xy_security_studies +- openai_mmmlu_default_ar_xy_sociology +- openai_mmmlu_default_ar_xy_us_foreign_policy +- openai_mmmlu_default_ar_xy_virology +- openai_mmmlu_default_ar_xy_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..db0587889c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_abstract_algebra" +"task_alias": "ar_xy abstract algebra" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..d7b12c0058 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_anatomy" +"task_alias": "ar_xy anatomy" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..840382bff5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_astronomy" +"task_alias": "ar_xy astronomy" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..ab44a39046 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_business_ethics" +"task_alias": "ar_xy business ethics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..fa1c0f0ec8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_clinical_knowledge" +"task_alias": "ar_xy clinical knowledge" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..308b7b010c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_college_biology" +"task_alias": "ar_xy college biology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..0aee42c85c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_college_chemistry" +"task_alias": "ar_xy college chemistry" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..a547a6b089 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_college_computer_science" +"task_alias": "ar_xy college computer science" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..f07bdc1770 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_college_mathematics" +"task_alias": "ar_xy college mathematics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..7923ff8b49 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_college_medicine" +"task_alias": "ar_xy college medicine" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..31483a5f80 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_college_physics" +"task_alias": "ar_xy college physics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..054da35cc9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_computer_security" +"task_alias": "ar_xy computer security" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..ba68370ce6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_conceptual_physics" +"task_alias": "ar_xy conceptual physics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..f62527e9ce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_econometrics" +"task_alias": "ar_xy econometrics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..b087c9fb6b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_electrical_engineering" +"task_alias": "ar_xy electrical engineering" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..ab586865c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_elementary_mathematics" +"task_alias": "ar_xy elementary mathematics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..e138278499 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_formal_logic" +"task_alias": "ar_xy formal logic" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..4ae8e755c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_global_facts" +"task_alias": "ar_xy global facts" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..17144057b6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_biology" +"task_alias": "ar_xy high school biology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..cb262b9721 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_chemistry" +"task_alias": "ar_xy high school chemistry" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..bd0b89d503 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_computer_science" +"task_alias": "ar_xy high school computer science" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..fec11886fc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_high_school_european_history" +"task_alias": "ar_xy high school european history" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..2dedde96b7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_high_school_geography" +"task_alias": "ar_xy high school geography" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..95ef59ddf4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_high_school_government_and_politics" +"task_alias": "ar_xy high school government and politics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..8791b2368b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_high_school_macroeconomics" +"task_alias": "ar_xy high school macroeconomics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..b29dc7a1a4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_mathematics" +"task_alias": "ar_xy high school mathematics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..1bc55f72f2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_high_school_microeconomics" +"task_alias": "ar_xy high school microeconomics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..3fe6a097e5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_physics" +"task_alias": "ar_xy high school physics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..9842a763ea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_high_school_psychology" +"task_alias": "ar_xy high school psychology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..3cacf8424d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_high_school_statistics" +"task_alias": "ar_xy high school statistics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..d382550c40 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_high_school_us_history" +"task_alias": "ar_xy high school us history" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..f082283949 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_high_school_world_history" +"task_alias": "ar_xy high school world history" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..0bcfa14989 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_human_aging" +"task_alias": "ar_xy human aging" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..b355f0d99b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_human_sexuality" +"task_alias": "ar_xy human sexuality" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..8c62b6f1ce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_international_law" +"task_alias": "ar_xy international law" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..02020abc19 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_jurisprudence" +"task_alias": "ar_xy jurisprudence" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..718baf20a4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_logical_fallacies" +"task_alias": "ar_xy logical fallacies" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..198b872e42 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_stem" +"task": "openai_mmmlu_default_ar_xy_machine_learning" +"task_alias": "ar_xy machine learning" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..b0331a359f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_management" +"task_alias": "ar_xy management" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..0a6ad359e5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_marketing" +"task_alias": "ar_xy marketing" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..6872d7f30c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_medical_genetics" +"task_alias": "ar_xy medical genetics" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..e823ffeb55 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_miscellaneous" +"task_alias": "ar_xy miscellaneous" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..0eb49db8cd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_moral_disputes" +"task_alias": "ar_xy moral disputes" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..12ed3f2e37 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_moral_scenarios" +"task_alias": "ar_xy moral scenarios" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..f50d702df1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_nutrition" +"task_alias": "ar_xy nutrition" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..f50583f06c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_philosophy" +"task_alias": "ar_xy philosophy" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..4d2120299a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_prehistory" +"task_alias": "ar_xy prehistory" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..26a147f8af --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_professional_accounting" +"task_alias": "ar_xy professional accounting" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..b4981a7b5d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_professional_law" +"task_alias": "ar_xy professional law" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..b19a27ba13 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_professional_medicine" +"task_alias": "ar_xy professional medicine" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..dde830bcf3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_professional_psychology" +"task_alias": "ar_xy professional psychology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..a460f20f7b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_public_relations" +"task_alias": "ar_xy public relations" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..26ab069edb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_security_studies" +"task_alias": "ar_xy security studies" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..84f55d1fdc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_sociology" +"task_alias": "ar_xy sociology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..e915f8c204 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_social_sciences" +"task": "openai_mmmlu_default_ar_xy_us_foreign_policy" +"task_alias": "ar_xy us foreign policy" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..b1bed66955 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_other" +"task": "openai_mmmlu_default_ar_xy_virology" +"task_alias": "ar_xy virology" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..9a0bf4d77a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ar_xy/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nالإجابة:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ar_xy_humanities" +"task": "openai_mmmlu_default_ar_xy_world_religions" +"task_alias": "ar_xy world religions" +"test_split": "AR_XY" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/_default_bn_bd_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/_default_bn_bd_group_info.yaml new file mode 100644 index 0000000000..52af6c0748 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/_default_bn_bd_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_bn_bd +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_bn_bd_abstract_algebra +- openai_mmmlu_default_bn_bd_anatomy +- openai_mmmlu_default_bn_bd_astronomy +- openai_mmmlu_default_bn_bd_business_ethics +- openai_mmmlu_default_bn_bd_clinical_knowledge +- openai_mmmlu_default_bn_bd_college_biology +- openai_mmmlu_default_bn_bd_college_chemistry +- openai_mmmlu_default_bn_bd_college_computer_science +- openai_mmmlu_default_bn_bd_college_mathematics +- openai_mmmlu_default_bn_bd_college_medicine +- openai_mmmlu_default_bn_bd_college_physics +- openai_mmmlu_default_bn_bd_computer_security +- openai_mmmlu_default_bn_bd_conceptual_physics +- openai_mmmlu_default_bn_bd_econometrics +- openai_mmmlu_default_bn_bd_electrical_engineering +- openai_mmmlu_default_bn_bd_elementary_mathematics +- openai_mmmlu_default_bn_bd_formal_logic +- openai_mmmlu_default_bn_bd_global_facts +- openai_mmmlu_default_bn_bd_high_school_biology +- openai_mmmlu_default_bn_bd_high_school_chemistry +- openai_mmmlu_default_bn_bd_high_school_computer_science +- openai_mmmlu_default_bn_bd_high_school_european_history +- openai_mmmlu_default_bn_bd_high_school_geography +- openai_mmmlu_default_bn_bd_high_school_government_and_politics +- openai_mmmlu_default_bn_bd_high_school_macroeconomics +- openai_mmmlu_default_bn_bd_high_school_mathematics +- openai_mmmlu_default_bn_bd_high_school_microeconomics +- openai_mmmlu_default_bn_bd_high_school_physics +- openai_mmmlu_default_bn_bd_high_school_psychology +- openai_mmmlu_default_bn_bd_high_school_statistics +- openai_mmmlu_default_bn_bd_high_school_us_history +- openai_mmmlu_default_bn_bd_high_school_world_history +- openai_mmmlu_default_bn_bd_human_aging +- openai_mmmlu_default_bn_bd_human_sexuality +- openai_mmmlu_default_bn_bd_international_law +- openai_mmmlu_default_bn_bd_jurisprudence +- openai_mmmlu_default_bn_bd_logical_fallacies +- openai_mmmlu_default_bn_bd_machine_learning +- openai_mmmlu_default_bn_bd_management +- openai_mmmlu_default_bn_bd_marketing +- openai_mmmlu_default_bn_bd_medical_genetics +- openai_mmmlu_default_bn_bd_miscellaneous +- openai_mmmlu_default_bn_bd_moral_disputes +- openai_mmmlu_default_bn_bd_moral_scenarios +- openai_mmmlu_default_bn_bd_nutrition +- openai_mmmlu_default_bn_bd_philosophy +- openai_mmmlu_default_bn_bd_prehistory +- openai_mmmlu_default_bn_bd_professional_accounting +- openai_mmmlu_default_bn_bd_professional_law +- openai_mmmlu_default_bn_bd_professional_medicine +- openai_mmmlu_default_bn_bd_professional_psychology +- openai_mmmlu_default_bn_bd_public_relations +- openai_mmmlu_default_bn_bd_security_studies +- openai_mmmlu_default_bn_bd_sociology +- openai_mmmlu_default_bn_bd_us_foreign_policy +- openai_mmmlu_default_bn_bd_virology +- openai_mmmlu_default_bn_bd_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..941df9fadd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_abstract_algebra" +"task_alias": "bn_bd abstract algebra" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..9af36d5883 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_anatomy" +"task_alias": "bn_bd anatomy" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..e878697a9a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_astronomy" +"task_alias": "bn_bd astronomy" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..efcacf9765 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_business_ethics" +"task_alias": "bn_bd business ethics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..b98b79450f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_clinical_knowledge" +"task_alias": "bn_bd clinical knowledge" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..a2772757f9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_college_biology" +"task_alias": "bn_bd college biology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..befe7a3e8e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_college_chemistry" +"task_alias": "bn_bd college chemistry" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..65a9fe1ce2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_college_computer_science" +"task_alias": "bn_bd college computer science" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..e0c669e899 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_college_mathematics" +"task_alias": "bn_bd college mathematics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..439ee588c0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_college_medicine" +"task_alias": "bn_bd college medicine" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..4102c6bd28 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_college_physics" +"task_alias": "bn_bd college physics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..5973355549 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_computer_security" +"task_alias": "bn_bd computer security" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..a0541db904 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_conceptual_physics" +"task_alias": "bn_bd conceptual physics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..5a1af088bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_econometrics" +"task_alias": "bn_bd econometrics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..035338a2c7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_electrical_engineering" +"task_alias": "bn_bd electrical engineering" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..cc75df8ddf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_elementary_mathematics" +"task_alias": "bn_bd elementary mathematics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..a2a5a875e3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_formal_logic" +"task_alias": "bn_bd formal logic" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..aea1daf868 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_global_facts" +"task_alias": "bn_bd global facts" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..b5f4bc3133 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_biology" +"task_alias": "bn_bd high school biology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..1960e51803 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_chemistry" +"task_alias": "bn_bd high school chemistry" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..b7ec1f7ed7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_computer_science" +"task_alias": "bn_bd high school computer science" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..7138fa4421 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_high_school_european_history" +"task_alias": "bn_bd high school european history" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..7decb3d602 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_high_school_geography" +"task_alias": "bn_bd high school geography" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..655b672465 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_high_school_government_and_politics" +"task_alias": "bn_bd high school government and politics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..8de2672716 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_high_school_macroeconomics" +"task_alias": "bn_bd high school macroeconomics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..8aa6677fd2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_mathematics" +"task_alias": "bn_bd high school mathematics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..478584b028 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_high_school_microeconomics" +"task_alias": "bn_bd high school microeconomics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..4d84de05c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_physics" +"task_alias": "bn_bd high school physics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..294e7d46cb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_high_school_psychology" +"task_alias": "bn_bd high school psychology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..787e72c857 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_high_school_statistics" +"task_alias": "bn_bd high school statistics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..25d0cee8b6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_high_school_us_history" +"task_alias": "bn_bd high school us history" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..83b1de7048 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_high_school_world_history" +"task_alias": "bn_bd high school world history" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..b5c0a92ea7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_human_aging" +"task_alias": "bn_bd human aging" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..ab3192f7fe --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_human_sexuality" +"task_alias": "bn_bd human sexuality" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..2cb2f2ba33 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_international_law" +"task_alias": "bn_bd international law" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..a1d978375e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_jurisprudence" +"task_alias": "bn_bd jurisprudence" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..f33cb9be11 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_logical_fallacies" +"task_alias": "bn_bd logical fallacies" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..9b6c75fdfb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_stem" +"task": "openai_mmmlu_default_bn_bd_machine_learning" +"task_alias": "bn_bd machine learning" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..45331449fd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_management" +"task_alias": "bn_bd management" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..7b8824f30d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_marketing" +"task_alias": "bn_bd marketing" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..72751719c8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_medical_genetics" +"task_alias": "bn_bd medical genetics" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..2b4c164920 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_miscellaneous" +"task_alias": "bn_bd miscellaneous" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..e460d51d97 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_moral_disputes" +"task_alias": "bn_bd moral disputes" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..b903e56572 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_moral_scenarios" +"task_alias": "bn_bd moral scenarios" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..ea27544b54 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_nutrition" +"task_alias": "bn_bd nutrition" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..3497cdb444 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_philosophy" +"task_alias": "bn_bd philosophy" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..105f341973 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_prehistory" +"task_alias": "bn_bd prehistory" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..fdf2414ffb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_professional_accounting" +"task_alias": "bn_bd professional accounting" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..24498a2867 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_professional_law" +"task_alias": "bn_bd professional law" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..33e1e67c0a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_professional_medicine" +"task_alias": "bn_bd professional medicine" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..3a4ea14722 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_professional_psychology" +"task_alias": "bn_bd professional psychology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..9a380fb8e0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_public_relations" +"task_alias": "bn_bd public relations" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..472ee2442e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_security_studies" +"task_alias": "bn_bd security studies" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..6cf8a8dc72 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_sociology" +"task_alias": "bn_bd sociology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..6ff2e1906f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_social_sciences" +"task": "openai_mmmlu_default_bn_bd_us_foreign_policy" +"task_alias": "bn_bd us foreign policy" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..ae1c962048 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_other" +"task": "openai_mmmlu_default_bn_bd_virology" +"task_alias": "bn_bd virology" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..2671defb65 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/bn_bd/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nউত্তর:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_bn_bd_humanities" +"task": "openai_mmmlu_default_bn_bd_world_religions" +"task_alias": "bn_bd world religions" +"test_split": "BN_BD" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/_default_de_de_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/_default_de_de_group_info.yaml new file mode 100644 index 0000000000..c361a889f5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/_default_de_de_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_de_de +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_de_de_abstract_algebra +- openai_mmmlu_default_de_de_anatomy +- openai_mmmlu_default_de_de_astronomy +- openai_mmmlu_default_de_de_business_ethics +- openai_mmmlu_default_de_de_clinical_knowledge +- openai_mmmlu_default_de_de_college_biology +- openai_mmmlu_default_de_de_college_chemistry +- openai_mmmlu_default_de_de_college_computer_science +- openai_mmmlu_default_de_de_college_mathematics +- openai_mmmlu_default_de_de_college_medicine +- openai_mmmlu_default_de_de_college_physics +- openai_mmmlu_default_de_de_computer_security +- openai_mmmlu_default_de_de_conceptual_physics +- openai_mmmlu_default_de_de_econometrics +- openai_mmmlu_default_de_de_electrical_engineering +- openai_mmmlu_default_de_de_elementary_mathematics +- openai_mmmlu_default_de_de_formal_logic +- openai_mmmlu_default_de_de_global_facts +- openai_mmmlu_default_de_de_high_school_biology +- openai_mmmlu_default_de_de_high_school_chemistry +- openai_mmmlu_default_de_de_high_school_computer_science +- openai_mmmlu_default_de_de_high_school_european_history +- openai_mmmlu_default_de_de_high_school_geography +- openai_mmmlu_default_de_de_high_school_government_and_politics +- openai_mmmlu_default_de_de_high_school_macroeconomics +- openai_mmmlu_default_de_de_high_school_mathematics +- openai_mmmlu_default_de_de_high_school_microeconomics +- openai_mmmlu_default_de_de_high_school_physics +- openai_mmmlu_default_de_de_high_school_psychology +- openai_mmmlu_default_de_de_high_school_statistics +- openai_mmmlu_default_de_de_high_school_us_history +- openai_mmmlu_default_de_de_high_school_world_history +- openai_mmmlu_default_de_de_human_aging +- openai_mmmlu_default_de_de_human_sexuality +- openai_mmmlu_default_de_de_international_law +- openai_mmmlu_default_de_de_jurisprudence +- openai_mmmlu_default_de_de_logical_fallacies +- openai_mmmlu_default_de_de_machine_learning +- openai_mmmlu_default_de_de_management +- openai_mmmlu_default_de_de_marketing +- openai_mmmlu_default_de_de_medical_genetics +- openai_mmmlu_default_de_de_miscellaneous +- openai_mmmlu_default_de_de_moral_disputes +- openai_mmmlu_default_de_de_moral_scenarios +- openai_mmmlu_default_de_de_nutrition +- openai_mmmlu_default_de_de_philosophy +- openai_mmmlu_default_de_de_prehistory +- openai_mmmlu_default_de_de_professional_accounting +- openai_mmmlu_default_de_de_professional_law +- openai_mmmlu_default_de_de_professional_medicine +- openai_mmmlu_default_de_de_professional_psychology +- openai_mmmlu_default_de_de_public_relations +- openai_mmmlu_default_de_de_security_studies +- openai_mmmlu_default_de_de_sociology +- openai_mmmlu_default_de_de_us_foreign_policy +- openai_mmmlu_default_de_de_virology +- openai_mmmlu_default_de_de_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..c20518f271 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_abstract_algebra" +"task_alias": "de_de abstract algebra" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..9573504f4a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_anatomy" +"task_alias": "de_de anatomy" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..e62163cfce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_astronomy" +"task_alias": "de_de astronomy" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..644e4e0729 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_business_ethics" +"task_alias": "de_de business ethics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..1b5bdb9a63 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_clinical_knowledge" +"task_alias": "de_de clinical knowledge" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..24f534e26d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_college_biology" +"task_alias": "de_de college biology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..549e1f4808 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_college_chemistry" +"task_alias": "de_de college chemistry" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..b1ba7f7fbf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_college_computer_science" +"task_alias": "de_de college computer science" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..c032038a5e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_college_mathematics" +"task_alias": "de_de college mathematics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..45289d0b6a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_college_medicine" +"task_alias": "de_de college medicine" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..8b585ceac9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_college_physics" +"task_alias": "de_de college physics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..f1ba458a9a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_computer_security" +"task_alias": "de_de computer security" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..ed18e33c05 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_conceptual_physics" +"task_alias": "de_de conceptual physics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..e5e6d99e1e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_econometrics" +"task_alias": "de_de econometrics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..de4114ca85 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_electrical_engineering" +"task_alias": "de_de electrical engineering" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..76651f173b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_elementary_mathematics" +"task_alias": "de_de elementary mathematics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..3ede2104de --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_formal_logic" +"task_alias": "de_de formal logic" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..52c35b74ea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_global_facts" +"task_alias": "de_de global facts" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..24342c0f08 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_biology" +"task_alias": "de_de high school biology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..aea55b61af --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_chemistry" +"task_alias": "de_de high school chemistry" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..6ca93ea5fd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_computer_science" +"task_alias": "de_de high school computer science" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..a4d8b5d734 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_high_school_european_history" +"task_alias": "de_de high school european history" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..6cf3d68394 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_high_school_geography" +"task_alias": "de_de high school geography" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..0eb0095716 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_high_school_government_and_politics" +"task_alias": "de_de high school government and politics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..a76f757a30 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_high_school_macroeconomics" +"task_alias": "de_de high school macroeconomics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..743cf5d354 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_mathematics" +"task_alias": "de_de high school mathematics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..925e48381f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_high_school_microeconomics" +"task_alias": "de_de high school microeconomics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..431de5c24a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_physics" +"task_alias": "de_de high school physics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..51833f786c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_high_school_psychology" +"task_alias": "de_de high school psychology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..606d8ee395 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_high_school_statistics" +"task_alias": "de_de high school statistics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..d9340113a7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_high_school_us_history" +"task_alias": "de_de high school us history" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..13357416a5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_high_school_world_history" +"task_alias": "de_de high school world history" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..523d48d33a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_human_aging" +"task_alias": "de_de human aging" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..e25dde00c5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_human_sexuality" +"task_alias": "de_de human sexuality" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..13bdcf00dc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_international_law" +"task_alias": "de_de international law" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..377effb1e6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_jurisprudence" +"task_alias": "de_de jurisprudence" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..8fa9acc66f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_logical_fallacies" +"task_alias": "de_de logical fallacies" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..128024dbf5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_stem" +"task": "openai_mmmlu_default_de_de_machine_learning" +"task_alias": "de_de machine learning" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..846a9d2db5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_management" +"task_alias": "de_de management" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..54b565a958 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_marketing" +"task_alias": "de_de marketing" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..9ae97531e3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_medical_genetics" +"task_alias": "de_de medical genetics" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..45ba44ab9f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_miscellaneous" +"task_alias": "de_de miscellaneous" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..41a9964b02 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_moral_disputes" +"task_alias": "de_de moral disputes" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..9b23d0f74b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_moral_scenarios" +"task_alias": "de_de moral scenarios" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..9ff92dd4c7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_nutrition" +"task_alias": "de_de nutrition" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..dfa44ad59b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_philosophy" +"task_alias": "de_de philosophy" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..37a3f7a66d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_prehistory" +"task_alias": "de_de prehistory" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..a34f0db0a5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_professional_accounting" +"task_alias": "de_de professional accounting" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..d1a2899f50 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_professional_law" +"task_alias": "de_de professional law" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..18f77f5ce4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_professional_medicine" +"task_alias": "de_de professional medicine" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..51af42f284 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_professional_psychology" +"task_alias": "de_de professional psychology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..a522166dd5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_public_relations" +"task_alias": "de_de public relations" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..7de398ae95 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_security_studies" +"task_alias": "de_de security studies" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..3df1972fc3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_sociology" +"task_alias": "de_de sociology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..8c60f54640 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_social_sciences" +"task": "openai_mmmlu_default_de_de_us_foreign_policy" +"task_alias": "de_de us foreign policy" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..8da437cec9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_other" +"task": "openai_mmmlu_default_de_de_virology" +"task_alias": "de_de virology" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..8a82c859a3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/de_de/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAntwort:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_de_de_humanities" +"task": "openai_mmmlu_default_de_de_world_religions" +"task_alias": "de_de world religions" +"test_split": "DE_DE" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/_default_en_us_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/_default_en_us_group_info.yaml new file mode 100644 index 0000000000..a7e09f6f31 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/_default_en_us_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_en_us +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_en_us_abstract_algebra +- openai_mmmlu_default_en_us_anatomy +- openai_mmmlu_default_en_us_astronomy +- openai_mmmlu_default_en_us_business_ethics +- openai_mmmlu_default_en_us_clinical_knowledge +- openai_mmmlu_default_en_us_college_biology +- openai_mmmlu_default_en_us_college_chemistry +- openai_mmmlu_default_en_us_college_computer_science +- openai_mmmlu_default_en_us_college_mathematics +- openai_mmmlu_default_en_us_college_medicine +- openai_mmmlu_default_en_us_college_physics +- openai_mmmlu_default_en_us_computer_security +- openai_mmmlu_default_en_us_conceptual_physics +- openai_mmmlu_default_en_us_econometrics +- openai_mmmlu_default_en_us_electrical_engineering +- openai_mmmlu_default_en_us_elementary_mathematics +- openai_mmmlu_default_en_us_formal_logic +- openai_mmmlu_default_en_us_global_facts +- openai_mmmlu_default_en_us_high_school_biology +- openai_mmmlu_default_en_us_high_school_chemistry +- openai_mmmlu_default_en_us_high_school_computer_science +- openai_mmmlu_default_en_us_high_school_european_history +- openai_mmmlu_default_en_us_high_school_geography +- openai_mmmlu_default_en_us_high_school_government_and_politics +- openai_mmmlu_default_en_us_high_school_macroeconomics +- openai_mmmlu_default_en_us_high_school_mathematics +- openai_mmmlu_default_en_us_high_school_microeconomics +- openai_mmmlu_default_en_us_high_school_physics +- openai_mmmlu_default_en_us_high_school_psychology +- openai_mmmlu_default_en_us_high_school_statistics +- openai_mmmlu_default_en_us_high_school_us_history +- openai_mmmlu_default_en_us_high_school_world_history +- openai_mmmlu_default_en_us_human_aging +- openai_mmmlu_default_en_us_human_sexuality +- openai_mmmlu_default_en_us_international_law +- openai_mmmlu_default_en_us_jurisprudence +- openai_mmmlu_default_en_us_logical_fallacies +- openai_mmmlu_default_en_us_machine_learning +- openai_mmmlu_default_en_us_management +- openai_mmmlu_default_en_us_marketing +- openai_mmmlu_default_en_us_medical_genetics +- openai_mmmlu_default_en_us_miscellaneous +- openai_mmmlu_default_en_us_moral_disputes +- openai_mmmlu_default_en_us_moral_scenarios +- openai_mmmlu_default_en_us_nutrition +- openai_mmmlu_default_en_us_philosophy +- openai_mmmlu_default_en_us_prehistory +- openai_mmmlu_default_en_us_professional_accounting +- openai_mmmlu_default_en_us_professional_law +- openai_mmmlu_default_en_us_professional_medicine +- openai_mmmlu_default_en_us_professional_psychology +- openai_mmmlu_default_en_us_public_relations +- openai_mmmlu_default_en_us_security_studies +- openai_mmmlu_default_en_us_sociology +- openai_mmmlu_default_en_us_us_foreign_policy +- openai_mmmlu_default_en_us_virology +- openai_mmmlu_default_en_us_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..97b12022c9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_abstract_algebra" +"task_alias": "en_us abstract algebra" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..3d6ff52db1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_anatomy" +"task_alias": "en_us anatomy" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..5027f77ec5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_astronomy" +"task_alias": "en_us astronomy" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..d30111629b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_business_ethics" +"task_alias": "en_us business ethics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..a0a1af140e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_clinical_knowledge" +"task_alias": "en_us clinical knowledge" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..6e6d73daf6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_college_biology" +"task_alias": "en_us college biology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..2c060df3ac --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_college_chemistry" +"task_alias": "en_us college chemistry" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..8523c83d83 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_college_computer_science" +"task_alias": "en_us college computer science" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..c1460920da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_college_mathematics" +"task_alias": "en_us college mathematics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..0e23355364 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_college_medicine" +"task_alias": "en_us college medicine" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..0377c7315f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_college_physics" +"task_alias": "en_us college physics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..a5369b0768 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_computer_security" +"task_alias": "en_us computer security" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..0f01e4f528 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_conceptual_physics" +"task_alias": "en_us conceptual physics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..3aca39d313 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_econometrics" +"task_alias": "en_us econometrics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..ae77ec138d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_electrical_engineering" +"task_alias": "en_us electrical engineering" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..688ea50c5d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_elementary_mathematics" +"task_alias": "en_us elementary mathematics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..7b9b8ced00 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_formal_logic" +"task_alias": "en_us formal logic" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..6dc66d4b25 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_global_facts" +"task_alias": "en_us global facts" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..4efeaa85ee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_biology" +"task_alias": "en_us high school biology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..d20e6c24ac --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_chemistry" +"task_alias": "en_us high school chemistry" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..e9d47fb7e4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_computer_science" +"task_alias": "en_us high school computer science" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..00fc790ac7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_high_school_european_history" +"task_alias": "en_us high school european history" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..f20e383f11 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_high_school_geography" +"task_alias": "en_us high school geography" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..19eb9aac2d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_high_school_government_and_politics" +"task_alias": "en_us high school government and politics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..3fe777d456 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_high_school_macroeconomics" +"task_alias": "en_us high school macroeconomics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..eddbb5ea96 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_mathematics" +"task_alias": "en_us high school mathematics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..037bfb37e8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_high_school_microeconomics" +"task_alias": "en_us high school microeconomics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..e062d7feb7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_physics" +"task_alias": "en_us high school physics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..05e7651261 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_high_school_psychology" +"task_alias": "en_us high school psychology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..ebb0d3b498 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_high_school_statistics" +"task_alias": "en_us high school statistics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..7ad398568a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_high_school_us_history" +"task_alias": "en_us high school us history" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..75dd7b5b6c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_high_school_world_history" +"task_alias": "en_us high school world history" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..0b279aa3d6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_human_aging" +"task_alias": "en_us human aging" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..c086c79181 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_human_sexuality" +"task_alias": "en_us human sexuality" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..77af4352b2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_international_law" +"task_alias": "en_us international law" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..67b52170d8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_jurisprudence" +"task_alias": "en_us jurisprudence" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..112d7716b3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_logical_fallacies" +"task_alias": "en_us logical fallacies" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..b3bb9856ee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_stem" +"task": "openai_mmmlu_default_en_us_machine_learning" +"task_alias": "en_us machine learning" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..32ea0f6311 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_management" +"task_alias": "en_us management" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..0b849cba6b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_marketing" +"task_alias": "en_us marketing" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..161ba107e1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_medical_genetics" +"task_alias": "en_us medical genetics" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..23ca8fe94c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_miscellaneous" +"task_alias": "en_us miscellaneous" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..65755be0ff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_moral_disputes" +"task_alias": "en_us moral disputes" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..3002104883 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_moral_scenarios" +"task_alias": "en_us moral scenarios" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..2a153c7cbe --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_nutrition" +"task_alias": "en_us nutrition" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..f6e1012a3a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_philosophy" +"task_alias": "en_us philosophy" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..72923d9055 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_prehistory" +"task_alias": "en_us prehistory" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..7a86b92126 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_professional_accounting" +"task_alias": "en_us professional accounting" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..0e7e3dc051 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_professional_law" +"task_alias": "en_us professional law" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..c29f39594d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_professional_medicine" +"task_alias": "en_us professional medicine" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..2f0797e18d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_professional_psychology" +"task_alias": "en_us professional psychology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..d06658a738 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_public_relations" +"task_alias": "en_us public relations" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..c27325eb12 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_security_studies" +"task_alias": "en_us security studies" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..c4c3efdb9d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_sociology" +"task_alias": "en_us sociology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..08cc68a40c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_social_sciences" +"task": "openai_mmmlu_default_en_us_us_foreign_policy" +"task_alias": "en_us us foreign policy" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..47ee51008f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_other" +"task": "openai_mmmlu_default_en_us_virology" +"task_alias": "en_us virology" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..04da7f519b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/en_us/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nAnswer:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_en_us_humanities" +"task": "openai_mmmlu_default_en_us_world_religions" +"task_alias": "en_us world religions" +"test_split": "EN_US" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/_default_es_la_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/_default_es_la_group_info.yaml new file mode 100644 index 0000000000..d02c65c0fd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/_default_es_la_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_es_la +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_es_la_abstract_algebra +- openai_mmmlu_default_es_la_anatomy +- openai_mmmlu_default_es_la_astronomy +- openai_mmmlu_default_es_la_business_ethics +- openai_mmmlu_default_es_la_clinical_knowledge +- openai_mmmlu_default_es_la_college_biology +- openai_mmmlu_default_es_la_college_chemistry +- openai_mmmlu_default_es_la_college_computer_science +- openai_mmmlu_default_es_la_college_mathematics +- openai_mmmlu_default_es_la_college_medicine +- openai_mmmlu_default_es_la_college_physics +- openai_mmmlu_default_es_la_computer_security +- openai_mmmlu_default_es_la_conceptual_physics +- openai_mmmlu_default_es_la_econometrics +- openai_mmmlu_default_es_la_electrical_engineering +- openai_mmmlu_default_es_la_elementary_mathematics +- openai_mmmlu_default_es_la_formal_logic +- openai_mmmlu_default_es_la_global_facts +- openai_mmmlu_default_es_la_high_school_biology +- openai_mmmlu_default_es_la_high_school_chemistry +- openai_mmmlu_default_es_la_high_school_computer_science +- openai_mmmlu_default_es_la_high_school_european_history +- openai_mmmlu_default_es_la_high_school_geography +- openai_mmmlu_default_es_la_high_school_government_and_politics +- openai_mmmlu_default_es_la_high_school_macroeconomics +- openai_mmmlu_default_es_la_high_school_mathematics +- openai_mmmlu_default_es_la_high_school_microeconomics +- openai_mmmlu_default_es_la_high_school_physics +- openai_mmmlu_default_es_la_high_school_psychology +- openai_mmmlu_default_es_la_high_school_statistics +- openai_mmmlu_default_es_la_high_school_us_history +- openai_mmmlu_default_es_la_high_school_world_history +- openai_mmmlu_default_es_la_human_aging +- openai_mmmlu_default_es_la_human_sexuality +- openai_mmmlu_default_es_la_international_law +- openai_mmmlu_default_es_la_jurisprudence +- openai_mmmlu_default_es_la_logical_fallacies +- openai_mmmlu_default_es_la_machine_learning +- openai_mmmlu_default_es_la_management +- openai_mmmlu_default_es_la_marketing +- openai_mmmlu_default_es_la_medical_genetics +- openai_mmmlu_default_es_la_miscellaneous +- openai_mmmlu_default_es_la_moral_disputes +- openai_mmmlu_default_es_la_moral_scenarios +- openai_mmmlu_default_es_la_nutrition +- openai_mmmlu_default_es_la_philosophy +- openai_mmmlu_default_es_la_prehistory +- openai_mmmlu_default_es_la_professional_accounting +- openai_mmmlu_default_es_la_professional_law +- openai_mmmlu_default_es_la_professional_medicine +- openai_mmmlu_default_es_la_professional_psychology +- openai_mmmlu_default_es_la_public_relations +- openai_mmmlu_default_es_la_security_studies +- openai_mmmlu_default_es_la_sociology +- openai_mmmlu_default_es_la_us_foreign_policy +- openai_mmmlu_default_es_la_virology +- openai_mmmlu_default_es_la_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..80763e2dcd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_abstract_algebra" +"task_alias": "es_la abstract algebra" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..221ee4ec78 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_anatomy" +"task_alias": "es_la anatomy" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..bbfa4ae778 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_astronomy" +"task_alias": "es_la astronomy" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..0d68b4fd57 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_business_ethics" +"task_alias": "es_la business ethics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..2eac42d6d8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_clinical_knowledge" +"task_alias": "es_la clinical knowledge" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..1d212c2f12 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_college_biology" +"task_alias": "es_la college biology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..f42b578f46 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_college_chemistry" +"task_alias": "es_la college chemistry" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..479024523c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_college_computer_science" +"task_alias": "es_la college computer science" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..5bd9bf49fc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_college_mathematics" +"task_alias": "es_la college mathematics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..3496f28d21 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_college_medicine" +"task_alias": "es_la college medicine" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..bfb03097d7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_college_physics" +"task_alias": "es_la college physics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..52c2049938 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_computer_security" +"task_alias": "es_la computer security" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..9a6c7af849 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_conceptual_physics" +"task_alias": "es_la conceptual physics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..18f8b74a2c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_econometrics" +"task_alias": "es_la econometrics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..1b770c042b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_electrical_engineering" +"task_alias": "es_la electrical engineering" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..a188cfa645 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_elementary_mathematics" +"task_alias": "es_la elementary mathematics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..af5373c804 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_formal_logic" +"task_alias": "es_la formal logic" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..dde673352e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_global_facts" +"task_alias": "es_la global facts" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..36b0352d60 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_biology" +"task_alias": "es_la high school biology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..3c68ea06f2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_chemistry" +"task_alias": "es_la high school chemistry" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..3154cfe610 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_computer_science" +"task_alias": "es_la high school computer science" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..b0d822c43d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_high_school_european_history" +"task_alias": "es_la high school european history" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..1f0148488d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_high_school_geography" +"task_alias": "es_la high school geography" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..f92cc386a0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_high_school_government_and_politics" +"task_alias": "es_la high school government and politics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..f72d8c553c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_high_school_macroeconomics" +"task_alias": "es_la high school macroeconomics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..d0ecb42d06 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_mathematics" +"task_alias": "es_la high school mathematics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..5225e8bc30 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_high_school_microeconomics" +"task_alias": "es_la high school microeconomics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..fe5da73de9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_physics" +"task_alias": "es_la high school physics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..dc4381ad4f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_high_school_psychology" +"task_alias": "es_la high school psychology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..89cbb667ec --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_high_school_statistics" +"task_alias": "es_la high school statistics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..fb911c6df5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_high_school_us_history" +"task_alias": "es_la high school us history" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..4fa550ba9f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_high_school_world_history" +"task_alias": "es_la high school world history" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..44472a26d5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_human_aging" +"task_alias": "es_la human aging" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..3eb113dd56 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_human_sexuality" +"task_alias": "es_la human sexuality" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..91257d6c61 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_international_law" +"task_alias": "es_la international law" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..c3107e6cbd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_jurisprudence" +"task_alias": "es_la jurisprudence" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..9f336adc03 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_logical_fallacies" +"task_alias": "es_la logical fallacies" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..ee111929ea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_stem" +"task": "openai_mmmlu_default_es_la_machine_learning" +"task_alias": "es_la machine learning" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..2e6e2db21a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_management" +"task_alias": "es_la management" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..2a4a83b774 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_marketing" +"task_alias": "es_la marketing" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..8d1ec223c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_medical_genetics" +"task_alias": "es_la medical genetics" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..5dd9a9c947 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_miscellaneous" +"task_alias": "es_la miscellaneous" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..692e3ef6dc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_moral_disputes" +"task_alias": "es_la moral disputes" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..0ceace3f62 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_moral_scenarios" +"task_alias": "es_la moral scenarios" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..c891436520 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_nutrition" +"task_alias": "es_la nutrition" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..dbf8bb2a22 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_philosophy" +"task_alias": "es_la philosophy" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..78035d7213 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_prehistory" +"task_alias": "es_la prehistory" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..1df409079a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_professional_accounting" +"task_alias": "es_la professional accounting" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..11f7dbc611 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_professional_law" +"task_alias": "es_la professional law" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..19f876c034 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_professional_medicine" +"task_alias": "es_la professional medicine" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..4ee0e15555 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_professional_psychology" +"task_alias": "es_la professional psychology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..6f2b47fee9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_public_relations" +"task_alias": "es_la public relations" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..841b7ba752 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_security_studies" +"task_alias": "es_la security studies" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..4849794667 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_sociology" +"task_alias": "es_la sociology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..633af8ce23 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_social_sciences" +"task": "openai_mmmlu_default_es_la_us_foreign_policy" +"task_alias": "es_la us foreign policy" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..731e5921cc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_other" +"task": "openai_mmmlu_default_es_la_virology" +"task_alias": "es_la virology" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..7cc5c1cdca --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/es_la/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRespuesta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_es_la_humanities" +"task": "openai_mmmlu_default_es_la_world_religions" +"task_alias": "es_la world religions" +"test_split": "ES_LA" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/_default_fr_fr_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/_default_fr_fr_group_info.yaml new file mode 100644 index 0000000000..0e2d31f984 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/_default_fr_fr_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_fr_fr +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_fr_fr_abstract_algebra +- openai_mmmlu_default_fr_fr_anatomy +- openai_mmmlu_default_fr_fr_astronomy +- openai_mmmlu_default_fr_fr_business_ethics +- openai_mmmlu_default_fr_fr_clinical_knowledge +- openai_mmmlu_default_fr_fr_college_biology +- openai_mmmlu_default_fr_fr_college_chemistry +- openai_mmmlu_default_fr_fr_college_computer_science +- openai_mmmlu_default_fr_fr_college_mathematics +- openai_mmmlu_default_fr_fr_college_medicine +- openai_mmmlu_default_fr_fr_college_physics +- openai_mmmlu_default_fr_fr_computer_security +- openai_mmmlu_default_fr_fr_conceptual_physics +- openai_mmmlu_default_fr_fr_econometrics +- openai_mmmlu_default_fr_fr_electrical_engineering +- openai_mmmlu_default_fr_fr_elementary_mathematics +- openai_mmmlu_default_fr_fr_formal_logic +- openai_mmmlu_default_fr_fr_global_facts +- openai_mmmlu_default_fr_fr_high_school_biology +- openai_mmmlu_default_fr_fr_high_school_chemistry +- openai_mmmlu_default_fr_fr_high_school_computer_science +- openai_mmmlu_default_fr_fr_high_school_european_history +- openai_mmmlu_default_fr_fr_high_school_geography +- openai_mmmlu_default_fr_fr_high_school_government_and_politics +- openai_mmmlu_default_fr_fr_high_school_macroeconomics +- openai_mmmlu_default_fr_fr_high_school_mathematics +- openai_mmmlu_default_fr_fr_high_school_microeconomics +- openai_mmmlu_default_fr_fr_high_school_physics +- openai_mmmlu_default_fr_fr_high_school_psychology +- openai_mmmlu_default_fr_fr_high_school_statistics +- openai_mmmlu_default_fr_fr_high_school_us_history +- openai_mmmlu_default_fr_fr_high_school_world_history +- openai_mmmlu_default_fr_fr_human_aging +- openai_mmmlu_default_fr_fr_human_sexuality +- openai_mmmlu_default_fr_fr_international_law +- openai_mmmlu_default_fr_fr_jurisprudence +- openai_mmmlu_default_fr_fr_logical_fallacies +- openai_mmmlu_default_fr_fr_machine_learning +- openai_mmmlu_default_fr_fr_management +- openai_mmmlu_default_fr_fr_marketing +- openai_mmmlu_default_fr_fr_medical_genetics +- openai_mmmlu_default_fr_fr_miscellaneous +- openai_mmmlu_default_fr_fr_moral_disputes +- openai_mmmlu_default_fr_fr_moral_scenarios +- openai_mmmlu_default_fr_fr_nutrition +- openai_mmmlu_default_fr_fr_philosophy +- openai_mmmlu_default_fr_fr_prehistory +- openai_mmmlu_default_fr_fr_professional_accounting +- openai_mmmlu_default_fr_fr_professional_law +- openai_mmmlu_default_fr_fr_professional_medicine +- openai_mmmlu_default_fr_fr_professional_psychology +- openai_mmmlu_default_fr_fr_public_relations +- openai_mmmlu_default_fr_fr_security_studies +- openai_mmmlu_default_fr_fr_sociology +- openai_mmmlu_default_fr_fr_us_foreign_policy +- openai_mmmlu_default_fr_fr_virology +- openai_mmmlu_default_fr_fr_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..a67c1163ad --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_abstract_algebra" +"task_alias": "fr_fr abstract algebra" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..b5971ac08f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_anatomy" +"task_alias": "fr_fr anatomy" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..5f5d7fc468 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_astronomy" +"task_alias": "fr_fr astronomy" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..8e1fc0cb67 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_business_ethics" +"task_alias": "fr_fr business ethics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..91d8efbb74 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_clinical_knowledge" +"task_alias": "fr_fr clinical knowledge" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..b04836cec6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_college_biology" +"task_alias": "fr_fr college biology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..7f6229a98a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_college_chemistry" +"task_alias": "fr_fr college chemistry" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..13bf7cf6f2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_college_computer_science" +"task_alias": "fr_fr college computer science" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..c9712f9aef --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_college_mathematics" +"task_alias": "fr_fr college mathematics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..55eca121f4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_college_medicine" +"task_alias": "fr_fr college medicine" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..725f05308a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_college_physics" +"task_alias": "fr_fr college physics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..b4d30344b9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_computer_security" +"task_alias": "fr_fr computer security" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..217b17ca02 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_conceptual_physics" +"task_alias": "fr_fr conceptual physics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..b007ee4bc6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_econometrics" +"task_alias": "fr_fr econometrics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..e1c2478967 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_electrical_engineering" +"task_alias": "fr_fr electrical engineering" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..397e347160 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_elementary_mathematics" +"task_alias": "fr_fr elementary mathematics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..af7ab0deff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_formal_logic" +"task_alias": "fr_fr formal logic" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..ae8baa980c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_global_facts" +"task_alias": "fr_fr global facts" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..2c5b7ae9c7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_biology" +"task_alias": "fr_fr high school biology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..079b9c5744 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_chemistry" +"task_alias": "fr_fr high school chemistry" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..20ce2382c3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_computer_science" +"task_alias": "fr_fr high school computer science" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..7fea12765e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_high_school_european_history" +"task_alias": "fr_fr high school european history" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..cf303c638d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_high_school_geography" +"task_alias": "fr_fr high school geography" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..9abe40051e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_high_school_government_and_politics" +"task_alias": "fr_fr high school government and politics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..d20afe6912 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_high_school_macroeconomics" +"task_alias": "fr_fr high school macroeconomics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..cd06205f6a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_mathematics" +"task_alias": "fr_fr high school mathematics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..94a9e52cfb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_high_school_microeconomics" +"task_alias": "fr_fr high school microeconomics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..8337b352bb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_physics" +"task_alias": "fr_fr high school physics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..e4bbfd1fc2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_high_school_psychology" +"task_alias": "fr_fr high school psychology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..4874c056a7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_high_school_statistics" +"task_alias": "fr_fr high school statistics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..726ce3ebb3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_high_school_us_history" +"task_alias": "fr_fr high school us history" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..76c3e32be6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_high_school_world_history" +"task_alias": "fr_fr high school world history" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..6547633a4b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_human_aging" +"task_alias": "fr_fr human aging" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..50bb0dd145 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_human_sexuality" +"task_alias": "fr_fr human sexuality" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..ba0e62f670 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_international_law" +"task_alias": "fr_fr international law" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..4a830d7cc4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_jurisprudence" +"task_alias": "fr_fr jurisprudence" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..0f8c83b8ed --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_logical_fallacies" +"task_alias": "fr_fr logical fallacies" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..008c515c42 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_stem" +"task": "openai_mmmlu_default_fr_fr_machine_learning" +"task_alias": "fr_fr machine learning" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..5cbb6646d3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_management" +"task_alias": "fr_fr management" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..152107e347 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_marketing" +"task_alias": "fr_fr marketing" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..1b6d8f9894 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_medical_genetics" +"task_alias": "fr_fr medical genetics" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..730a0fd3da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_miscellaneous" +"task_alias": "fr_fr miscellaneous" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..7b4562bc26 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_moral_disputes" +"task_alias": "fr_fr moral disputes" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..cba1c74fae --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_moral_scenarios" +"task_alias": "fr_fr moral scenarios" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..5d6b2318b8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_nutrition" +"task_alias": "fr_fr nutrition" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..533d689efd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_philosophy" +"task_alias": "fr_fr philosophy" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..e5a65ff8bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_prehistory" +"task_alias": "fr_fr prehistory" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..539964b93c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_professional_accounting" +"task_alias": "fr_fr professional accounting" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..89310e9de2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_professional_law" +"task_alias": "fr_fr professional law" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..ee9a1dc311 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_professional_medicine" +"task_alias": "fr_fr professional medicine" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..f88b88cc03 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_professional_psychology" +"task_alias": "fr_fr professional psychology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..f47b48950f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_public_relations" +"task_alias": "fr_fr public relations" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..4dc19ec01c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_security_studies" +"task_alias": "fr_fr security studies" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..13f59fa168 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_sociology" +"task_alias": "fr_fr sociology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..ecbe2cb039 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_social_sciences" +"task": "openai_mmmlu_default_fr_fr_us_foreign_policy" +"task_alias": "fr_fr us foreign policy" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..40093ddfe4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_other" +"task": "openai_mmmlu_default_fr_fr_virology" +"task_alias": "fr_fr virology" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..41c2148f29 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/fr_fr/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRéponse:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_fr_fr_humanities" +"task": "openai_mmmlu_default_fr_fr_world_religions" +"task_alias": "fr_fr world religions" +"test_split": "FR_FR" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/_default_hi_in_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/_default_hi_in_group_info.yaml new file mode 100644 index 0000000000..94cb8d1d72 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/_default_hi_in_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_hi_in +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_hi_in_abstract_algebra +- openai_mmmlu_default_hi_in_anatomy +- openai_mmmlu_default_hi_in_astronomy +- openai_mmmlu_default_hi_in_business_ethics +- openai_mmmlu_default_hi_in_clinical_knowledge +- openai_mmmlu_default_hi_in_college_biology +- openai_mmmlu_default_hi_in_college_chemistry +- openai_mmmlu_default_hi_in_college_computer_science +- openai_mmmlu_default_hi_in_college_mathematics +- openai_mmmlu_default_hi_in_college_medicine +- openai_mmmlu_default_hi_in_college_physics +- openai_mmmlu_default_hi_in_computer_security +- openai_mmmlu_default_hi_in_conceptual_physics +- openai_mmmlu_default_hi_in_econometrics +- openai_mmmlu_default_hi_in_electrical_engineering +- openai_mmmlu_default_hi_in_elementary_mathematics +- openai_mmmlu_default_hi_in_formal_logic +- openai_mmmlu_default_hi_in_global_facts +- openai_mmmlu_default_hi_in_high_school_biology +- openai_mmmlu_default_hi_in_high_school_chemistry +- openai_mmmlu_default_hi_in_high_school_computer_science +- openai_mmmlu_default_hi_in_high_school_european_history +- openai_mmmlu_default_hi_in_high_school_geography +- openai_mmmlu_default_hi_in_high_school_government_and_politics +- openai_mmmlu_default_hi_in_high_school_macroeconomics +- openai_mmmlu_default_hi_in_high_school_mathematics +- openai_mmmlu_default_hi_in_high_school_microeconomics +- openai_mmmlu_default_hi_in_high_school_physics +- openai_mmmlu_default_hi_in_high_school_psychology +- openai_mmmlu_default_hi_in_high_school_statistics +- openai_mmmlu_default_hi_in_high_school_us_history +- openai_mmmlu_default_hi_in_high_school_world_history +- openai_mmmlu_default_hi_in_human_aging +- openai_mmmlu_default_hi_in_human_sexuality +- openai_mmmlu_default_hi_in_international_law +- openai_mmmlu_default_hi_in_jurisprudence +- openai_mmmlu_default_hi_in_logical_fallacies +- openai_mmmlu_default_hi_in_machine_learning +- openai_mmmlu_default_hi_in_management +- openai_mmmlu_default_hi_in_marketing +- openai_mmmlu_default_hi_in_medical_genetics +- openai_mmmlu_default_hi_in_miscellaneous +- openai_mmmlu_default_hi_in_moral_disputes +- openai_mmmlu_default_hi_in_moral_scenarios +- openai_mmmlu_default_hi_in_nutrition +- openai_mmmlu_default_hi_in_philosophy +- openai_mmmlu_default_hi_in_prehistory +- openai_mmmlu_default_hi_in_professional_accounting +- openai_mmmlu_default_hi_in_professional_law +- openai_mmmlu_default_hi_in_professional_medicine +- openai_mmmlu_default_hi_in_professional_psychology +- openai_mmmlu_default_hi_in_public_relations +- openai_mmmlu_default_hi_in_security_studies +- openai_mmmlu_default_hi_in_sociology +- openai_mmmlu_default_hi_in_us_foreign_policy +- openai_mmmlu_default_hi_in_virology +- openai_mmmlu_default_hi_in_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..1bac20bfc0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_abstract_algebra" +"task_alias": "hi_in abstract algebra" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..3182763719 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_anatomy" +"task_alias": "hi_in anatomy" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..9b14fa5df2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_astronomy" +"task_alias": "hi_in astronomy" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..e56a15a9d5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_business_ethics" +"task_alias": "hi_in business ethics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..93391a7197 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_clinical_knowledge" +"task_alias": "hi_in clinical knowledge" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..be1d3aa62f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_college_biology" +"task_alias": "hi_in college biology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..21ac092cd0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_college_chemistry" +"task_alias": "hi_in college chemistry" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..82144108b2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_college_computer_science" +"task_alias": "hi_in college computer science" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..b9d0bf8f4d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_college_mathematics" +"task_alias": "hi_in college mathematics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..a5faa90374 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_college_medicine" +"task_alias": "hi_in college medicine" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..969b6f9244 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_college_physics" +"task_alias": "hi_in college physics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..4e4e320f32 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_computer_security" +"task_alias": "hi_in computer security" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..064b265c64 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_conceptual_physics" +"task_alias": "hi_in conceptual physics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..f8ed1ed902 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_econometrics" +"task_alias": "hi_in econometrics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..349b9e0b2c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_electrical_engineering" +"task_alias": "hi_in electrical engineering" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..2c2e7541de --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_elementary_mathematics" +"task_alias": "hi_in elementary mathematics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..127ae83357 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_formal_logic" +"task_alias": "hi_in formal logic" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..fc7cf0e149 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_global_facts" +"task_alias": "hi_in global facts" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..b015848ea8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_biology" +"task_alias": "hi_in high school biology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..21c9d4cc44 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_chemistry" +"task_alias": "hi_in high school chemistry" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..72f0768595 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_computer_science" +"task_alias": "hi_in high school computer science" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..ce760731e9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_high_school_european_history" +"task_alias": "hi_in high school european history" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..58858408ec --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_high_school_geography" +"task_alias": "hi_in high school geography" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..cd5e73a65a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_high_school_government_and_politics" +"task_alias": "hi_in high school government and politics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..2322a339cf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_high_school_macroeconomics" +"task_alias": "hi_in high school macroeconomics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..9640d0706c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_mathematics" +"task_alias": "hi_in high school mathematics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..f456d7bec4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_high_school_microeconomics" +"task_alias": "hi_in high school microeconomics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..4187802007 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_physics" +"task_alias": "hi_in high school physics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..e657d31790 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_high_school_psychology" +"task_alias": "hi_in high school psychology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..3ba90852c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_high_school_statistics" +"task_alias": "hi_in high school statistics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..1eca93d6da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_high_school_us_history" +"task_alias": "hi_in high school us history" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..f0ccee2bdb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_high_school_world_history" +"task_alias": "hi_in high school world history" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..4479d3f9a0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_human_aging" +"task_alias": "hi_in human aging" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..3c68dc45ce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_human_sexuality" +"task_alias": "hi_in human sexuality" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..dc8927f5b6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_international_law" +"task_alias": "hi_in international law" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..080554fa50 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_jurisprudence" +"task_alias": "hi_in jurisprudence" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..76dae19621 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_logical_fallacies" +"task_alias": "hi_in logical fallacies" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..0c863b0bee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_stem" +"task": "openai_mmmlu_default_hi_in_machine_learning" +"task_alias": "hi_in machine learning" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..5a322795b0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_management" +"task_alias": "hi_in management" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..b856dca40a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_marketing" +"task_alias": "hi_in marketing" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..f83976cb19 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_medical_genetics" +"task_alias": "hi_in medical genetics" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..968d0c074e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_miscellaneous" +"task_alias": "hi_in miscellaneous" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..a99beca7c6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_moral_disputes" +"task_alias": "hi_in moral disputes" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..090e1e3f93 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_moral_scenarios" +"task_alias": "hi_in moral scenarios" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..1eda4ec9d4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_nutrition" +"task_alias": "hi_in nutrition" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..83938864e2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_philosophy" +"task_alias": "hi_in philosophy" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..7906502165 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_prehistory" +"task_alias": "hi_in prehistory" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..3abb7489ba --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_professional_accounting" +"task_alias": "hi_in professional accounting" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..875efe047e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_professional_law" +"task_alias": "hi_in professional law" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..f943855010 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_professional_medicine" +"task_alias": "hi_in professional medicine" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..95e304df03 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_professional_psychology" +"task_alias": "hi_in professional psychology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..696a4434cb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_public_relations" +"task_alias": "hi_in public relations" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..7987221aaa --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_security_studies" +"task_alias": "hi_in security studies" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..577c70dc1d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_sociology" +"task_alias": "hi_in sociology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..115ef15847 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_social_sciences" +"task": "openai_mmmlu_default_hi_in_us_foreign_policy" +"task_alias": "hi_in us foreign policy" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..4d15a2579e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_other" +"task": "openai_mmmlu_default_hi_in_virology" +"task_alias": "hi_in virology" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..85cbaac1e7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/hi_in/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nउत्तर:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_hi_in_humanities" +"task": "openai_mmmlu_default_hi_in_world_religions" +"task_alias": "hi_in world religions" +"test_split": "HI_IN" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/_default_id_id_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/_default_id_id_group_info.yaml new file mode 100644 index 0000000000..e34a9ae839 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/_default_id_id_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_id_id +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_id_id_abstract_algebra +- openai_mmmlu_default_id_id_anatomy +- openai_mmmlu_default_id_id_astronomy +- openai_mmmlu_default_id_id_business_ethics +- openai_mmmlu_default_id_id_clinical_knowledge +- openai_mmmlu_default_id_id_college_biology +- openai_mmmlu_default_id_id_college_chemistry +- openai_mmmlu_default_id_id_college_computer_science +- openai_mmmlu_default_id_id_college_mathematics +- openai_mmmlu_default_id_id_college_medicine +- openai_mmmlu_default_id_id_college_physics +- openai_mmmlu_default_id_id_computer_security +- openai_mmmlu_default_id_id_conceptual_physics +- openai_mmmlu_default_id_id_econometrics +- openai_mmmlu_default_id_id_electrical_engineering +- openai_mmmlu_default_id_id_elementary_mathematics +- openai_mmmlu_default_id_id_formal_logic +- openai_mmmlu_default_id_id_global_facts +- openai_mmmlu_default_id_id_high_school_biology +- openai_mmmlu_default_id_id_high_school_chemistry +- openai_mmmlu_default_id_id_high_school_computer_science +- openai_mmmlu_default_id_id_high_school_european_history +- openai_mmmlu_default_id_id_high_school_geography +- openai_mmmlu_default_id_id_high_school_government_and_politics +- openai_mmmlu_default_id_id_high_school_macroeconomics +- openai_mmmlu_default_id_id_high_school_mathematics +- openai_mmmlu_default_id_id_high_school_microeconomics +- openai_mmmlu_default_id_id_high_school_physics +- openai_mmmlu_default_id_id_high_school_psychology +- openai_mmmlu_default_id_id_high_school_statistics +- openai_mmmlu_default_id_id_high_school_us_history +- openai_mmmlu_default_id_id_high_school_world_history +- openai_mmmlu_default_id_id_human_aging +- openai_mmmlu_default_id_id_human_sexuality +- openai_mmmlu_default_id_id_international_law +- openai_mmmlu_default_id_id_jurisprudence +- openai_mmmlu_default_id_id_logical_fallacies +- openai_mmmlu_default_id_id_machine_learning +- openai_mmmlu_default_id_id_management +- openai_mmmlu_default_id_id_marketing +- openai_mmmlu_default_id_id_medical_genetics +- openai_mmmlu_default_id_id_miscellaneous +- openai_mmmlu_default_id_id_moral_disputes +- openai_mmmlu_default_id_id_moral_scenarios +- openai_mmmlu_default_id_id_nutrition +- openai_mmmlu_default_id_id_philosophy +- openai_mmmlu_default_id_id_prehistory +- openai_mmmlu_default_id_id_professional_accounting +- openai_mmmlu_default_id_id_professional_law +- openai_mmmlu_default_id_id_professional_medicine +- openai_mmmlu_default_id_id_professional_psychology +- openai_mmmlu_default_id_id_public_relations +- openai_mmmlu_default_id_id_security_studies +- openai_mmmlu_default_id_id_sociology +- openai_mmmlu_default_id_id_us_foreign_policy +- openai_mmmlu_default_id_id_virology +- openai_mmmlu_default_id_id_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..b468b0ff52 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_abstract_algebra" +"task_alias": "id_id abstract algebra" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..a18375d033 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_anatomy" +"task_alias": "id_id anatomy" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..6a57d7cfda --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_astronomy" +"task_alias": "id_id astronomy" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..cd7bdd6c5f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_business_ethics" +"task_alias": "id_id business ethics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..a8dce10c84 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_clinical_knowledge" +"task_alias": "id_id clinical knowledge" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..e9591c404b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_college_biology" +"task_alias": "id_id college biology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..5046b6e46c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_college_chemistry" +"task_alias": "id_id college chemistry" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..f3d1b1a51f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_college_computer_science" +"task_alias": "id_id college computer science" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..acc542586e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_college_mathematics" +"task_alias": "id_id college mathematics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..75b15800bb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_college_medicine" +"task_alias": "id_id college medicine" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..7276aa17c8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_college_physics" +"task_alias": "id_id college physics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..d2a1ab52c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_computer_security" +"task_alias": "id_id computer security" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..fd22a0ccff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_conceptual_physics" +"task_alias": "id_id conceptual physics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..3339d09dfd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_econometrics" +"task_alias": "id_id econometrics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..b5c254971c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_electrical_engineering" +"task_alias": "id_id electrical engineering" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..35ea3cae72 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_elementary_mathematics" +"task_alias": "id_id elementary mathematics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..229ecdce0b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_formal_logic" +"task_alias": "id_id formal logic" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..3c6538db18 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_global_facts" +"task_alias": "id_id global facts" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..c9bbf287e6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_biology" +"task_alias": "id_id high school biology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..1b82f0b171 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_chemistry" +"task_alias": "id_id high school chemistry" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..83126aca31 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_computer_science" +"task_alias": "id_id high school computer science" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..0e7e82dc0b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_high_school_european_history" +"task_alias": "id_id high school european history" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..017d596d81 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_high_school_geography" +"task_alias": "id_id high school geography" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..590afb470d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_high_school_government_and_politics" +"task_alias": "id_id high school government and politics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..4791d4ec00 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_high_school_macroeconomics" +"task_alias": "id_id high school macroeconomics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..069ce7e035 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_mathematics" +"task_alias": "id_id high school mathematics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..9bdad50351 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_high_school_microeconomics" +"task_alias": "id_id high school microeconomics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..9843463e53 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_physics" +"task_alias": "id_id high school physics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..c8f2920a3c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_high_school_psychology" +"task_alias": "id_id high school psychology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..4fb37318ef --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_high_school_statistics" +"task_alias": "id_id high school statistics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..498f5482b6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_high_school_us_history" +"task_alias": "id_id high school us history" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..09fcac5776 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_high_school_world_history" +"task_alias": "id_id high school world history" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..ca981d6153 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_human_aging" +"task_alias": "id_id human aging" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..e584a3434b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_human_sexuality" +"task_alias": "id_id human sexuality" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..bdbbac3257 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_international_law" +"task_alias": "id_id international law" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..d8c55c0a26 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_jurisprudence" +"task_alias": "id_id jurisprudence" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..f5ea4ad1e8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_logical_fallacies" +"task_alias": "id_id logical fallacies" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..fdbbff6bf1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_stem" +"task": "openai_mmmlu_default_id_id_machine_learning" +"task_alias": "id_id machine learning" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..2fe23082a6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_management" +"task_alias": "id_id management" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..05fcec831f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_marketing" +"task_alias": "id_id marketing" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..5ef0f78739 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_medical_genetics" +"task_alias": "id_id medical genetics" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..9ea93e7ee0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_miscellaneous" +"task_alias": "id_id miscellaneous" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..fa8a3e07a7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_moral_disputes" +"task_alias": "id_id moral disputes" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..1d83b3c893 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_moral_scenarios" +"task_alias": "id_id moral scenarios" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..e25f6af38c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_nutrition" +"task_alias": "id_id nutrition" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..f786804933 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_philosophy" +"task_alias": "id_id philosophy" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..6f04a4ac67 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_prehistory" +"task_alias": "id_id prehistory" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..f0afde66a2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_professional_accounting" +"task_alias": "id_id professional accounting" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..427491d986 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_professional_law" +"task_alias": "id_id professional law" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..bd73ebe8ba --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_professional_medicine" +"task_alias": "id_id professional medicine" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..1e3dbc0103 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_professional_psychology" +"task_alias": "id_id professional psychology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..dd526807bc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_public_relations" +"task_alias": "id_id public relations" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..6c40e7857c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_security_studies" +"task_alias": "id_id security studies" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..cd79852ebb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_sociology" +"task_alias": "id_id sociology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..297c9a0237 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_social_sciences" +"task": "openai_mmmlu_default_id_id_us_foreign_policy" +"task_alias": "id_id us foreign policy" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..6fb5d3773f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_other" +"task": "openai_mmmlu_default_id_id_virology" +"task_alias": "id_id virology" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..aa6cfa4616 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/id_id/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawaban:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_id_id_humanities" +"task": "openai_mmmlu_default_id_id_world_religions" +"task_alias": "id_id world religions" +"test_split": "ID_ID" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/_default_it_it_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/_default_it_it_group_info.yaml new file mode 100644 index 0000000000..a06985b5ee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/_default_it_it_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_it_it +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_it_it_abstract_algebra +- openai_mmmlu_default_it_it_anatomy +- openai_mmmlu_default_it_it_astronomy +- openai_mmmlu_default_it_it_business_ethics +- openai_mmmlu_default_it_it_clinical_knowledge +- openai_mmmlu_default_it_it_college_biology +- openai_mmmlu_default_it_it_college_chemistry +- openai_mmmlu_default_it_it_college_computer_science +- openai_mmmlu_default_it_it_college_mathematics +- openai_mmmlu_default_it_it_college_medicine +- openai_mmmlu_default_it_it_college_physics +- openai_mmmlu_default_it_it_computer_security +- openai_mmmlu_default_it_it_conceptual_physics +- openai_mmmlu_default_it_it_econometrics +- openai_mmmlu_default_it_it_electrical_engineering +- openai_mmmlu_default_it_it_elementary_mathematics +- openai_mmmlu_default_it_it_formal_logic +- openai_mmmlu_default_it_it_global_facts +- openai_mmmlu_default_it_it_high_school_biology +- openai_mmmlu_default_it_it_high_school_chemistry +- openai_mmmlu_default_it_it_high_school_computer_science +- openai_mmmlu_default_it_it_high_school_european_history +- openai_mmmlu_default_it_it_high_school_geography +- openai_mmmlu_default_it_it_high_school_government_and_politics +- openai_mmmlu_default_it_it_high_school_macroeconomics +- openai_mmmlu_default_it_it_high_school_mathematics +- openai_mmmlu_default_it_it_high_school_microeconomics +- openai_mmmlu_default_it_it_high_school_physics +- openai_mmmlu_default_it_it_high_school_psychology +- openai_mmmlu_default_it_it_high_school_statistics +- openai_mmmlu_default_it_it_high_school_us_history +- openai_mmmlu_default_it_it_high_school_world_history +- openai_mmmlu_default_it_it_human_aging +- openai_mmmlu_default_it_it_human_sexuality +- openai_mmmlu_default_it_it_international_law +- openai_mmmlu_default_it_it_jurisprudence +- openai_mmmlu_default_it_it_logical_fallacies +- openai_mmmlu_default_it_it_machine_learning +- openai_mmmlu_default_it_it_management +- openai_mmmlu_default_it_it_marketing +- openai_mmmlu_default_it_it_medical_genetics +- openai_mmmlu_default_it_it_miscellaneous +- openai_mmmlu_default_it_it_moral_disputes +- openai_mmmlu_default_it_it_moral_scenarios +- openai_mmmlu_default_it_it_nutrition +- openai_mmmlu_default_it_it_philosophy +- openai_mmmlu_default_it_it_prehistory +- openai_mmmlu_default_it_it_professional_accounting +- openai_mmmlu_default_it_it_professional_law +- openai_mmmlu_default_it_it_professional_medicine +- openai_mmmlu_default_it_it_professional_psychology +- openai_mmmlu_default_it_it_public_relations +- openai_mmmlu_default_it_it_security_studies +- openai_mmmlu_default_it_it_sociology +- openai_mmmlu_default_it_it_us_foreign_policy +- openai_mmmlu_default_it_it_virology +- openai_mmmlu_default_it_it_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..eebdf10d9b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_abstract_algebra" +"task_alias": "it_it abstract algebra" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..26e36a2472 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_anatomy" +"task_alias": "it_it anatomy" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..acbe2c85ee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_astronomy" +"task_alias": "it_it astronomy" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..434be19351 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_business_ethics" +"task_alias": "it_it business ethics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..1f479ca5f7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_clinical_knowledge" +"task_alias": "it_it clinical knowledge" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..62169c25ed --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_college_biology" +"task_alias": "it_it college biology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..c58a2aa535 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_college_chemistry" +"task_alias": "it_it college chemistry" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..576b3262f3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_college_computer_science" +"task_alias": "it_it college computer science" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..bcec4d934c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_college_mathematics" +"task_alias": "it_it college mathematics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..5811189599 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_college_medicine" +"task_alias": "it_it college medicine" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..cb2917034d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_college_physics" +"task_alias": "it_it college physics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..fe2f5463fc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_computer_security" +"task_alias": "it_it computer security" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..ac7afffed9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_conceptual_physics" +"task_alias": "it_it conceptual physics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..43288369d4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_econometrics" +"task_alias": "it_it econometrics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..6429d9c9bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_electrical_engineering" +"task_alias": "it_it electrical engineering" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..9606ad3ee2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_elementary_mathematics" +"task_alias": "it_it elementary mathematics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..574fee4f1f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_formal_logic" +"task_alias": "it_it formal logic" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..dd0bd7bad1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_global_facts" +"task_alias": "it_it global facts" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..22af72dfc6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_biology" +"task_alias": "it_it high school biology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..56027967a9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_chemistry" +"task_alias": "it_it high school chemistry" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..c0a372c806 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_computer_science" +"task_alias": "it_it high school computer science" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..1a5d1154d3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_high_school_european_history" +"task_alias": "it_it high school european history" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..4f766183ba --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_high_school_geography" +"task_alias": "it_it high school geography" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..2129702c66 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_high_school_government_and_politics" +"task_alias": "it_it high school government and politics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..ec19b916fd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_high_school_macroeconomics" +"task_alias": "it_it high school macroeconomics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..d6c93e791c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_mathematics" +"task_alias": "it_it high school mathematics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..b152c29253 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_high_school_microeconomics" +"task_alias": "it_it high school microeconomics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..04acfa9f08 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_physics" +"task_alias": "it_it high school physics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..5adaaf8ebf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_high_school_psychology" +"task_alias": "it_it high school psychology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..6a244b8b5b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_high_school_statistics" +"task_alias": "it_it high school statistics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..8c4e45c71b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_high_school_us_history" +"task_alias": "it_it high school us history" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..7c0a8d7a43 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_high_school_world_history" +"task_alias": "it_it high school world history" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..fcbbb71a8c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_human_aging" +"task_alias": "it_it human aging" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..b643017c02 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_human_sexuality" +"task_alias": "it_it human sexuality" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..0ca3b289a0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_international_law" +"task_alias": "it_it international law" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..9d62eadc73 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_jurisprudence" +"task_alias": "it_it jurisprudence" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..e2716fef3b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_logical_fallacies" +"task_alias": "it_it logical fallacies" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..40e4f86147 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_stem" +"task": "openai_mmmlu_default_it_it_machine_learning" +"task_alias": "it_it machine learning" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..7af3536531 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_management" +"task_alias": "it_it management" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..71238a4c32 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_marketing" +"task_alias": "it_it marketing" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..1731f6d343 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_medical_genetics" +"task_alias": "it_it medical genetics" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..f71d9883db --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_miscellaneous" +"task_alias": "it_it miscellaneous" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..364d8b1905 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_moral_disputes" +"task_alias": "it_it moral disputes" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..186cf31b04 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_moral_scenarios" +"task_alias": "it_it moral scenarios" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..8316313641 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_nutrition" +"task_alias": "it_it nutrition" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..2d9290488c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_philosophy" +"task_alias": "it_it philosophy" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..dd030ac054 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_prehistory" +"task_alias": "it_it prehistory" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..0e66c1f5d2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_professional_accounting" +"task_alias": "it_it professional accounting" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..f9240b9356 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_professional_law" +"task_alias": "it_it professional law" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..e578aeeb86 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_professional_medicine" +"task_alias": "it_it professional medicine" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..cfd73f67c2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_professional_psychology" +"task_alias": "it_it professional psychology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..9eb7171685 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_public_relations" +"task_alias": "it_it public relations" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..bcbcaff68f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_security_studies" +"task_alias": "it_it security studies" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..4b3db355c4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_sociology" +"task_alias": "it_it sociology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..8e38cfc481 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_social_sciences" +"task": "openai_mmmlu_default_it_it_us_foreign_policy" +"task_alias": "it_it us foreign policy" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..1b96720fb9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_other" +"task": "openai_mmmlu_default_it_it_virology" +"task_alias": "it_it virology" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..3fb5d1c828 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/it_it/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nRisposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_it_it_humanities" +"task": "openai_mmmlu_default_it_it_world_religions" +"task_alias": "it_it world religions" +"test_split": "IT_IT" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/_default_ja_jp_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/_default_ja_jp_group_info.yaml new file mode 100644 index 0000000000..436d3b378b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/_default_ja_jp_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_ja_jp +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_ja_jp_abstract_algebra +- openai_mmmlu_default_ja_jp_anatomy +- openai_mmmlu_default_ja_jp_astronomy +- openai_mmmlu_default_ja_jp_business_ethics +- openai_mmmlu_default_ja_jp_clinical_knowledge +- openai_mmmlu_default_ja_jp_college_biology +- openai_mmmlu_default_ja_jp_college_chemistry +- openai_mmmlu_default_ja_jp_college_computer_science +- openai_mmmlu_default_ja_jp_college_mathematics +- openai_mmmlu_default_ja_jp_college_medicine +- openai_mmmlu_default_ja_jp_college_physics +- openai_mmmlu_default_ja_jp_computer_security +- openai_mmmlu_default_ja_jp_conceptual_physics +- openai_mmmlu_default_ja_jp_econometrics +- openai_mmmlu_default_ja_jp_electrical_engineering +- openai_mmmlu_default_ja_jp_elementary_mathematics +- openai_mmmlu_default_ja_jp_formal_logic +- openai_mmmlu_default_ja_jp_global_facts +- openai_mmmlu_default_ja_jp_high_school_biology +- openai_mmmlu_default_ja_jp_high_school_chemistry +- openai_mmmlu_default_ja_jp_high_school_computer_science +- openai_mmmlu_default_ja_jp_high_school_european_history +- openai_mmmlu_default_ja_jp_high_school_geography +- openai_mmmlu_default_ja_jp_high_school_government_and_politics +- openai_mmmlu_default_ja_jp_high_school_macroeconomics +- openai_mmmlu_default_ja_jp_high_school_mathematics +- openai_mmmlu_default_ja_jp_high_school_microeconomics +- openai_mmmlu_default_ja_jp_high_school_physics +- openai_mmmlu_default_ja_jp_high_school_psychology +- openai_mmmlu_default_ja_jp_high_school_statistics +- openai_mmmlu_default_ja_jp_high_school_us_history +- openai_mmmlu_default_ja_jp_high_school_world_history +- openai_mmmlu_default_ja_jp_human_aging +- openai_mmmlu_default_ja_jp_human_sexuality +- openai_mmmlu_default_ja_jp_international_law +- openai_mmmlu_default_ja_jp_jurisprudence +- openai_mmmlu_default_ja_jp_logical_fallacies +- openai_mmmlu_default_ja_jp_machine_learning +- openai_mmmlu_default_ja_jp_management +- openai_mmmlu_default_ja_jp_marketing +- openai_mmmlu_default_ja_jp_medical_genetics +- openai_mmmlu_default_ja_jp_miscellaneous +- openai_mmmlu_default_ja_jp_moral_disputes +- openai_mmmlu_default_ja_jp_moral_scenarios +- openai_mmmlu_default_ja_jp_nutrition +- openai_mmmlu_default_ja_jp_philosophy +- openai_mmmlu_default_ja_jp_prehistory +- openai_mmmlu_default_ja_jp_professional_accounting +- openai_mmmlu_default_ja_jp_professional_law +- openai_mmmlu_default_ja_jp_professional_medicine +- openai_mmmlu_default_ja_jp_professional_psychology +- openai_mmmlu_default_ja_jp_public_relations +- openai_mmmlu_default_ja_jp_security_studies +- openai_mmmlu_default_ja_jp_sociology +- openai_mmmlu_default_ja_jp_us_foreign_policy +- openai_mmmlu_default_ja_jp_virology +- openai_mmmlu_default_ja_jp_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..f90a83ff25 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_abstract_algebra" +"task_alias": "ja_jp abstract algebra" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..9348cbfb74 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_anatomy" +"task_alias": "ja_jp anatomy" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..9185626aab --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_astronomy" +"task_alias": "ja_jp astronomy" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..0b351f92fb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_business_ethics" +"task_alias": "ja_jp business ethics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..879800d21c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_clinical_knowledge" +"task_alias": "ja_jp clinical knowledge" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..7bef4df728 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_college_biology" +"task_alias": "ja_jp college biology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..26a197762a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_college_chemistry" +"task_alias": "ja_jp college chemistry" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..edf6f60753 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_college_computer_science" +"task_alias": "ja_jp college computer science" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..ebe49b82c0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_college_mathematics" +"task_alias": "ja_jp college mathematics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..87c1060274 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_college_medicine" +"task_alias": "ja_jp college medicine" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..e2c2d7df0d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_college_physics" +"task_alias": "ja_jp college physics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..22c7dd5fdd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_computer_security" +"task_alias": "ja_jp computer security" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..405bb727a6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_conceptual_physics" +"task_alias": "ja_jp conceptual physics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..129747974a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_econometrics" +"task_alias": "ja_jp econometrics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..e2e350b813 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_electrical_engineering" +"task_alias": "ja_jp electrical engineering" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..7bce5c51d2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_elementary_mathematics" +"task_alias": "ja_jp elementary mathematics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..3839be5089 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_formal_logic" +"task_alias": "ja_jp formal logic" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..8d2d290d7d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_global_facts" +"task_alias": "ja_jp global facts" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..596343f4d9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_biology" +"task_alias": "ja_jp high school biology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..a87232e068 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_chemistry" +"task_alias": "ja_jp high school chemistry" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..ba8181cc2a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_computer_science" +"task_alias": "ja_jp high school computer science" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..5afbbea594 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_high_school_european_history" +"task_alias": "ja_jp high school european history" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..0fafec8cb2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_high_school_geography" +"task_alias": "ja_jp high school geography" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..1d9107cec4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_high_school_government_and_politics" +"task_alias": "ja_jp high school government and politics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..2266576c6a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_high_school_macroeconomics" +"task_alias": "ja_jp high school macroeconomics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..51aca8d420 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_mathematics" +"task_alias": "ja_jp high school mathematics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..b0c2002032 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_high_school_microeconomics" +"task_alias": "ja_jp high school microeconomics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..43e799ba6d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_physics" +"task_alias": "ja_jp high school physics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..3df5ef3973 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_high_school_psychology" +"task_alias": "ja_jp high school psychology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..9817a5569b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_high_school_statistics" +"task_alias": "ja_jp high school statistics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..9166364762 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_high_school_us_history" +"task_alias": "ja_jp high school us history" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..e46f0f9363 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_high_school_world_history" +"task_alias": "ja_jp high school world history" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..e9082a277b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_human_aging" +"task_alias": "ja_jp human aging" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..adf416da1c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_human_sexuality" +"task_alias": "ja_jp human sexuality" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..ec3e6eb162 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_international_law" +"task_alias": "ja_jp international law" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..c01f03ce39 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_jurisprudence" +"task_alias": "ja_jp jurisprudence" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..74a163e97f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_logical_fallacies" +"task_alias": "ja_jp logical fallacies" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..8207efcbf6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_stem" +"task": "openai_mmmlu_default_ja_jp_machine_learning" +"task_alias": "ja_jp machine learning" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..b6c7c7e8b8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_management" +"task_alias": "ja_jp management" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..ae9741da17 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_marketing" +"task_alias": "ja_jp marketing" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..3eb11bc8da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_medical_genetics" +"task_alias": "ja_jp medical genetics" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..c4d1b786bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_miscellaneous" +"task_alias": "ja_jp miscellaneous" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..2d1f458a07 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_moral_disputes" +"task_alias": "ja_jp moral disputes" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..6965749950 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_moral_scenarios" +"task_alias": "ja_jp moral scenarios" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..e36a9d46f8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_nutrition" +"task_alias": "ja_jp nutrition" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..50e467f44d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_philosophy" +"task_alias": "ja_jp philosophy" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..a4b4fea486 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_prehistory" +"task_alias": "ja_jp prehistory" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..95e2cfd4ca --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_professional_accounting" +"task_alias": "ja_jp professional accounting" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..c994852702 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_professional_law" +"task_alias": "ja_jp professional law" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..afdc20f563 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_professional_medicine" +"task_alias": "ja_jp professional medicine" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..3255bc09db --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_professional_psychology" +"task_alias": "ja_jp professional psychology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..e8aedbf9ec --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_public_relations" +"task_alias": "ja_jp public relations" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..a562bdc1e0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_security_studies" +"task_alias": "ja_jp security studies" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..d2f011d280 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_sociology" +"task_alias": "ja_jp sociology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..82008667f8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_social_sciences" +"task": "openai_mmmlu_default_ja_jp_us_foreign_policy" +"task_alias": "ja_jp us foreign policy" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..0ea5df89b1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_other" +"task": "openai_mmmlu_default_ja_jp_virology" +"task_alias": "ja_jp virology" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..ef216b0599 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ja_jp/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n答え:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ja_jp_humanities" +"task": "openai_mmmlu_default_ja_jp_world_religions" +"task_alias": "ja_jp world religions" +"test_split": "JA_JP" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/_default_ko_kr_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/_default_ko_kr_group_info.yaml new file mode 100644 index 0000000000..202504b76b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/_default_ko_kr_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_ko_kr +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_ko_kr_abstract_algebra +- openai_mmmlu_default_ko_kr_anatomy +- openai_mmmlu_default_ko_kr_astronomy +- openai_mmmlu_default_ko_kr_business_ethics +- openai_mmmlu_default_ko_kr_clinical_knowledge +- openai_mmmlu_default_ko_kr_college_biology +- openai_mmmlu_default_ko_kr_college_chemistry +- openai_mmmlu_default_ko_kr_college_computer_science +- openai_mmmlu_default_ko_kr_college_mathematics +- openai_mmmlu_default_ko_kr_college_medicine +- openai_mmmlu_default_ko_kr_college_physics +- openai_mmmlu_default_ko_kr_computer_security +- openai_mmmlu_default_ko_kr_conceptual_physics +- openai_mmmlu_default_ko_kr_econometrics +- openai_mmmlu_default_ko_kr_electrical_engineering +- openai_mmmlu_default_ko_kr_elementary_mathematics +- openai_mmmlu_default_ko_kr_formal_logic +- openai_mmmlu_default_ko_kr_global_facts +- openai_mmmlu_default_ko_kr_high_school_biology +- openai_mmmlu_default_ko_kr_high_school_chemistry +- openai_mmmlu_default_ko_kr_high_school_computer_science +- openai_mmmlu_default_ko_kr_high_school_european_history +- openai_mmmlu_default_ko_kr_high_school_geography +- openai_mmmlu_default_ko_kr_high_school_government_and_politics +- openai_mmmlu_default_ko_kr_high_school_macroeconomics +- openai_mmmlu_default_ko_kr_high_school_mathematics +- openai_mmmlu_default_ko_kr_high_school_microeconomics +- openai_mmmlu_default_ko_kr_high_school_physics +- openai_mmmlu_default_ko_kr_high_school_psychology +- openai_mmmlu_default_ko_kr_high_school_statistics +- openai_mmmlu_default_ko_kr_high_school_us_history +- openai_mmmlu_default_ko_kr_high_school_world_history +- openai_mmmlu_default_ko_kr_human_aging +- openai_mmmlu_default_ko_kr_human_sexuality +- openai_mmmlu_default_ko_kr_international_law +- openai_mmmlu_default_ko_kr_jurisprudence +- openai_mmmlu_default_ko_kr_logical_fallacies +- openai_mmmlu_default_ko_kr_machine_learning +- openai_mmmlu_default_ko_kr_management +- openai_mmmlu_default_ko_kr_marketing +- openai_mmmlu_default_ko_kr_medical_genetics +- openai_mmmlu_default_ko_kr_miscellaneous +- openai_mmmlu_default_ko_kr_moral_disputes +- openai_mmmlu_default_ko_kr_moral_scenarios +- openai_mmmlu_default_ko_kr_nutrition +- openai_mmmlu_default_ko_kr_philosophy +- openai_mmmlu_default_ko_kr_prehistory +- openai_mmmlu_default_ko_kr_professional_accounting +- openai_mmmlu_default_ko_kr_professional_law +- openai_mmmlu_default_ko_kr_professional_medicine +- openai_mmmlu_default_ko_kr_professional_psychology +- openai_mmmlu_default_ko_kr_public_relations +- openai_mmmlu_default_ko_kr_security_studies +- openai_mmmlu_default_ko_kr_sociology +- openai_mmmlu_default_ko_kr_us_foreign_policy +- openai_mmmlu_default_ko_kr_virology +- openai_mmmlu_default_ko_kr_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..10ae1e3d3d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_abstract_algebra" +"task_alias": "ko_kr abstract algebra" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..f79c1ef735 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_anatomy" +"task_alias": "ko_kr anatomy" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..450eba7fe6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_astronomy" +"task_alias": "ko_kr astronomy" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..8b4782b089 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_business_ethics" +"task_alias": "ko_kr business ethics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..13b660f6d7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_clinical_knowledge" +"task_alias": "ko_kr clinical knowledge" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..c9e353e79c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_college_biology" +"task_alias": "ko_kr college biology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..c66bbd7904 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_college_chemistry" +"task_alias": "ko_kr college chemistry" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..6c0c55f58e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_college_computer_science" +"task_alias": "ko_kr college computer science" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..9518c03df5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_college_mathematics" +"task_alias": "ko_kr college mathematics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..a4b2cebfcc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_college_medicine" +"task_alias": "ko_kr college medicine" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..ba5c987d76 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_college_physics" +"task_alias": "ko_kr college physics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..6b903baa36 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_computer_security" +"task_alias": "ko_kr computer security" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..eb859cceea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_conceptual_physics" +"task_alias": "ko_kr conceptual physics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..864c4222c5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_econometrics" +"task_alias": "ko_kr econometrics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..5f5a9eae7f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_electrical_engineering" +"task_alias": "ko_kr electrical engineering" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..5f46808469 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_elementary_mathematics" +"task_alias": "ko_kr elementary mathematics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..74e621e8d1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_formal_logic" +"task_alias": "ko_kr formal logic" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..a58cf1521a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_global_facts" +"task_alias": "ko_kr global facts" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..9bd39ea837 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_biology" +"task_alias": "ko_kr high school biology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..9e232ed619 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_chemistry" +"task_alias": "ko_kr high school chemistry" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..1910a150ae --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_computer_science" +"task_alias": "ko_kr high school computer science" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..2226b97948 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_high_school_european_history" +"task_alias": "ko_kr high school european history" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..2bc28479f3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_high_school_geography" +"task_alias": "ko_kr high school geography" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..7b01b61184 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_high_school_government_and_politics" +"task_alias": "ko_kr high school government and politics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..00451721ea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_high_school_macroeconomics" +"task_alias": "ko_kr high school macroeconomics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..68d4423fe4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_mathematics" +"task_alias": "ko_kr high school mathematics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..8427933a37 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_high_school_microeconomics" +"task_alias": "ko_kr high school microeconomics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..63aac8bb57 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_physics" +"task_alias": "ko_kr high school physics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..6f613391d6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_high_school_psychology" +"task_alias": "ko_kr high school psychology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..a311251863 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_high_school_statistics" +"task_alias": "ko_kr high school statistics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..2ad2f944ed --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_high_school_us_history" +"task_alias": "ko_kr high school us history" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..98ee27e6b2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_high_school_world_history" +"task_alias": "ko_kr high school world history" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..b8c6559bee --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_human_aging" +"task_alias": "ko_kr human aging" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..f3f77f6032 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_human_sexuality" +"task_alias": "ko_kr human sexuality" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..88f7759e4f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_international_law" +"task_alias": "ko_kr international law" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..c7486af609 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_jurisprudence" +"task_alias": "ko_kr jurisprudence" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..9f7446bb25 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_logical_fallacies" +"task_alias": "ko_kr logical fallacies" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..e2a2ebb2fc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_stem" +"task": "openai_mmmlu_default_ko_kr_machine_learning" +"task_alias": "ko_kr machine learning" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..1b334d6399 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_management" +"task_alias": "ko_kr management" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..9ab41d18a5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_marketing" +"task_alias": "ko_kr marketing" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..b8797df7a2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_medical_genetics" +"task_alias": "ko_kr medical genetics" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..43ee9f9990 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_miscellaneous" +"task_alias": "ko_kr miscellaneous" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..3f7a6fef31 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_moral_disputes" +"task_alias": "ko_kr moral disputes" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..b336f34647 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_moral_scenarios" +"task_alias": "ko_kr moral scenarios" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..55c46cae24 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_nutrition" +"task_alias": "ko_kr nutrition" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..972ad7d266 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_philosophy" +"task_alias": "ko_kr philosophy" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..d2c006c6da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_prehistory" +"task_alias": "ko_kr prehistory" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..c86b39b082 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_professional_accounting" +"task_alias": "ko_kr professional accounting" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..fb90022b23 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_professional_law" +"task_alias": "ko_kr professional law" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..0009145609 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_professional_medicine" +"task_alias": "ko_kr professional medicine" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..d95241e532 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_professional_psychology" +"task_alias": "ko_kr professional psychology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..ea09567fd3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_public_relations" +"task_alias": "ko_kr public relations" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..0e07d213d9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_security_studies" +"task_alias": "ko_kr security studies" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..20087678d4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_sociology" +"task_alias": "ko_kr sociology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..af9354986e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_social_sciences" +"task": "openai_mmmlu_default_ko_kr_us_foreign_policy" +"task_alias": "ko_kr us foreign policy" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..309cda2c8c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_other" +"task": "openai_mmmlu_default_ko_kr_virology" +"task_alias": "ko_kr virology" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..26cde4775a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/ko_kr/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n답변:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_ko_kr_humanities" +"task": "openai_mmmlu_default_ko_kr_world_religions" +"task_alias": "ko_kr world religions" +"test_split": "KO_KR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/_default_pt_br_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/_default_pt_br_group_info.yaml new file mode 100644 index 0000000000..2be6090b3c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/_default_pt_br_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_pt_br +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_pt_br_abstract_algebra +- openai_mmmlu_default_pt_br_anatomy +- openai_mmmlu_default_pt_br_astronomy +- openai_mmmlu_default_pt_br_business_ethics +- openai_mmmlu_default_pt_br_clinical_knowledge +- openai_mmmlu_default_pt_br_college_biology +- openai_mmmlu_default_pt_br_college_chemistry +- openai_mmmlu_default_pt_br_college_computer_science +- openai_mmmlu_default_pt_br_college_mathematics +- openai_mmmlu_default_pt_br_college_medicine +- openai_mmmlu_default_pt_br_college_physics +- openai_mmmlu_default_pt_br_computer_security +- openai_mmmlu_default_pt_br_conceptual_physics +- openai_mmmlu_default_pt_br_econometrics +- openai_mmmlu_default_pt_br_electrical_engineering +- openai_mmmlu_default_pt_br_elementary_mathematics +- openai_mmmlu_default_pt_br_formal_logic +- openai_mmmlu_default_pt_br_global_facts +- openai_mmmlu_default_pt_br_high_school_biology +- openai_mmmlu_default_pt_br_high_school_chemistry +- openai_mmmlu_default_pt_br_high_school_computer_science +- openai_mmmlu_default_pt_br_high_school_european_history +- openai_mmmlu_default_pt_br_high_school_geography +- openai_mmmlu_default_pt_br_high_school_government_and_politics +- openai_mmmlu_default_pt_br_high_school_macroeconomics +- openai_mmmlu_default_pt_br_high_school_mathematics +- openai_mmmlu_default_pt_br_high_school_microeconomics +- openai_mmmlu_default_pt_br_high_school_physics +- openai_mmmlu_default_pt_br_high_school_psychology +- openai_mmmlu_default_pt_br_high_school_statistics +- openai_mmmlu_default_pt_br_high_school_us_history +- openai_mmmlu_default_pt_br_high_school_world_history +- openai_mmmlu_default_pt_br_human_aging +- openai_mmmlu_default_pt_br_human_sexuality +- openai_mmmlu_default_pt_br_international_law +- openai_mmmlu_default_pt_br_jurisprudence +- openai_mmmlu_default_pt_br_logical_fallacies +- openai_mmmlu_default_pt_br_machine_learning +- openai_mmmlu_default_pt_br_management +- openai_mmmlu_default_pt_br_marketing +- openai_mmmlu_default_pt_br_medical_genetics +- openai_mmmlu_default_pt_br_miscellaneous +- openai_mmmlu_default_pt_br_moral_disputes +- openai_mmmlu_default_pt_br_moral_scenarios +- openai_mmmlu_default_pt_br_nutrition +- openai_mmmlu_default_pt_br_philosophy +- openai_mmmlu_default_pt_br_prehistory +- openai_mmmlu_default_pt_br_professional_accounting +- openai_mmmlu_default_pt_br_professional_law +- openai_mmmlu_default_pt_br_professional_medicine +- openai_mmmlu_default_pt_br_professional_psychology +- openai_mmmlu_default_pt_br_public_relations +- openai_mmmlu_default_pt_br_security_studies +- openai_mmmlu_default_pt_br_sociology +- openai_mmmlu_default_pt_br_us_foreign_policy +- openai_mmmlu_default_pt_br_virology +- openai_mmmlu_default_pt_br_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..f3cd577d2f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_abstract_algebra" +"task_alias": "pt_br abstract algebra" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..2c41989560 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_anatomy" +"task_alias": "pt_br anatomy" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..284c3056aa --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_astronomy" +"task_alias": "pt_br astronomy" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..a5a32c1b6c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_business_ethics" +"task_alias": "pt_br business ethics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..b7a581f05a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_clinical_knowledge" +"task_alias": "pt_br clinical knowledge" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..c56b35b5fb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_college_biology" +"task_alias": "pt_br college biology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..2fae4cd5c7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_college_chemistry" +"task_alias": "pt_br college chemistry" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..2950cabc2f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_college_computer_science" +"task_alias": "pt_br college computer science" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..c822b74725 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_college_mathematics" +"task_alias": "pt_br college mathematics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..81229900d9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_college_medicine" +"task_alias": "pt_br college medicine" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..5d4e90a1d0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_college_physics" +"task_alias": "pt_br college physics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..27bab4af15 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_computer_security" +"task_alias": "pt_br computer security" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..c06dc7c042 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_conceptual_physics" +"task_alias": "pt_br conceptual physics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..1185656bea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_econometrics" +"task_alias": "pt_br econometrics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..682a82dccd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_electrical_engineering" +"task_alias": "pt_br electrical engineering" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..32d35f393d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_elementary_mathematics" +"task_alias": "pt_br elementary mathematics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..5e14d6facc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_formal_logic" +"task_alias": "pt_br formal logic" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..e7467c41b2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_global_facts" +"task_alias": "pt_br global facts" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..03d3f1509e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_biology" +"task_alias": "pt_br high school biology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..d5e3f24040 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_chemistry" +"task_alias": "pt_br high school chemistry" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..31290539c5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_computer_science" +"task_alias": "pt_br high school computer science" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..8cfbc4a77d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_high_school_european_history" +"task_alias": "pt_br high school european history" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..ad9039d2a6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_high_school_geography" +"task_alias": "pt_br high school geography" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..5fb3d8100c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_high_school_government_and_politics" +"task_alias": "pt_br high school government and politics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..2ded5e0e71 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_high_school_macroeconomics" +"task_alias": "pt_br high school macroeconomics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..f57338b741 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_mathematics" +"task_alias": "pt_br high school mathematics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..ab34870b61 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_high_school_microeconomics" +"task_alias": "pt_br high school microeconomics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..48dfb313e3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_physics" +"task_alias": "pt_br high school physics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..8a5a7296ff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_high_school_psychology" +"task_alias": "pt_br high school psychology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..d202a07cd8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_high_school_statistics" +"task_alias": "pt_br high school statistics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..eaf2c5d880 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_high_school_us_history" +"task_alias": "pt_br high school us history" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..99657af69a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_high_school_world_history" +"task_alias": "pt_br high school world history" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..c113b415a9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_human_aging" +"task_alias": "pt_br human aging" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..c3b460c99d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_human_sexuality" +"task_alias": "pt_br human sexuality" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..f337e8195d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_international_law" +"task_alias": "pt_br international law" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..e2ff9f9558 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_jurisprudence" +"task_alias": "pt_br jurisprudence" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..57b7777459 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_logical_fallacies" +"task_alias": "pt_br logical fallacies" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..b79e395daa --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_stem" +"task": "openai_mmmlu_default_pt_br_machine_learning" +"task_alias": "pt_br machine learning" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..12bb97552a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_management" +"task_alias": "pt_br management" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..28a9f2ea5b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_marketing" +"task_alias": "pt_br marketing" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..b52146d7f8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_medical_genetics" +"task_alias": "pt_br medical genetics" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..8d552ace93 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_miscellaneous" +"task_alias": "pt_br miscellaneous" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..59029173ac --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_moral_disputes" +"task_alias": "pt_br moral disputes" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..a87e2c8c61 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_moral_scenarios" +"task_alias": "pt_br moral scenarios" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..ee094f1bf1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_nutrition" +"task_alias": "pt_br nutrition" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..7c8fe5afeb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_philosophy" +"task_alias": "pt_br philosophy" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..3bfaaa573c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_prehistory" +"task_alias": "pt_br prehistory" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..6b9f8a1dd2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_professional_accounting" +"task_alias": "pt_br professional accounting" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..8e82173fe5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_professional_law" +"task_alias": "pt_br professional law" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..c22a35a889 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_professional_medicine" +"task_alias": "pt_br professional medicine" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..730e96ff4a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_professional_psychology" +"task_alias": "pt_br professional psychology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..363ee0577e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_public_relations" +"task_alias": "pt_br public relations" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..f56d32d76f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_security_studies" +"task_alias": "pt_br security studies" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..447142126c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_sociology" +"task_alias": "pt_br sociology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..7e32d6d907 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_social_sciences" +"task": "openai_mmmlu_default_pt_br_us_foreign_policy" +"task_alias": "pt_br us foreign policy" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..7f2f20def3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_other" +"task": "openai_mmmlu_default_pt_br_virology" +"task_alias": "pt_br virology" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..3f313dc4aa --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/pt_br/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nResposta:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_pt_br_humanities" +"task": "openai_mmmlu_default_pt_br_world_religions" +"task_alias": "pt_br world religions" +"test_split": "PT_BR" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/_default_sw_ke_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/_default_sw_ke_group_info.yaml new file mode 100644 index 0000000000..06ec8f2691 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/_default_sw_ke_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_sw_ke +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_sw_ke_abstract_algebra +- openai_mmmlu_default_sw_ke_anatomy +- openai_mmmlu_default_sw_ke_astronomy +- openai_mmmlu_default_sw_ke_business_ethics +- openai_mmmlu_default_sw_ke_clinical_knowledge +- openai_mmmlu_default_sw_ke_college_biology +- openai_mmmlu_default_sw_ke_college_chemistry +- openai_mmmlu_default_sw_ke_college_computer_science +- openai_mmmlu_default_sw_ke_college_mathematics +- openai_mmmlu_default_sw_ke_college_medicine +- openai_mmmlu_default_sw_ke_college_physics +- openai_mmmlu_default_sw_ke_computer_security +- openai_mmmlu_default_sw_ke_conceptual_physics +- openai_mmmlu_default_sw_ke_econometrics +- openai_mmmlu_default_sw_ke_electrical_engineering +- openai_mmmlu_default_sw_ke_elementary_mathematics +- openai_mmmlu_default_sw_ke_formal_logic +- openai_mmmlu_default_sw_ke_global_facts +- openai_mmmlu_default_sw_ke_high_school_biology +- openai_mmmlu_default_sw_ke_high_school_chemistry +- openai_mmmlu_default_sw_ke_high_school_computer_science +- openai_mmmlu_default_sw_ke_high_school_european_history +- openai_mmmlu_default_sw_ke_high_school_geography +- openai_mmmlu_default_sw_ke_high_school_government_and_politics +- openai_mmmlu_default_sw_ke_high_school_macroeconomics +- openai_mmmlu_default_sw_ke_high_school_mathematics +- openai_mmmlu_default_sw_ke_high_school_microeconomics +- openai_mmmlu_default_sw_ke_high_school_physics +- openai_mmmlu_default_sw_ke_high_school_psychology +- openai_mmmlu_default_sw_ke_high_school_statistics +- openai_mmmlu_default_sw_ke_high_school_us_history +- openai_mmmlu_default_sw_ke_high_school_world_history +- openai_mmmlu_default_sw_ke_human_aging +- openai_mmmlu_default_sw_ke_human_sexuality +- openai_mmmlu_default_sw_ke_international_law +- openai_mmmlu_default_sw_ke_jurisprudence +- openai_mmmlu_default_sw_ke_logical_fallacies +- openai_mmmlu_default_sw_ke_machine_learning +- openai_mmmlu_default_sw_ke_management +- openai_mmmlu_default_sw_ke_marketing +- openai_mmmlu_default_sw_ke_medical_genetics +- openai_mmmlu_default_sw_ke_miscellaneous +- openai_mmmlu_default_sw_ke_moral_disputes +- openai_mmmlu_default_sw_ke_moral_scenarios +- openai_mmmlu_default_sw_ke_nutrition +- openai_mmmlu_default_sw_ke_philosophy +- openai_mmmlu_default_sw_ke_prehistory +- openai_mmmlu_default_sw_ke_professional_accounting +- openai_mmmlu_default_sw_ke_professional_law +- openai_mmmlu_default_sw_ke_professional_medicine +- openai_mmmlu_default_sw_ke_professional_psychology +- openai_mmmlu_default_sw_ke_public_relations +- openai_mmmlu_default_sw_ke_security_studies +- openai_mmmlu_default_sw_ke_sociology +- openai_mmmlu_default_sw_ke_us_foreign_policy +- openai_mmmlu_default_sw_ke_virology +- openai_mmmlu_default_sw_ke_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..b17024d6dd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_abstract_algebra" +"task_alias": "sw_ke abstract algebra" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..8610c31891 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_anatomy" +"task_alias": "sw_ke anatomy" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..d675455e66 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_astronomy" +"task_alias": "sw_ke astronomy" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..47a9919b72 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_business_ethics" +"task_alias": "sw_ke business ethics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..02ee256819 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_clinical_knowledge" +"task_alias": "sw_ke clinical knowledge" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..32e32fe8ca --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_college_biology" +"task_alias": "sw_ke college biology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..7d5a089add --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_college_chemistry" +"task_alias": "sw_ke college chemistry" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..611e2ceaa9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_college_computer_science" +"task_alias": "sw_ke college computer science" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..f8e0331256 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_college_mathematics" +"task_alias": "sw_ke college mathematics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..98b70ec130 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_college_medicine" +"task_alias": "sw_ke college medicine" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..b20595cdd3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_college_physics" +"task_alias": "sw_ke college physics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..2678893cff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_computer_security" +"task_alias": "sw_ke computer security" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..ad758d6578 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_conceptual_physics" +"task_alias": "sw_ke conceptual physics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..1863148691 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_econometrics" +"task_alias": "sw_ke econometrics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..52e8d58ada --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_electrical_engineering" +"task_alias": "sw_ke electrical engineering" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..4d44e12b02 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_elementary_mathematics" +"task_alias": "sw_ke elementary mathematics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..8992ac20f1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_formal_logic" +"task_alias": "sw_ke formal logic" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..60786095ca --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_global_facts" +"task_alias": "sw_ke global facts" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..afa9b911cc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_biology" +"task_alias": "sw_ke high school biology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..833020b102 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_chemistry" +"task_alias": "sw_ke high school chemistry" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..d87c1c80d3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_computer_science" +"task_alias": "sw_ke high school computer science" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..5c4d8e484c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_high_school_european_history" +"task_alias": "sw_ke high school european history" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..0bc69098e6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_high_school_geography" +"task_alias": "sw_ke high school geography" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..7a281951c1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_high_school_government_and_politics" +"task_alias": "sw_ke high school government and politics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..cadb17f9fc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_high_school_macroeconomics" +"task_alias": "sw_ke high school macroeconomics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..bda9e8cb35 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_mathematics" +"task_alias": "sw_ke high school mathematics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..c70d3bc162 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_high_school_microeconomics" +"task_alias": "sw_ke high school microeconomics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..cda674d39b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_physics" +"task_alias": "sw_ke high school physics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..3fc737b62a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_high_school_psychology" +"task_alias": "sw_ke high school psychology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..167fcb691f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_high_school_statistics" +"task_alias": "sw_ke high school statistics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..f7aa3fbd76 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_high_school_us_history" +"task_alias": "sw_ke high school us history" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..2796fd949f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_high_school_world_history" +"task_alias": "sw_ke high school world history" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..cc72317a2a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_human_aging" +"task_alias": "sw_ke human aging" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..b7ae235b13 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_human_sexuality" +"task_alias": "sw_ke human sexuality" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..79df2f92a2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_international_law" +"task_alias": "sw_ke international law" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..f67914a7bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_jurisprudence" +"task_alias": "sw_ke jurisprudence" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..952b9c0c58 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_logical_fallacies" +"task_alias": "sw_ke logical fallacies" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..c768092cd2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_stem" +"task": "openai_mmmlu_default_sw_ke_machine_learning" +"task_alias": "sw_ke machine learning" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..5576d2d630 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_management" +"task_alias": "sw_ke management" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..2f1c22f664 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_marketing" +"task_alias": "sw_ke marketing" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..b04dae4348 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_medical_genetics" +"task_alias": "sw_ke medical genetics" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..c04dc84f01 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_miscellaneous" +"task_alias": "sw_ke miscellaneous" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..89fec2c4ce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_moral_disputes" +"task_alias": "sw_ke moral disputes" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..4717606857 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_moral_scenarios" +"task_alias": "sw_ke moral scenarios" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..3fb5ad8025 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_nutrition" +"task_alias": "sw_ke nutrition" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..07941858be --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_philosophy" +"task_alias": "sw_ke philosophy" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..9a6e2b5b83 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_prehistory" +"task_alias": "sw_ke prehistory" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..24182795ce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_professional_accounting" +"task_alias": "sw_ke professional accounting" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..d1f2b79743 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_professional_law" +"task_alias": "sw_ke professional law" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..467175a33c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_professional_medicine" +"task_alias": "sw_ke professional medicine" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..2d95bae681 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_professional_psychology" +"task_alias": "sw_ke professional psychology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..dc6e8631b3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_public_relations" +"task_alias": "sw_ke public relations" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..9e8d80ea99 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_security_studies" +"task_alias": "sw_ke security studies" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..58fe9e56bf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_sociology" +"task_alias": "sw_ke sociology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..c60f462bce --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_social_sciences" +"task": "openai_mmmlu_default_sw_ke_us_foreign_policy" +"task_alias": "sw_ke us foreign policy" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..ad967894f1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_other" +"task": "openai_mmmlu_default_sw_ke_virology" +"task_alias": "sw_ke virology" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..7d255332c8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/sw_ke/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nJawabu:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_sw_ke_humanities" +"task": "openai_mmmlu_default_sw_ke_world_religions" +"task_alias": "sw_ke world religions" +"test_split": "SW_KE" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/_default_yo_ng_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/_default_yo_ng_group_info.yaml new file mode 100644 index 0000000000..3116e83956 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/_default_yo_ng_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_yo_ng +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_yo_ng_abstract_algebra +- openai_mmmlu_default_yo_ng_anatomy +- openai_mmmlu_default_yo_ng_astronomy +- openai_mmmlu_default_yo_ng_business_ethics +- openai_mmmlu_default_yo_ng_clinical_knowledge +- openai_mmmlu_default_yo_ng_college_biology +- openai_mmmlu_default_yo_ng_college_chemistry +- openai_mmmlu_default_yo_ng_college_computer_science +- openai_mmmlu_default_yo_ng_college_mathematics +- openai_mmmlu_default_yo_ng_college_medicine +- openai_mmmlu_default_yo_ng_college_physics +- openai_mmmlu_default_yo_ng_computer_security +- openai_mmmlu_default_yo_ng_conceptual_physics +- openai_mmmlu_default_yo_ng_econometrics +- openai_mmmlu_default_yo_ng_electrical_engineering +- openai_mmmlu_default_yo_ng_elementary_mathematics +- openai_mmmlu_default_yo_ng_formal_logic +- openai_mmmlu_default_yo_ng_global_facts +- openai_mmmlu_default_yo_ng_high_school_biology +- openai_mmmlu_default_yo_ng_high_school_chemistry +- openai_mmmlu_default_yo_ng_high_school_computer_science +- openai_mmmlu_default_yo_ng_high_school_european_history +- openai_mmmlu_default_yo_ng_high_school_geography +- openai_mmmlu_default_yo_ng_high_school_government_and_politics +- openai_mmmlu_default_yo_ng_high_school_macroeconomics +- openai_mmmlu_default_yo_ng_high_school_mathematics +- openai_mmmlu_default_yo_ng_high_school_microeconomics +- openai_mmmlu_default_yo_ng_high_school_physics +- openai_mmmlu_default_yo_ng_high_school_psychology +- openai_mmmlu_default_yo_ng_high_school_statistics +- openai_mmmlu_default_yo_ng_high_school_us_history +- openai_mmmlu_default_yo_ng_high_school_world_history +- openai_mmmlu_default_yo_ng_human_aging +- openai_mmmlu_default_yo_ng_human_sexuality +- openai_mmmlu_default_yo_ng_international_law +- openai_mmmlu_default_yo_ng_jurisprudence +- openai_mmmlu_default_yo_ng_logical_fallacies +- openai_mmmlu_default_yo_ng_machine_learning +- openai_mmmlu_default_yo_ng_management +- openai_mmmlu_default_yo_ng_marketing +- openai_mmmlu_default_yo_ng_medical_genetics +- openai_mmmlu_default_yo_ng_miscellaneous +- openai_mmmlu_default_yo_ng_moral_disputes +- openai_mmmlu_default_yo_ng_moral_scenarios +- openai_mmmlu_default_yo_ng_nutrition +- openai_mmmlu_default_yo_ng_philosophy +- openai_mmmlu_default_yo_ng_prehistory +- openai_mmmlu_default_yo_ng_professional_accounting +- openai_mmmlu_default_yo_ng_professional_law +- openai_mmmlu_default_yo_ng_professional_medicine +- openai_mmmlu_default_yo_ng_professional_psychology +- openai_mmmlu_default_yo_ng_public_relations +- openai_mmmlu_default_yo_ng_security_studies +- openai_mmmlu_default_yo_ng_sociology +- openai_mmmlu_default_yo_ng_us_foreign_policy +- openai_mmmlu_default_yo_ng_virology +- openai_mmmlu_default_yo_ng_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..be8db6a8f5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_abstract_algebra" +"task_alias": "yo_ng abstract algebra" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..e220cdee49 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_anatomy" +"task_alias": "yo_ng anatomy" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..2265b0c9b7 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_astronomy" +"task_alias": "yo_ng astronomy" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..997b37fce2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_business_ethics" +"task_alias": "yo_ng business ethics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..a47ad1380b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_clinical_knowledge" +"task_alias": "yo_ng clinical knowledge" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..d00ff1d4b3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_college_biology" +"task_alias": "yo_ng college biology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..d4b0c2e110 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_college_chemistry" +"task_alias": "yo_ng college chemistry" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..8edd013b8d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_college_computer_science" +"task_alias": "yo_ng college computer science" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..2521616258 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_college_mathematics" +"task_alias": "yo_ng college mathematics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..515df717ec --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_college_medicine" +"task_alias": "yo_ng college medicine" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..d9a118f60e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_college_physics" +"task_alias": "yo_ng college physics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..a2e3c88522 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_computer_security" +"task_alias": "yo_ng computer security" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..da03f95228 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_conceptual_physics" +"task_alias": "yo_ng conceptual physics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..f409e773b8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_econometrics" +"task_alias": "yo_ng econometrics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..0e6406c336 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_electrical_engineering" +"task_alias": "yo_ng electrical engineering" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..56b75ea9fb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_elementary_mathematics" +"task_alias": "yo_ng elementary mathematics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..c0211e56fe --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_formal_logic" +"task_alias": "yo_ng formal logic" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..f4fbf079be --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_global_facts" +"task_alias": "yo_ng global facts" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..93d96edb40 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_biology" +"task_alias": "yo_ng high school biology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..da21d5c72e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_chemistry" +"task_alias": "yo_ng high school chemistry" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..d190985d78 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_computer_science" +"task_alias": "yo_ng high school computer science" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..6473966719 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_high_school_european_history" +"task_alias": "yo_ng high school european history" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..507d7dc0b6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_high_school_geography" +"task_alias": "yo_ng high school geography" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..5182f6887e --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_high_school_government_and_politics" +"task_alias": "yo_ng high school government and politics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..f2b1869747 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_high_school_macroeconomics" +"task_alias": "yo_ng high school macroeconomics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..d06b05bcf5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_mathematics" +"task_alias": "yo_ng high school mathematics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..49fe9ff007 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_high_school_microeconomics" +"task_alias": "yo_ng high school microeconomics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..6dd75f1d27 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_physics" +"task_alias": "yo_ng high school physics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..b0b7589e7c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_high_school_psychology" +"task_alias": "yo_ng high school psychology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..aa81c80f6a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_high_school_statistics" +"task_alias": "yo_ng high school statistics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..6247d0302a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_high_school_us_history" +"task_alias": "yo_ng high school us history" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..4ac049d6a8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_high_school_world_history" +"task_alias": "yo_ng high school world history" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..136466c7f0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_human_aging" +"task_alias": "yo_ng human aging" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..11b8c1083c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_human_sexuality" +"task_alias": "yo_ng human sexuality" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..12fba5f49f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_international_law" +"task_alias": "yo_ng international law" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..320e2dd466 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_jurisprudence" +"task_alias": "yo_ng jurisprudence" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..f8416fea43 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_logical_fallacies" +"task_alias": "yo_ng logical fallacies" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..2712319d56 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_stem" +"task": "openai_mmmlu_default_yo_ng_machine_learning" +"task_alias": "yo_ng machine learning" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..97237a71d0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_management" +"task_alias": "yo_ng management" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..7db12772c1 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_marketing" +"task_alias": "yo_ng marketing" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..6b645d4a77 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_medical_genetics" +"task_alias": "yo_ng medical genetics" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..3b0ab517e3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_miscellaneous" +"task_alias": "yo_ng miscellaneous" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..99d514c48b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_moral_disputes" +"task_alias": "yo_ng moral disputes" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..187e7c034a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_moral_scenarios" +"task_alias": "yo_ng moral scenarios" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..2b73c37a64 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_nutrition" +"task_alias": "yo_ng nutrition" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..6dd5a8d28b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_philosophy" +"task_alias": "yo_ng philosophy" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..740b324fbe --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_prehistory" +"task_alias": "yo_ng prehistory" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..3f87630c8b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_professional_accounting" +"task_alias": "yo_ng professional accounting" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..3eb2905a34 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_professional_law" +"task_alias": "yo_ng professional law" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..af6ce0d9dd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_professional_medicine" +"task_alias": "yo_ng professional medicine" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..da2ef6181c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_professional_psychology" +"task_alias": "yo_ng professional psychology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..e357f015ea --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_public_relations" +"task_alias": "yo_ng public relations" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..f8df522478 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_security_studies" +"task_alias": "yo_ng security studies" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..312eee6fef --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_sociology" +"task_alias": "yo_ng sociology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..809c756355 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_social_sciences" +"task": "openai_mmmlu_default_yo_ng_us_foreign_policy" +"task_alias": "yo_ng us foreign policy" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..5090923169 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_other" +"task": "openai_mmmlu_default_yo_ng_virology" +"task_alias": "yo_ng virology" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..ab756bbcc8 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/yo_ng/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\nIdahun:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_yo_ng_humanities" +"task": "openai_mmmlu_default_yo_ng_world_religions" +"task_alias": "yo_ng world religions" +"test_split": "YO_NG" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/_default_zh_cn_group_info.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/_default_zh_cn_group_info.yaml new file mode 100644 index 0000000000..2c765de1fb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/_default_zh_cn_group_info.yaml @@ -0,0 +1,64 @@ +aggregate_metric_list: +- metric: acc + weight_by_size: true +group: openai_mmmlu_default_zh_cn +metadata: + version: 1.0.0 +task: +- openai_mmmlu_default_zh_cn_abstract_algebra +- openai_mmmlu_default_zh_cn_anatomy +- openai_mmmlu_default_zh_cn_astronomy +- openai_mmmlu_default_zh_cn_business_ethics +- openai_mmmlu_default_zh_cn_clinical_knowledge +- openai_mmmlu_default_zh_cn_college_biology +- openai_mmmlu_default_zh_cn_college_chemistry +- openai_mmmlu_default_zh_cn_college_computer_science +- openai_mmmlu_default_zh_cn_college_mathematics +- openai_mmmlu_default_zh_cn_college_medicine +- openai_mmmlu_default_zh_cn_college_physics +- openai_mmmlu_default_zh_cn_computer_security +- openai_mmmlu_default_zh_cn_conceptual_physics +- openai_mmmlu_default_zh_cn_econometrics +- openai_mmmlu_default_zh_cn_electrical_engineering +- openai_mmmlu_default_zh_cn_elementary_mathematics +- openai_mmmlu_default_zh_cn_formal_logic +- openai_mmmlu_default_zh_cn_global_facts +- openai_mmmlu_default_zh_cn_high_school_biology +- openai_mmmlu_default_zh_cn_high_school_chemistry +- openai_mmmlu_default_zh_cn_high_school_computer_science +- openai_mmmlu_default_zh_cn_high_school_european_history +- openai_mmmlu_default_zh_cn_high_school_geography +- openai_mmmlu_default_zh_cn_high_school_government_and_politics +- openai_mmmlu_default_zh_cn_high_school_macroeconomics +- openai_mmmlu_default_zh_cn_high_school_mathematics +- openai_mmmlu_default_zh_cn_high_school_microeconomics +- openai_mmmlu_default_zh_cn_high_school_physics +- openai_mmmlu_default_zh_cn_high_school_psychology +- openai_mmmlu_default_zh_cn_high_school_statistics +- openai_mmmlu_default_zh_cn_high_school_us_history +- openai_mmmlu_default_zh_cn_high_school_world_history +- openai_mmmlu_default_zh_cn_human_aging +- openai_mmmlu_default_zh_cn_human_sexuality +- openai_mmmlu_default_zh_cn_international_law +- openai_mmmlu_default_zh_cn_jurisprudence +- openai_mmmlu_default_zh_cn_logical_fallacies +- openai_mmmlu_default_zh_cn_machine_learning +- openai_mmmlu_default_zh_cn_management +- openai_mmmlu_default_zh_cn_marketing +- openai_mmmlu_default_zh_cn_medical_genetics +- openai_mmmlu_default_zh_cn_miscellaneous +- openai_mmmlu_default_zh_cn_moral_disputes +- openai_mmmlu_default_zh_cn_moral_scenarios +- openai_mmmlu_default_zh_cn_nutrition +- openai_mmmlu_default_zh_cn_philosophy +- openai_mmmlu_default_zh_cn_prehistory +- openai_mmmlu_default_zh_cn_professional_accounting +- openai_mmmlu_default_zh_cn_professional_law +- openai_mmmlu_default_zh_cn_professional_medicine +- openai_mmmlu_default_zh_cn_professional_psychology +- openai_mmmlu_default_zh_cn_public_relations +- openai_mmmlu_default_zh_cn_security_studies +- openai_mmmlu_default_zh_cn_sociology +- openai_mmmlu_default_zh_cn_us_foreign_policy +- openai_mmmlu_default_zh_cn_virology +- openai_mmmlu_default_zh_cn_world_religions diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_abstract_algebra.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_abstract_algebra.yaml new file mode 100644 index 0000000000..5204e25767 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_abstract_algebra.yaml @@ -0,0 +1,16 @@ +"dataset_name": "abstract_algebra" +"description": "The following are multiple choice questions (with answers) about abstract_algebra.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_abstract_algebra" +"task_alias": "zh_cn abstract algebra" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_anatomy.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_anatomy.yaml new file mode 100644 index 0000000000..47d7452ed5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_anatomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "anatomy" +"description": "The following are multiple choice questions (with answers) about anatomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_anatomy" +"task_alias": "zh_cn anatomy" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_astronomy.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_astronomy.yaml new file mode 100644 index 0000000000..87f4f4b6a6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_astronomy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "astronomy" +"description": "The following are multiple choice questions (with answers) about astronomy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_astronomy" +"task_alias": "zh_cn astronomy" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_business_ethics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_business_ethics.yaml new file mode 100644 index 0000000000..8acd922361 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_business_ethics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "business_ethics" +"description": "The following are multiple choice questions (with answers) about business_ethics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_business_ethics" +"task_alias": "zh_cn business ethics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_clinical_knowledge.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_clinical_knowledge.yaml new file mode 100644 index 0000000000..72ae1413f4 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_clinical_knowledge.yaml @@ -0,0 +1,16 @@ +"dataset_name": "clinical_knowledge" +"description": "The following are multiple choice questions (with answers) about clinical_knowledge.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_clinical_knowledge" +"task_alias": "zh_cn clinical knowledge" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_biology.yaml new file mode 100644 index 0000000000..1051e6b539 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_biology" +"description": "The following are multiple choice questions (with answers) about college_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_college_biology" +"task_alias": "zh_cn college biology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_chemistry.yaml new file mode 100644 index 0000000000..a24903d3da --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_chemistry" +"description": "The following are multiple choice questions (with answers) about college_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_college_chemistry" +"task_alias": "zh_cn college chemistry" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_computer_science.yaml new file mode 100644 index 0000000000..0c0fa16e38 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_computer_science" +"description": "The following are multiple choice questions (with answers) about college_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_college_computer_science" +"task_alias": "zh_cn college computer science" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_mathematics.yaml new file mode 100644 index 0000000000..f9b6dc91ff --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_mathematics" +"description": "The following are multiple choice questions (with answers) about college_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_college_mathematics" +"task_alias": "zh_cn college mathematics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_medicine.yaml new file mode 100644 index 0000000000..ffd653e8fd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_medicine" +"description": "The following are multiple choice questions (with answers) about college_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_college_medicine" +"task_alias": "zh_cn college medicine" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_physics.yaml new file mode 100644 index 0000000000..160ab527b2 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_college_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "college_physics" +"description": "The following are multiple choice questions (with answers) about college_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_college_physics" +"task_alias": "zh_cn college physics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_computer_security.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_computer_security.yaml new file mode 100644 index 0000000000..2ccd209985 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_computer_security.yaml @@ -0,0 +1,16 @@ +"dataset_name": "computer_security" +"description": "The following are multiple choice questions (with answers) about computer_security.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_computer_security" +"task_alias": "zh_cn computer security" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_conceptual_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_conceptual_physics.yaml new file mode 100644 index 0000000000..314aba1e62 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_conceptual_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "conceptual_physics" +"description": "The following are multiple choice questions (with answers) about conceptual_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_conceptual_physics" +"task_alias": "zh_cn conceptual physics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_econometrics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_econometrics.yaml new file mode 100644 index 0000000000..3423681194 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_econometrics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "econometrics" +"description": "The following are multiple choice questions (with answers) about econometrics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_econometrics" +"task_alias": "zh_cn econometrics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_electrical_engineering.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_electrical_engineering.yaml new file mode 100644 index 0000000000..7fd225f9cb --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_electrical_engineering.yaml @@ -0,0 +1,16 @@ +"dataset_name": "electrical_engineering" +"description": "The following are multiple choice questions (with answers) about electrical_engineering.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_electrical_engineering" +"task_alias": "zh_cn electrical engineering" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_elementary_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_elementary_mathematics.yaml new file mode 100644 index 0000000000..9a57d297cf --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_elementary_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "elementary_mathematics" +"description": "The following are multiple choice questions (with answers) about elementary_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_elementary_mathematics" +"task_alias": "zh_cn elementary mathematics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_formal_logic.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_formal_logic.yaml new file mode 100644 index 0000000000..d43f198fb5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_formal_logic.yaml @@ -0,0 +1,16 @@ +"dataset_name": "formal_logic" +"description": "The following are multiple choice questions (with answers) about formal_logic.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_formal_logic" +"task_alias": "zh_cn formal logic" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_global_facts.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_global_facts.yaml new file mode 100644 index 0000000000..535d3efb33 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_global_facts.yaml @@ -0,0 +1,16 @@ +"dataset_name": "global_facts" +"description": "The following are multiple choice questions (with answers) about global_facts.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_global_facts" +"task_alias": "zh_cn global facts" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_biology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_biology.yaml new file mode 100644 index 0000000000..c9bb0d9299 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_biology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_biology" +"description": "The following are multiple choice questions (with answers) about high_school_biology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_biology" +"task_alias": "zh_cn high school biology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_chemistry.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_chemistry.yaml new file mode 100644 index 0000000000..e4c870d415 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_chemistry.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_chemistry" +"description": "The following are multiple choice questions (with answers) about high_school_chemistry.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_chemistry" +"task_alias": "zh_cn high school chemistry" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_computer_science.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_computer_science.yaml new file mode 100644 index 0000000000..269acb0e6f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_computer_science.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_computer_science" +"description": "The following are multiple choice questions (with answers) about high_school_computer_science.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_computer_science" +"task_alias": "zh_cn high school computer science" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_european_history.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_european_history.yaml new file mode 100644 index 0000000000..ee26f7d0de --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_european_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_european_history" +"description": "The following are multiple choice questions (with answers) about high_school_european_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_high_school_european_history" +"task_alias": "zh_cn high school european history" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_geography.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_geography.yaml new file mode 100644 index 0000000000..6a54096ab0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_geography.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_geography" +"description": "The following are multiple choice questions (with answers) about high_school_geography.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_high_school_geography" +"task_alias": "zh_cn high school geography" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_government_and_politics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_government_and_politics.yaml new file mode 100644 index 0000000000..c22efdee9f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_government_and_politics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_government_and_politics" +"description": "The following are multiple choice questions (with answers) about high_school_government_and_politics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_high_school_government_and_politics" +"task_alias": "zh_cn high school government and politics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_macroeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_macroeconomics.yaml new file mode 100644 index 0000000000..42b29a1592 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_macroeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_macroeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_macroeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_high_school_macroeconomics" +"task_alias": "zh_cn high school macroeconomics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_mathematics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_mathematics.yaml new file mode 100644 index 0000000000..a45f42429c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_mathematics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_mathematics" +"description": "The following are multiple choice questions (with answers) about high_school_mathematics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_mathematics" +"task_alias": "zh_cn high school mathematics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_microeconomics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_microeconomics.yaml new file mode 100644 index 0000000000..9c6e92b309 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_microeconomics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_microeconomics" +"description": "The following are multiple choice questions (with answers) about high_school_microeconomics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_high_school_microeconomics" +"task_alias": "zh_cn high school microeconomics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_physics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_physics.yaml new file mode 100644 index 0000000000..2638a44e02 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_physics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_physics" +"description": "The following are multiple choice questions (with answers) about high_school_physics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_physics" +"task_alias": "zh_cn high school physics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_psychology.yaml new file mode 100644 index 0000000000..6a72fc8f92 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_psychology" +"description": "The following are multiple choice questions (with answers) about high_school_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_high_school_psychology" +"task_alias": "zh_cn high school psychology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_statistics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_statistics.yaml new file mode 100644 index 0000000000..e7ce1c9433 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_statistics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_statistics" +"description": "The following are multiple choice questions (with answers) about high_school_statistics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_high_school_statistics" +"task_alias": "zh_cn high school statistics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_us_history.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_us_history.yaml new file mode 100644 index 0000000000..1eabb21217 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_us_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_us_history" +"description": "The following are multiple choice questions (with answers) about high_school_us_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_high_school_us_history" +"task_alias": "zh_cn high school us history" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_world_history.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_world_history.yaml new file mode 100644 index 0000000000..cff4c134a0 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_high_school_world_history.yaml @@ -0,0 +1,16 @@ +"dataset_name": "high_school_world_history" +"description": "The following are multiple choice questions (with answers) about high_school_world_history.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_high_school_world_history" +"task_alias": "zh_cn high school world history" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_aging.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_aging.yaml new file mode 100644 index 0000000000..bd114d5701 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_aging.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_aging" +"description": "The following are multiple choice questions (with answers) about human_aging.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_human_aging" +"task_alias": "zh_cn human aging" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_sexuality.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_sexuality.yaml new file mode 100644 index 0000000000..47b978ef3f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_human_sexuality.yaml @@ -0,0 +1,16 @@ +"dataset_name": "human_sexuality" +"description": "The following are multiple choice questions (with answers) about human_sexuality.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_human_sexuality" +"task_alias": "zh_cn human sexuality" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_international_law.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_international_law.yaml new file mode 100644 index 0000000000..893957bd55 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_international_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "international_law" +"description": "The following are multiple choice questions (with answers) about international_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_international_law" +"task_alias": "zh_cn international law" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_jurisprudence.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_jurisprudence.yaml new file mode 100644 index 0000000000..df5886729c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_jurisprudence.yaml @@ -0,0 +1,16 @@ +"dataset_name": "jurisprudence" +"description": "The following are multiple choice questions (with answers) about jurisprudence.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_jurisprudence" +"task_alias": "zh_cn jurisprudence" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_logical_fallacies.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_logical_fallacies.yaml new file mode 100644 index 0000000000..419c216a8a --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_logical_fallacies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "logical_fallacies" +"description": "The following are multiple choice questions (with answers) about logical_fallacies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_logical_fallacies" +"task_alias": "zh_cn logical fallacies" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_machine_learning.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_machine_learning.yaml new file mode 100644 index 0000000000..8add74194f --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_machine_learning.yaml @@ -0,0 +1,16 @@ +"dataset_name": "machine_learning" +"description": "The following are multiple choice questions (with answers) about machine_learning.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_stem" +"task": "openai_mmmlu_default_zh_cn_machine_learning" +"task_alias": "zh_cn machine learning" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_management.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_management.yaml new file mode 100644 index 0000000000..cba9024583 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_management.yaml @@ -0,0 +1,16 @@ +"dataset_name": "management" +"description": "The following are multiple choice questions (with answers) about management.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_management" +"task_alias": "zh_cn management" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_marketing.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_marketing.yaml new file mode 100644 index 0000000000..0219a68963 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_marketing.yaml @@ -0,0 +1,16 @@ +"dataset_name": "marketing" +"description": "The following are multiple choice questions (with answers) about marketing.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_marketing" +"task_alias": "zh_cn marketing" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_medical_genetics.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_medical_genetics.yaml new file mode 100644 index 0000000000..1d8bf34800 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_medical_genetics.yaml @@ -0,0 +1,16 @@ +"dataset_name": "medical_genetics" +"description": "The following are multiple choice questions (with answers) about medical_genetics.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_medical_genetics" +"task_alias": "zh_cn medical genetics" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_miscellaneous.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_miscellaneous.yaml new file mode 100644 index 0000000000..cf7c7e86df --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_miscellaneous.yaml @@ -0,0 +1,16 @@ +"dataset_name": "miscellaneous" +"description": "The following are multiple choice questions (with answers) about miscellaneous.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_miscellaneous" +"task_alias": "zh_cn miscellaneous" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_disputes.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_disputes.yaml new file mode 100644 index 0000000000..398bb5899d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_disputes.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_disputes" +"description": "The following are multiple choice questions (with answers) about moral_disputes.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_moral_disputes" +"task_alias": "zh_cn moral disputes" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_scenarios.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_scenarios.yaml new file mode 100644 index 0000000000..367991b3f5 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_moral_scenarios.yaml @@ -0,0 +1,16 @@ +"dataset_name": "moral_scenarios" +"description": "The following are multiple choice questions (with answers) about moral_scenarios.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_moral_scenarios" +"task_alias": "zh_cn moral scenarios" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_nutrition.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_nutrition.yaml new file mode 100644 index 0000000000..84877e4b6d --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_nutrition.yaml @@ -0,0 +1,16 @@ +"dataset_name": "nutrition" +"description": "The following are multiple choice questions (with answers) about nutrition.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_nutrition" +"task_alias": "zh_cn nutrition" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_philosophy.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_philosophy.yaml new file mode 100644 index 0000000000..e6e51c60d3 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_philosophy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "philosophy" +"description": "The following are multiple choice questions (with answers) about philosophy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_philosophy" +"task_alias": "zh_cn philosophy" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_prehistory.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_prehistory.yaml new file mode 100644 index 0000000000..b017ea884b --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_prehistory.yaml @@ -0,0 +1,16 @@ +"dataset_name": "prehistory" +"description": "The following are multiple choice questions (with answers) about prehistory.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_prehistory" +"task_alias": "zh_cn prehistory" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_accounting.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_accounting.yaml new file mode 100644 index 0000000000..ba7ef048bc --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_accounting.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_accounting" +"description": "The following are multiple choice questions (with answers) about professional_accounting.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_professional_accounting" +"task_alias": "zh_cn professional accounting" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_law.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_law.yaml new file mode 100644 index 0000000000..b4489d8b76 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_law.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_law" +"description": "The following are multiple choice questions (with answers) about professional_law.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_professional_law" +"task_alias": "zh_cn professional law" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_medicine.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_medicine.yaml new file mode 100644 index 0000000000..88c83dcef9 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_medicine.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_medicine" +"description": "The following are multiple choice questions (with answers) about professional_medicine.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_professional_medicine" +"task_alias": "zh_cn professional medicine" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_psychology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_psychology.yaml new file mode 100644 index 0000000000..9c3f86d963 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_professional_psychology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "professional_psychology" +"description": "The following are multiple choice questions (with answers) about professional_psychology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_professional_psychology" +"task_alias": "zh_cn professional psychology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_public_relations.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_public_relations.yaml new file mode 100644 index 0000000000..54b8f217bd --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_public_relations.yaml @@ -0,0 +1,16 @@ +"dataset_name": "public_relations" +"description": "The following are multiple choice questions (with answers) about public_relations.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_public_relations" +"task_alias": "zh_cn public relations" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_security_studies.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_security_studies.yaml new file mode 100644 index 0000000000..eb753ec824 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_security_studies.yaml @@ -0,0 +1,16 @@ +"dataset_name": "security_studies" +"description": "The following are multiple choice questions (with answers) about security_studies.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_security_studies" +"task_alias": "zh_cn security studies" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_sociology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_sociology.yaml new file mode 100644 index 0000000000..24b086ce98 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_sociology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "sociology" +"description": "The following are multiple choice questions (with answers) about sociology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_sociology" +"task_alias": "zh_cn sociology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_us_foreign_policy.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_us_foreign_policy.yaml new file mode 100644 index 0000000000..4a4ad5a79c --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_us_foreign_policy.yaml @@ -0,0 +1,16 @@ +"dataset_name": "us_foreign_policy" +"description": "The following are multiple choice questions (with answers) about us_foreign_policy.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_social_sciences" +"task": "openai_mmmlu_default_zh_cn_us_foreign_policy" +"task_alias": "zh_cn us foreign policy" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_virology.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_virology.yaml new file mode 100644 index 0000000000..9b79f8d6e6 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_virology.yaml @@ -0,0 +1,16 @@ +"dataset_name": "virology" +"description": "The following are multiple choice questions (with answers) about virology.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_other" +"task": "openai_mmmlu_default_zh_cn_virology" +"task_alias": "zh_cn virology" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_world_religions.yaml b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_world_religions.yaml new file mode 100644 index 0000000000..5265a3d1fa --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/default/zh_cn/openai_mmmlu_default_world_religions.yaml @@ -0,0 +1,16 @@ +"dataset_name": "world_religions" +"description": "The following are multiple choice questions (with answers) about world_religions.\n\ + \n" +"doc_to_choice": +- "A" +- "B" +- "C" +- "D" +"doc_to_target": "{{Answer.strip()}}" +"doc_to_text": "{{Question.strip()}}\nA. {{A.strip()}}\nB. {{B.strip()}}\nC. {{C.strip()}}\n\ + D. {{D.strip()}}\n回答:" +"include": "../../_default_template_yaml" +"tag": "openai_mmmlu_default_zh_cn_humanities" +"task": "openai_mmmlu_default_zh_cn_world_religions" +"task_alias": "zh_cn world religions" +"test_split": "ZH_CN" diff --git a/lm_eval/tasks/openai_mmmlu/utils.py b/lm_eval/tasks/openai_mmmlu/utils.py new file mode 100644 index 0000000000..bae362f063 --- /dev/null +++ b/lm_eval/tasks/openai_mmmlu/utils.py @@ -0,0 +1,128 @@ +QUERY_TEMPLATE_MULTICHOICE = """ +Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering. + +{{Question}} + +A) {{A}} +B) {{B}} +C) {{C}} +D) {{D}} +""".strip() + + +SUBJECTS = { + "abstract_algebra": "stem", + "anatomy": "stem", + "astronomy": "stem", + "business_ethics": "other", + "clinical_knowledge": "other", + "college_biology": "stem", + "college_chemistry": "stem", + "college_computer_science": "stem", + "college_mathematics": "stem", + "college_medicine": "other", + "college_physics": "stem", + "computer_security": "stem", + "conceptual_physics": "stem", + "econometrics": "social_sciences", + "electrical_engineering": "stem", + "elementary_mathematics": "stem", + "formal_logic": "humanities", + "global_facts": "other", + "high_school_biology": "stem", + "high_school_chemistry": "stem", + "high_school_computer_science": "stem", + "high_school_european_history": "humanities", + "high_school_geography": "social_sciences", + "high_school_government_and_politics": "social_sciences", + "high_school_macroeconomics": "social_sciences", + "high_school_mathematics": "stem", + "high_school_microeconomics": "social_sciences", + "high_school_physics": "stem", + "high_school_psychology": "social_sciences", + "high_school_statistics": "stem", + "high_school_us_history": "humanities", + "high_school_world_history": "humanities", + "human_aging": "other", + "human_sexuality": "social_sciences", + "international_law": "humanities", + "jurisprudence": "humanities", + "logical_fallacies": "humanities", + "machine_learning": "stem", + "management": "other", + "marketing": "other", + "medical_genetics": "other", + "miscellaneous": "other", + "moral_disputes": "humanities", + "moral_scenarios": "humanities", + "nutrition": "other", + "philosophy": "humanities", + "prehistory": "humanities", + "professional_accounting": "other", + "professional_law": "humanities", + "professional_medicine": "other", + "professional_psychology": "social_sciences", + "public_relations": "social_sciences", + "security_studies": "social_sciences", + "sociology": "social_sciences", + "us_foreign_policy": "social_sciences", + "virology": "other", + "world_religions": "humanities", +} + +LANGUAGES = { + "AR_XY": "Arabic (Generic)", + "BN_BD": "Bengali (Bangladesh)", + "DE_DE": "German (Germany)", + "ES_LA": "Spanish (Latin America)", + "FR_FR": "French (France)", + "HI_IN": "Hindi (India)", + "ID_ID": "Indonesian (Indonesia)", + "IT_IT": "Italian (Italy)", + "JA_JP": "Japanese (Japan)", + "KO_KR": "Korean (South Korea)", + "PT_BR": "Portuguese (Brazil)", + "ZH_CN": "Chinese (China)", + "SW_KE": "Swahili (Kenya)", + "YO_NG": "Yoruba (Nigeria)", + "EN_US": "English (United States)", +} + +ANSWERS = { + "AR_XY": "الإجابة:", + "BN_BD": "উত্তর:", + "DE_DE": "Antwort:", + "ES_LA": "Respuesta:", + "FR_FR": "Réponse:", + "HI_IN": "उत्तर:", + "ID_ID": "Jawaban:", + "IT_IT": "Risposta:", + "JA_JP": "答え:", + "KO_KR": "답변:", + "PT_BR": "Resposta:", + "ZH_CN": "回答:", + "SW_KE": "Jawabu:", + "YO_NG": "Idahun:", + "EN_US": "Answer:", +} + + +def normalize_extracted_answer(extracted_answer: str) -> str: + return ( + # In arabic these are the letters used for A-D in multiple choice questions + extracted_answer.replace("أ", " A") + .replace("ب", " B") + .replace("ج", " C") + .replace("د", " D") + # In Bengali these are the letters used for A-D in multiple choice questions + .replace("অ", " A") + .replace("ব", " B") + .replace("ড", " C") + .replace("ঢ", " D") + # In Japanese these are the letters sometimes used for A-D in multiple choice questions + .replace("Ａ", " A") + .replace("Ｂ", " B") + .replace("Ｃ", " C") + .replace("Ｄ", " D") + .strip() + )