[MMLU redux] Do not use samples which do not have error_type="ok" (#3410)

* fix dataset - do not use samples which do not have error_type='ok'

* increase version

Author: fxmarty-amd

lm_eval/tasks/mmlu-redux/generative/README.md

@@ -57,5 +57,7 @@ If other tasks on this dataset are already supported:
 - [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
 - [ ] Have you noted which, if any, published evaluation setups are matched by this variant?
 
-ver 1: PR #2705
-First implementation
+
+
+- version 3: First implementation from PR https://github.com/EleutherAI/lm-evaluation-harness/pull/2705.
+- version 4: Filter out the answers not marked as `error_type="ok"` in https://github.com/EleutherAI/lm-evaluation-harness/pull/3410 (~6.5% of questions, 370 out of 5700 are filtered out).

lm_eval/tasks/mmlu-redux/generative/_default_template_yaml

@@ -1,4 +1,4 @@
-dataset_path: "edinburgh-dawg/mmlu-redux-2.0"
+dataset_path: "fxmarty/mmlu-redux-2.0-ok"
 test_split: test
 dataset_kwargs:
   trust_remote_code: true
@@ -29,4 +29,4 @@ filter_list:
       - function: take_first
 
 metadata:
-  version: 3.0
+  version: 4.0

lm_eval/tasks/mmlu-redux/generative/_mmlu.yaml

@@ -30,4 +30,4 @@ aggregate_metric_list:
     metric: exact_match
     weight_by_size: true
 metadata:
-  version: 3
+  version: 4