Datasets `v4` is not compatible with current implmentaion

[Samoed]

Describe the bug

Datasets v4 is not compatible with our implementation

FAILED tests/test_benchmark/test_benchmark.py::test_prompt_name_passed_to_all_encodes_with_prompts[False-task17] - TypeError: Wrong key type: '0' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
FAILED tests/test_benchmark/test_benchmark.py::test_mulitple_mteb_tasks[model0-tasks0] - TypeError: Wrong key type: '1' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
FAILED tests/test_benchmark/test_benchmark.py::test_prompt_name_passed_to_all_encodes_with_prompts[False-task18] - TypeError: Wrong key type: '0' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
FAILED tests/test_benchmark/test_benchmark.py::test_benchmark_encoders_on_task[model0-task3] - TypeError: Wrong key type: '1' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
FAILED tests/test_benchmark/test_benchmark.py::test_benchmark_encoders_on_task[model0-task4] - TypeError: Wrong key type: '1' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.
FAILED tests/test_benchmark/test_benchmark.py::test_benchmark_encoders_on_task[model0-task17] - TypeError: Wrong key type: '0' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.

1. They've deprecated trust_remote_code, so some of our datasets are not compatible with this version ref [v2] Reupload datasets with trust_remote_code #1877
2. It seems now we can't pass numpy objects as index

To reproduce

CI:

• v2 branch https://github.com/embeddings-benchmark/mteb/actions/runs/16190269265/job/45704205049?pr=2833
• v1 branch https://github.com/embeddings-benchmark/mteb/actions/runs/16187630417/job/45706531895?pr=2889

Additional information

No response

Are you interested to contribute a fix for this bug?

Yes

[KennethEnevoldsen]

Hmm, is it easy to replace the numpy array with just a list? trust_remote_code seems doable, though time consuming to implement.

[Samoed]

though time consuming to implement.

Yes. I will try to continue to reupload all datasets