Skip to content

A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation files prepare/cards/news_category_classification_headline.py and prepare/cards/tablerow_classify.py #1774

Open
@dafnapension

Description

@dafnapension
Collaborator

'lmsys/arena-hard-browser' was gone from HF. test_preparation does not catch faulty prepare/cards/arena_hard/generation/english_gpt-4-0314_reference.py that tries to access this (gone) dataset, because a missing dataset is considered by test_preparation an error to be ignored.
That card participates in performance test, which does not ignore missing dataset. For that participation in performance-test, the card was fixed to use a similar dataset read from a github repository. The fix has been committed to main, the PR is: #1757

However, additional prepare-files exist that generate cards that look (in vain) for 'lmsys/arena-hard-browser':
prepare/cards/arena_hard/response_assessment/pairwise_comparative_rating/both_games_gpt4_judge.py
prepare/cards/arena_hard/response_assessment/pairwise_comparative_rating/both_games_mean_judgment_gpt4_judge.py
prepare/cards/arena_hard/response_assessment/pairwise_comparative_rating/first_game_only_gpt4_judge.py

A modification of the above prepare files is needed, in the spirit of the above mentioned PR.

Among others, a mapping between the data-files mentioned in the (currently faulty) prepares files and the data-files found in the github repository is not clear for these prepare-files, as they use * in its paths, for example.

Also, and independently, a missing key is failing the run of: unitxt/prepare/cards/news_category_classification_headline.py, raising an exception (that is ignored by test_preparation): Please obtain kaggle credentials https://christianjmills.com/posts/kaggle-obtain-api-key-tutorial/ and save them to local ./kaggle.json file

The same goes for:
Testing preparation file: /home/runner/work/unitxt/unitxt/prepare/cards/tablerow_classify.py failed with ignored error: Please obtain kaggle credentials https://christianjmills.com/posts/kaggle-obtain-api-key-tutorial/ and save them to local ./kaggle.json file

Activity

OfirArviv

OfirArviv commented on May 8, 2025

@OfirArviv
Collaborator

It seems the data is moved here:
https://huggingface.co/datasets/lmarena-ai/arena-hard-auto/tree/main/data/arena-hard-v0.1

Please see if the file structure is the same and if we can just point the card to the new directory

dafnapension

dafnapension commented on May 8, 2025

@dafnapension
CollaboratorAuthor

Nice catch! (douze points as they say today..) It is a new dataset, younger than 20 day old!
For prepare/cards/arena_hard/generation/english_gpt-4-0314_reference.py, just returning to the older version of LoadFromHFSpace (with updated paths, of course)

Yielded:
[Unitxt|CRITICAL|test_preparation.py:89] 2025-05-08 11:03:31,604 >> Testing preparation file: /home/runner/work/unitxt/unitxt/prepare/cards/arena_hard/generation/english_gpt-4-0314_reference.py failed with ignored error: The Huggingface space 'lmarena-ai/arena-hard-auto' was not found. Please check if the name is correct and you have access to the space.

and changing to LoadHF yielded: ,
raise DatasetNotFoundError(f"Dataset '{path}' doesn't exist on the Hub or cannot be accessed.") from e datasets.exceptions.DatasetNotFoundError: Dataset 'lmarena-ai' doesn't exist on the Hub or cannot be accessed

LoadHF with path="lmarena-ai/arena-hard-auto" finally made it, but then the fields do not match. So I changed them per the previous version that I prepared for the github version. Now test_performance, which does consume the card, fails. The same error that is reported on the HF site when looking why a dataset viewer is not available.

I am stuck.

github-actions

github-actions commented on Jun 8, 2025

@github-actions

This issue is stale because it has been open for 30 days with no activity.

changed the title [-]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF[/-] [+]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation file unitxt/prepare/cards/news_category_classification_headline.py[/+] on Jun 9, 2025
changed the title [-]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation file unitxt/prepare/cards/news_category_classification_headline.py[/-] [+]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation file prepare/cards/news_category_classification_headline.py and prepare/cards/tablerow_classify.py[/+] on Jun 9, 2025
changed the title [-]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation file prepare/cards/news_category_classification_headline.py and prepare/cards/tablerow_classify.py[/-] [+]A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation files prepare/cards/news_category_classification_headline.py and prepare/cards/tablerow_classify.py[/+] on Jun 9, 2025
github-actions

github-actions commented on Jul 10, 2025

@github-actions

This issue is stale because it has been open for 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @OfirArviv@dafnapension

        Issue actions

          A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF; And, a key is needed for running preparation files prepare/cards/news_category_classification_headline.py and prepare/cards/tablerow_classify.py · Issue #1774 · IBM/unitxt