Issue: Missing Public Artifacts Block Full Nejumi Leaderboard Runs

[lhl]

Hey @olachinkei - per our chat last week, I'm trying to run the Nejumi leaderboard on against our Shisa V2 models, but so far have run into a number of blockers.

I've cloned the repo into my own fork and followed along w/ the README.

I've also mirrored the artifacts to my own W&B project, eg:

uv run wandb artifact get <llm-leaderboard/nejumi-leaderboard4/<name>:<version>>
uv run python scripts/data_uploader/upload_dataset.py -e augmxnt -p nejumi-leaderboard4 \
  -n <name> -d <local path> -m "Mirrored from llm-leaderboard/nejumi-leaderboard4"

However, it looks like I've run into some blockers as these are private only:

jbbq.artifacts_path = llm-leaderboard/nejumi-leaderboard4-private/jbbq:production
toxicity.artifact_path = llm-leaderboard/nejumi-leaderboard4-private/toxicity_dataset_full:production
toxicity.judge_prompts_path = llm-leaderboard/nejumi-leaderboard4-private/toxicity_judge_prompts:production

There's another open issue #257 on one of the datasets (but there are more!) - I looked at #243 as well, but the issue isn't just permissions, it's that the datasets aren't publicly available?

I know replicability has been an issue for quite a while #43 - let me know if I should contact the support email directly or if there's someone better to coordinate w/. Happy to help validate/help getting this working with third parties, both from a documentation, dataset,and script reproducibility (eg updating the data_uploader).

Is it possible to just make the artifacts for Nejumi versioned and publicly accessible from an official nejumi4 account? That might be best from making things replicable perpective?

Also, how do model submissions look like? I'll be running on our V2 and V2.1 models and might be interested in submitting them. Some models, like our 405B are resource intensive/hard to run, so I don't mind submitting some if so.

[olachinkei]

@lhl
Thanks for following up — and for taking the time to fork the repo and mirror the artifacts. You’re right: the current blockers are not about permissions but rather about dataset availability and compliance restrictions.

To clarify the situation:

JBBQ and LINE Yahoo Toxicity datasets are both classified as sensitive.

JBBQ requires direct application to the original repository’s maintainers before access is granted: https://github.com/ynklab/JBBQ_data

The LINE Yahoo dataset cannot currently be redistributed. We’re working with LINE Yahoo and our legal team to create a formal application and approval process for external researchers.

Because of these restrictions, the private artifacts you mentioned —
nejumi-leaderboard4-private/jbbq and nejumi-leaderboard4-private/toxicity_dataset_full —
cannot yet be made public.

We will update the Readme to make it clearer.

On model submissions:

To avoid the kind of overfitting and leaderboard gaming seen on open platforms like OpenLLM Leaderboard, we’re not enabling open bulk submissions.
Instead, we handle submissions through individual coordination, ensuring evaluation fairness and consistency across runs.

If you’re interested in submitting your Shisa V2 or V2.1 models, please reach out directly via email so we can discuss evaluation logistics and dataset access in detail:
📧 keisuke.kamata@wandb.com