Skip to content

Create synthetic MMLU via GPT-4#367

Closed
nouhadziri wants to merge 55 commits intomainfrom
mmlu_synthetic
Closed

Create synthetic MMLU via GPT-4#367
nouhadziri wants to merge 55 commits intomainfrom
mmlu_synthetic

Conversation

@nouhadziri
Copy link
Contributor

@nouhadziri nouhadziri commented Sep 24, 2024

Creation process:

  1. Select random few-shot examples from each category in MMLU.
  2. Prompt GPT4 with the few-shot examples to generate similar (but not identical) questions, options and responses.
  3. Parse GPT4 outputs to extract responses and filter malformed outputs.
  4. Upload data to HF.

Copy link
Collaborator

@natolambert natolambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nouhadziri this looks good. We need to move the location though to match other data scripts
scripts/data/...

@hamishivi
Copy link
Collaborator

Gonna close this since I think we've moved on -- feel free to reopen + prep for merging if you think otherwise!

@hamishivi hamishivi closed this May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants