Roadmap for v0.4.0

[zimmski]

The v0.4.0 is mainly meant for introducing Java to the benchmark. There are two main goals

1. Simply to the same evaluation we did with 0.3.0 with plain.go, but this time with a plain.java.
2. Automate the interpretation of the evaluation result as much as possible so we can iterate on new releases faster (every release needs a blog post about the results)

Tasks:

• Make assessments (and metrics) of tasks more generic and automated:
  • Refactor metrics into a common assessment map Refactor metrics to assessment #34 Assessment for generation #33
  • Move all metrics to be automatically interpreted:
    • "Execute" Move existing metrics to assessments #38
    • "Problem" Move existing metrics to assessments #38
    • "Coverage" Move existing metrics to assessments #38
    • "Extra text" Assessment for generation #33
    • Automatic categorization of models Automatic response categorization #32 More assessments based on a model response #39
  • Models should have "points" so they can be compared and sorted
    • Introduce score and sort Scoring and Ranking #41
• Generate .md with a automatically written report
  • General generation of .md file with SVG Automatic Markdown export #49
  • Link models right away to correct file
    • Update current blog post to bring in the directory of the repository for the evaluation results.
    • Update current blog post to link to the evaluation files of the models
• Test all the code we have, so from now on we only allow changes with tests
  • Evaluation package Test "evaluate" package #40
  • Binary Integration testing for the binary #46
• Split results into separate files
  • Split logs of models Separate evaluation logs from the model logs by generating separate log files in a reports directory #25
  • Use relative file paths in logs (not absolute, because absolute paths are hard to verify and compare by others)
• Automatic tool installation with fixed version
  • Symflower Automatic Symflower installation with fixed version #47 Automatically install Symflower with a fixed version for the CI, development environment and evaluation benchmark #50
• "plain.java" Java language implementation #61
• Release
  • Do a full evaluation with the new version
  • Tag version
  • Blog post
  • Adapt README
  • Announce and eat cake

[bauersimon]

Scoring, Categorization, Bar Charts split by language.

[zimmski]

Check determinism of models e.g. execute each plain repository X-times, and then check if they are stable.

[zimmski]

Save the descriptons of the models as well: https://openrouter.ai/api/v1/models

The reason is that these can change over time, and we need to know after a while what they where. e.g right now i would like to know if mistral-7b-instruct for the last evaluation was v0.1. or not

[zimmski]

Order models by open.weight, allows commercial-use, closed, and price(!) and size: e.g. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 is great because open-weight, and Apache2 so commerical-use allowed. Should be better rated than GPT4

[zimmski]

Write down a playbook for evaluations

• e.g. one thing that should happen is that we let the benchmark play 5 times and then sum up points, but ... the runs should have at least one hour berak in between to not run into cached responses.

[zimmski]

Bar charts should have have their value on the bar. The axis values do not work that well

[zimmski]

Pick an example or several examples per category: goal is to find interesting results automatically, because it will get harder and harder to go manually through results.

[zimmski]

Do test file paths through

• symflower symbols
• Task for models

[zimmski]

Added all follow ups to #79 so this issue is officially closed for changes. We only do the last tasks and then close it

CC @bauersimon

[zimmski]

Finally. Eating that cake!