forked from zeno-ml/zeno
-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Currently, we use the default order to show examples. We want to switch to a re-ranking system where more informative examples are shown first.
Specifically, we will assign scores to each example using the following criteria:
- Does the model critique negatively on the example output? Are new requirements suggested with the critique?
- How many existing requirements does the example violate? How much degradation from the last version (overall / per requirement)?
- How are examples textually different from the examples above (i.e., diversity ranking)?
1 is used to help users identify new requirements, 2 for identifying regressions, 3 for diversifying what to look at.
Metadata
Metadata
Assignees
Labels
No labels