Skip to content

[Task] Rerank example shown to users #12

@malusamayo

Description

@malusamayo

Currently, we use the default order to show examples. We want to switch to a re-ranking system where more informative examples are shown first.

Specifically, we will assign scores to each example using the following criteria:

  1. Does the model critique negatively on the example output? Are new requirements suggested with the critique?
  2. How many existing requirements does the example violate? How much degradation from the last version (overall / per requirement)?
  3. How are examples textually different from the examples above (i.e., diversity ranking)?

1 is used to help users identify new requirements, 2 for identifying regressions, 3 for diversifying what to look at.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions