Skip to content

Add LUQ (Long-text Uncertainty Quantification) scorer #46

@dylanbouchard

Description

@dylanbouchard

Is your feature request related to a problem? Please describe.
We would like to incorporate the LUQ (Longform Uncertainty Quantification) scorer proposed by Zhang et al., 2024. While our current NLI-based scorers (Semantic Negentropy and Non-contradiction probability) are highly effective, their performance may be less effective on very long responses.

Describe the solution you'd like
To include the LUQ scorers, we would should include this in the NLIScorer class and offer it in the BlackBoxUQ scorer list.

Additional context
Note: Please do not ask to work on this issue unless you have read and fully understand the paper linked above.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions