LongForm Semantic Entropy

[dylanbouchard]

Is your feature request related to a problem? Please describe.
We would like to enable the longform version of Semantic Entropy as proposed by Farquhar et al., 2024. This approach takes the following steps:

1. Deconstruct an LLM response into individual 'factoids'
2. For each factoid, create a small set of questions for which the factoid is the answer.
3. For each of the questions created in step (2) above, generate $m$ sampled responses from which semantic entropy is computed.
4. Average the semantic entropy for each factoid in the response to get longform semantic entropy score.

Describe the solution you'd like
Assigning this issue to @mohitcek and trust his decision-making for this one.

Describe alternatives you've considered
Only offering short-form (regular) semantic entropy

Additional context
The claim-to-question will require an LLM. Let's use the same prompt used in the original paper.

[mohitcek]

Hi Dylan, I can work on this issue, please assign it to me.