Support more aggregations of numeric eval output besides average and histogram

[zsiegel92]

Suppose you have a function that uses an LLM to generate a score of 1, 2, or 3 from some text.

You label some data with ground truth and upload it to Laminar to evaluate with:

async function getScoreFromOneToThree(inputString: string): Promise<number> {
  return 1 // Placeholder for something that uses LLM
}

const dataset = new LaminarDataset<string, { score: number }>("my dataset")

evaluate({
  data: dataset,
  executor: getScoreFromOneToThree,
  evaluators: {
    correctness: async (o, t) => {
      if (!t) {
        throw new Error("No test case target")
      }
      return {
        exactlyCorrect: Number(o === t.score), // Average of this is % correct
        scoreMinusTarget: o - t.score, // Average of this is mean error
        scoreMinusTargetSquared: (o - t.score) ** 2, // Average of this is MSE
        scoreMinusTargetAbsolute: Math.abs(o - t.score), // Average of this is MAE
      }
    },
  },
})

When we run the eval we get an average (arithmetic mean) of each of those metrics along with a histogram:

Problem: There is no way to get some more useful aggregation than the average like RMSE (square root of MSE), standard deviation, max error, etc.

There are a few straightforward ways to address this:

• eject full evaluation results back to runtime where evaluation was run, i.e. const results = evaluate({...}) instead of evaluate being :void. Then users can log their own preferred metrics. Not as good as having it in the web UI 🤷‍♂️
• UI automatically displays some of these like RMSE, etc.
• add some API like reporters: { exactlyCorrect: async (outputs: o[], targets: t[]) => RMSE(outputs, targets) } and include these as datapoints in the line graph of the other averaged fields (the graph at /project/[projectId]/evaluations/[evaluationId]] that has p99 and p95 etc.)

[skull8888888]

@zsiegel92 good points, do you have exact list of aggregations that you want to see in the UI. I will try to add them all.

[zsiegel92]

@zsiegel92 good points, do you have exact list of aggregations that you want to see in the UI. I will try to add them all.

@skull8888888 I would say RMSE, the problem is that not all numbers in this dashboard represent errors, so RMSE would not be well-defined. Maybe some metadata on each evaluation indicating whether or not it represents a loss/error would be necessary...

The more I think about it the more I think some kind of "eject my results from evaluate" interface would be best since users are going to have their own ideas about loss/error/metrics, and then they can calculate whatever they want in their own runtime.

I actually wrote an evaluateAndEject function that wraps evaluate and does that, it wasn't too hard, would be happy to share.

[skull8888888]

@zsiegel92 makes sense, you can also download evaluation runs as .csv and run any analytics on it.

[zsiegel92]

@zsiegel92 makes sense, you can also download evaluation runs as .csv and run any analytics on it.

Expose this in the sdk! Something like

import {listEvalResults, getEvalResult, evaluate} from "@lmnr-ai/lmnr"
evaluate({...groupName: "eval-group"})
const evalResultIds = await listEvalResults("eval-group")
const mostRecentEvalResultId = evalResultIds[0]
const evalResult = await getEvalResult(mostRecentEvalResultId)
...

or

const { evalResultId } = evaluate({...})
...

[skull8888888]

@zsiegel92 excellent idea, will do it. We're actually shipped eval via API feature in python lmnr-ai/lmnr-python#129. And we plan to cover as much of our UI functionality in SDK.