Description
Feature request
Add an option to disable LLM inference for the OUTPUT column when running an experiment in the LangSmith playground. This would allow users to evaluate datasets that already contain a REFERENCED OUTPUT without triggering model inference.
Use case:
I have a dataset with two fields: INPUT and REFERENCED OUTPUT. I’ve built a custom evaluator that evaluates only the REFERENCED OUTPUT. However, when I try to run the experiment in the playground, LangSmith still attempts to populate the OUTPUT column via an LLM call, which is unnecessary for my use case and consumes extra resources.
Proposal:
Introduce a toggle or configuration flag in the experiment settings that allows users to bypass the generation of the OUTPUT field. This way, the experiment will rely solely on the REFERENCED OUTPUT and custom evaluators.
Benefit:
This would make LangSmith more flexible for evaluation-only use cases where users have pre-existing model outputs or references and want to benchmark or analyze them without re-generating predictions
Motivation
I want to evaluate existing REFERENCED OUTPUTs without triggering LLM inference. Automatically generating the OUTPUT column adds unnecessary cost and slows down evaluation when only custom evaluation is needed. A toggle to disable inference would streamline this process.