[Feature Request] Add support for running a pairwise experiment #37
Description
Currently, all evaluations are run in isolation. However, it is often beneficial to compare the results of against another evaluation.
LangSmith supports evaluating existing experiments in a comparative manner. Under the hood, this can be achieved via the SDK by using aevaluate() with two existing experiments.
I'm not quite sure of the best way to implement this via the config file. I was thinking that it could be something that is performed after a new experiment has been created if an experiment to compare against is provided via the config file - i.e. create a new experiment and then create a pairwise evaluation for it against a base. It could also be that the config has an option for "compare to previous matching experiment" and performs a look-up of the last evaluation that matches the prefix, for example.