Conversation
juho-y
force-pushed
the
list-judges-tool
branch
8 times, most recently
from
8a8561f to
3177a07
Compare
juho-y
force-pushed
the
run-judge-tool
branch
4 times, most recently
from
20001bc to
54db952
Compare
src/root_signals_mcp/judge.py
Outdated
| result = await self.async_client.run_judge( | ||
| judge_id=request.judge_id, | ||
| request=request.request, | ||
| response=request.response, | ||
| ) |
There was a problem hiding this comment.
Nitpick: could pass the **request here if needed, but ideally we just pass RunJudgeRequest downstream. Maintaining this mapping seems redundant.
There was a problem hiding this comment.
Done, passing it downstream.
| payload = { | ||
| "request": request, | ||
| "response": response, | ||
| } |
There was a problem hiding this comment.
similarily here, ideally we map From a RS API domain object for a Judge Request to our local JudgeRequest, which can be a simplified version with exclude_none=True in the basemodel
There was a problem hiding this comment.
Sure, would need to change all the evaluator calls as well, so will leave that into a follow-up refactoring PR.
TensorTemplar
left a comment
TensorTemplar
left a comment
There was a problem hiding this comment.
A cleaner domain boundary could use pydantic models explicitly, instead of enforcing it via selective mapping in the run judge client logic. Not a blocker while the judges are WIP, but good to revisit later
RAG not yet supported by judge endpoint
2 participants
Continuation to #16
Includes the run judge tool call and updates README.
Judges do not yet support the RAG case; hence, the RAG-specific Judge tools have been omitted.