Description
Hi @bilgeyucel and team,
I've just explored the DevOps HitL cookbook notebook. Building on this cookbook recipe, I'd like to suggest an extension that could showcase another interesting application of HitL principles.
The DevOps recipe demonstrates interactive human intervention when an agent faces uncertainty. From my point-of-view, a nice evolution of this use case would be a cookbook example that implements a continuous improvement cycle by collecting user feedback on RAG responses, ideally in a WebUI. This could be achieved by two distinct approaches:
- a) By integrating Hayhooks to expose feedback endpoints with Open WebUI to provide intuitive thumbs up/down rating buttons for each LLM-generated response. For example, when users rate an answer poorly, the system could automatically generate an alternative response and store both versions along with user preferences. The collected feedback could then be structured, creating a complete loop where an system could learn from human preferences over time.
- b) As an alternative to simple binary feedback from the user, the cookbook could demonstrate an agentic RAG pipeline that proactively engages users when facing uncertainty, triggering clarification questions when ambiguity is detected. This could help to improve the retrieval process itself, gathering additional context before generating a final response (similar to the DevOps HitL tutorial, but specifically applied to the RAG context).
This is just an out-of-the-box idea that might exceed the scope of typical cookbook tutorials. Perhaps it could be simplified to focus on just a few key components to demonstrate the basic principle of using HitL in a RAG pipeline. From my point-of-view, such a tutorial would be particularly valuable as it demonstrates the complete AI feedback, either through collecting explicit user ratings that can feed into a learning process, or by implementing an agentic system that adapts in real-time based on user clarifications.