Choosing the Right Approach for OWASP NestBot AI Assistant

[arkid15r]

We're planning to extend NestBot with an AI-powered Slack assistant for OWASP, to help answer common questions, redirect discussions to the right channels, and improve overall engagement within the community.

There are multiple ways we can build this, and we'd like to gather input on the best approach. Below are three primary options under consideration, along with a possible hybrid approach.

---

1. Custom NLP/ML Model Training

This approach involves training an in-house machine learning model using historical Slack messages, documentation, and structured knowledge sources.

Pros

• Tailored to OWASP-specific terminology and questions
• Full privacy and control over data (no external APIs)
• Can be optimized for specific types of queries

Cons

• High effort to collect, preprocess, and train data
• Requires ongoing maintenance and retraining
• May struggle with new or unexpected questions

---

2. LangChain with LLMs (Agentic Workflow)

This approach uses pre-trained Large Language Models (LLMs) (e.g., OpenAI GPT, Claude, Mistral) via LangChain to provide intelligent responses, combined with Retrieval-Augmented Generation (RAG) for fetching accurate OWASP-specific answers.

Pros

• Faster development (no need to train a model from scratch)
• More adaptable to new and evolving questions
• Easier integration with Slack via LangChain

Cons

• Possible privacy concerns if sending queries to external APIs
• Recurring costs for API usage
• May require additional tuning to prevent inaccurate responses

---

3. Rule-Based System with Predefined Responses

This approach uses a structured decision tree or keyword-based system to detect common patterns in questions and provide predefined responses, along with links to relevant OWASP resources.

Pros

• Fully controlled responses with no risk of AI-generated misinformation
• No dependency on external APIs or training data
• Minimal computational cost

Cons

• Limited flexibility; cannot handle new or unexpected questions
• Requires frequent manual updates as the community evolves
• Harder to scale for more complex queries

---

4. Hybrid Approach: Combining Rule-Based System with LLMs

This approach would start with a rule-based system for handling frequently asked questions, while falling back on an LLM when a question does not match predefined patterns. A confidence threshold could determine whether an AI-generated response is appropriate or if the question should be escalated to a human or redirected.

Pros

• Balances control and flexibility
• Reduces API costs by limiting LLM usage to more complex queries
• Provides reliable responses while still allowing adaptability

Cons

• Requires development of both rule-based and AI-driven components
• Complexity in deciding when to use predefined vs. AI-generated responses
• Ongoing maintenance needed for both systems

---

Key Factors to Consider

1. Accuracy and Relevance – Can the approach ensure OWASP-specific responses without noise?
2. Cost and Maintenance – What are the long-term costs and effort required to keep it running?
3. Slack Integration – How smoothly can we integrate with Slack’s API and maintain real-time interactions?
4. Privacy and Security – Do we need to avoid sending Slack data to external services?
5. Flexibility – Will the solution be able to adapt as OWASP discussions evolve?

---

Looking for Community Feedback

We want to hear from developers, AI/ML enthusiasts, and OWASP community members:

• Which approach do you think is more suitable?
• Are there any alternative solutions we should consider?
• How can we balance privacy, cost, and accuracy?

Please share your thoughts in the discussion.

[harsh3dev]

considering all the options available here's my take:
I have taken into consideration these metrics:

Accuracy and Relevance

• ML model will be accurate but long term relevancy can not be promised as data and rules change dynamically overtime
• Both LLM and Rule Based systems will be accurate and relevant overtime

Cost and Maintenance

• Initial development of ML model will incur hosting and training costs along with redundant cost in retraining and long term maintenance
• LLMs will incur costs in the long run due to API charges and minimal data storage costs

Slack Integration and real time interaction

• If we go with ML model, we would have to train the model and then craft APIs out of it, and then on eery message event those APIs would be called (it will need more effort but response will be real time)
• Our existing rule based system along with LLM will be easier to integrate with slack due to langchain's inbuilt slack support

Privacy and Security

• ML model will be utmost private and secure
• Integrating LLM will compromise with data privacy but this could also be mitigated if we use self hosted models.

Flexibility

• As OWASP discussions evolve - rule based systems have to be updated and models need to be retrained on a frequent basis
• combining rule based systems along with llms would be a better solution towards the need of flexibility

Overall I think:
The best is to combine existing rule based system along with LLM powered AI solution, this way development and long term maintenance would be easy along with minimal retraining costs but it lacks privacy and has long term API charges.
Hence it provides a balanced approach.

In short:

Metric	ML Model	Rule-Based System + LLM	Combined Approach
Accuracy and Relevance	Accurate but long-term relevance is uncertain due to dynamic changes in data and rules.	Accurate and relevant over time.	Combines the strengths of both, providing accuracy with adaptive relevance.
Cost and Maintenance	High initial costs for development, hosting, and retraining.	Ongoing API charges and minimal data storage costs.	Reduced retraining costs but long-term API costs remain.
Slack Integration & Real-Time	Requires training, creating APIs, and calling them on each event (more effort).	Easier to integrate with Slack using Langchain’s built-in support.	Combines real-time interaction with easier integration.
Privacy and Security	High privacy and security as it’s self-hosted.	Potential data privacy issues with LLMs unless self-hosted.	Balanced privacy with better flexibility if LLM is self-hosted.
Flexibility	Needs frequent retraining as OWASP discussions evolve.	Needs regular updates for changing rules.	Flexible solution with minimal retraining by combining both approaches.
Overall Recommendation	High accuracy and privacy but costly and less adaptable.	Easier to maintain but potential privacy issues and ongoing costs.	Balanced approach with good accuracy, flexibility, and easier maintenance, but potential privacy trade-offs.

[Yashgupta9330]

yes I think rag and rule based system would be best as it provides a hybrid approach involving all the metrics we are looking for

[Dishant1804]

I would like to share my experience, I have worked with RAG chatbots, i have used LLM APIs with langchain and langraph and also used local models for fun projects :)

Based on the key factors mentioned here's my opinion:

1. Accuracy and relevance : If we provide the data source to the model we can surely get correct and accurate information by tuning the RAG approach which will improve the accuracy of the output and give answer which is even very specific to OWASP organization
2. Cost and Maintenance: The long term costs will depend mainly on 2 factors,first is the vector Database (we can use Pg Vector DB) cost and second is the API key cost, both of these mainly depend on the amount of queries the contributors asks with the nestbot and the amount of data we have to store for slack messages.
3. Slack Integration: It is very easy to integrate slack with langchain
4. Privacy and Security: The context or the data source which we will be giving to the LLM will not contain sensitive information upto my knowledge, as the data source or the slack messages we will be storing are just the queries of the users, we can also find a way to filter that whether the user is sharing any sensitive information and we can give them a feedback of not to do so, in that way we will address privacy and security of users (we can discuss this further)
5. Flexibility: By again using a tuned approach we can make our RAG capable of adapting the response to long discussions without loosing the actual context

Langchain being a flexible and compatible with python and smooth slack integration makes it a great fit for AI assistance nestbot

[KaranNegi20Feb]

I’d go with a hybrid approach, leaning heavily on a fine-tuned LLM trained on OWASP’s Slack data and documentation. If privacy and accuracy are key, self-hosting a fine-tuned Mistral or Llama 3 model with vector search (FAISS) for better retrieval would be ideal.

That way, we:

1. Ensure security & control (no external APIs).
2. Improve accuracy over time with fine-tuning.
3. Reduce costs compared to using OpenAI APIs.

P.S. The only limitation I see is how we’re going to train the model since a conventional one like Llama 3.1 requires significant resources. Maybe we could use Unsloth to optimize the process. I have worked previously on fine tuning of data with respect to medical/ mental health based data (custom dataset with collection of 3 datasets from kaggle) (more of conversation based and suggestive stuff aswell )which took me nealry 50 mins to train on free google colab cpu.

https://unsloth.ai/

[Li-ChangC]

Many enterprise-grade bots and Q&A systems today use LangChain + LLMs or similar approaches (e.g., Copilot Agents). Among the four approaches considered, LangChain with LLMs (Agentic Workflow) is the optimal choice, especially with a sensitive information filter, as suggested by others.

However, for the OWASP community Slackbot, cost control is also an important factor.
The Slackbot must handle a wide range of questions, from general principles to specific vulnerability code analysis. Repeated API calls for common questions (especially when certain events trigger a rapid short-term surge in specific queries) can significantly increase costs (especially under token-based billing models).

Proposed Solution: Layered Response Strategy
To balance accuracy, cost, and efficiency, we can implement a tiered response mechanism:

1. Rule-Based System: Handle explicit, repetitive questions.
2. RAG: Retrieve relevant OWASP documentation for context.
3. LLM: Generate final answers for complex queries, using a confidence threshold to determine when LLM invocation is necessary.

Tiered Question Handling
Apply an Identification and classification logic to questions.
Simple Questions: Processed by the Rule-Based System (e.g., "What is ssh?").
Privacy-Sensitive Questions: Return predefined local responses or route to a local LLM.
Complex Questions: RAG + LLM for detailed responses.

Development Roadmap
Beta Phase: Start with a Rule-Based System + LLM to control costs. Focus on answering common questions locally, monitor API call frequency, and assess costs. If LLM calls exceed budget (e.g., too frequent), adjust triggering rules (e.g., set a confidence threshold, invoking LLM only for low-confidence queries).
Later Phase: Gradually introduce RAG to enhance handling of complex questions, improving diversity and maintainability. Continuously optimize the RAG knowledge base to reduce LLM dependency and increase local processing capabilities.

Key Decision Factors
The final choice depends on the following considerations:

• LLM Model Preference: Subscription-based vs. token-based pricing, or local deployment.
• Privacy and Compliance.
• Cost Comparison via A/B Testing.
• Development Effort: Hybrid approaches increase complexity and human resource costs compared to a pure LangChain + RAG solution.
• Sustainability and Maintainability