A comprehensive CivicTech platform connecting Tulsa residents with city government through AI-powered tools, automated notifications, and intelligent meeting analytics.
π Deployed Demo: https://d1s9nkkr0t3pmn.cloudfront.net
- Interactive chatbot with real-time city council knowledge
- RAG-Enhanced Responses: Document-based answers using city budgets, legislation, and policies
- Natural language queries about Tulsa government with contextual document search
- Meeting summary generation and analysis
- Automated alerts for city council meetings
- Topic-based subscriptions (housing, transportation, etc.)
- SMS and email delivery with AI-categorized content
- AI categorization of 42+ civic topics
- Automated agenda extraction and impact assessment
- Searchable meeting minutes with keyword analysis
- AI-powered email generation to contact officials
- District-based representative lookup
- Pre-built templates for common civic issues
- Campaign tracking and petition management
- Neighborhood-based organizing tools
- User preference and notification management
The RAG system enables the chatbot to search and reference actual city documents, budgets, legislation, and policies to provide accurate, source-backed answers instead of generic responses.
- Vector Database: ChromaDB (dev) / FAISS (production) for semantic search
- Document Processing: AI-powered text extraction, chunking, and summarization
- Embedding Generation: OpenAI text-embedding-3-small for vector representations
- Smart Search: Contextual document retrieval with relevance scoring
- π° Budgets: City financial documents and allocations
- π Legislation: Ordinances, resolutions, and policies
- π Meeting Minutes: City council and committee records
- π Reports: Studies, analyses, and official reports
- π Policies: Administrative procedures and guidelines
# Install RAG dependencies
pip install -r backend/requirements.txt
# Run database migration
cd backend && python -m alembic upgrade head
# Test the system
python scripts/test_rag_system.py
π Full Documentation: See docs/RAG_SYSTEM_README.md
Advanced evaluation framework using GPT-4 to assess chatbot response quality beyond traditional keyword matching.
- Smart Quality Assessment: GPT-4 evaluates helpfulness, accuracy, and civic appropriateness
- Semantic Understanding: Goes beyond keywords to understand context and meaning
- Comparative Analysis: Traditional metrics vs. LLM evaluation with combined scoring
- Actionable Feedback: Specific improvement suggestions and response strengths
- Accuracy: Factual correctness and Tulsa civic relevance
- Helpfulness: How well responses assist users with civic questions
- Completeness: Information sufficiency without verbosity
- Civic Appropriateness: Tone and context suitability for government use
# LLM-as-Judge evaluation
python scripts/run_eval.py --mode llm-judge
# Compare traditional vs LLM metrics
python scripts/llm_judge_evaluator.py
# Performance benchmarking
python scripts/run_eval.py --mode performance
LLM-AS-JUDGE EVALUATION SUMMARY
Combined Score: 0.847/1.0 (Grade: B)
Traditional Score: 0.789/1.0
LLM Judge Score: 0.873/1.0
Score Improvement: +0.084
Grade Distribution: A: 2, B: 6, C: 2, D: 0, F: 0
π Full Documentation: See docs/CHATBOT_EVALUATION_README.md
# Clone repository
git clone https://github.com/kaizengrowth/CityCamp_AI.git
cd CityCamp_AI
# Option 1: Automated setup (recommended)
./scripts/start-dev.sh
# Option 2: Manual setup
# Backend
cd backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp env.example .env
python -m app.main
# Frontend (new terminal)
cd frontend
npm install
npm run dev
Access Points:
- π¨ Frontend: http://localhost:3007
- βοΈ Backend API: http://localhost:8000
- π API Docs: http://localhost:8000/docs
- π Live Application: https://d1s9nkkr0t3pmn.cloudfront.net
- π AWS Console: ECS Services
- Frontend: React 18 + TypeScript + Vite + Tailwind CSS
- Backend: FastAPI + Python 3.11 + PostgreSQL + Redis
- AI/ML: OpenAI GPT-4 + Custom categorization models + RAG System (ChromaDB/FAISS)
- Document Processing: Multi-format support (PDF, DOCX, TXT) with vector embeddings
- Evaluation: LLM-as-Judge system with GPT-4 intelligent response assessment
- Infrastructure: AWS (ECS Fargate, RDS, ElastiCache, S3, CloudFront)
- CI/CD: GitHub Actions with automated testing and deployment
The application follows a layered architecture with clear separation of concerns, dependency injection, and standardized patterns across all components.
graph TB
subgraph "π Frontend"
FE[React + TypeScript SPA]
end
subgraph "π API Layer"
API[FastAPI Backend]
ENDPOINTS[REST Endpoints]
API --> ENDPOINTS
end
subgraph "βοΈ Services"
CHAT[AI Chatbot Service]
RAG[RAG Document Search]
NOTIF[Notification Service]
SCRAPER[Meeting Scrapers]
end
subgraph "πΎ Data Layer"
DB[(PostgreSQL)]
VECTOR[(Vector Store)]
CACHE[(Redis Cache)]
S3[(File Storage)]
end
subgraph "π€ External APIs"
OPENAI[OpenAI GPT-4]
TWILIO[Twilio SMS]
end
%% Main connections
FE --> API
API --> CHAT
API --> RAG
API --> NOTIF
API --> SCRAPER
CHAT --> OPENAI
RAG --> VECTOR
NOTIF --> TWILIO
SCRAPER --> DB
API --> DB
API --> CACHE
API --> S3
π― Key Architecture Components:
- Frontend: React 18 SPA with TypeScript for type safety
- API Layer: FastAPI with standardized responses and error handling
- AI Services: GPT-4 chatbot with RAG document search capabilities
- Data Storage: PostgreSQL + Vector store + Redis caching + S3 files
- External Integrations: OpenAI for AI, Twilio for SMS notifications
- Dependency Injection: Services use constructor injection for better testability
- Base Service Pattern: All services inherit from
BaseService
for consistency - Standardized Responses: Unified API response formats with proper error handling
- Unified Data Models: Consolidated notification preferences and user management
- RAG Architecture: Document processing with vector embeddings for semantic search
- Event-Driven Notifications: Async notification processing with queue management
graph TB
subgraph "π Frontend Layer"
FE[React 18 + TypeScript SPA]
FE_COMP[Components Layer]
FE_PAGES[Pages Layer]
FE_CTX[Contexts Layer]
FE_API[API Config]
FE --> FE_COMP
FE --> FE_PAGES
FE --> FE_CTX
FE --> FE_API
end
subgraph "π API Gateway Layer"
CF[CloudFront CDN]
ALB[Application Load Balancer]
CORS[CORS Middleware]
CF --> ALB
ALB --> CORS
end
subgraph "π FastAPI Application"
MAIN[main.py - App Entry]
ROUTER[API Router v1]
MIDDLEWARE[Middleware Stack]
EXCEPTION[Exception Handlers]
MAIN --> ROUTER
MAIN --> MIDDLEWARE
MAIN --> EXCEPTION
end
subgraph "π‘ API Endpoints"
EP_AUTH[Auth Endpoints]
EP_MEET[Meetings Endpoints]
EP_ORG[Organizations Endpoints]
EP_CAMP[Campaigns Endpoints]
EP_CHAT[Chatbot Endpoints]
EP_DOC[Documents Endpoints]
EP_SUB[Subscriptions Endpoints]
EP_REP[Representatives Endpoints]
ROUTER --> EP_AUTH
ROUTER --> EP_MEET
ROUTER --> EP_ORG
ROUTER --> EP_CAMP
ROUTER --> EP_CHAT
ROUTER --> EP_DOC
ROUTER --> EP_SUB
ROUTER --> EP_REP
end
subgraph "βοΈ Service Layer"
BASE_SVC[BaseService Abstract Class]
CHAT_SVC[ChatbotService]
DOC_SVC[DocumentProcessingService]
VEC_SVC[VectorService]
NOTIF_SVC[NotificationService]
TWILIO_SVC[TwilioService]
S3_SVC[S3Service]
GEO_SVC[GeocodingService]
AUTH_SVC[AuthService]
RESEARCH_SVC[ResearchService]
BASE_SVC -.-> CHAT_SVC
BASE_SVC -.-> DOC_SVC
BASE_SVC -.-> VEC_SVC
BASE_SVC -.-> NOTIF_SVC
BASE_SVC -.-> TWILIO_SVC
BASE_SVC -.-> S3_SVC
BASE_SVC -.-> GEO_SVC
BASE_SVC -.-> RESEARCH_SVC
CHAT_SVC --> VEC_SVC
CHAT_SVC --> RESEARCH_SVC
DOC_SVC --> VEC_SVC
NOTIF_SVC --> TWILIO_SVC
end
subgraph "ποΈ Data Models Layer"
USER_MODEL[User Model]
MEET_MODEL[Meeting Model]
ORG_MODEL[Organization Model]
CAMP_MODEL[Campaign Model]
DOC_MODEL[Document Model]
NOTIF_PREF[NotificationPreferences Model]
SUB_MODEL[Subscription Models]
NOTIF_PREF --> USER_MODEL
end
subgraph "πΎ Data Storage Layer"
POSTGRES[(PostgreSQL Database)]
REDIS[(Redis Cache)]
CHROMA[(ChromaDB Vector Store)]
FAISS[(FAISS Vector Index)]
S3[(AWS S3 Storage)]
VEC_SVC --> CHROMA
VEC_SVC --> FAISS
S3_SVC --> S3
end
subgraph "π€ AI & External Services"
OPENAI[OpenAI GPT-4 API]
EMBEDDING[OpenAI Embeddings]
TWILIO_API[Twilio SMS API]
GOOGLE_API[Google Search API]
GEOCODING[Geocoding APIs]
CHAT_SVC --> OPENAI
VEC_SVC --> EMBEDDING
TWILIO_SVC --> TWILIO_API
RESEARCH_SVC --> GOOGLE_API
GEO_SVC --> GEOCODING
end
subgraph "π Data Collection"
SCRAPERS[Meeting Scrapers]
TGOV_SCRAPER[TGOV Scraper]
ARCHIVE_SCRAPER[Archive Scraper]
SCRAPERS --> TGOV_SCRAPER
SCRAPERS --> ARCHIVE_SCRAPER
end
subgraph "βοΈ AWS Infrastructure"
ECS[ECS Fargate]
RDS[RDS PostgreSQL]
ELASTICACHE[ElastiCache Redis]
CLOUDFRONT[CloudFront CDN]
ECS --> RDS
ECS --> ELASTICACHE
CLOUDFRONT --> ECS
end
subgraph "π Response Standardization"
STD_LIST[StandardListResponse]
STD_RESP[StandardResponse]
ERR_RESP[ErrorResponse]
PAGINATION[PaginationParams]
EP_MEET --> STD_LIST
EP_ORG --> STD_LIST
EP_CAMP --> STD_LIST
EP_DOC --> STD_LIST
end
%% Connections
FE_API --> CF
CF --> MAIN
EP_AUTH --> AUTH_SVC
EP_MEET --> CHAT_SVC
EP_ORG --> BASE_SVC
EP_CAMP --> BASE_SVC
EP_CHAT --> CHAT_SVC
EP_DOC --> DOC_SVC
EP_SUB --> NOTIF_SVC
USER_MODEL --> POSTGRES
MEET_MODEL --> POSTGRES
ORG_MODEL --> POSTGRES
CAMP_MODEL --> POSTGRES
DOC_MODEL --> POSTGRES
NOTIF_PREF --> POSTGRES
SUB_MODEL --> POSTGRES
SCRAPERS --> MEET_MODEL
POSTGRES --> RDS
REDIS --> ELASTICACHE
S3 --> S3
CF --> CLOUDFRONT
- Single Responsibility: Each service has a focused, well-defined purpose
- Dependency Inversion: High-level modules don't depend on low-level modules
- Interface Segregation: Clients depend only on interfaces they use
- Open/Closed: Open for extension, closed for modification
- DRY (Don't Repeat Yourself): Shared functionality in base classes
- Consistent Error Handling: Standardized exception handling across all layers
- Type Safety: Full TypeScript and Python type coverage
The codebase has been significantly refactored to improve maintainability and consistency:
β Standardization Improvements:
- Unified Response Format: All API endpoints now use
StandardListResponse
andPaginationParams
- Base Service Pattern: All services inherit from
BaseService
with dependency injection - Centralized Error Handling: Custom exception classes with structured error responses
- Unified Notification System: Consolidated user and subscription notification preferences
β Code Quality Improvements:
- Reduced Duplication: Eliminated ~200 lines of redundant code
- Better Type Safety: Full type coverage with generic response types
- Improved Testability: Dependency injection enables easier mocking
- Consistent Logging: Standardized logging patterns across all services
β Database Improvements:
- Unified Models:
NotificationPreferences
model consolidates user notification settings - Proper Migrations: Alembic migrations preserve existing data during schema changes
- Better Relationships: Cleaner model relationships with proper foreign keys
See docs/ARCHITECTURE_IMPROVEMENTS.md
for detailed implementation notes.
CityCamp_AI/
βββ π¨ frontend/ # React TypeScript application
β βββ src/
β β βββ components/ # React components
β β βββ pages/ # Route components
β β βββ contexts/ # React contexts
β β βββ config/ # API configuration
β βββ package.json # Dependencies
β βββ vite.config.ts # Vite configuration
β
βββ βοΈ backend/ # FastAPI Python backend
β βββ app/
β β βββ api/v1/ # API endpoints with standardized responses
β β β βββ endpoints/ # Individual endpoint modules
β β βββ core/ # Core application components
β β β βββ config.py # Application configuration
β β β βββ database.py # Database connection and setup
β β β βββ exceptions.py # Custom exception handling
β β βββ models/ # SQLAlchemy database models
β β β βββ user.py # User and authentication models
β β β βββ meeting.py # Meeting and agenda models
β β β βββ organization.py # Community organization models
β β β βββ campaign.py # Civic campaign models
β β β βββ document.py # RAG document models
β β β βββ notification_preferences.py # Unified notification system
β β βββ schemas/ # Pydantic schemas for API
β β β βββ base.py # Standardized response schemas
β β β βββ *.py # Individual endpoint schemas
β β βββ services/ # Business logic layer
β β β βββ base.py # BaseService abstract class
β β β βββ chatbot_service.py # AI chatbot with RAG
β β β βββ vector_service.py # Vector embeddings and search
β β β βββ document_processing_service.py # Document processing
β β β βββ notification_service.py # Meeting notifications
β β β βββ *.py # Other specialized services
β β βββ scrapers/ # Data collection modules
β βββ alembic/ # Database migrations
β βββ requirements.txt # Python dependencies
β βββ main.py # Application entry point with exception handlers
β
βββ βοΈ aws/ # Infrastructure as Code
β βββ terraform/ # Terraform configurations
β βββ scripts/ # Deployment scripts
β βββ README.md # AWS deployment guide
β
βββ π docs/ # Comprehensive documentation
β βββ QUICKSTART.md # 5-minute setup guide
β βββ RAG_SYSTEM_README.md # RAG architecture & usage guide
β βββ CHATBOT_EVALUATION_README.md # LLM-as-Judge evaluation guide
β βββ ARCHITECTURE_IMPROVEMENTS.md # Recent refactoring details
β βββ aws-deployment-guide.md
β βββ TROUBLESHOOTING.md # Issue resolution
β βββ API_DOCUMENTATION.md # API reference
β
βββ π§ͺ tests/ # Centralized testing
β βββ backend/ # Python API tests
β βββ frontend/ # React component tests
β βββ README.md # Testing instructions
β
βββ π§ scripts/ # Development & deployment
βββ start-dev.sh # Local development setup
βββ eval_chatbot_system.py # Traditional evaluation system
βββ llm_judge_evaluator.py # LLM-as-Judge evaluation
βββ run_eval.py # Evaluation runner with multiple modes
βββ test_rag_system.py # RAG system testing
βββ test_production_api.sh # Production diagnostics
βββ fix_production_api.sh # Production issue fixes
# Check requirements
node --version # >= 18.0.0
python --version # >= 3.11.0
docker --version # Latest
# Setup with automation
./scripts/start-dev.sh
# Manual database setup (if needed)
./scripts/setup_database.py
# Frontend
cd frontend
npm run dev # Start development server
npm run build # Build for production
npm run test # Run component tests
npm run lint # Code linting
# Backend
cd backend
source venv/bin/activate
python -m app.main # Start development server
python -m pytest ../tests/backend/ -v # Run API tests
python -m app.services.meeting_scraper # Live meeting discovery
# Full system
./scripts/start-dev.sh # Start all services
./scripts/test-all.sh # Run all tests
# Deploy to AWS (requires AWS CLI configured)
./aws/scripts/deploy.sh
# Check production health
./scripts/test_production_api.sh
# Fix production issues
./scripts/fix_production_api.sh
- Infrastructure: Deploy with Terraform (
aws/terraform/
) - Backend: ECS Fargate service with auto-scaling
- Frontend: S3 + CloudFront distribution
- Database: RDS PostgreSQL with automated backups
- Monitoring: CloudWatch logs and alerts
- π API Response Time: < 500ms average
- π Database: 40+ meetings with full AI categorization
- π Uptime: 99%+ availability
- πΎ Data: AI-processed meeting minutes and agenda items
# Automated testing
npm run test # Frontend unit tests
python -m pytest tests/backend/ -v # Backend API tests
# Manual testing
./scripts/test_production_api.sh # Production diagnostics
curl http://localhost:8000/health # Health checks
- β API Endpoints: All CRUD operations
- β Database Models: Meeting, User, Notification schemas
- β AI Services: Text extraction and categorization
- β Frontend Components: Key UI components
- β Production Health: End-to-end API testing
Meeting Details Not Loading:
./scripts/fix_production_api.sh # Automated fix
Backend Won't Start:
lsof -ti:8000 | xargs kill -9 # Kill conflicting processes
cd backend && source venv/bin/activate && python -m app.main
Frontend Dependencies Issues:
cd frontend && rm -rf node_modules package-lock.json && npm install
Production API Issues:
./scripts/test_production_api.sh # Diagnose issues
./scripts/fix_production_api.sh # Apply fixes
More solutions: See docs/TROUBLESHOOTING.md
- π Quick Start Guide - 5-minute setup
- π§ Environment Setup - Detailed configuration
- π§ͺ Testing Guide - Testing procedures
- βοΈ AWS Deployment - Production setup
- π CI/CD Setup - Automated workflows
- π‘οΈ Security Guide - Security best practices
- π API Documentation - Interactive API docs
- π€ Chatbot Guide - AI assistant setup
- π RAG System Guide - Document processing & vector search
- π§ͺ Evaluation System Guide - LLM-as-Judge testing framework
- π§ Scraper Documentation - Meeting discovery & PDF import
- Fork the repository
- Clone your fork:
git clone https://github.com/YOUR_USERNAME/CityCamp_AI.git
- Setup: Run
./scripts/start-dev.sh
- Develop: Make changes and test locally
- Test: Run
npm test
andpython -m pytest tests/backend/ -v
- Submit: Create pull request
- β TypeScript: Strict mode enabled
- β Python: Type hints and docstrings
- β Testing: Unit tests for new features
- β Linting: Pre-commit hooks configured
- β Documentation: Update README for new features
- π CloudWatch: Application and infrastructure metrics
- π Logs: Centralized logging with search capabilities
- π¨ Alerts: Automated notifications for issues
- π Dashboard: Real-time system health metrics
- π Web Traffic: CloudFront analytics
- π± API Usage: Request volume and response times
- πΎ Database: Query performance and connection health
- π€ AI Services: API usage and categorization accuracy
- π‘οΈ HTTPS: All communication encrypted
- π Environment Variables: Secrets management
- ποΈ VPC: Network isolation in AWS
- π€ IAM: Minimal permission roles
- π Security Scanning: Automated vulnerability checks
- π Database Encryption: At rest and in transit
- π API Authentication: JWT tokens
- π‘οΈ Input Validation: XSS and injection prevention
- π Audit Logs: Comprehensive activity tracking
MIT License - see LICENSE file for details.
- π Documentation: Check
docs/
directory - π Issues: Create GitHub issue with details
- π¬ Discussions: Use GitHub Discussions for questions
- π οΈ Emergency: Run
./scripts/fix_production_api.sh
for production issues
- π§ Email: [email protected]
- π GitHub: @kaizengrowth
RAG System & Evaluation Architecture Changes (August 5, 2025):
- β Implemented LLM-as-Judge evaluation system using GPT-4 for intelligent response assessment
- β Comprehensive RAG system for document-based chatbot responses with vector search
- β Added advanced evaluation framework with traditional + LLM metrics comparison
- β Created document processing pipeline with multi-format support (PDF, DOCX, TXT)
- β Integrated RAG into existing chatbot with OpenAI function calling
- β Added complete document management API with upload/search endpoints
- β Built evaluation suite with multiple modes (quick, full, RAG, performance, LLM-judge)
- β Fixed production meeting details loading issues
- β Added comprehensive production troubleshooting scripts
- β Resolved security vulnerabilities (esbuild, dependencies)
- β Enhanced CI/CD pipeline with automated testing
- β Updated documentation and README files
Next Planned Features:
- π RAG System Expansion: Upload city budgets, ordinances, and legislation
- π Advanced Document Search: Semantic search across all city documents
- π§ͺ Evaluation System Enhancement: A/B testing, user satisfaction metrics, continuous monitoring
- π€ Multi-Judge Evaluation: Compare GPT-4 vs Claude vs other LLMs as judges
- π Enhanced meeting data scraping automation
- π± Mobile app development
- π€ Advanced AI categorization improvements
- π User analytics dashboard
Ready to get started? Follow the Quick Start Guide or run ./scripts/start-dev.sh
! π