Envoy AI Gateway Demos Repository

Comprehensive demos and examples for the Envoy AI Gateway

Showcasing how to deploy, configure, and use AI Gateway features in Kubernetes environments

✨ Key Features of Envoy AI Gateway

• 🔌 Multi-Provider Support - Route traffic to OpenAI, AWS Bedrock, Azure OpenAI, and more
• 🔒 Token-Based Rate Limiting - Advanced rate limiting based on AI tokens, not just requests
• 🔄 Provider Fallback - Automatic failover between AI providers for reliability
• 📊 OpenAI-Compatible API - Drop-in replacement for OpenAI API clients
• 🛡️ Built on Envoy - Leverages battle-tested Envoy Proxy technology
• 📈 Observability - Rich metrics, tracing, and logging for AI workloads
• 🔧 Kubernetes Native - Designed for cloud-native environments

🎯 What's Inside

• Complete Demo Environments: Ready-to-run demos with automated setup and testing
• Infrastructure Automation: Taskfile-based automation for cluster setup and management
• CI/CD Integration: GitHub Actions workflows for automated testing and validation
• Production-Ready Examples: Real-world configurations and best practices

📁 Repository Structure

├── demos/                           # Individual demo environments
│   ├── 01-getting-started/          # Basic Envoy AI Gateway setup with LLM-D simulator  
│   └── 02-usage-based-rate-limiting/ # Advanced token-based rate limiting
├── scripts/                         # Automation scripts for setup and management
├── .github/workflows/               # CI/CD workflows for automated testing
└── Taskfile.yml                     # Main automation tasks

🚀 Available Demos

💡 Each demo includes its own comprehensive README with detailed setup instructions, configuration options, and usage examples. Always refer to the individual demo README for complete guidance.

01-getting-started

A comprehensive introduction to Envoy AI Gateway featuring:

• LLM-D Inference Simulator as a lightweight AI backend
• Qwen3 model configured in echo mode for testing
• Complete API endpoints (chat, models, streaming)
• Automated testing suite with GitHub Actions integration
• Performance tuning (10ms TTFT, 20ms inter-token latency)

📖 Read the full demo README for step-by-step instructions and detailed configuration.

02-usage-based-rate-limiting

Advanced token-based rate limiting for AI workloads featuring:

• Token-based rate limiting with different quotas per model (qwen3: 50/hour, gpt-4: 1000/hour, gpt-3.5-turbo: 100/hour)
• Per-user and per-model enforcement using x-user-id and x-ai-eg-model headers
• Automatic token tracking from LLM responses with input/output/total token metrics
• Raw metrics collection via task metrics with Prometheus-compatible output
• Rate limit enforcement with 429 status codes and comprehensive testing

📖 Read the full demo README for usage-based rate limiting setup and metrics analysis.

🛠️ Prerequisites

Before running any demos, ensure you have:

• Taskfile - Task runner for automation Installation

  sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin/

• kind - Kubernetes in Docker (installed automatically)

• kubectl - Kubernetes CLI (installed automatically)

• helm - Kubernetes package manager (installed automatically)

• jq - JSON processor (recommended for testing)

• Docker - Container runtime

⚡ Quick Start

Option 1: Full Environment Setup

Set up the complete Envoy AI Gateway environment:

task setup-all

This will:

1. Install all required dependencies (kind, helm, kubectl)
2. Create a kind cluster with proper configuration
3. Install Envoy Gateway (latest version)
4. Install Envoy AI Gateway (latest version)
5. Configure AI Gateway integration
6. Verify the complete installation

Option 2: Run a Specific Demo

Jump directly into a demo:

cd demos/01-getting-started
# Read the demo README first for detailed instructions
cat README.md
task setup

📝 Important: Each demo has its own README with specific setup instructions, configuration details, and usage examples. Always check the demo's README before running tasks.

📋 Available Tasks

View all available tasks:

task --list

🔧 Configuration

The following environment variables can be customized in Taskfile.yml:

• CLUSTER_NAME (default: envoy-ai-gateway-demo)
• KIND_VERSION (default: v0.29.0)
• ENVOY_GATEWAY_VERSION (default: v0.0.0-latest)
• ENVOY_AI_GATEWAY_VERSION (default: v0.0.0-latest)

🏗️ Core Setup Tasks

• task setup-all - Complete environment setup from scratch
• task create-cluster - Create kind cluster with Envoy Gateway
• task install-envoy-gateway - Install Envoy Gateway only
• task install-envoy-ai-gateway - Install Envoy AI Gateway only
• task verify-installation - Verify all components are running

🔍 Monitoring & Status Tasks

• task port-forward - Port forward to access gateway (localhost:8080)
• task status - Check status of all components
• task logs - View logs from AI Gateway components

🧹 Cleanup Tasks

• task cleanup - Remove all resources and cluster
• task reset - Reset environment for fresh start

Utility Tasks

• task cleanup - Remove k3s cluster and cleanup
• task logs-envoy-gateway - View Envoy Gateway logs
• task logs-ai-gateway - View AI Gateway logs
• task port-forward - Port forward to access gateway (localhost:8080)
• task verify-installation - Verify installation status

🐛 Troubleshooting

Common Issues

1. Kind cluster creation fails

   # Ensure Docker is running and has sufficient resources
   docker info
   task cleanup && task create-cluster

2. Gateway installation fails

   # Verify cluster readiness
   kubectl get nodes
   kubectl get pods -A
   task verify-installation

3. Port forwarding issues

   # Kill existing port forwards and restart
   pkill -f "kubectl.*port-forward"
   task port-forward

4. Demo-specific issues

   # Check demo logs and status
   cd demos/01-getting-started
   task logs
   task test

Getting Help

• Task Status: task --status <task-name>
• Component Logs: kubectl logs -n envoy-gateway-system -l app=envoy-gateway
• Installation Check: task verify-installation
• Demo Diagnostics: Each demo includes comprehensive logging and testing

🚀 Development & Contributing

Adding New Demos

1. Create directory: demos/<demo-name>/
2. Include required files:
   • README.md - Comprehensive demo documentation
   • Taskfile.yml - Demo-specific automation tasks
   • Kubernetes manifests and configurations
3. Add GitHub Actions workflow: .github/workflows/demo-<name>.yml
4. Test thoroughly with task test

Best Practices

• Documentation: Each demo should be self-contained with clear README
• Automation: Use Taskfile for all setup, testing, and cleanup operations
• Testing: Include comprehensive test suites with error diagnostics
• CI/CD: Add GitHub Actions workflows for automated validation
• Resource Management: Ensure proper cleanup and resource limits

Contributing Guidelines

1. Fork the repository
2. Create a feature branch: git checkout -b feature/new-demo
3. Develop your demo following the established patterns
4. Test locally: task setup-all && cd demos/<your-demo> && task test
5. Ensure CI passes: Check GitHub Actions workflows
6. Submit a pull request with detailed description

📚 Resources & Documentation

• Envoy AI Gateway - Official documentation and guides
• Envoy Gateway - Core Envoy Gateway project
• Taskfile - Task runner documentation
• Kind - Kubernetes in Docker
• LLM-D Inference Simulator - Lightweight AI backend for testing

Ready to get started?

Jump into the 01-getting-started demo or run task setup-all to set up the complete environment! 🎉

🔗 Official Documentation & Resources

📖 Documentation

• Envoy AI Gateway Docs - Complete documentation and guides
• Getting Started Guide - Quick start tutorial
• Basic Usage - Core concepts and examples
• LLM Provider Integrations - Supported AI services
• Release Notes - Latest updates and features

🌐 Community & Support

• GitHub Repository - Source code and issues
• Slack Community - Join the conversation
• Weekly Community Meetings - Thursdays
• GitHub Discussions - Community Q&A

🔧 Related Projects

• Envoy Gateway - Core gateway functionality
• Envoy Proxy - The underlying proxy technology
• LLM-D Inference Simulator - Lightweight testing backend