|
| 1 | +# Valkey Distributed Cache Configuration |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +Valkey distributed cache enables horizontal scaling of Trino Gateway by sharing query metadata across multiple gateway instances. When disabled, each gateway maintains its own local cache. |
| 6 | + |
| 7 | +## Quick Start (Minimal Configuration) |
| 8 | + |
| 9 | +```yaml |
| 10 | +valkeyConfiguration: |
| 11 | + enabled: true |
| 12 | + host: localhost |
| 13 | + port: 6379 |
| 14 | + # password: ${VALKEY_PASSWORD} # Optional: if AUTH required |
| 15 | +``` |
| 16 | + |
| 17 | +**That's it!** Sensible defaults are provided for all other settings. |
| 18 | + |
| 19 | +--- |
| 20 | + |
| 21 | +## Configuration Reference |
| 22 | + |
| 23 | +### Basic Settings |
| 24 | + |
| 25 | +| Parameter | Type | Default | Description | |
| 26 | +|-----------|------|---------|-------------| |
| 27 | +| `enabled` | boolean | `false` | Enable/disable distributed caching | |
| 28 | +| `host` | string | `localhost` | Valkey server hostname | |
| 29 | +| `port` | int | `6379` | Valkey server port | |
| 30 | +| `password` | string | `null` | Optional password for AUTH | |
| 31 | +| `database` | int | `0` | Database index (0-15) | |
| 32 | + |
| 33 | +### Advanced Settings (Optional) |
| 34 | + |
| 35 | +These settings have sensible defaults and should only be changed for specific performance tuning needs. |
| 36 | + |
| 37 | +| Parameter | Type | Default | Description | |
| 38 | +|-----------|------|---------|-------------| |
| 39 | +| `maxTotal` | int | `20` | Maximum total connections in pool | |
| 40 | +| `maxIdle` | int | `10` | Maximum idle connections | |
| 41 | +| `minIdle` | int | `5` | Minimum idle connections | |
| 42 | +| `timeoutMs` | int | `2000` | Connection timeout in milliseconds | |
| 43 | +| `cacheTtlSeconds` | long | `1800` | Cache entry TTL (30 minutes) | |
| 44 | +| `healthCheckIntervalMs` | long | `30000` | Health check interval (30 seconds) | |
| 45 | + |
| 46 | +--- |
| 47 | + |
| 48 | +## Environment Variables |
| 49 | + |
| 50 | +Use environment variable substitution for sensitive values: |
| 51 | + |
| 52 | +```yaml |
| 53 | +valkeyConfiguration: |
| 54 | + enabled: true |
| 55 | + host: ${VALKEY_HOST:localhost} |
| 56 | + port: ${VALKEY_PORT:6379} |
| 57 | + password: ${VALKEY_PASSWORD} |
| 58 | +``` |
| 59 | +
|
| 60 | +--- |
| 61 | +
|
| 62 | +## Deployment Scenarios |
| 63 | +
|
| 64 | +### Single Gateway Instance |
| 65 | +
|
| 66 | +```yaml |
| 67 | +valkeyConfiguration: |
| 68 | + enabled: false # Not needed - local cache is sufficient |
| 69 | +``` |
| 70 | +
|
| 71 | +### Multiple Gateway Instances (Recommended) |
| 72 | +
|
| 73 | +```yaml |
| 74 | +valkeyConfiguration: |
| 75 | + enabled: true |
| 76 | + host: valkey.internal.prod |
| 77 | + port: 6379 |
| 78 | + password: ${VALKEY_PASSWORD} |
| 79 | +``` |
| 80 | +
|
| 81 | +### High-Traffic Production (Advanced Tuning) |
| 82 | +
|
| 83 | +```yaml |
| 84 | +valkeyConfiguration: |
| 85 | + enabled: true |
| 86 | + host: valkey.internal.prod |
| 87 | + port: 6379 |
| 88 | + password: ${VALKEY_PASSWORD} |
| 89 | + maxTotal: 100 # More connections for high concurrency |
| 90 | + maxIdle: 50 |
| 91 | + minIdle: 25 |
| 92 | + timeoutMs: 5000 # Longer timeout for slower networks |
| 93 | + cacheTtlSeconds: 3600 # 1 hour for long-running queries |
| 94 | +``` |
| 95 | +
|
| 96 | +--- |
| 97 | +
|
| 98 | +## Connection Pool Sizing Guidelines |
| 99 | +
|
| 100 | +| Deployment Size | Gateway Instances | Recommended `maxTotal` | Recommended `maxIdle` | |
| 101 | +|-----------------|-------------------|------------------------|----------------------| |
| 102 | +| Small | 1-2 | 20 (default) | 10 (default) | |
| 103 | +| Medium | 3-5 | 50 | 25 | |
| 104 | +| Large | 6-10 | 100 | 50 | |
| 105 | +| Enterprise | 10+ | 200 | 100 | |
| 106 | + |
| 107 | +**Formula:** `maxTotal = (number of gateways) × 10` is a good starting point. |
| 108 | + |
| 109 | +--- |
| 110 | + |
| 111 | +## Performance Tuning |
| 112 | + |
| 113 | +### Cache TTL (`cacheTtlSeconds`) |
| 114 | + |
| 115 | +- **Default (1800s / 30min):** Good for typical workloads |
| 116 | +- **Short-lived queries (<5min):** Use 600s (10min) |
| 117 | +- **Long-running queries (hours):** Use 3600s (1 hour) or more |
| 118 | +- **Interactive development:** Use 300s (5min) |
| 119 | + |
| 120 | +### Health Check Interval (`healthCheckIntervalMs`) |
| 121 | + |
| 122 | +- **Default (30000ms / 30s):** Balanced check frequency |
| 123 | +- **Unstable network:** Increase to 60000ms (1 min) |
| 124 | +- **Critical systems:** Decrease to 10000ms (10s) |
| 125 | + |
| 126 | +### Connection Timeouts (`timeoutMs`) |
| 127 | + |
| 128 | +- **Default (2000ms):** Good for local/same-datacenter Valkey |
| 129 | +- **Cross-region:** Increase to 5000ms |
| 130 | +- **High latency network:** Increase to 10000ms |
| 131 | + |
| 132 | +--- |
| 133 | + |
| 134 | +## Monitoring |
| 135 | + |
| 136 | +Valkey cache exposes the following metrics (accessible via `ValkeyDistributedCache` instance): |
| 137 | + |
| 138 | +```java |
| 139 | +long hits = cache.getCacheHits(); |
| 140 | +long misses = cache.getCacheMisses(); |
| 141 | +long writes = cache.getCacheWrites(); |
| 142 | +long errors = cache.getCacheErrors(); |
| 143 | +double hitRate = cache.getCacheHitRate(); // Percentage |
| 144 | +``` |
| 145 | + |
| 146 | +### Expected Metrics (Healthy System) |
| 147 | + |
| 148 | +- **Cache Hit Rate:** 85-95% |
| 149 | +- **Cache Errors:** 0 (or very low) |
| 150 | +- **Cache Writes:** ~Equal to query submission rate |
| 151 | + |
| 152 | +### Troubleshooting |
| 153 | + |
| 154 | +**Low Hit Rate (<70%)** |
| 155 | +- Check TTL settings (may be too short) |
| 156 | +- Verify Valkey isn't evicting entries (check memory) |
| 157 | +- Check if multiple gateway versions deployed (cache key mismatch) |
| 158 | + |
| 159 | +**High Error Rate** |
| 160 | +- Check Valkey connectivity |
| 161 | +- Verify password/AUTH configuration |
| 162 | +- Review Valkey server logs |
| 163 | + |
| 164 | +**Connection Pool Exhaustion** |
| 165 | +- Increase `maxTotal` setting |
| 166 | +- Check for connection leaks (should be none with try-with-resources) |
| 167 | + |
| 168 | +--- |
| 169 | + |
| 170 | +## Security Considerations |
| 171 | + |
| 172 | +### Production Deployment Checklist |
| 173 | + |
| 174 | +- [ ] **Enable AUTH:** Set `password` in configuration |
| 175 | +- [ ] **Use Environment Variables:** Don't hardcode passwords |
| 176 | +- [ ] **Network Security:** Deploy Valkey in private VPC/network |
| 177 | +- [ ] **Encryption at Rest:** Enable Valkey persistence encryption |
| 178 | +- [ ] **TLS/SSL:** (Future enhancement - not yet supported) |
| 179 | +- [ ] **Access Control:** Restrict Valkey port (6379) to gateway instances only |
| 180 | + |
| 181 | +### Example Production Setup |
| 182 | + |
| 183 | +```yaml |
| 184 | +# config.yaml |
| 185 | +valkeyConfiguration: |
| 186 | + enabled: true |
| 187 | + host: ${VALKEY_INTERNAL_HOST} |
| 188 | + port: 6379 |
| 189 | + password: ${VALKEY_PASSWORD} |
| 190 | +``` |
| 191 | + |
| 192 | +```bash |
| 193 | +# Environment variables (set in deployment) |
| 194 | +export VALKEY_INTERNAL_HOST=valkey.vpc.internal |
| 195 | +export VALKEY_PASSWORD=$(vault read -field=password secret/valkey) |
| 196 | +``` |
| 197 | + |
| 198 | +--- |
| 199 | + |
| 200 | +## Architecture |
| 201 | + |
| 202 | +### 3-Tier Caching |
| 203 | + |
| 204 | +``` |
| 205 | +Request Flow: |
| 206 | +1. Check L1 (Local Guava Cache) → 10k entries, 30min TTL |
| 207 | + ├─ Hit: Return immediately (~1ms) |
| 208 | + └─ Miss: Continue to L2 |
| 209 | + |
| 210 | +2. Check L2 (Valkey Distributed Cache) → Shared across gateways |
| 211 | + ├─ Hit: Populate L1, return (~5ms) |
| 212 | + └─ Miss: Continue to L3 |
| 213 | + |
| 214 | +3. Check L3 (PostgreSQL Database) → Source of truth |
| 215 | + ├─ Found: Populate L2 + L1, return (~50ms) |
| 216 | + └─ Not Found: Search all backends via HTTP (~200ms) |
| 217 | +``` |
| 218 | +
|
| 219 | +### Cache Keys |
| 220 | +
|
| 221 | +``` |
| 222 | +Backend: trino:query:backend:{queryId} |
| 223 | +Routing Group: trino:query:routinggroup:{queryId} |
| 224 | +External URL: trino:query:externalurl:{queryId} |
| 225 | +``` |
| 226 | +
|
| 227 | +--- |
| 228 | +
|
| 229 | +## Migration Guide |
| 230 | +
|
| 231 | +### From Single Gateway to Multi-Gateway |
| 232 | +
|
| 233 | +1. **Deploy Valkey server** (standalone or cluster) |
| 234 | +2. **Update config.yaml** on all gateways: |
| 235 | + ```yaml |
| 236 | + valkeyConfiguration: |
| 237 | + enabled: true |
| 238 | + host: valkey.internal |
| 239 | + port: 6379 |
| 240 | + password: ${VALKEY_PASSWORD} |
| 241 | + ``` |
| 242 | +3. **Restart gateways** (rolling restart recommended) |
| 243 | +4. **Monitor metrics** to verify cache hit rates |
| 244 | + |
| 245 | +No data migration needed - cache will populate automatically. |
| 246 | + |
| 247 | +--- |
| 248 | + |
| 249 | +## FAQ |
| 250 | + |
| 251 | +**Q: Do I need Valkey if I only have one gateway?** |
| 252 | +A: No. Local Guava cache is sufficient for single-instance deployments. |
| 253 | + |
| 254 | +**Q: What happens if Valkey goes down?** |
| 255 | +A: Graceful degradation - queries continue working, falling back to database. Performance may degrade slightly. |
| 256 | + |
| 257 | +**Q: Can I use Redis instead of Valkey?** |
| 258 | +A: Yes! Valkey is a Redis fork with compatible protocol. Just point to your Redis server. |
| 259 | + |
| 260 | +**Q: How much memory does Valkey need?** |
| 261 | +A: Rough estimate: `(queries per minute) × (average query lifetime in minutes) × 500 bytes` |
| 262 | + Example: 1000 q/min × 30 min × 500 bytes = ~15 MB |
| 263 | + |
| 264 | +**Q: Can I clear the cache?** |
| 265 | +A: Yes, via Valkey CLI: `redis-cli -h <host> -a <password> FLUSHDB` |
| 266 | + Or selectively: `redis-cli DEL trino:query:backend:*` |
| 267 | + |
| 268 | +--- |
| 269 | + |
| 270 | +## Support |
| 271 | + |
| 272 | +For issues or questions: |
| 273 | +- GitHub Issues: https://github.com/trinodb/trino-gateway/issues |
| 274 | +- Trino Community Slack: #trino-gateway channel |
0 commit comments