|
| 1 | +# OpenTelemetry SDK Implementation Notes |
| 2 | + |
| 3 | +This document describes the implementation details and design decisions for the OpenTelemetry OTLP exporter. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +This implementation provides full OpenTelemetry Protocol (OTLP) support using the official OpenTelemetry SDK. It bridges the `stats` library's metric interface to OpenTelemetry's metric API. |
| 8 | + |
| 9 | +## Architecture |
| 10 | + |
| 11 | +### Core Components |
| 12 | + |
| 13 | +1. **SDKHandler** - Main handler implementing `stats.Handler` |
| 14 | +2. **Protocol Support** - Both gRPC and HTTP/Protobuf transports |
| 15 | +3. **Instrument Management** - Efficient caching of OpenTelemetry instruments |
| 16 | +4. **Gauge Value Tracking** - Delta calculation for absolute gauge semantics |
| 17 | + |
| 18 | +## Design Decisions |
| 19 | + |
| 20 | +### 1. Gauge Implementation |
| 21 | + |
| 22 | +**Challenge**: OpenTelemetry's stable SDK doesn't have a true Gauge instrument yet. |
| 23 | + |
| 24 | +**Solution**: Use `Float64UpDownCounter` with delta calculation to maintain absolute value semantics. |
| 25 | + |
| 26 | +```go |
| 27 | +// When stats.Set("metric", 42) is called: |
| 28 | +// 1. Calculate delta: newValue - previousValue |
| 29 | +// 2. Call UpDownCounter.Add(delta) |
| 30 | +// 3. Store newValue for next delta calculation |
| 31 | +``` |
| 32 | + |
| 33 | +**Why**: Users expect `stats.Set("metric", 42)` to set the metric to 42, not add 42 to the previous value. By tracking previous values and calculating deltas, we maintain this semantic while using UpDownCounter. |
| 34 | + |
| 35 | +**Trade-off**: Requires additional memory to track gauge values per metric+attribute combination. |
| 36 | + |
| 37 | +### 2. Context Management |
| 38 | + |
| 39 | +**Challenge**: Stored contexts can be cancelled, causing metric recording to fail. |
| 40 | + |
| 41 | +**Solution**: |
| 42 | +- Use `context.Background()` for metric recording operations |
| 43 | +- Store the initialization context as `shutdownCtx` only for shutdown operations |
| 44 | +- This ensures metrics continue to be recorded even if the original context is cancelled |
| 45 | + |
| 46 | +**Why**: Metric recording should be resilient and not fail due to context cancellation. The handler should continue working throughout the application lifecycle. |
| 47 | + |
| 48 | +### 3. Instrument Caching |
| 49 | + |
| 50 | +**Implementation**: Thread-safe two-level locking pattern |
| 51 | +```go |
| 52 | +// Fast path: read lock for lookup |
| 53 | +h.mu.RLock() |
| 54 | +inst, exists := h.instruments[metricName] |
| 55 | +h.mu.RUnlock() |
| 56 | + |
| 57 | +// Slow path: write lock only if creating new instrument |
| 58 | +if !exists { |
| 59 | + h.mu.Lock() |
| 60 | + // Double-check after acquiring write lock |
| 61 | + inst, exists = h.instruments[metricName] |
| 62 | + if !exists { |
| 63 | + inst = h.createInstruments(meter, metricName, field.Type()) |
| 64 | + h.instruments[metricName] = inst |
| 65 | + } |
| 66 | + h.mu.Unlock() |
| 67 | +} |
| 68 | +``` |
| 69 | + |
| 70 | +**Why**: Instruments are created once per metric name and reused. This pattern minimizes lock contention in the hot path (metric recording) while ensuring thread-safety during instrument creation. |
| 71 | + |
| 72 | +### 4. Attribute Handling |
| 73 | + |
| 74 | +**Implementation**: Direct conversion from `stats.Tag` to `attribute.KeyValue` |
| 75 | +```go |
| 76 | +func (h *SDKHandler) tagsToAttributes(tags []stats.Tag) []attribute.KeyValue { |
| 77 | + attrs := make([]attribute.KeyValue, 0, len(tags)) |
| 78 | + for _, tag := range tags { |
| 79 | + attrs = append(attrs, attribute.String(tag.Name, tag.Value)) |
| 80 | + } |
| 81 | + return attrs |
| 82 | +} |
| 83 | +``` |
| 84 | + |
| 85 | +**Why**: Simple 1:1 mapping preserves all user-provided metadata without transformation. |
| 86 | + |
| 87 | +### 5. Resource Detection |
| 88 | + |
| 89 | +**Pattern**: Leverage official OpenTelemetry resource detectors |
| 90 | +```go |
| 91 | +resource.New(ctx, |
| 92 | + resource.WithDetectors(ec2.NewResourceDetector()), |
| 93 | + resource.WithFromEnv(), |
| 94 | + resource.WithHost(), |
| 95 | + resource.WithProcess(), |
| 96 | +) |
| 97 | +``` |
| 98 | + |
| 99 | +**Why**: Automatic detection of cloud provider, Kubernetes, host, and process metadata without manual configuration. |
| 100 | + |
| 101 | +## Performance Considerations |
| 102 | + |
| 103 | +### Instrument Reuse |
| 104 | +- Instruments are created once and cached |
| 105 | +- RWMutex allows concurrent reads (the common case) |
| 106 | +- Write locks only taken during initial instrument creation |
| 107 | + |
| 108 | +### Gauge Delta Calculation |
| 109 | +- Memory overhead: O(unique metric × unique attribute sets) |
| 110 | +- Computational overhead: One map lookup + one subtraction per gauge recording |
| 111 | +- Trade-off: Necessary to maintain correct gauge semantics |
| 112 | + |
| 113 | +### Batching and Export Strategy |
| 114 | + |
| 115 | +**Decision**: Delegate all batching to OpenTelemetry SDK's `PeriodicReader` |
| 116 | + |
| 117 | +**Implementation**: No custom buffering or batching logic in the handler |
| 118 | +```go |
| 119 | +provider := sdkmetric.NewMeterProvider( |
| 120 | + sdkmetric.WithResource(res), |
| 121 | + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter, |
| 122 | + sdkmetric.WithInterval(config.ExportInterval), // Default: 10s |
| 123 | + sdkmetric.WithTimeout(config.ExportTimeout), // Default: 30s |
| 124 | + )), |
| 125 | +) |
| 126 | +``` |
| 127 | + |
| 128 | +**Why**: |
| 129 | +- The OTel SDK provides production-ready batching with in-memory aggregation |
| 130 | +- `PeriodicReader` handles timing, aggregation reset, and export lifecycle |
| 131 | +- Avoids reinventing batching logic and potential bugs |
| 132 | +- Provides standard OTel behavior that users expect |
| 133 | + |
| 134 | +**How it works**: |
| 135 | +1. Metrics are recorded immediately to OTel instruments (no blocking) |
| 136 | +2. SDK aggregates metrics in memory (e.g., summing counters, collecting histogram samples) |
| 137 | +3. Every `ExportInterval`, the reader exports aggregated data and resets aggregations |
| 138 | +4. Reduces network overhead and collector load automatically |
| 139 | + |
| 140 | +**Trade-offs**: |
| 141 | +- Metrics are not real-time (delayed by up to `ExportInterval`) |
| 142 | +- Memory grows proportionally to metric cardinality until export |
| 143 | +- Users must call `Flush()` before shutdown to export remaining metrics |
| 144 | + |
| 145 | +## Error Handling |
| 146 | + |
| 147 | +### Instrument Creation Failures |
| 148 | +- Logged but don't block other metrics |
| 149 | +- Silent no-op if instrument is nil |
| 150 | +- Prevents cascade failures |
| 151 | + |
| 152 | +### Export Failures |
| 153 | +- Logged but don't stop metric collection |
| 154 | +- Retries handled by OpenTelemetry SDK exporters |
| 155 | +- Backoff and timeout configured at SDK level |
| 156 | + |
| 157 | +### Context Cancellation |
| 158 | +- Metric recording uses background context |
| 159 | +- Unaffected by user context cancellation |
| 160 | +- Shutdown still respects user-provided context |
| 161 | + |
| 162 | +## Testing Strategy |
| 163 | + |
| 164 | +### Unit Tests |
| 165 | +- Instrument creation and caching |
| 166 | +- Gauge delta calculation |
| 167 | +- Value type conversions |
| 168 | +- Protocol selection (HTTP vs gRPC) |
| 169 | + |
| 170 | +### Integration Tests |
| 171 | +- Environment variable configuration |
| 172 | +- Multiple concurrent metrics |
| 173 | +- Gauge absolute value semantics |
| 174 | + |
| 175 | +### Benchmarks |
| 176 | +- Metric recording performance |
| 177 | +- Lock contention under load |
| 178 | + |
| 179 | +## Limitations and Known Issues |
| 180 | + |
| 181 | +### 1. Gauge Implementation |
| 182 | +- Requires tracking previous values in memory |
| 183 | +- High cardinality can increase memory usage |
| 184 | +- Memory is never freed (instruments are cached forever) |
| 185 | + |
| 186 | +### 2. No Exemplars |
| 187 | +- Current implementation doesn't support exemplars |
| 188 | +- Could be added in future versions |
| 189 | + |
| 190 | +### 3. No Custom Views |
| 191 | +- Uses default aggregation and views |
| 192 | +- Advanced users may want custom histogram buckets or aggregations |
| 193 | + |
| 194 | +## Future Enhancements |
| 195 | + |
| 196 | +### Potential Improvements |
| 197 | +1. **Memory Management**: Add LRU eviction for unused instruments |
| 198 | +2. **Exemplar Support**: Bridge to trace context for exemplars |
| 199 | +3. **Custom Views**: Allow users to configure aggregations |
| 200 | +4. **Metric Metadata**: Expose units and descriptions via OTel API |
| 201 | +5. **Delta vs Cumulative**: Support both temporality modes |
| 202 | + |
| 203 | +### OpenTelemetry SDK Evolution |
| 204 | +- **Gauge Support**: When stable SDK adds Gauge, migrate from UpDownCounter |
| 205 | +- **New Instrument Types**: Support ExponentialHistogram when available |
| 206 | +- **Protocol Extensions**: Support new OTLP features as they're added |
| 207 | + |
| 208 | +## Migration from Legacy Handler |
| 209 | + |
| 210 | +The legacy `Handler` in this package is marked as Alpha and has limitations: |
| 211 | + |
| 212 | +**Legacy Handler Issues:** |
| 213 | +- Custom OTLP implementation (not using official SDK) |
| 214 | +- Only HTTP transport (despite having gRPC dependencies) |
| 215 | +- No environment variable support |
| 216 | +- No resource detection |
| 217 | + |
| 218 | +**SDKHandler Advantages:** |
| 219 | +- Official OpenTelemetry SDK |
| 220 | +- Both HTTP and gRPC |
| 221 | +- Full environment variable support |
| 222 | +- Automatic resource detection |
| 223 | +- Production-ready and well-tested |
| 224 | + |
| 225 | +**Migration Path:** |
| 226 | +```go |
| 227 | +// Old (legacy) |
| 228 | +handler := &otlp.Handler{ |
| 229 | + Client: otlp.NewHTTPClient(endpoint), |
| 230 | + // ... |
| 231 | +} |
| 232 | + |
| 233 | +// New (recommended) |
| 234 | +handler, err := otlp.NewSDKHandler(ctx, otlp.SDKConfig{ |
| 235 | + Protocol: otlp.ProtocolHTTPProtobuf, |
| 236 | + Endpoint: endpoint, |
| 237 | +}) |
| 238 | +``` |
| 239 | + |
| 240 | +## References |
| 241 | + |
| 242 | +- [OpenTelemetry Metrics Specification](https://opentelemetry.io/docs/specs/otel/metrics/) |
| 243 | +- [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/) |
| 244 | +- [Go SDK Documentation](https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric) |
| 245 | +- [Resource Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/resource/) |
0 commit comments