|
| 1 | +# SQLite Implementation Documentation |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +Eyelet's SQLite implementation provides high-performance, concurrent logging for multiple Claude Code instances with rich metadata and powerful query capabilities. The system supports flexible logging formats (JSON files, SQLite, or both) based on user configuration. |
| 6 | + |
| 7 | +## Architecture |
| 8 | + |
| 9 | +### 1. Connection Management (`sqlite_connection.py`) |
| 10 | + |
| 11 | +**Process-Local Connections** |
| 12 | +- Each process maintains its own connection (fork-safe) |
| 13 | +- Automatically recreates connection when process ID changes |
| 14 | +- No shared connection pool between processes (avoids multi-process issues) |
| 15 | + |
| 16 | +```python |
| 17 | +class ProcessLocalConnection: |
| 18 | + def __init__(self, db_path: Path): |
| 19 | + self._pid = None |
| 20 | + self._conn = None |
| 21 | + |
| 22 | + @property |
| 23 | + def connection(self): |
| 24 | + if self._pid != os.getpid(): |
| 25 | + # New process detected, create new connection |
| 26 | + self._conn = self._create_connection() |
| 27 | + self._pid = os.getpid() |
| 28 | + return self._conn |
| 29 | +``` |
| 30 | + |
| 31 | +**Optimizations Applied** |
| 32 | +- WAL mode for concurrent reads/writes |
| 33 | +- 64MB cache size |
| 34 | +- Memory-mapped I/O (256MB) |
| 35 | +- 60-second busy timeout |
| 36 | +- Autocommit mode for logging |
| 37 | + |
| 38 | +### 2. Retry Logic |
| 39 | + |
| 40 | +**Exponential Backoff with Jitter** |
| 41 | +```python |
| 42 | +@sqlite_retry(max_attempts=10, base_delay=0.05) |
| 43 | +def log_hook(self, hook_data: HookData) -> bool: |
| 44 | + # Automatic retry on database lock |
| 45 | +``` |
| 46 | + |
| 47 | +- Base delay: 50ms |
| 48 | +- Exponential growth: 0.05s → 0.1s → 0.2s → 0.4s → 0.8s... |
| 49 | +- Random jitter prevents thundering herd |
| 50 | +- Maximum 10 attempts before failure |
| 51 | + |
| 52 | +### 3. Database Schema |
| 53 | + |
| 54 | +**Hybrid Design**: Indexed fields + full JSON data |
| 55 | + |
| 56 | +```sql |
| 57 | +CREATE TABLE hooks ( |
| 58 | + -- Core indexed fields for fast queries |
| 59 | + id INTEGER PRIMARY KEY, |
| 60 | + timestamp REAL NOT NULL, |
| 61 | + session_id TEXT NOT NULL, |
| 62 | + hook_type TEXT NOT NULL, |
| 63 | + tool_name TEXT, |
| 64 | + status TEXT, |
| 65 | + |
| 66 | + -- Metadata fields |
| 67 | + hostname TEXT, |
| 68 | + ip_address TEXT, |
| 69 | + project_dir TEXT, |
| 70 | + |
| 71 | + -- Full data as BLOB (JSONB optimization) |
| 72 | + data BLOB NOT NULL CHECK(json_valid(data)), |
| 73 | + |
| 74 | + -- Generated columns for JSON fields |
| 75 | + error_code TEXT GENERATED ALWAYS AS |
| 76 | + (json_extract(data, '$.execution.error_message')) STORED, |
| 77 | + git_branch TEXT GENERATED ALWAYS AS |
| 78 | + (json_extract(data, '$.metadata.git.branch')) STORED |
| 79 | +); |
| 80 | +``` |
| 81 | + |
| 82 | +**Indexes** |
| 83 | +- Single column indexes on core fields |
| 84 | +- Composite index for time-based queries: `(hook_type, timestamp DESC)` |
| 85 | +- Conditional indexes on generated columns |
| 86 | + |
| 87 | +### 4. Logging Flow |
| 88 | + |
| 89 | +``` |
| 90 | +1. Hook Triggered → execute.py |
| 91 | +2. ConfigService loads eyelet.yaml settings |
| 92 | +3. HookLogger determines format (json/sqlite/both) and scope (project/global/both) |
| 93 | +4. For SQLite: |
| 94 | + - GitMetadata enriches log data |
| 95 | + - SQLiteLogger writes with retry logic |
| 96 | + - ProcessLocalConnection handles concurrency |
| 97 | +5. For JSON: |
| 98 | + - Traditional file-based logging to eyelet-hooks/ |
| 99 | +``` |
| 100 | + |
| 101 | +### 5. Query System (`query_service.py`) |
| 102 | + |
| 103 | +**Flexible Filtering** |
| 104 | +- By hook type, tool name, session ID |
| 105 | +- Time-based (since/until) |
| 106 | +- By status, git branch, error presence |
| 107 | +- Full-text search across JSON data |
| 108 | + |
| 109 | +**CLI Commands** |
| 110 | +```bash |
| 111 | +# Search with filters |
| 112 | +eyelet query search --hook-type PreToolUse --tool Bash --since 1h |
| 113 | + |
| 114 | +# Summary statistics |
| 115 | +eyelet query summary --since 24h |
| 116 | + |
| 117 | +# Recent errors |
| 118 | +eyelet query errors --limit 10 |
| 119 | + |
| 120 | +# Session timeline |
| 121 | +eyelet query session <session-id> |
| 122 | + |
| 123 | +# Search in data |
| 124 | +eyelet query grep "git push" |
| 125 | +``` |
| 126 | + |
| 127 | +### 6. Configuration (`eyelet.yaml`) |
| 128 | + |
| 129 | +**Global Configuration** (`~/.claude/eyelet.yaml`) |
| 130 | +```yaml |
| 131 | +logging: |
| 132 | + format: json # json, sqlite, or both |
| 133 | + enabled: true |
| 134 | + scope: project # project, global, or both |
| 135 | + global_path: ~/.claude/eyelet-logging |
| 136 | + project_path: .eyelet-logging |
| 137 | + add_to_gitignore: true |
| 138 | + |
| 139 | +metadata: |
| 140 | + include_hostname: true |
| 141 | + include_ip: true |
| 142 | + custom_fields: |
| 143 | + team: engineering |
| 144 | +``` |
| 145 | +
|
| 146 | +**Project Override** (`./eyelet.yaml`) |
| 147 | +```yaml |
| 148 | +logging: |
| 149 | + format: sqlite # Override to use SQLite |
| 150 | + scope: both # Log to both locations |
| 151 | +``` |
| 152 | + |
| 153 | +## Performance Characteristics |
| 154 | + |
| 155 | +### Write Performance |
| 156 | +- Single insert: ~5-10ms with retry logic |
| 157 | +- Batch insert: 100-200 inserts/second |
| 158 | +- Concurrent writes: Handled via WAL mode + retry |
| 159 | +- Lock contention: Mitigated by exponential backoff |
| 160 | + |
| 161 | +### Query Performance |
| 162 | +- Indexed queries: <10ms for most operations |
| 163 | +- Full-text search: Linear scan (optimizable with FTS5) |
| 164 | +- Summary statistics: ~50ms for 10,000 records |
| 165 | + |
| 166 | +### Storage |
| 167 | +- Overhead: ~1KB per log entry |
| 168 | +- Compression: Via BLOB storage for JSON |
| 169 | +- Growth rate: ~100MB per 100,000 hooks |
| 170 | + |
| 171 | +## Migration Strategy |
| 172 | + |
| 173 | +Schema migrations use SQLite's `PRAGMA user_version`: |
| 174 | + |
| 175 | +```python |
| 176 | +MIGRATIONS = [ |
| 177 | + (1, "Initial schema", "..."), |
| 178 | + (2, "Add user_id column", "ALTER TABLE hooks ADD COLUMN user_id TEXT;") |
| 179 | +] |
| 180 | +
|
| 181 | +def migrate_database(conn): |
| 182 | + current_version = conn.execute("PRAGMA user_version").fetchone()[0] |
| 183 | + for version, description, sql in MIGRATIONS: |
| 184 | + if version > current_version: |
| 185 | + conn.executescript(sql) |
| 186 | + conn.execute(f"PRAGMA user_version = {version}") |
| 187 | +``` |
| 188 | + |
| 189 | +## Monitoring & Health |
| 190 | + |
| 191 | +**Database Health Checks** |
| 192 | +- Integrity verification |
| 193 | +- WAL file size monitoring |
| 194 | +- Table statistics and growth tracking |
| 195 | +- Index usage analysis |
| 196 | + |
| 197 | +**Performance Monitoring** |
| 198 | +- Insert/query timing statistics |
| 199 | +- Retry counts and lock wait times |
| 200 | +- Success rates and error tracking |
| 201 | + |
| 202 | +## Security Considerations |
| 203 | + |
| 204 | +1. **Git Credentials**: Sanitized from remote URLs |
| 205 | +2. **Environment Variables**: Only CLAUDE*/EYELET* vars logged |
| 206 | +3. **File Permissions**: Database created with user-only access |
| 207 | +4. **SQL Injection**: Parameterized queries throughout |
| 208 | + |
| 209 | +## Future Enhancements |
| 210 | + |
| 211 | +1. **Full-Text Search**: Add FTS5 for better search performance |
| 212 | +2. **Compression**: zlib compression for older entries |
| 213 | +3. **Partitioning**: Time-based partitioning for large databases |
| 214 | +4. **Replication**: Optional sync to remote database |
| 215 | +5. **Analytics**: Pre-computed statistics tables |
| 216 | +6. **Retention**: Automatic cleanup of old entries |
0 commit comments