Skip to content

Commit 6112605

Browse files
committed
feat: Complete SQLite logging integration
Major implementation completed: - Integrated new unified logging system with execute command - Support for json, sqlite, or both logging formats based on config - Rich metadata collection including Git repository information - Powerful query interface with multiple commands and filters - Process-safe SQLite implementation with retry logic Key features: 1. Configuration-driven logging (eyelet.yaml) - Global config: ~/.claude/eyelet.yaml - Project config: ./eyelet.yaml - Flexible format and scope options 2. SQLite implementation - Process-local connections (fork-safe) - WAL mode with optimized pragmas - Exponential backoff retry logic - Modern schema with generated columns - Performance monitoring and health checks 3. Query interface (eyelet query) - search: Filter by type, tool, session, time, status - summary: Activity statistics - errors: Recent error analysis - session: Timeline view - grep: Full-text search 4. Git metadata enrichment - Branch, commit, dirty state - Repository information - Sanitized remote URLs Fixes: - JSON serialization for Path objects - Console stderr output handling - Query commands handle missing execution data The system now supports high-volume concurrent logging from multiple Claude Code instances with powerful analysis capabilities.
1 parent 35db390 commit 6112605

17 files changed

+2730
-86
lines changed

.ai/research/2025-01-29-sqlite-best-practices.md

Lines changed: 583 additions & 0 deletions
Large diffs are not rendered by default.

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,8 @@ uv.lock
6969
workflows/user/*
7070
.claude/settings.json.backup
7171
eyelet-hooks/
72+
.eyelet-logs/
73+
.eyelet-logging/
7274
test-*.json
7375
tmp-test/claude-code/
7476
claude-code/
Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
# SQLite Implementation Documentation
2+
3+
## Overview
4+
5+
Eyelet's SQLite implementation provides high-performance, concurrent logging for multiple Claude Code instances with rich metadata and powerful query capabilities. The system supports flexible logging formats (JSON files, SQLite, or both) based on user configuration.
6+
7+
## Architecture
8+
9+
### 1. Connection Management (`sqlite_connection.py`)
10+
11+
**Process-Local Connections**
12+
- Each process maintains its own connection (fork-safe)
13+
- Automatically recreates connection when process ID changes
14+
- No shared connection pool between processes (avoids multi-process issues)
15+
16+
```python
17+
class ProcessLocalConnection:
18+
def __init__(self, db_path: Path):
19+
self._pid = None
20+
self._conn = None
21+
22+
@property
23+
def connection(self):
24+
if self._pid != os.getpid():
25+
# New process detected, create new connection
26+
self._conn = self._create_connection()
27+
self._pid = os.getpid()
28+
return self._conn
29+
```
30+
31+
**Optimizations Applied**
32+
- WAL mode for concurrent reads/writes
33+
- 64MB cache size
34+
- Memory-mapped I/O (256MB)
35+
- 60-second busy timeout
36+
- Autocommit mode for logging
37+
38+
### 2. Retry Logic
39+
40+
**Exponential Backoff with Jitter**
41+
```python
42+
@sqlite_retry(max_attempts=10, base_delay=0.05)
43+
def log_hook(self, hook_data: HookData) -> bool:
44+
# Automatic retry on database lock
45+
```
46+
47+
- Base delay: 50ms
48+
- Exponential growth: 0.05s → 0.1s → 0.2s → 0.4s → 0.8s...
49+
- Random jitter prevents thundering herd
50+
- Maximum 10 attempts before failure
51+
52+
### 3. Database Schema
53+
54+
**Hybrid Design**: Indexed fields + full JSON data
55+
56+
```sql
57+
CREATE TABLE hooks (
58+
-- Core indexed fields for fast queries
59+
id INTEGER PRIMARY KEY,
60+
timestamp REAL NOT NULL,
61+
session_id TEXT NOT NULL,
62+
hook_type TEXT NOT NULL,
63+
tool_name TEXT,
64+
status TEXT,
65+
66+
-- Metadata fields
67+
hostname TEXT,
68+
ip_address TEXT,
69+
project_dir TEXT,
70+
71+
-- Full data as BLOB (JSONB optimization)
72+
data BLOB NOT NULL CHECK(json_valid(data)),
73+
74+
-- Generated columns for JSON fields
75+
error_code TEXT GENERATED ALWAYS AS
76+
(json_extract(data, '$.execution.error_message')) STORED,
77+
git_branch TEXT GENERATED ALWAYS AS
78+
(json_extract(data, '$.metadata.git.branch')) STORED
79+
);
80+
```
81+
82+
**Indexes**
83+
- Single column indexes on core fields
84+
- Composite index for time-based queries: `(hook_type, timestamp DESC)`
85+
- Conditional indexes on generated columns
86+
87+
### 4. Logging Flow
88+
89+
```
90+
1. Hook Triggered → execute.py
91+
2. ConfigService loads eyelet.yaml settings
92+
3. HookLogger determines format (json/sqlite/both) and scope (project/global/both)
93+
4. For SQLite:
94+
- GitMetadata enriches log data
95+
- SQLiteLogger writes with retry logic
96+
- ProcessLocalConnection handles concurrency
97+
5. For JSON:
98+
- Traditional file-based logging to eyelet-hooks/
99+
```
100+
101+
### 5. Query System (`query_service.py`)
102+
103+
**Flexible Filtering**
104+
- By hook type, tool name, session ID
105+
- Time-based (since/until)
106+
- By status, git branch, error presence
107+
- Full-text search across JSON data
108+
109+
**CLI Commands**
110+
```bash
111+
# Search with filters
112+
eyelet query search --hook-type PreToolUse --tool Bash --since 1h
113+
114+
# Summary statistics
115+
eyelet query summary --since 24h
116+
117+
# Recent errors
118+
eyelet query errors --limit 10
119+
120+
# Session timeline
121+
eyelet query session <session-id>
122+
123+
# Search in data
124+
eyelet query grep "git push"
125+
```
126+
127+
### 6. Configuration (`eyelet.yaml`)
128+
129+
**Global Configuration** (`~/.claude/eyelet.yaml`)
130+
```yaml
131+
logging:
132+
format: json # json, sqlite, or both
133+
enabled: true
134+
scope: project # project, global, or both
135+
global_path: ~/.claude/eyelet-logging
136+
project_path: .eyelet-logging
137+
add_to_gitignore: true
138+
139+
metadata:
140+
include_hostname: true
141+
include_ip: true
142+
custom_fields:
143+
team: engineering
144+
```
145+
146+
**Project Override** (`./eyelet.yaml`)
147+
```yaml
148+
logging:
149+
format: sqlite # Override to use SQLite
150+
scope: both # Log to both locations
151+
```
152+
153+
## Performance Characteristics
154+
155+
### Write Performance
156+
- Single insert: ~5-10ms with retry logic
157+
- Batch insert: 100-200 inserts/second
158+
- Concurrent writes: Handled via WAL mode + retry
159+
- Lock contention: Mitigated by exponential backoff
160+
161+
### Query Performance
162+
- Indexed queries: <10ms for most operations
163+
- Full-text search: Linear scan (optimizable with FTS5)
164+
- Summary statistics: ~50ms for 10,000 records
165+
166+
### Storage
167+
- Overhead: ~1KB per log entry
168+
- Compression: Via BLOB storage for JSON
169+
- Growth rate: ~100MB per 100,000 hooks
170+
171+
## Migration Strategy
172+
173+
Schema migrations use SQLite's `PRAGMA user_version`:
174+
175+
```python
176+
MIGRATIONS = [
177+
(1, "Initial schema", "..."),
178+
(2, "Add user_id column", "ALTER TABLE hooks ADD COLUMN user_id TEXT;")
179+
]
180+
181+
def migrate_database(conn):
182+
current_version = conn.execute("PRAGMA user_version").fetchone()[0]
183+
for version, description, sql in MIGRATIONS:
184+
if version > current_version:
185+
conn.executescript(sql)
186+
conn.execute(f"PRAGMA user_version = {version}")
187+
```
188+
189+
## Monitoring & Health
190+
191+
**Database Health Checks**
192+
- Integrity verification
193+
- WAL file size monitoring
194+
- Table statistics and growth tracking
195+
- Index usage analysis
196+
197+
**Performance Monitoring**
198+
- Insert/query timing statistics
199+
- Retry counts and lock wait times
200+
- Success rates and error tracking
201+
202+
## Security Considerations
203+
204+
1. **Git Credentials**: Sanitized from remote URLs
205+
2. **Environment Variables**: Only CLAUDE*/EYELET* vars logged
206+
3. **File Permissions**: Database created with user-only access
207+
4. **SQL Injection**: Parameterized queries throughout
208+
209+
## Future Enhancements
210+
211+
1. **Full-Text Search**: Add FTS5 for better search performance
212+
2. **Compression**: zlib compression for older entries
213+
3. **Partitioning**: Time-based partitioning for large databases
214+
4. **Replication**: Optional sync to remote database
215+
5. **Analytics**: Pre-computed statistics tables
216+
6. **Retention**: Automatic cleanup of old entries

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ dependencies = [
2626
"jsonschema>=4.20.0",
2727
"sqlalchemy>=2.0.23",
2828
"claude-code-sdk>=0.0.17",
29+
"pyyaml>=6.0.1",
2930
]
3031

3132
[project.optional-dependencies]

0 commit comments

Comments
 (0)