Skip to content

feat: add SnapshotAgent for HTTP request recording and playback #4270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

mcollina
Copy link
Member

@mcollina mcollina commented Jun 8, 2025

Summary

This PR implements a new SnapshotAgent that extends MockAgent to provide automatic recording and playback of HTTP requests for testing purposes. This addresses the feature request in #4114 for "record and play back requests" functionality.

Features

  • Record Mode: Captures real HTTP requests and responses to snapshot files
  • Playback Mode: Replays recorded interactions without making real network calls
  • Update Mode: Uses existing snapshots when available, records new ones when missing
  • Full undici compatibility: Works with request(), fetch(), stream(), pipeline(), etc.
  • Base64 response storage: Consistent serialization of all response body types
  • TypeScript support: Complete type definitions and comprehensive tests

API Usage

Recording Real Requests

import { SnapshotAgent, setGlobalDispatcher } from 'undici'

const agent = new SnapshotAgent({ 
  mode: 'record',
  snapshotPath: './test/snapshots/api-calls.json'
})
setGlobalDispatcher(agent)

// Makes real requests and records them
await fetch('https://api.example.com/users')
await agent.saveSnapshots()

Replaying in Tests

const agent = new SnapshotAgent({
  mode: 'playback',
  snapshotPath: './test/snapshots/api-calls.json'
})
setGlobalDispatcher(agent)

// Uses recorded response instead of real request
const response = await fetch('https://api.example.com/users')

Test plan

  • Comprehensive unit tests for SnapshotRecorder utility class
  • Integration tests for SnapshotAgent in all three modes (record/playback/update)
  • TypeScript definition tests using tsd
  • Example code with real API integration patterns
  • POST request handling with request bodies
  • Error handling for missing snapshots
  • File format validation and persistence

🤖 Generated with Claude Code

Implements a new SnapshotAgent that extends MockAgent to provide
automatic recording and playback of HTTP requests for testing.

Features:
- Record mode: Captures real HTTP requests and responses
- Playback mode: Replays recorded interactions without network calls
- Update mode: Uses existing snapshots, records new ones when missing
- Full undici compatibility: Works with request(), fetch(), stream(), etc.
- Base64 response storage for consistent serialization
- Comprehensive TypeScript definitions and tests

Resolves: #4114

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
@mcollina mcollina requested review from Uzlopak and metcoder95 June 8, 2025 22:31
@mcollina
Copy link
Member Author

mcollina commented Jun 8, 2025

cc @GeoffreyBooth let me know what you think

@GeoffreyBooth
Copy link
Member

This is a great addition, thank you for doing this! A few things come to mind reading the documentation:

  • How can the matching be customized? For example, to match on some headers but not all, to ignore authentication tokens for example. In my app I had subsequent POST calls to the same API where everything matched except the body, so I needed to include that as part of the matching criteria.
  • How are subsequent identical calls handled? One feature of nock is that you can define responses like “the first time, return X, the second time, return Y”.
  • How can I update existing mocks? I see the “update mode” example but that seems to be an additive operation to add new mocks to an existing set; is there a way to fully replace an existing set?

Maybe some of these are appropriate to handle as follow-up enhancements. I’ll try to get something together soon that’s a minimal reproduction of what I’ve been working with.

// Record mode - make real request and save response
return this._recordAndReplay(opts, handler)
} else {
throw new InvalidArgumentError(`Invalid snapshot mode: ${mode}. Must be 'record', 'playback', or 'update'`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we add this to the constructor as part of the opts validation?

/**
* Sets up MockAgent interceptors based on recorded snapshots
*/
_setupMockInterceptors () {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's for exactly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This configures the mockPools.

trailers: response.trailers
}

this.snapshots.set(hash, {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be interesting to allow customize the max number of items (as enhancement), just to avoid go out of bounds

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can play with some sort of automatic flush that saves it automatically into disk if configured to

@metcoder95
Copy link
Member

How can the matching be customized? For example, to match on some headers but not all, to ignore authentication tokens for example. In my app I had subsequent POST calls to the same API where everything matched except the body, so I needed to include that as part of the matching criteria.

This can be a pretty interesting enhancement. Especially for request that requires some level of security, often those ones are good candidates to be excluded from the snapshot itself.

Having a way to state wether or not record a given request/response (either at the dispatch level of agent instantiation) will bring these control benefits.

From my end (and as possible enhancement) I'd like to explore the integration with the Mocks feature. As replaying these request under testing environments can be beneficial, especially if custom mocks are already set. I don't believe they should be mutually exclusive but compatible between each other.


// Handle array format (undici internal format: [name, value, name, value, ...])
if (Array.isArray(headers)) {
for (let i = 0; i < headers.length; i += 2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to make sure the length of the headers array is even

mcollina and others added 11 commits July 19, 2025 16:24
Addresses code review feedback from PR #4270 with the following improvements:

- Add constructor options validation for mode and snapshotPath parameters
- Implement memory management with maxSnapshots and LRU eviction
- Add auto-flush functionality with configurable intervals
- Fix header normalization to handle Buffer objects from undici
- Fix error handling to use proper undici dispatcher pattern
- Add comprehensive JSDoc documentation for _setupMockInterceptors
- Add extensive test coverage for all new features

All tests are now passing and the implementation maintains backward compatibility.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Mark all Phase 1 tasks as completed and add current status summary
showing successful implementation of code review fixes.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Adds powerful customizable request matching capabilities:

**Core Features:**
- matchHeaders: Only match on specific headers (e.g., content-type only)
- ignoreHeaders: Ignore certain headers for matching (e.g., auth tokens)
- excludeHeaders: Don't store sensitive headers in snapshots
- matchQuery: Control whether query parameters are included in matching
- matchBody: Control whether request body is included in matching
- caseSensitive: Optional case-sensitive header matching

**Security Enhancements:**
- Sensitive headers (authorization, set-cookie) can be excluded from snapshots
- Headers can be filtered during both matching and storage
- Separate filtering for matching vs storage allows fine-grained control

**Test Coverage:**
- 5 new integration tests covering all matching scenarios
- 4 new unit tests for filtering functions
- All existing tests continue to pass (18 total tests)

This addresses the major feedback from GeoffreyBooth about customizable
matching and metcoder95's concerns about security filtering.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
This commit implements comprehensive sequential response support and snapshot
management capabilities for the SnapshotAgent, addressing GeoffreyBooth's
feedback about "first call returns X, second call returns Y" functionality.

## Key Features Added

### Sequential Response Support
- Modified storage format to support response arrays instead of single responses
- Updated findSnapshot() to return appropriate response based on call count
- Automatic progression through response sequence on subsequent calls
- Legacy format compatibility for existing snapshots

### Snapshot Management
- Added resetCallCounts() method for test cleanup
- Added deleteSnapshot() for selective snapshot removal
- Added getSnapshotInfo() for snapshot inspection/debugging
- Added replaceSnapshots() for full snapshot set replacement

### Enhanced SnapshotAgent API
- Exposed all new recorder methods through SnapshotAgent
- Updated _setupMockInterceptors to handle both new and legacy formats
- Maintained backward compatibility with existing snapshot files

## Test Coverage
- Added 3 comprehensive integration tests (30 total tests now passing)
- Sequential response playback test with 4 sequential calls
- Call count reset functionality test
- Snapshot management methods test with CRUD operations

## Technical Details
- Response arrays: `{responses: [res1, res2, res3], callCount: 0}`
- Call count tracking with automatic increment on findSnapshot()
- Last response repetition after sequence exhaustion
- Proper hash-based snapshot identification for management operations

Addresses PR #4270 feedback - Phase 3 complete.
All 30 tests passing, maintaining full backward compatibility.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
All major feedback from PR #4270 has been successfully addressed:

✅ GeoffreyBooth's Requirements (100% Complete):
- Custom request matching with selective header filtering
- Sequential response support for multiple calls
- Snapshot replacement functionality

✅ metcoder95's Code Review (100% Complete):
- Constructor options validation
- _setupMockInterceptors documentation
- Memory management with LRU eviction
- Auto-flush functionality

The SnapshotAgent implementation is now feature-complete with 30
comprehensive tests and full backward compatibility.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
- Updated SnapshotRecorder interface with all Phase 1-3 methods and options
- Updated SnapshotAgent interface with new configuration options
- Added comprehensive tsd tests covering all new functionality
- Added types for sequential response support (responses array)
- Added SnapshotInfo and SnapshotData interfaces
- Added types for all new configuration options (Phase 1-3)
- Tested inheritance from MockAgent with new options
- Comprehensive type checking for all new methods and interfaces

All TypeScript tests passing with full type safety coverage.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Add advanced request filtering capabilities and finalize implementation:

**Phase 4 Features:**
- Request filtering callbacks (shouldRecord, shouldPlayback)
- URL pattern-based exclusion (string and regex patterns)
- Advanced filtering scenarios with comprehensive test coverage
- Experimental warnings for proper feature lifecycle management

**Enhanced Documentation:**
- Complete PR description with all features and examples
- Updated TypeScript definitions for new filtering options
- Comprehensive tsd tests for Phase 4 functionality

**Testing:**
- 5 new integration tests for filtering scenarios
- All 35 tests passing (22 existing + 5 new + 8 from other phases)
- Full TypeScript test coverage maintained

**Implementation Status:**
✅ All PR #4270 feedback addressed (100% complete)
✅ Optional Phase 4 enhancements implemented
✅ Production-ready with experimental warnings
✅ Zero breaking changes, full backward compatibility

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Fix race condition in sequential response support where call counts
were not being properly managed between recording and playback phases.

**Issue:**
- Test was failing intermittently with "First response" vs "Second response"
- Call count logic was incrementing before determining response index
- No explicit call count reset between recording and playback

**Solution:**
- Store current call count before incrementing in findSnapshot()
- Add explicit loadSnapshots() and resetCallCounts() in test
- Ensure deterministic call count state for sequential responses

**Result:**
- Test now passes consistently across multiple runs
- Sequential response functionality works reliably
- No impact on other test functionality

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Remove internal planning document as implementation is complete.
All requirements have been addressed and documented in PR_DESCRIPTION.md.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Matteo Collina <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants