|
| 1 | +# WARP.md |
| 2 | + |
| 3 | +This file provides guidance to WARP (warp.dev) when working with code in this repository. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +**acai_aws** is a DRY, configurable, declarative Python library for working with Amazon Web Services Lambdas. It provides: |
| 8 | +- Highly configurable API Gateway internal routing |
| 9 | +- OpenAPI schema adherence and generation |
| 10 | +- Extensible middleware for validation |
| 11 | +- Support for multiple AWS event sources (S3, DynamoDB, Kinesis, SNS, SQS, MSK, MQ, DocumentDB, Firehose) |
| 12 | +- Happy Path Programming (HPP) philosophy - inputs validated before operations |
| 13 | + |
| 14 | +## Common Commands |
| 15 | + |
| 16 | +### Development Environment |
| 17 | +```bash |
| 18 | +# Install dependencies (using pipenv) |
| 19 | +pipenv install |
| 20 | + |
| 21 | +# Install development dependencies |
| 22 | +pipenv install --dev |
| 23 | +``` |
| 24 | + |
| 25 | +### Testing |
| 26 | +```bash |
| 27 | +# Run all tests |
| 28 | +pipenv run test |
| 29 | + |
| 30 | +# Run with coverage report |
| 31 | +pipenv run coverage |
| 32 | +``` |
| 33 | + |
| 34 | +### Linting |
| 35 | +```bash |
| 36 | +# Run pylint with custom rules |
| 37 | +pipenv run lint |
| 38 | +``` |
| 39 | + |
| 40 | +### OpenAPI Generation |
| 41 | +```bash |
| 42 | +# Generate OpenAPI docs from handlers |
| 43 | +pipenv run generate |
| 44 | +# This generates OpenAPI documentation from handler files in tests/mocks/apigateway/openapi/**/*.py |
| 45 | +``` |
| 46 | + |
| 47 | +### Running Single Tests |
| 48 | +```bash |
| 49 | +# Run specific test file |
| 50 | +python -m unittest tests.acai_aws.<module>.<test_file> |
| 51 | + |
| 52 | +# Example: |
| 53 | +python -m unittest tests.acai_aws.apigateway.test_router |
| 54 | +``` |
| 55 | + |
| 56 | +## Architecture Overview |
| 57 | + |
| 58 | +### Core Philosophy: Happy Path Programming (HPP) |
| 59 | +All inputs are validated before being operated on. The library uses layers of customizable middleware to allow developers to dictate valid inputs without nested conditionals or try/catch blocks. |
| 60 | + |
| 61 | +### Module Structure |
| 62 | + |
| 63 | +The codebase is organized by AWS service event types: |
| 64 | + |
| 65 | +#### Event Handler Pattern |
| 66 | +All event handlers follow a common inheritance pattern from `acai_aws.base.event.BaseRecordsEvent`: |
| 67 | + |
| 68 | +``` |
| 69 | +BaseRecordsEvent (base/event.py) |
| 70 | +├── DynamoDBEvent (dynamodb/event.py) |
| 71 | +├── S3Event (s3/event.py) |
| 72 | +├── KinesisEvent (kinesis/event.py) |
| 73 | +├── SNSEvent (sns/event.py) |
| 74 | +├── SQSEvent (sqs/event.py) |
| 75 | +├── MSKEvent (msk/event.py) |
| 76 | +├── MQEvent (mq/event.py) |
| 77 | +├── DocumentDBEvent (documentdb/event.py) |
| 78 | +└── FirehoseEvent (firehose/event.py) |
| 79 | +``` |
| 80 | + |
| 81 | +Each event class: |
| 82 | +1. Inherits from `BaseRecordsEvent` |
| 83 | +2. Sets `self._record_class` to its specific Record class |
| 84 | +3. Implements `records` property that returns validated, parsed records |
| 85 | +4. Supports optional `data_class` for custom data transformations |
| 86 | + |
| 87 | +#### API Gateway Architecture |
| 88 | + |
| 89 | +The API Gateway module (`acai_aws/apigateway/`) provides a complete routing and validation system: |
| 90 | + |
| 91 | +**Key Components:** |
| 92 | +- `Router` - Main routing orchestrator with middleware pipeline |
| 93 | +- `Request` - Normalized request object with multi-format body parsing (JSON, XML, form, GraphQL) |
| 94 | +- `Response` - Response builder with error tracking |
| 95 | +- `Endpoint` - Wrapper around handler functions with requirements metadata |
| 96 | +- `Resolver` - Maps incoming requests to handler functions (supports directory and pattern-based routing) |
| 97 | +- `Validator` - Request/response validation against OpenAPI specs or inline schemas |
| 98 | + |
| 99 | +**Router Middleware Pipeline:** |
| 100 | +1. `before_all` - Pre-processing hook |
| 101 | +2. `with_auth` - Authentication/authorization |
| 102 | +3. Request validation (OpenAPI or requirements-based) |
| 103 | +4. Handler execution |
| 104 | +5. Response validation (optional) |
| 105 | +6. `after_all` - Post-processing hook |
| 106 | +7. Error handling (`on_error`, `on_timeout`) |
| 107 | + |
| 108 | +**Handler Requirements Decorator:** |
| 109 | +Handlers use the `@requirements()` decorator to declare validation rules: |
| 110 | + |
| 111 | +```python |
| 112 | +from acai_aws.apigateway.requirements import requirements |
| 113 | + |
| 114 | +@requirements( |
| 115 | + auth_required=True, |
| 116 | + required_headers=['x-api-key'], |
| 117 | + required_query=['user_id'], |
| 118 | + required_body={'$ref': '#/components/schemas/User'}, |
| 119 | + timeout=30, |
| 120 | + before=pre_hook, |
| 121 | + after=post_hook |
| 122 | +) |
| 123 | +def post(request, response): |
| 124 | + response.body = {'result': 'success'} |
| 125 | + return response |
| 126 | +``` |
| 127 | + |
| 128 | +#### Common Utilities |
| 129 | + |
| 130 | +**Validator** (`acai_aws/common/validator.py`): |
| 131 | +- Supports JSON Schema (Draft 7) and Pydantic model validation |
| 132 | +- Validates request headers, query params, and body |
| 133 | +- Validates response schemas |
| 134 | +- Used by both API Gateway and event record validation |
| 135 | + |
| 136 | +**Schema** (`acai_aws/common/schema.py`): |
| 137 | +- Manages OpenAPI spec loading and reference resolution |
| 138 | +- Extracts route specifications for validation |
| 139 | + |
| 140 | +**JSON Helper** (`acai_aws/common/json_helper.py`): |
| 141 | +- Handles JSON encoding/decoding with DynamoDB JSON format support |
| 142 | + |
| 143 | +### Testing Structure |
| 144 | + |
| 145 | +Tests mirror the source structure under `tests/acai_aws/`: |
| 146 | +- Each module has corresponding test files |
| 147 | +- Mocks are in `tests/mocks/` organized by service |
| 148 | +- Mock functions demonstrate handler patterns |
| 149 | + |
| 150 | +### OpenAPI Generation |
| 151 | + |
| 152 | +The `apigateway` module includes a CLI tool for generating OpenAPI documentation from handler code: |
| 153 | +- Scans handler files for decorated functions |
| 154 | +- Extracts requirements metadata |
| 155 | +- Generates/updates OpenAPI spec files in JSON/YAML |
| 156 | +- Invoked via: `python -m acai_aws.apigateway generate-openapi` |
| 157 | + |
| 158 | +## Code Style |
| 159 | + |
| 160 | +### Linting Rules (from .pylintrc) |
| 161 | +- Max line length: 140 characters |
| 162 | +- String quotes: single quotes for strings, double quotes for docstrings |
| 163 | +- Pylint score must be >= 10 to pass CI |
| 164 | +- Private attributes use double underscore: `self.__attribute` |
| 165 | +- Protected attributes use single underscore: `self._attribute` |
| 166 | + |
| 167 | +### Naming Conventions |
| 168 | +- Functions: `snake_case` (min 3 chars) |
| 169 | +- Variables: `snake_case` (min 3 chars) |
| 170 | +- Classes: `PascalCase` |
| 171 | +- Constants: `UPPER_SNAKE_CASE` |
| 172 | +- Methods: `snake_case` (min 3 chars) |
| 173 | + |
| 174 | +## CI/CD |
| 175 | + |
| 176 | +### CircleCI Pipeline |
| 177 | +- Python 3.9 environment |
| 178 | +- Automated testing with coverage reports |
| 179 | +- Pylint execution with score thresholds |
| 180 | +- SonarCloud integration for code quality |
| 181 | +- PyPI deployment on git tags |
| 182 | + |
| 183 | +### Deployment |
| 184 | +Package is deployed to PyPI automatically on tagged releases. Version is read from `CIRCLE_TAG` environment variable. |
| 185 | + |
| 186 | +## Working with Events |
| 187 | + |
| 188 | +### Record-based Event Handlers |
| 189 | +When creating handlers for AWS event sources: |
| 190 | + |
| 191 | +1. **Validation Options:** |
| 192 | + - `operations`: Filter by operation type (e.g., `['INSERT', 'MODIFY']` for DynamoDB) |
| 193 | + - `raise_operation_error`: Raise exception if operation doesn't match |
| 194 | + - `required_body`: JSON schema for record body validation |
| 195 | + - `raise_body_error`: Raise exception if body validation fails |
| 196 | + |
| 197 | +2. **Data Classes:** |
| 198 | + - Set `event.data_class = YourDataClass` to transform records |
| 199 | + - Data class receives `record` parameter in constructor |
| 200 | + - `event.records` returns list of data class instances |
| 201 | + |
| 202 | +3. **S3-Specific Features:** |
| 203 | + - `get_object=True`: Automatically fetch S3 object bodies |
| 204 | + - `data_type='json'|'csv'`: Parse object contents |
| 205 | + - `delimiter=','`: CSV delimiter (default: comma) |
| 206 | + |
| 207 | +### API Gateway Handlers |
| 208 | + |
| 209 | +1. **File-based Routing:** |
| 210 | + - Directory structure maps to routes: `handlers/user/_user_id.py` → `/user/{user_id}` |
| 211 | + - Pattern-based: `_param` becomes `{param}` path parameter |
| 212 | + - Method name matches HTTP method: `def get(request, response)` |
| 213 | + |
| 214 | +2. **Request Object:** |
| 215 | + - Auto-detects content type and parses body |
| 216 | + - Access via: `request.body`, `request.json`, `request.xml`, `request.form`, `request.graphql` |
| 217 | + - Path params: `request.path_params['user_id']` |
| 218 | + - Query params: `request.query_params['filter']` |
| 219 | + |
| 220 | +3. **Response Object:** |
| 221 | + - Set body: `response.body = {'data': 'value'}` |
| 222 | + - Set status: `response.code = 201` |
| 223 | + - Add errors: `response.set_error('field', 'error message')` |
| 224 | + - Check errors: `response.has_errors` |
| 225 | + |
| 226 | +## Dependencies |
| 227 | + |
| 228 | +Core dependencies: |
| 229 | +- `boto3` - AWS SDK |
| 230 | +- `jsonschema` - JSON Schema validation |
| 231 | +- `pydantic` - Data validation with type hints |
| 232 | +- `dynamodb_json` - DynamoDB JSON format handling |
| 233 | + |
| 234 | +Development dependencies: |
| 235 | +- `pytest` - Testing framework |
| 236 | +- `pylint` - Linting |
| 237 | +- `moto` - AWS service mocking |
0 commit comments