Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5998c26
build: matching Python versions with pytorch/pytorch:2.8.0-cuda12.8-c…
deanq Oct 16, 2025
48d3e6c
chore: vscode type-hint helpers
deanq Oct 16, 2025
020412d
refactor: handler --> live_serverless.handler
deanq Oct 16, 2025
3756420
build: `make setup` where it applies
deanq Oct 16, 2025
9780147
refactor: remote_execution --> live_serverless.remote_execution
deanq Oct 16, 2025
96ef6ed
refactor: remote_executor --> live_serverless.remote_executor
deanq Oct 16, 2025
ed1bc90
refactor: dependency_installer --> live_serverless.dependency_installer
deanq Oct 16, 2025
93848d8
refactor: cache_sync_manager --> live_serverless.cache_sync_manager
deanq Oct 16, 2025
40c8543
refactor: subprocess_utils --> live_serverless.subprocess_utils
deanq Oct 16, 2025
ae0b7ee
refactor: constants --> live_serverless.constants
deanq Oct 16, 2025
7fde6d0
refactor: log_streamer --> live_serverless.log_streamer
deanq Oct 16, 2025
5e9b01c
refactor: class_executor --> live_serverless.class_executor
deanq Oct 16, 2025
6c5cbc3
refactor: function_executor --> live_serverless.function_executor
deanq Oct 16, 2025
d37fb14
refactor: serialization_utils --> live_serverless.serialization_utils
deanq Oct 16, 2025
edbd96e
refactor: logger --> live_serverless.logger
deanq Oct 16, 2025
5d021ae
refactor: relative imports from live_serverless module
deanq Oct 16, 2025
8c0934e
refactor: setup_logging in live_serverless handler
deanq Oct 16, 2025
5beb799
chore: ruff format
deanq Oct 16, 2025
1543c06
docs: updated to reflect the live_serverless module
deanq Oct 16, 2025
398d625
feat: add dynamic handler loading via env var `HANDLER_MODULE`
deanq Oct 16, 2025
0a9a00c
feat: add dynamic handler loading and event protocol
deanq Oct 16, 2025
cdcc940
docs: Dynamic Handler Loading architecture
deanq Nov 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 5 additions & 20 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build CPU Docker image
uses: docker/build-push-action@v6
Expand Down Expand Up @@ -186,10 +183,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build and push GPU Docker image (main)
uses: docker/build-push-action@v6
Expand Down Expand Up @@ -239,10 +233,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build and push CPU Docker image (main)
uses: docker/build-push-action@v6
Expand Down Expand Up @@ -302,10 +293,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build and push GPU Docker image (prod)
uses: docker/build-push-action@v6
Expand Down Expand Up @@ -365,10 +353,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build and push CPU Docker image (prod)
uses: docker/build-push-action@v6
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,7 @@ jobs:
enable-cache: true

- name: Setup dependencies
run: |
uv sync
git submodule update
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
run: make setup

- name: Build and push GPU Docker image
uses: docker/build-push-action@v6
Expand Down
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.12
3.11
3 changes: 2 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"python.testing.pytestEnabled": true,
"python.envFile": "${workspaceFolder}/.env",
"python.analysis.extraPaths": [
"${workspaceFolder}/src"
"${workspaceFolder}/src",
"${workspaceFolder}/tetra-rp/src",
]
}
60 changes: 35 additions & 25 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,36 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

This is `worker-tetra`, a RunPod Serverless worker template that provides dynamic GPU provisioning for ML workloads with transparent execution and persistent workspace management. The project consists of two main components:

1. **RunPod Worker Handler** (`src/handler.py`) - A serverless function that executes remote Python functions with dependency management and workspace support
2. **Tetra SDK** (`tetra-rp/` submodule) - Python library for distributed inference and serving of ML models
1. **RunPod Worker Handler** (`src/handler.py`) - A lightweight RunPod wrapper that imports and starts the serverless handler from the `live_serverless` module
2. **Live Serverless Module** (`src/live_serverless/`) - Core implementation that executes remote Python functions with dependency management and workspace support
3. **Tetra SDK** (`tetra-rp/` submodule) - Python library for distributed inference and serving of ML models

## Key Areas of Responsibility

### 1. Remote Function Execution Engine (`src/`)
- **Core Handler** (`src/handler.py:18`): Main RunPod serverless entry point that orchestrates remote execution
- **Remote Executor** (`src/remote_executor.py:11`): Central orchestrator that coordinates all execution components using composition pattern
- **Function Executor** (`src/function_executor.py:12`): Handles individual function execution with full output capture (stdout, stderr, logs)
- **Class Executor** (`src/class_executor.py:14`): Manages class instantiation and method execution with instance persistence and metadata tracking
### 1. Remote Function Execution Engine (`src/live_serverless/`)
- **Core Handler** (`src/live_serverless/__init__.py`): Main serverless handler that orchestrates remote execution
- **Remote Executor** (`src/live_serverless/remote_executor.py`): Central orchestrator that coordinates all execution components using composition pattern
- **Function Executor** (`src/live_serverless/function_executor.py`): Handles individual function execution with full output capture (stdout, stderr, logs)
- **Class Executor** (`src/live_serverless/class_executor.py`): Manages class instantiation and method execution with instance persistence and metadata tracking

### 2. Dependency Management System (`src/dependency_installer.py:14`)
### 2. Dependency Management System (`src/live_serverless/dependency_installer.py`)
- **Python Package Installation**: UV-based package management with environment-aware configuration (Docker vs local)
- **System Package Installation**: APT/Nala-based system dependency handling with acceleration support
- **Differential Installation**: Optimized package installation that skips already-installed packages
- **Environment Detection**: Automatic Docker vs local environment detection for appropriate installation methods
- **System Package Filtering**: Intelligent detection of system-available packages to avoid redundant installation
- **Universal Subprocess Integration**: All subprocess operations use centralized logging utility

### 3. Universal Subprocess Utility (`src/subprocess_utils.py`)
### 3. Universal Subprocess Utility (`src/live_serverless/subprocess_utils.py`)
- **Centralized Subprocess Operations**: All subprocess calls use `run_logged_subprocess` for consistency
- **Automatic Logging Integration**: All subprocess output flows through log streamer at DEBUG level
- **Environment-Aware Execution**: Handles Docker vs local environment differences automatically
- **Standardized Error Handling**: Consistent FunctionResponse pattern for all subprocess operations
- **Timeout Management**: Configurable timeouts with proper cleanup on timeout/cancellation

### 4. Serialization & Protocol Management
- **Protocol Definitions** (`src/remote_execution.py:13`): Pydantic models for request/response with validation
- **Serialization Utils** (`src/serialization_utils.py`): CloudPickle-based data serialization for function arguments and results
- **Base Executor** (`src/base_executor.py`): Common execution interface and environment setup
- **Protocol Definitions** (`src/live_serverless/remote_execution.py`): Pydantic models for request/response with validation
- **Serialization Utils** (`src/live_serverless/serialization_utils.py`): CloudPickle-based data serialization for function arguments and results

### 5. Tetra SDK Integration (`tetra-rp/` submodule)
- **Client Interface**: `@remote` decorator for marking functions for remote execution
Expand All @@ -58,20 +58,25 @@ This is `worker-tetra`, a RunPod Serverless worker template that provides dynami
- **Release Management**: Automated semantic versioning and Docker Hub deployment

### 8. Configuration & Constants
- **Constants** (`src/constants.py`): System-wide configuration values (NAMESPACE, LARGE_SYSTEM_PACKAGES)
- **Constants** (`src/live_serverless/constants.py`): System-wide configuration values (NAMESPACE, LARGE_SYSTEM_PACKAGES)
- **Environment Configuration**: RunPod API integration

## Architecture

### Core Components

- **`src/handler.py`**: Main RunPod serverless handler implementing composition pattern
- **`src/handler.py`**: Lightweight RunPod wrapper that starts the serverless handler
- Imports and initializes the handler from `live_serverless` module
- Entry point for RunPod Serverless execution

- **`src/live_serverless/`**: Core serverless implementation module
- `__init__.py`: Main handler implementing composition pattern
- Executes arbitrary Python functions remotely with workspace support
- Handles dynamic installation of Python and system dependencies with differential updates
- Serializes/deserializes function arguments and results using cloudpickle
- Captures stdout, stderr, and logs from remote execution

- **`src/remote_execution.py`**: Protocol definitions using Pydantic models
- **`src/live_serverless/remote_execution.py`**: Protocol definitions using Pydantic models
- `FunctionRequest`: Defines function execution requests with dependencies
- `FunctionResponse`: Standardized response format with success/error handling

Expand Down Expand Up @@ -188,16 +193,21 @@ gpu_config = LiveServerless(
## File Structure Highlights

```
├── src/ # Core implementation
│ ├── handler.py # Main serverless function handler
│ ├── remote_executor.py # Central execution orchestrator
│ ├── remote_execution.py # Protocol definitions
│ ├── function_executor.py # Function execution with output capture
│ ├── class_executor.py # Class execution with persistence
│ ├── dependency_installer.py # Python and system dependency management
│ ├── serialization_utils.py # CloudPickle serialization utilities
│ ├── base_executor.py # Common execution interface
│ ├── constants.py # System-wide configuration constants
├── src/ # Source directory
│ ├── handler.py # RunPod wrapper (entry point)
│ ├── live_serverless/ # Core serverless implementation module
│ │ ├── __init__.py # Main handler with composition pattern
│ │ ├── remote_executor.py # Central execution orchestrator
│ │ ├── remote_execution.py # Protocol definitions
│ │ ├── function_executor.py # Function execution with output capture
│ │ ├── class_executor.py # Class execution with persistence
│ │ ├── dependency_installer.py # Python and system dependency management
│ │ ├── subprocess_utils.py # Centralized subprocess operations
│ │ ├── log_streamer.py # Log streaming system
│ │ ├── logger.py # Unified logging configuration
│ │ ├── serialization_utils.py # CloudPickle serialization utilities
│ │ ├── cache_sync_manager.py # Workspace sync management
│ │ └── constants.py # System-wide configuration constants
│ └── tests/ # Handler test JSON files
├── tests/ # Comprehensive test suite
│ ├── conftest.py # Shared test fixtures
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile-cpu
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.12-slim
FROM python:3.11-slim

WORKDIR /app

Expand Down
5 changes: 3 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,15 @@ clean: # Remove build artifacts and cache files
find . -type f -name "*.pyc" -delete
find . -type f -name "*.pkl" -delete

setup: dev # Initialize project, sync deps, update submodules
setup: # Initialize project, sync deps, update submodules
uv sync
@if [ ! -f "tetra-rp/.git" ]; then \
git submodule update --init --recursive; \
fi
make protocols

protocols: # Copy remote_execution protocol from submodule
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/
cp tetra-rp/src/tetra_rp/protos/remote_execution.py src/live_serverless/

build: # Build both GPU and CPU Docker images
make build-gpu
Expand Down
14 changes: 7 additions & 7 deletions docs/Centralized_Log_Streaming_System.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This design implements a comprehensive log streaming architecture that captures

## Key Components

### 1. LogStreamer (`src/log_streamer.py`)
### 1. LogStreamer (`src/live_serverless/log_streamer.py`)
Thread-safe log capture system that buffers logs and streams them to the response output.

```mermaid
Expand All @@ -21,7 +21,7 @@ graph TB
H[System Operations] --> B
```

### 2. Centralized Logging (`src/logger.py`)
### 2. Centralized Logging (`src/live_serverless/logger.py`)
Unified logging configuration with:
- **Debug Format**: `timestamp | level | name | file:line | message`
- **Production Format**: `timestamp | level | message`
Expand Down Expand Up @@ -67,11 +67,11 @@ sequenceDiagram

| Component | Purpose |
|-----------|---------|
| `src/log_streamer.py` | New centralized log streaming system |
| `src/logger.py` | New unified logging configuration |
| `src/remote_executor.py` | Integration with log streaming |
| `src/dependency_installer.py` | Log capture during installation |
| `src/test_log_streaming.json` | Test case for log visibility |
| `src/live_serverless/log_streamer.py` | New centralized log streaming system |
| `src/live_serverless/logger.py` | New unified logging configuration |
| `src/live_serverless/remote_executor.py` | Integration with log streaming |
| `src/live_serverless/dependency_installer.py` | Log capture during installation |
| `src/tests/test_log_streaming.json` | Test case for log visibility |

## Benefits

Expand Down
Loading
Loading