Skip to content

Commit df7514a

Browse files
vijayvammiclaude
andauthored
feat: Loop node implementation (#247)
* docs: Add loop node design document Design for a new loop composite node that iterates until a break condition is met or max iterations is reached. Includes: - Do-while execution style - Parameter flow between iterations - Lineage tracking via separate branch logs per iteration - Argo compatibility via recursive templates Co-Authored-By: Claude Opus 4.5 <[email protected]> * docs: Update loop node design - branch must always be Pipeline Clarify that loop node branch parameter must always be a Pipeline, consistent with other composite nodes (parallel, map, conditional). Update SDK examples to wrap tasks in Pipeline objects. Co-Authored-By: Claude Opus 4.5 <[email protected]> * docs: added plan for loop node * feat: add LOOP_PLACEHOLDER constant Add loop placeholder constant following MAP_PLACEHOLDER pattern for iteration index replacement in loop node names * feat: refactor placeholder resolution for map and loop - Rename _resolve_map_placeholders to _resolve_iter_placeholders - Add support for LOOP_PLACEHOLDER resolution - Handle nested map and loop placeholders correctly - Keep old method for backward compatibility Co-Authored-By: Claude Sonnet 4 <[email protected]> * feat: add basic LoopNode class with placeholder support - Create LoopNode class following composite node patterns - Use LOOP_PLACEHOLDER for branch name generation - Add validation for break_on and index_as parameters - Implement _get_iteration_branch_name using refactored placeholder resolution Co-Authored-By: Claude Sonnet 4 <[email protected]> * feat: add loop node parameter and branch log management - Add get_break_condition_value() method with proper validation - Add _create_iteration_branch_log() using placeholder resolution - Add _build_iteration_iter_variable() for iteration context - Proper error handling for missing/invalid parameters - Add comprehensive tests for all new methods Co-Authored-By: Claude Sonnet 4 <[email protected]> * feat: implement LoopNode fan_out method - Create branch logs for current iteration - Copy parameters from parent (iteration 0) or previous iteration - Use proper placeholder resolution for branch names - Support both initial and subsequent iterations Co-Authored-By: Claude Sonnet 4 <[email protected]> * feat: implement LoopNode fan_in method - Check break condition and max iterations correctly - Roll back parameters to parent on loop exit - Set final step status based on branch execution - Extract current iteration from iter_variable properly Co-Authored-By: Claude Sonnet 4 <[email protected]> * feat: loop node, missing argo implementation --------- Co-authored-by: Claude Opus 4.5 <[email protected]>
1 parent 3a6bcb0 commit df7514a

File tree

21 files changed

+4047
-55
lines changed

21 files changed

+4047
-55
lines changed

docs/advanced-patterns/async-streaming.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
1-
# Async & Streaming Execution
1+
# 🧪 Async & Streaming Execution (Experimental)
22

3-
Execute async functions and stream results in real-time with runnable's native async capabilities.
3+
Execute async functions and stream results in real-time with runnable's experimental async capabilities.
44

5-
## The Core Insight
5+
!!! warning "Experimental Feature"
66

7-
**Runnable natively supports both batch and streaming workflows in a single framework**. While traditional orchestrators like Kedro, Metaflow, and Airflow excel at batch processing, runnable uniquely combines:
7+
**Async and streaming capabilities are experimental** and under active development. Features may change, and stability is not guaranteed for production use.
88

9-
- **Batch Processing**: Traditional data pipelines with full reproducibility and orchestration
10-
- **Real-Time Streaming**: Native AsyncGenerator support for LLM inference, APIs, and streaming data
9+
## Overview
1110

12-
**AsyncPipeline and AsyncPythonTask** bring this streaming capability natively to runnable, enabling real-time workflows while maintaining all of runnable's core features: parameter management, catalog system, reproducibility, and multi-environment execution.
11+
**Runnable provides experimental async and streaming support** for specialized use cases requiring real-time processing:
12+
13+
- **Batch Processing**: Core production-ready data pipelines with full reproducibility and orchestration
14+
- **Experimental Streaming**: AsyncGenerator support for LLM inference, APIs, and streaming data
15+
16+
**AsyncPipeline and AsyncPythonTask** are experimental features that enable streaming workflows while maintaining runnable's core features: parameter management, catalog system, reproducibility, and multi-environment execution.
1317

1418
!!! warning "Local Execution Only"
1519

0 commit comments

Comments
 (0)