ETL Pipeline基盤整備: 設定ローダー(config_loader.py)実装

[yu23ki14]

ETL Pipeline基盤整備: 設定ローダー実装

関連Issue: #154 (ETLの柔軟性向上)

概要

設定駆動型ETLパイプラインの実現のため、YAML/JSON設定ファイルを読み込み、パイプライン設定オブジェクトに変換する設定ローダーを実装します。

実装内容

1. 設定ローダー実装

• pipeline/config/config_loader.py: メイン実装
• pipeline/config/models.py: 設定データクラス定義
• YAML/JSON両対応の設定ファイル読み込み

2. 設定データ構造定義

@dataclass
class ETLConfig:
    pipeline_name: str
    components: List[ComponentConfig]
    filters: List[FilterConfig]
    ai_settings: AIConfig
    extraction_settings: ExtractionConfig

3. 設定ファイルフォーマット例

pipeline_name: "language_first_pattern"
components:
  - name: "note_extractor"
    type: "NoteExtractorComponent"
  - name: "note_language_filter"
    type: "LanguageFilterComponent"

技術要件

PyYAML使用
型安全な設定読み込み
バリデーション機能
デフォルト設定対応

受け入れ条件

• 設定ローダーが実装されている
• YAML/JSON両方の形式に対応
• 設定バリデーション機能がある
• エラーハンドリングが適切
• 単体テストが実装されている

[yu23ki14]

@claude
Start work on this

[claude]

Claude finished @yu23ki14's task in 6m 20s —— View job • claude/issue-156-20250611_041228 • Create PR ➔

---

ETL Configuration Loader Implementation ✅

I've successfully implemented the configuration loader for the ETL pipeline as requested in issue #156.

Todo List

• Analyze existing ETL codebase structure
• Check dependencies and project setup
• Add PyYAML dependency to pyproject.toml
• Implement configuration data models (pipeline/config/models.py)
• Implement configuration loader (pipeline/config/config_loader.py)
• Add comprehensive unit tests
• Run linting and type checking (unable to run - no bash permissions)
• Commit changes and provide PR link

Implementation Summary

✅ Configuration Models (models.py):

• ETLConfig: Main pipeline configuration with validation
• ComponentConfig: Individual component configuration
• FilterConfig: Filter configuration
• AIConfig: AI model settings with validation
• ExtractionConfig: Data extraction settings with validation

✅ Configuration Loader (config_loader.py):

• Support for both YAML and JSON formats
• Auto-format detection for unknown file extensions
• Comprehensive validation and error handling
• Save/load functionality with type safety
• Default configuration creation
• Sample configuration generation

✅ Comprehensive Tests:

• test_config_models.py: 25+ test cases for all configuration models
• test_config_loader.py: 20+ test cases for configuration loader
• Coverage includes validation, error handling, file I/O, and edge cases

Example Usage

from birdxplorer_etl.pipeline.config import load_config, create_sample_config

# Load configuration from file
config = load_config("pipeline_config.yaml")

# Create sample configuration
create_sample_config("sample.yaml", format="yaml")

# Access configuration
for component in config.get_enabled_components():
    print(f"Component: {component.name} ({component.type})")

Next Steps

Code Quality Check: Please run the following commands from the etl/ directory to verify code quality:

cd etl
black src/birdxplorer_etl/pipeline/config/ tests/test_config_*.py
isort src/birdxplorer_etl/pipeline/config/ tests/test_config_*.py
pflake8 src/birdxplorer_etl/pipeline/config/ tests/test_config_*.py
mypy src/birdxplorer_etl/pipeline/config/ --strict
pytest tests/test_config_*.py -v

---