feat: add daily rating count recalculation to ETL pipeline#227
Merged
ayuki-joto merged 6 commits intomainfrom Mar 9, 2026
Merged
feat: add daily rating count recalculation to ETL pipeline#227ayuki-joto merged 6 commits intomainfrom
ayuki-joto merged 6 commits intomainfrom
Conversation
…d readability - Standardized exception message formatting by collapsing multiline strings to single lines. - Updated dictionary reformatting for cleaner structure and readability.
Add recalculate_rating_counts() that aggregates rating data from row_note_ratings and bulk-updates notes table columns (rate_count, helpful_count, somewhat_helpful_count, not_helpful_count) after the daily ratings extraction phase. Closes #220. Co-Authored-By: Claude Opus 4.6 <[email protected]>
The ETL test job only installed etl[dev] dependencies, but birdxplorer_common is a prod-only dependency. Tests that import extract_ecs.py (which uses birdxplorer_common.storage) failed with ModuleNotFoundError in CI. This matches what tox.ini already does with `-e ../common`. Co-Authored-By: Claude Opus 4.6 <[email protected]>
…rmance and maintainability - Moved validation logic for rating rows to a dedicated `_validate_rating_row` function. - Refactored `_process_rating_rows` to perform bulk loading using COPY for improved efficiency. - Introduced staging table with deduplication and atomic swap to ensure data integrity during ratings reload. - Added helper functions for staging table creation, deduplication, swap, and cleanup.
- Resolved potential naming conflicts by dynamically querying and renaming PK indexes during staging table swaps. - Ensured accurate normalization of PK index names for both old and new tables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
recalculate_rating_counts()関数がrow_note_ratingsテーブルから集計し、notesテーブルのrate_count,helpful_count,somewhat_helpful_count,not_helpful_countをバルク更新Changes
etl/src/birdxplorer_etl/extract_ecs.py:recalculate_rating_counts()関数の追加とextract_data()への統合etl/tests/test_recalculate_rating_counts.py: 4件のユニットテスト（正常系、0行更新、ログ出力、エラー伝播）etl/tests/test_postlookup_lambda.py: Blackフォーマッタによる整形のみTest plan
🤖 Generated with Claude Code