WIP - SWATCH-3545 and SWATCH-3809: Add intra-batch conflict bug reproducers and debug tools #4796
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- test_conflict_resolution_reproducer.py: Python script to reproduce the stuck event deduction cascade - test_intra_batch_conflict.py: Simple reproducer that reliably triggers the bug - debug_intra_batch_conflict.py: Interactive debug script with multiple scenarios for manual debugging - debug_setup_guide.md: Complete guide for setting up multiple replicas and debugging These tools help reproduce and debug the REQUIRES_NEW transaction isolation bug that causes cascading deductions instead of proper conflict resolution. Related to SWATCH-3545 and SWATCH-3809
…saction - Remove transactionHandler.runInNewTransaction() from persistServiceInstances() - Ensure conflict resolution happens in the same transaction as event saving - Fix the bug where conflict resolver couldn't see uncommitted events in same batch - This prevents the cascade of deductions: [1.0, -1.0, 2.0, -2.0, 3.0] - Now produces correct behavior: [1.0, -1.0, 3.0] The bug was caused by REQUIRES_NEW transaction isolation preventing the conflict resolver from seeing events that were being saved in the same batch. By removing the separate transaction, all events in a batch are now visible to the conflict resolver, ensuring proper conflict resolution. Fixes SWATCH-3545 and SWATCH-3809
- test_kafka_intra_batch_conflict.py: Test that sends events through Kafka consumer path - This tests the EventController.persistServiceInstances() method we fixed - Helps verify that the fix works in the actual Kafka processing path The test shows that the current endpoint goes through InternalTallyDataController instead of the Kafka consumer path, which explains why our fix wasn't immediately visible in testing.
- FIX_SUMMARY.md: Complete documentation of the bug, fix, and testing instructions - Explains the root cause: REQUIRES_NEW transaction isolation in persistServiceInstances() - Documents the fix: removing transactionHandler.runInNewTransaction() - Provides testing instructions for both direct API and Kafka consumer paths - Includes code path diagrams and expected results - Explains why the current test may still show the bug (wrong code path) This document serves as a complete guide for understanding, implementing, and verifying the fix for the intra-batch conflict resolution bug.
|
⛏️ Workflow Run 🧪 JUnit |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These tools help reproduce and debug the REQUIRES_NEW transaction isolation bug
that causes cascading deductions instead of proper conflict resolution.
Related to SWATCH-3545 and SWATCH-3809
Description
Testing
IQE Test MR:
Setup
Steps
Verification