Stream Reactor 8.1.19
·
1534 commits
to master
since this release
Fix: Prevent ElasticSearch from Skipping Records After Tombstone (#172) * Fix: Prevent ElasticSearch from Skipping Records After Tombstone Overview This pull request addresses a critical bug in ElasticSearch versions 6 (ES6) and 7 (ES7) where records following a tombstone are inadvertently skipped during the insertion process. The issue stemmed from an erroneous return statement that halted the processing of subsequent records. Background In the current implementation, when a tombstone record is encountered within a sequence of records to be written to ElasticSearch, the insertion process prematurely exits due to a return instruction. This results in all records following the tombstone being ignored, leading to incomplete data ingestion and potential inconsistencies within the ElasticSearch indices. Changes Made Refactored Insert Method: Modularization: The original insert method has been decomposed into smaller, more focused functions. This enhances code readability, maintainability, and facilitates easier testing. Detailed Log Entries: Added log statements at key points within the insertion workflow ES Error not handled: Previously the response from ElasticSearch ignored failures. With this change, if any of the batch fail, the sink will raise an exception. * Avoid sending empty requests * Fix the unit tests --------- Co-authored-by: stheppi <[email protected]>