This repository was archived by the owner on Jul 19, 2023. It is now read-only.
Feature/per stream high water #92
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A few changes to resolve #74
The issue was that the existing underlying model stored the high water mark at the log group level, e.g.
However, in a situation where there were a lot of log events across multiple streams in the same log group, this could cause some events to disappear.
The ultimate cause here is that the
:interleaved => true
parameter in the@cloudwatch.filter_log_events
is a best-effort request, and not guaranteed. So, as a result, if you got two streams worth of messages a single log group, they might not get interleaved completelye.g. given two streams
It's possible that the first stream could be loaded before the second, and as a result, when that is queried, we only get events from 12:06 (and so lose a minute's worth of events)
This change was to widen @sincedb to also store a high water mark at the stream level
And the algorithm was changed to:
The file format still has the old
group position
entries to allow for the previous high-water mark to be maintained on an upgrade