feat(redis sink): add Redis Sentinel support for redis sink #23355
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for Redis Sentinel in the Redis sink. This is a feature that changes how the underlying connection is established and maintained for the sink.
At the core of the change is a new connection abstraction that handles both "direct connections" to a redis server or "sentinel connections" that can resolve to a redis server. With the sentinel connections, they will attempt to establish a new connection from sentinel when it suspects the primary is unreachable. This is determined by the
RetryLogic
of the sink.The redis integration tests will now run a sentinel instance to run checks that it can be reached and that modifications reach the redis server it owns.
The redis sink's config was modified to add new, optional fields to configure how the sink should connect to a primary from sentinel. Sentinel connections will be used when
sentinel_service
is specified. Config names were picked to convey meaning, but better suggestions are welcome.There is a small fix for the sink's
RetryLogic
as it is currently handling the wrong error type (present:RedisError
, fix:RedisSinkError
). Luckily, none of the default logic was overridden in the existing implementation so this wasn't felt.Sadly this PR became larger than expected, but Sentinel required a large overhaul and needed to be more connected to other components within the sink.
Vector configuration
How did you test this PR?
Via the existing unit tests, writing new integration tests, and against a larger sentinel cluster and simulated node failures.
Sentinel cluster setup:
Tested failing primary, then failing replica that was promoted to primary.
Tested failing sentinel instances.
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changelog
label to this PR.References
Related: #13535
Related: #23234 (1 of 4)
Notes
@vectordotdev/vector
to reach out to us regarding this PR.pre-push
hook, please see this template.cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings
cargo nextest run --workspace
(alternatively, you can runcargo test --all
)git merge origin master
andgit push
.Cargo.lock
), pleaserun
cargo vdev build licenses
to regenerate the license inventory and commit the changes (if any). More details here.