Skip to content

feat(redis sink): add Redis Sentinel support for redis sink #23355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

5Dev24
Copy link

@5Dev24 5Dev24 commented Jul 10, 2025

Summary

This PR adds support for Redis Sentinel in the Redis sink. This is a feature that changes how the underlying connection is established and maintained for the sink.

At the core of the change is a new connection abstraction that handles both "direct connections" to a redis server or "sentinel connections" that can resolve to a redis server. With the sentinel connections, they will attempt to establish a new connection from sentinel when it suspects the primary is unreachable. This is determined by the RetryLogic of the sink.

The redis integration tests will now run a sentinel instance to run checks that it can be reached and that modifications reach the redis server it owns.

The redis sink's config was modified to add new, optional fields to configure how the sink should connect to a primary from sentinel. Sentinel connections will be used when sentinel_service is specified. Config names were picked to convey meaning, but better suggestions are welcome.

There is a small fix for the sink's RetryLogic as it is currently handling the wrong error type (present: RedisError, fix: RedisSinkError). Luckily, none of the default logic was overridden in the existing implementation so this wasn't felt.

Sadly this PR became larger than expected, but Sentinel required a large overhaul and needed to be more connected to other components within the sink.

Vector configuration

sinks:
  valkey:
    type: redis
    endpoint:
    - redis://127.0.0.1:26379/
    - redis://127.0.0.1:26380/
    - redis://127.0.0.1:26381/
    sentinel_service: primary
    inputs: ...
    encoding:
      codec: text
    key: "testing"
    data_type: list
    list_option:
      method: rpush
    batch:
      max_bytes: 2048
      max_events: 1
      timeout_secs: 5

How did you test this PR?

Via the existing unit tests, writing new integration tests, and against a larger sentinel cluster and simulated node failures.

Sentinel cluster setup:

  • Three sentinel instances
  • One primary redis instance
  • Two replica redis instances

Tested failing primary, then failing replica that was promoted to primary.
Tested failing sentinel instances.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Related: #13535
Related: #23234 (1 of 4)

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • cargo fmt --all
      • cargo clippy --workspace --all-targets -- -D warnings
      • cargo nextest run --workspace (alternatively, you can run cargo test --all)
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

@5Dev24 5Dev24 requested review from a team as code owners July 10, 2025 16:23
@github-actions github-actions bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants