Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement CLUSTER PUBSUB #866

Merged
merged 34 commits into from
Jan 31, 2025
Merged

Implement CLUSTER PUBSUB #866

merged 34 commits into from
Jan 31, 2025

Conversation

vazois
Copy link
Collaborator

@vazois vazois commented Dec 9, 2024 •
edited
Loading

This PR adds support for PUBSUB in Cluster mode.

  • Implement SPUBLISH and SSUBSCRIBE APIs
  • Add published message forwarding using the active gossip connections
  • Expose parameters to allow configuring max outstanding PubSub forwarding tasks
  • Augment latency and throughput benchmarks with PUBLISH and SPUBLISH support.

@vazois vazois force-pushed the vazois/cluster-pubsub branch 4 times, most recently from 2f789f5 to 422532b Compare December 12, 2024 01:28
@vazois vazois force-pushed the vazois/cluster-pubsub branch 9 times, most recently from 381fb96 to cd57d5a Compare January 13, 2025 18:44
@vazois vazois marked this pull request as ready for review January 13, 2025 19:06
@badrishc
Copy link
Collaborator

Why was the BDN removed as it seemed useful?

@vazois
Copy link
Collaborator Author

vazois commented Jan 14, 2025

Why was the BDN removed as it seemed useful?

Because it was not related to cluster. It was only exercising the publish path with no connected nodes. I am not sure how communication over socket will affect BDN.

@vazois vazois force-pushed the vazois/cluster-pubsub branch 2 times, most recently from f1938bf to 74325aa Compare January 17, 2025 22:02
vazois added 17 commits January 23, 2025 09:26
@vazois
PUBLISH and SPUBLISH benchmark addition
c7208bc
@vazois
fix check for getting most recent config instance
9db32e7
@vazois
optimization 1: calculate candidates for published message forwarding…
c9345aa 
… only when config gets updated
@vazois
optimization2: ignore response from cluster publish
412ae03
@vazois
add publish BDN
95db9b0
@vazois
minimize task creation
7816c5e
@vazois
load no response task
5ef16a3
@vazois
change to pinned array
8fc56d1
@vazois
simplify config update for publish forward
bd1e3c8
@vazois
wip; wait for response vs fire and forget
120edee
@vazois
add protected connection initialization for gossip-publish
5788de0
@vazois
remove BDN because not useful
f42a1f4
@vazois
remove unused NoResponse flag
e5b48b6
@vazois
remove option for tasks
eecb867
@vazois
ensure noResponse for slot verification of cluster commands
8026a30
@vazois
add SPUBLISH and PUBLISH docs
8c17dd9
@vazois
correct typo
4fa2ddd
@vazois vazois force-pushed the vazois/cluster-pubsub branch from 599a832 to 4fa2ddd Compare January 23, 2025 17:54
TalZaccai
TalZaccai requested changes Jan 28, 2025
View reviewed changes
Copy link
Contributor

@TalZaccai TalZaccai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good to me! Added some comments...

@vazois vazois requested a review from TalZaccai January 30, 2025 18:23
@vazois vazois merged commit 8bfb8c6 into microsoft:main Jan 31, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 1, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@badrishc badrishc badrishc approved these changes

@TalZaccai TalZaccai TalZaccai approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vazois @badrishc @TalZaccai