Skip to content

Conversation

@robholland
Copy link
Contributor

@robholland robholland commented Jan 14, 2026

What changed?

Pause replication streams when the scheduler is under pressure.

Why?

This allows us to apply backpressure to replication streams when we cannot keep up with the load. The schedulers are shared amongst streams so just keep tracking of tasked tasks is not enough. We use a timer to notify us if a submit is taking too long rather than recording the time after the fact so that the backpressure is more reactive.

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Note

Implements latency-aware backpressure for replication stream receiving.

  • Add history.ReplicationReceiverSubmissionLatencyThreshold and history.ReplicationReceiverSlowSubmissionWindow dynamic configs; plumbed through configs.Config
  • Stream receiver now measures scheduler Submit latency, records per-priority slow submission timestamps, and exposes them to flow control
  • Flow controller pauses when taskTrackingCount exceeds ReplicationReceiverMaxOutstandingTaskCount or a slow submission occurred within the configured window
  • Refactor scheduler routing: new getTaskSchedulerPriority plus existing getTaskScheduler; add thread-safe tracking of last slow submissions
  • Add unit tests for flow control slow-submit behavior and update stream receiver tests; minor stability tweak to panic-capture test

Written by Cursor Bugbot for commit 6cc7a01. This will update automatically on new commits. Configure here.

@robholland robholland requested review from a team as code owners January 14, 2026 14:20
Copy link
Contributor

@yux0 yux0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think about pending task count

scheduler.Submit(task)
end := time.Now()
if end.Sub(start) > submissionThreshold {
r.recordSlowSubmission(schedulerPriority, end)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can we add a feature flag on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants