fix: batch recovery jobs to avoid 16MB read limit#172
fix: batch recovery jobs to avoid 16MB read limit#172sethconvex wants to merge 5 commits intomainfrom
Conversation
Recovery was sending all stale running jobs to a single `recover` mutation, which could exceed Convex's 16MB read limit when many jobs needed recovery at once (e.g. high maxParallelism + server restart). Batch into chunks of 50, matching the pattern used for cancellations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughUpdates recovery to process old jobs in sequential batches of 50. The recovery handler now returns whether more candidates remain; the main flow sets Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit: |
The initial batch fix only batched the scheduled `recover` call, but `handleRecovery` inside `main` was still reading work docs for every old running job unbounded. Now it processes at most RECOVERY_BATCH_SIZE candidates per iteration and signals `main` to re-run recovery immediately if more remain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 96380fe.
| if (r.started >= oldEnoughToConsider) { | ||
| return null; | ||
| } | ||
| const work = await ctx.db.get(r.workId); |
There was a problem hiding this comment.
this is still loading the old work - which may be big
Work documents can store arbitrarily large fnArgs, so use a conservative batch size to stay well under the 16MB read limit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2 participants
Summary
recovermutation, which could exceed Convex's 16MB read limit when many jobs needed recovery simultaneously (e.g. highmaxParallelism+ server restart)CANCELLATION_BATCH_SIZE = 64)Test plan
maxParallelismdeployment that recovery no longer hits the 16MB limit🤖 Generated with Claude Code
Summary by CodeRabbit