Skip to content

perf: paginate and throttle clearLocal checkpoint deletion#268

Open
sleidig wants to merge 2 commits into
masterfrom
perf/clear-local-throttling
Open

perf: paginate and throttle clearLocal checkpoint deletion#268
sleidig wants to merge 2 commits into
masterfrom
perf/clear-local-throttling

Conversation

@sleidig

@sleidig sleidig commented Jun 12, 2026

Copy link
Copy Markdown
Member

Summary

Part of #261 (item 7). Based on #262 (e2e suite) — merge that first; GitHub retargets automatically. Independent of the other improvement PRs.

clearLocal (triggered on every permission change and via /admin/clear_local/:db) fetched all _local docs in one buffered response — one replication checkpoint doc exists per client, so large deployments have thousands — and fired every DELETE in parallel via Promise.all. That momentarily floods CouchDB, and a single failure rejected the whole operation while the remaining deletes kept running detached.

Changes

  • _local_docs is fetched in pages of 500; since deleted docs disappear from the index, re-fetching naturally pages through the remainder (an attempted-ids set guards against endless loops on undeletable docs).
  • Deletes run with bounded concurrency (10 in flight).
  • Failures are logged individually, all docs are still attempted, and a summary error is thrown at the end so the admin endpoint still reports failure.

Test plan

  • new unit tests: internal-doc skipping, bounded concurrency (max in-flight ≤ 10 across 25 docs), failure handling (all attempted, no endless retry, summary error)
  • e2e clear_local tests unchanged and green
  • 177 unit / 40 e2e tests green

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Abhinegi2, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3a81054b-c959-45d5-b26b-1142de060ebf

📥 Commits

Reviewing files that changed from the base of the PR and between ccdfd25 and bcbb8eb.

📒 Files selected for processing (2)
  • src/admin/admin.service.spec.ts
  • src/admin/admin.service.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/clear-local-throttling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Base automatically changed from test/e2e-suite to master June 17, 2026 16:19
clearLocal fetched all _local docs at once (one checkpoint doc exists
per client replication, so this can be large) and fired every DELETE
in parallel via Promise.all - momentarily flooding CouchDB on big
deployments, and rejecting wholesale on the first failure while the
remaining deletes kept running detached.

Now:
- _local_docs is fetched in pages of 500 (deleted docs disappear from
  the index, so re-fetching pages through the remainder)
- deletes run with bounded concurrency (10 in flight)
- failures are logged, all docs are still attempted, and a summary
  error is thrown at the end (endpoint still reports failure)

Part of #261

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Page forward through _local_docs by id cursor so undeletable docs cannot
block progress to later pages, and drop the unbounded attemptedIds set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants