Commit f4a7c76
feat: DbAdapter API + pg adapter (without CF) (#44)
* WIP
WIP
WIP
WIP
Uses mock adapter
WIP
WIP
WIP
WIP
WIP
WIP
* fix: ignore node_modules everywhere
* Adds split_db_adapter to CI run
* feat(cf-adapter): add Cloudflare Vectorize adapter (#28)
* feat(cf-adapter): add Cloudflare Vectorize adapter
* feat(cf-adapter): enhance Cloudflare Vectorize integration with config-based bindings and add tests
* feat(cf-adapter): refactor Cloudflare Vectorize integration to use config-based bindings and update tests
* chore: update pnpm-lock.yaml
* Preparing for automated pubishes. This one beta will be done by hand but hopefully the rest will be done automatically
* Bumps version since we added deleteEmbeddings. Also runs tsc so that we can be sure the whole project compiles
* Adds the type check to ci
* fixes type check
* removes silly double checking on split_db_adapter for push
* Adds root pnpm workspace
* feat(cf-adapter): update query parameters and method for deleting embeddings in Cloudflare Vectorize integration (#31)
* Bumps version to release
* Better typings (#34)
* Adds better id tracking for deletion and does only one search instead of many for querying (#35)
* Deduplicate shared logic across plugin and adapter packages (#36)
* Deduplicate shared logic across plugin and adapter packages
Extract repeated production patterns (chunk validation, delete embeddings,
task slug constants) into shared utilities exported from the root plugin.
Consolidate test helpers via vitest path aliases so adapter tests import
from the canonical root dev/ copies. Remove CF adapter dead test code
(unused utils, constants, helpers). Fix chunkRichText join bug in CF
adapter tests (was joining child nodes without spaces).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* cf adapter limitation acknowledgement and more DRY
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Removes dead code (#37)
* Bumps version for rollout
* Merge main (#40)
* adds should embed (#38)
* adds should embed - merged
* Merge main into split_db_adapter (beta.5) (#42)
* adds should embed (#38)
* adds should embed
* Ups version to get ready for release
* splits the job into one per batch (#41)
* splits the job into one per batch
* fix: remove waitUntil delay and persist failedChunkData on batch records
- Remove 30s waitUntil delay from per-batch task re-queue (was causing
test timeouts since the original code had no such delay)
- Add failedChunkData JSON field to batch collection so per-batch tasks
can store chunk-level failure data independently
- Aggregate failedChunkData from batch records in finalizeRunIfComplete()
instead of relying on in-memory accumulation from the old single-task flow
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add batchLimit to CollectionVectorizeOption with coordinator/worker architecture
Splits prepare-bulk-embedding into coordinator + per-collection workers.
Each worker processes one page of one collection, queuing a continuation
job before processing to ensure crash safety. Default batchLimit is 1000
when not explicitly set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rewrite batchLimit test 2 to reuse same Payload instance
The second test was creating a separate Payload instance sharing the same
DB and job queues, causing two crons to compete for jobs. This led to
double-execution and mock state inconsistency (expected 4 to be 2).
Now both tests use the single beforeAll instance with cleanup between.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add payload.destroy() in afterAll to prevent OOM from leaked crons
Every test file that creates a Payload instance now calls
payload.destroy() in afterAll (or try/finally for in-test instances).
This stops background cron jobs from accumulating across tests, which
was causing heap exhaustion in CI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Trying to not destroy our heap
* Runs tests in parallel now that each test gets its own db
* WIP
* fix: fix OOM, polling test assertions, and add diagnostic logging
- Add --max-old-space-size=8192 to test:int NODE_OPTIONS (cross-env was
overriding the CI env var, so the heap limit never took effect)
- Fix polling.spec.ts queueSpy assertions: coordinator/worker adds an
extra queue call, so poll-or-complete-single-batch is now call 3 and 4
instead of 2 and 3
- Add extensive [vectorize-debug] console.log throughout task handlers
(coordinator, worker, poll-single, finalize, streamAndBatchDocs) to
diagnose any remaining CI hangs
- Remove redundant NODE_OPTIONS from CI workflow (now in the script)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: remove poll-or-complete-bulk-embedding task and aggregate incrementally
Remove the backward-compatible fan-out task since the per-batch architecture
hasn't been released yet. Refactor finalizeRunIfComplete to aggregate batch
counts incrementally during pagination instead of collecting all batch objects
into memory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump to 0.5.5, update changelog, remove debug logging
- Bump version 0.5.4 → 0.5.5
- Add 0.5.5 entry to CHANGELOG.md (coordinator/worker, batchLimit, per-batch polling)
- Document batchLimit in README CollectionVectorizeOption section
- Remove all diagnostic console.log statements from bulkEmbedAll.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Adds upgrade note
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump version to 0.6.0-beta.5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve 4 CI test failures from merge
- chunkers.spec.ts: remove getPayload() call that crashes on dummy db,
pass SanitizedConfig directly to chunkRichText
- batchLimit.spec.ts: add missing dbAdapter (createMockAdapter) required
by split_db_adapter architecture
- extensionFieldsVectorSearch.spec.ts: pass adapter as second arg to
createVectorSearchHandlers (new signature from split_db_adapter)
- versionBump.spec.ts: destroy payload0 before creating payload1 to
prevent cron worker race condition between two instances
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Cleans a nit double line
* Undoes a weird test fix done by the bot
* fix: harden versionBump test with sequential steps and queue isolation
- Use test.step() to enforce sequential execution of each phase
- Add separate realtimeQueueName per payload instance to prevent
cron worker cross-talk on the default queue
- Use dynamic Date.now() keys to avoid cached state interference
- Increase waitForBulkJobs timeout to 30s for CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prevent waitForBulkJobs from returning prematurely
waitForBulkJobs could return early in the coordinator/worker fan-out
pattern when there's a brief window with 0 pending jobs between job
transitions. Now it also checks the bulk embeddings run status — only
returns when both no pending jobs exist AND no runs are in queued/running
state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove test.step() — not available in Vitest
test.step() is a Playwright API, not Vitest. Reverted to flat
sequential code with phase comments for readability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rewrite versionBump test with single Payload instance
Instead of creating two Payload instances (which caused cron cross-talk,
timeout, and queue isolation issues on CI), use one instance and mutate
the knowledgePools config version between bulk embed runs. Tests the
same code path (versionMismatch in streamAndBatchDocs) without the
multi-instance fragility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update beta version
* Remove Cloudflare adapter to unblock main branch merge
Split the CF adapter work out so the core DbAdapter API and pg adapter
can be merged to main independently. The CF adapter will continue in
a separate branch.
* docs: call for help adding more database adapters
* chore: bump version to 0.7.0
---------
Co-authored-by: dejan-velimirovic-calendly <dejan.velimirovic@calendly.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>1 parent 82ac6a1 commit f4a7c76
File tree
75 files changed
+3767
-1704
lines changed- .changeset
- .github/workflows
- adapters
- pg
- dev/specs
- src
- dev
- app/(frontend)
- helpers
- specs
- bulkEmbed
- helpers
- src
- endpoints
- tasks
- utils
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
75 files changed
+3767
-1704
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
10 | 30 | | |
11 | 31 | | |
12 | 32 | | |
| |||
33 | 53 | | |
34 | 54 | | |
35 | 55 | | |
36 | | - | |
37 | | - | |
| 56 | + | |
| 57 | + | |
38 | 58 | | |
39 | 59 | | |
40 | 60 | | |
| |||
53 | 73 | | |
54 | 74 | | |
55 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
56 | 122 | | |
57 | 123 | | |
58 | 124 | | |
| |||
79 | 145 | | |
80 | 146 | | |
81 | 147 | | |
82 | | - | |
83 | | - | |
| 148 | + | |
| 149 | + | |
84 | 150 | | |
85 | 151 | | |
86 | 152 | | |
| |||
101 | 167 | | |
102 | 168 | | |
103 | 169 | | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
5 | 107 | | |
6 | 108 | | |
7 | 109 | | |
| |||
0 commit comments