-
Notifications
You must be signed in to change notification settings - Fork 531
Description
Add User Data Sharing Pipeline (Kuzu + LanceDB β Kuzu + LanceDB)
π Summary
Create a pipeline that transfers data from one user's Kuzu + LanceDB databases to another user's Kuzu + LanceDB databases with proper metastore updates for ownership and permissions.
π Background
Users need to share processed knowledge graphs between Cognee instances using the same database stack (Kuzu + LanceDB). The pipeline must transfer graph data, vector embeddings, and update PostgreSQL metastore to reflect new user ownership.
π― Acceptance Criteria
- Source Data Export
Extract from source user's databases:
Export all nodes/edges from source Kuzu database
Export vector embeddings from source LanceDB collections
Export dataset metadata from PostgreSQL metastore
Data packaging: Bundle into transfer format with user context
- Target Data Import
Import to target user's databases:
Load graph data into target user's Kuzu database
Load vectors into target user's LanceDB collections
Update PostgreSQL metastore with new user ownership
- Metastore Updates
User mapping: Update all references from source_user_id to target_user_id
Dataset ownership: Transfer dataset ownership and permissions
Preserve relationships: Maintain data integrity across databases
- Pipeline Implementation
Add /v1/data/export/{dataset_id} endpoint
Add /v1/data/import endpoint accepting transfer bundle
Handle user authentication and authorization
Validate target user has sufficient permissions