Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: add record operation truncate #1654

Open
lovromazgon opened this issue Jun 13, 2024 · 0 comments
Open

Feature: add record operation truncate #1654

lovromazgon opened this issue Jun 13, 2024 · 0 comments
Labels
feature New feature or request
Milestone

Comments

@lovromazgon
Copy link
Member

Feature description

This feature request originated here ConduitIO/conduit-connector-postgres#168.

Some resources (databases) support a truncate operation to delete all records from a collection. In this case, we don't get the full list of records that were deleted, but rather just the information that the collection was truncated. With the current set of operations (snapshot, create, update, delete) it's impossible to represent this action. We would have to spawn a delete record for every row that was truncated, but the connector doesn't have the information about which rows existed before. Even without this limitation, this approach wouldn't really be performant on big collections.

One solution is to create a truncate operation that has to contain the metadata field for which collection is affected. The destination connector could apply this operation as best it can.

We need to figure out what side effects we could experience if we introduce such an operation. For instance, processors working on single records would likely fail to process a truncate operation, since the payload and key would likely be empty.

@lovromazgon lovromazgon added feature New feature or request triage Needs to be triaged labels Jun 13, 2024
@lovromazgon lovromazgon removed the triage Needs to be triaged label Jun 17, 2024
@lovromazgon lovromazgon added this to the Next milestone Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant