Skip to content

Feature: add record operation truncate #1654

Open
@lovromazgon

Description

@lovromazgon

Feature description

This feature request originated here ConduitIO/conduit-connector-postgres#168.

Some resources (databases) support a truncate operation to delete all records from a collection. In this case, we don't get the full list of records that were deleted, but rather just the information that the collection was truncated. With the current set of operations (snapshot, create, update, delete) it's impossible to represent this action. We would have to spawn a delete record for every row that was truncated, but the connector doesn't have the information about which rows existed before. Even without this limitation, this approach wouldn't really be performant on big collections.

One solution is to create a truncate operation that has to contain the metadata field for which collection is affected. The destination connector could apply this operation as best it can.

We need to figure out what side effects we could experience if we introduce such an operation. For instance, processors working on single records would likely fail to process a truncate operation, since the payload and key would likely be empty.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions