You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some resources (databases) support a truncate operation to delete all records from a collection. In this case, we don't get the full list of records that were deleted, but rather just the information that the collection was truncated. With the current set of operations (snapshot, create, update, delete) it's impossible to represent this action. We would have to spawn a delete record for every row that was truncated, but the connector doesn't have the information about which rows existed before. Even without this limitation, this approach wouldn't really be performant on big collections.
One solution is to create a truncate operation that has to contain the metadata field for which collection is affected. The destination connector could apply this operation as best it can.
We need to figure out what side effects we could experience if we introduce such an operation. For instance, processors working on single records would likely fail to process a truncate operation, since the payload and key would likely be empty.
The text was updated successfully, but these errors were encountered:
Feature description
This feature request originated here ConduitIO/conduit-connector-postgres#168.
Some resources (databases) support a truncate operation to delete all records from a collection. In this case, we don't get the full list of records that were deleted, but rather just the information that the collection was truncated. With the current set of operations (
snapshot
,create
,update
,delete
) it's impossible to represent this action. We would have to spawn adelete
record for every row that was truncated, but the connector doesn't have the information about which rows existed before. Even without this limitation, this approach wouldn't really be performant on big collections.One solution is to create a
truncate
operation that has to contain the metadata field for which collection is affected. The destination connector could apply this operation as best it can.We need to figure out what side effects we could experience if we introduce such an operation. For instance, processors working on single records would likely fail to process a
truncate
operation, since the payload and key would likely be empty.The text was updated successfully, but these errors were encountered: