Skip to content

Add acceptance tests for S3 #62

@maha-mstfa

Description

@maha-mstfa

Feature description

We already had a PR for this but it had many problems {all these are just things specific for acceptance tests to work on S3, they are not S3 related bugs}:
#44

  • The acceptance tests expect that the destination writes a record, and the source will retrieve the same record containing the same key and payload.. but this is not the case here because the key would be a random number.. and the payload would be the whole record {S3 destination has a batch size.. so the file name at the destination bucket can't have the same key.. so it's a random number.json}

  • the createdAt timestamp in the record is an int timestamp: ex: 1655131862580631000, and when we try to parse the file body into sdk.Record we get this error:

"CreatedAt":1655131862580631000
err: parsing time "1655131862580631000" as "\"2006-01-02T15:04:05Z07:00\"": cannot parse "1655131862580631000" as "\""
  • if the payload has any special characters, a back dash will be added to the string, which will make it not match, ex:
want: "Payload":{"'8B2\u0026U
got : "Payload":"{\"'8B2\\u0026U
  • the isEqualRecords method in acceptance tests assumes that payloads are unique, but in our case, the payload is the whole record.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions