Incremental streams #40
Conversation
Chargify API allow 200 records per page. It is more efficient to use as many records as possible.
This is more accurate to use the id for the bookmark. If there is no state persisted, we use the start_date from the configuration. Once a state is persisted (biggest id) we filter the transactions on this id, with the since_id parameter.
Add created_at and updated_at fields
|
We currently use the Chargify integration on Stitch and experiencing a major data discrepancy issue that 20% of transactions are not synced from Chargify to Redshift. After talking to @cirotix on Slack, this PR seems to be the rescue to the issue. We look forward to seeing it merged. Thanks! |
|
Hoping to see this merged as well. |
There was a problem hiding this comment.
Thank you for the work on this, we are interested in merging this but first there were some API parameters that looked to be out of place.
Before we can merge this we also need logs demonstrating that the tap has been tested locally with the changes and it functions as expected
|
|
||
| def customers(self, bookmark=None): | ||
| for i in self.get("customers.json"): | ||
| for i in self.get("customers.json", sort="asc", date_field="updated_at", start_datetime=bookmark): |
There was a problem hiding this comment.
The docs indicate the customers.json endpoint accepts a direction but not a sort parameter:
https://reference.chargify.com/v1/customers/list-customers
|
|
||
| def product_families(self, bookmark=None): | ||
| for i in self.get("product_families.json"): | ||
| for i in self.get("product_families.json", sort="asc", date_field="updated_at", start_datetime=bookmark): |
There was a problem hiding this comment.
The docs for the product_families.json endpoint do not include the sort parameter
https://reference.chargify.com/v1/product-families/list-product-family-via-site
| for j in self.get("product_families/{product_family_id}/products.json".format(product_family_id=k["product_family"]["id"]), | ||
| sort="asc", date_field="updated_at", start_datetime=bookmark): |
There was a problem hiding this comment.
The docs for the product_families/{product_family_id}/products.json do not include a sort parameter:
https://reference.chargify.com/v1/products/list-products
|
Hi @cirotix The simplest and safest way for you to share credentials with Stitch is to create a connection in Stitch and then open a conversation with the support staff informing them that you want to share the credentials on the connection with the engineering team for testing purposes. |
|
Hi @cosimon |
|
@cirotix Do you have any updates on this? |
5 participants
This PR provides incremental replication for:
Also some schema fixes
It has been tested with the big-query target.
fixes #35