-
Notifications
You must be signed in to change notification settings - Fork 950
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add event_time page #6383
base: current
Are you sure you want to change the base?
add event_time page #6383
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! A few suggested tweaks and clarifications but overall :chef_kiss:
|
||
When configured, `event_time` enables compare changes to: | ||
|
||
- Compare data in CI versus production for overlapping times only, reducing false discrepancies. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Compare data in CI versus production for overlapping times only, reducing false discrepancies. | |
- Compare data in CI against production for only the overlapping times, avoiding false positives and returning results faster. |
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hiya @mirnawong1
I've approved this one for you and left some suggestions and non-blockers.
Kind Regards
Natalie
Co-authored-by: nataliefiann <[email protected]>
Co-authored-by: nataliefiann <[email protected]>
Co-authored-by: nataliefiann <[email protected]>
Co-authored-by: nataliefiann <[email protected]>
### Speeding up comparisons | ||
It's common for CI jobs to only [build a subset of data](/best-practices/best-practice-workflows#limit-the-data-processed-when-in-development), for example only the last 7 days of data. When an [`event_time`](/reference/resource-configs/event-time) column is specified on your model, compare changes can: | ||
|
||
- Compare data in CI against production for only the overlapping times, avoiding false positives and returning results faster. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think both of these bullets have the same benefit of "using only the overlapping timeframe, which avoids incorrect row-count changes and returns results faster"
I would distinguish the 2 scenarios as:
- scenarios where your CI job only builds a subset of data
- scenarios where your CI job contains fresher data than production
Rather than nesting the second scenario within the first - lmk if that makes sense!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed it to this;
It's common for CI jobs to only build a subset of data (for example only the last 7 days of data).
When an event_time
column is specified on your model, compare changes can optimize comparisons by using only the overlapping timeframe (meaning the timeframe exists in both the CI and production environment), helping you avoid incorrect row-count changes to return results faster.
This is useful in scenarios like:
- Subset of data in CI — When CI builds only a subset of data (like the most recent 7 days), compare changes might interpret the excluded data as "deleted rows." Configuring
event_time
allows you to avoid this issue by limiting comparisons to the overlapping timeframe, preventing false alerts about data deletions that are just filtered out in CI. - Fresher data in CI than in production — When your CI job includes fresher data than production, compare changes might flag the additional rows as "new" data, even though they’re just fresher data in CI. With
event_time
configured, the comparison only includes the shared timeframe and correctly reflects actual changes in the data.
Co-authored-by: Grace Goheen <[email protected]>
this pr adds a new page for the
event_time
config for models, sources, snapshots, and sources.also adds to the advanced ci and links from other pages.
this PR #6384 will merge into this pr.
docs notion
some. outstanding items:
🚀 Deployment available! Here are the direct links to the updated files: