Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add event_time page #6383

Open
wants to merge 35 commits into
base: current
Choose a base branch
from
Open

add event_time page #6383

wants to merge 35 commits into from

Conversation

mirnawong1
Copy link
Contributor

@mirnawong1 mirnawong1 commented Oct 30, 2024

this pr adds a new page for the event_time config for models, sources, snapshots, and sources.

also adds to the advanced ci and links from other pages.

this PR #6384 will merge into this pr.

docs notion

some. outstanding items:

  • should we change the source page config as it only says 'enabled' is supported?

🚀 Deployment available! Here are the direct links to the updated files:

@mirnawong1 mirnawong1 requested a review from a team as a code owner October 30, 2024 11:45
Copy link

vercel bot commented Oct 30, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview Nov 11, 2024 4:54pm

@github-actions github-actions bot added content Improvements or additions to content size: large This change will more than a week to address and might require more than one person Docs team Authored by the Docs team @dbt Labs labels Oct 30, 2024
Copy link
Contributor

@joellabes joellabes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! A few suggested tweaks and clarifications but overall :chef_kiss:

website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/static/img/docs/deploy/apples_to_apples.png Outdated Show resolved Hide resolved
website/docs/docs/dbt-versions/release-notes.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/docs/deploy/advanced-ci.md Outdated Show resolved Hide resolved
website/docs/docs/deploy/advanced-ci.md Outdated Show resolved Hide resolved

When configured, `event_time` enables compare changes to:

- Compare data in CI versus production for overlapping times only, reducing false discrepancies.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Compare data in CI versus production for overlapping times only, reducing false discrepancies.
- Compare data in CI against production for only the overlapping times, avoiding false positives and returning results faster.

website/docs/docs/deploy/advanced-ci.md Outdated Show resolved Hide resolved
website/docs/docs/deploy/advanced-ci.md Show resolved Hide resolved
mirnawong1 and others added 6 commits October 30, 2024 17:32
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Co-authored-by: Joel Labes <[email protected]>
Copy link
Contributor

@nataliefiann nataliefiann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hiya @mirnawong1

I've approved this one for you and left some suggestions and non-blockers.

Kind Regards
Natalie

website/docs/docs/deploy/advanced-ci.md Outdated Show resolved Hide resolved
### Speeding up comparisons
It's common for CI jobs to only [build a subset of data](/best-practices/best-practice-workflows#limit-the-data-processed-when-in-development), for example only the last 7 days of data. When an [`event_time`](/reference/resource-configs/event-time) column is specified on your model, compare changes can:

- Compare data in CI against production for only the overlapping times, avoiding false positives and returning results faster.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think both of these bullets have the same benefit of "using only the overlapping timeframe, which avoids incorrect row-count changes and returns results faster"

I would distinguish the 2 scenarios as:

  • scenarios where your CI job only builds a subset of data
  • scenarios where your CI job contains fresher data than production

Rather than nesting the second scenario within the first - lmk if that makes sense!

Copy link
Contributor Author

@mirnawong1 mirnawong1 Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it to this;

It's common for CI jobs to only build a subset of data (for example only the last 7 days of data).

When an event_time column is specified on your model, compare changes can optimize comparisons by using only the overlapping timeframe (meaning the timeframe exists in both the CI and production environment), helping you avoid incorrect row-count changes to return results faster.

This is useful in scenarios like:

  • Subset of data in CI — When CI builds only a subset of data (like the most recent 7 days), compare changes might interpret the excluded data as "deleted rows." Configuring event_time allows you to avoid this issue by limiting comparisons to the overlapping timeframe, preventing false alerts about data deletions that are just filtered out in CI.
  • Fresher data in CI than in production — When your CI job includes fresher data than production, compare changes might flag the additional rows as "new" data, even though they’re just fresher data in CI. With event_time configured, the comparison only includes the shared timeframe and correctly reflects actual changes in the data.

website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
website/docs/reference/resource-configs/event-time.md Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content Docs team Authored by the Docs team @dbt Labs size: large This change will more than a week to address and might require more than one person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants