-
Notifications
You must be signed in to change notification settings - Fork 1.1k
add event_time page #6383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
add event_time page #6383
Changes from 2 commits
Commits
Show all changes
71 commits
Select commit
Hold shift + click to select a range
46231b0
add event_time page
mirnawong1 33a66a8
Merge branch 'current' into add-event-time
mirnawong1 6ebf5eb
update source/snapshots
mirnawong1 501d948
add to model
mirnawong1 4f2c6dc
add img and rn
mirnawong1 57ee608
fix link
mirnawong1 451fc46
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3354c9d
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 603c21c
fix link again
mirnawong1 0c68f62
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 488460c
Update event-time.md
mirnawong1 2fb62c5
Update release-notes.md
mirnawong1 69ba339
Update event-time.md
mirnawong1 1ebbbdb
Update advanced-ci.md
mirnawong1 2b713ee
Update advanced-ci.md
mirnawong1 c789601
Update advanced-ci.md
mirnawong1 5708119
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 903c5d1
Merge branch 'current' into add-event-time
mirnawong1 2dd873a
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 b7a07be
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 12cdffa
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 016c555
Update event-time.md
mirnawong1 9c49664
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 2910914
Merge branch 'current' into add-event-time
mirnawong1 5ba059e
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 79128fe
Merge branch 'current' into add-event-time
mirnawong1 cc34575
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 551821d
update img
mirnawong1 735ae38
fix img size
mirnawong1 ac7616b
Merge branch 'current' into add-event-time
mirnawong1 bdc037e
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 0363051
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 d693c9b
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 809f2a7
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 81e2318
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 14632b3
Merge branch 'current' into add-event-time
mirnawong1 aad3987
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 e92c9db
Update website/docs/reference/source-configs.md
mirnawong1 2b98454
Merge branch 'current' into add-event-time
mirnawong1 3ad1bb6
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3da521f
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 edd1123
add scenarios
mirnawong1 52c0db9
add scenarios
mirnawong1 f461ffa
fold in grace's feedback
mirnawong1 a4f3b23
Merge branch 'current' into add-event-time
mirnawong1 a1c8166
Merge branch 'add-event-time' of github.com:dbt-labs/docs.getdbt.com …
mirnawong1 f1969f4
remove redundant
mirnawong1 c170a3b
Merge branch 'current' into add-event-time
mirnawong1 d6a309b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3a8dee5
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 5851c2b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 0656327
Update release-notes.md
mirnawong1 57679b2
Merge branch 'current' into add-event-time
mirnawong1 556249a
Update website/docs/docs/dbt-versions/release-notes.md
mirnawong1 0bd8584
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 bd233ad
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 b9e4be0
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 ff3416a
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 613f1ef
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 85f181d
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 4644684
Merge branch 'current' into add-event-time
mirnawong1 8cf073b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 46763d8
Merge branch 'current' into add-event-time
mirnawong1 d2bf5af
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 6015dee
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 4250c9d
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 76b12e9
update header adn link
mirnawong1 de8f752
Merge branch 'current' into add-event-time
mirnawong1 4b28bbc
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 337248b
add event _time to sources/snapshots/models/seeds (#6384)
mirnawong1 0e16ca6
Update incremental-microbatch.md
mirnawong1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,247 @@ | ||
--- | ||
title: "event_time" | ||
id: "event-time" | ||
sidebar_label: "event_time" | ||
resource_types: [models, seeds, source] | ||
description: "Setting the event_time config allows you to specify when an event occurred. It's required for microbatch and can added to ensure you're comparing overlapping times in compare changes" | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
datatype: string | ||
--- | ||
|
||
Available in dbt Cloud Versionless and dbt Core v1.9 and higher. | ||
Check warning on line 10 in website/docs/reference/resource-configs/event-time.md
|
||
|
||
<Tabs> | ||
<TabItem value="model" label="Models"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
models: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
models: | ||
- name: model_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name="models/modelname.sql"> | ||
|
||
```sql | ||
{{ config( | ||
event_time='my_time_field' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
</TabItem> | ||
|
||
<TabItem value="seeds" label="Seeds"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
seeds: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name='seeds/properties.yml'> | ||
|
||
```yml | ||
seeds: | ||
- name: seed_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
|
||
</File> | ||
</TabItem> | ||
|
||
<TabItem value="snapshot" label="Snapshots"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
snapshots: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<VersionBlock firstVersion="1.9"> | ||
<File name='snapshots/properties.yml'> | ||
|
||
```yml | ||
snapshots: | ||
- name: snapshot_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
</File> | ||
</VersionBlock> | ||
|
||
<VersionBlock lastVersion="1.8"> | ||
|
||
<File name="models/modlename.sql"> | ||
|
||
```sql | ||
|
||
{{ config( | ||
event_time: 'my_time_field' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
|
||
import SnapshotYaml from '/snippets/_snapshot-yaml-spec.md'; | ||
|
||
<SnapshotYaml/> | ||
</VersionBlock> | ||
|
||
|
||
|
||
</TabItem> | ||
|
||
<TabItem value="sources" label="Sources"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
sources: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
sources: | ||
- name: source_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
|
||
</File> | ||
</TabItem> | ||
</Tabs> | ||
|
||
## Definition | ||
|
||
Set the `event_time` to the name of the field that represents the timestamp of the event, as opposed to a date like data loading date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block. | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
`event_time` is required for [Incremental mircrobatch](/docs/build/incremental-microbatch) and [Advanced CI compare changes](/docs/deploy/advanced-ci) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. | ||
|
||
When you configure `event_time`, it enables compare changes to: | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
- Compare data in CI versus production for overlapping times only, reducing false discrepancies. | ||
- Handle scenarios where CI has "fresher" data than production by using only the overlapping timeframe, allowing you to avoid incorrect row-count changes. | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
- Account for subset data builds in CI without flagging filtered-out rows as "deleted" when compared with production. | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
## Examples | ||
|
||
<Tabs> | ||
|
||
<TabItem value="model" label="Models"> | ||
|
||
Here's an example in the `dbt_project.yml` file: | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
models: | ||
my_project: | ||
user_sessions: | ||
+event_time: session_start_time | ||
``` | ||
</File> | ||
|
||
Example in a properties YAML file: | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
models: | ||
- name: user_sessions | ||
config: | ||
event_time: session_start_time | ||
``` | ||
|
||
</File> | ||
|
||
Example in sql model config block: | ||
|
||
<File name="models/user_sessions.sql"> | ||
|
||
```sql | ||
{{ config( | ||
event_time='session_start_time' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
This setup sets `session_start_time` as the `event_time` for the `user_sessions` model, which makes sure the compare changes process uses this timestamp for time-slice comparisons or incremental microbatching. | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
</TabItem> | ||
|
||
<TabItem value="seeds" label="Seeds"> | ||
|
||
Here's an example in the `dbt_project.yml` file: | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
seeds: | ||
my_project: | ||
my_seed: | ||
+event_time: record_timestamp | ||
``` | ||
|
||
</File> | ||
|
||
Example in a seed properties YAML: | ||
|
||
<File name='seeds/properties.yml'> | ||
|
||
```yml | ||
seeds: | ||
- name: my_seed | ||
config: | ||
event_time: record_timestamp | ||
``` | ||
</File> | ||
|
||
This setup sets `record_timestamp` as the `event_time` for `my_seed`. This ensures that the `record_timestamp` is used consistently for compare changes processes or incremental microbatching. | ||
mirnawong1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
</TabItem> | ||
<TabItem value="sources" label="Sources"> | ||
|
||
Here's an example of source properties YAML file: | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
sources: | ||
- name: source_name | ||
tables: | ||
- name: table_name | ||
config: | ||
event_time: event_timestamp | ||
``` | ||
</File> | ||
|
||
This setup sets `event_timestamp` as the `event_time` for the specified source table. | ||
|
||
</TabItem> | ||
</Tabs> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.