-
Notifications
You must be signed in to change notification settings - Fork 1.1k
add event_time page #6383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
add event_time page #6383
Changes from 48 commits
Commits
Show all changes
71 commits
Select commit
Hold shift + click to select a range
46231b0
add event_time page
mirnawong1 33a66a8
Merge branch 'current' into add-event-time
mirnawong1 6ebf5eb
update source/snapshots
mirnawong1 501d948
add to model
mirnawong1 4f2c6dc
add img and rn
mirnawong1 57ee608
fix link
mirnawong1 451fc46
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3354c9d
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 603c21c
fix link again
mirnawong1 0c68f62
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 488460c
Update event-time.md
mirnawong1 2fb62c5
Update release-notes.md
mirnawong1 69ba339
Update event-time.md
mirnawong1 1ebbbdb
Update advanced-ci.md
mirnawong1 2b713ee
Update advanced-ci.md
mirnawong1 c789601
Update advanced-ci.md
mirnawong1 5708119
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 903c5d1
Merge branch 'current' into add-event-time
mirnawong1 2dd873a
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 b7a07be
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 12cdffa
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 016c555
Update event-time.md
mirnawong1 9c49664
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 2910914
Merge branch 'current' into add-event-time
mirnawong1 5ba059e
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 79128fe
Merge branch 'current' into add-event-time
mirnawong1 cc34575
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 551821d
update img
mirnawong1 735ae38
fix img size
mirnawong1 ac7616b
Merge branch 'current' into add-event-time
mirnawong1 bdc037e
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 0363051
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 d693c9b
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 809f2a7
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 81e2318
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 14632b3
Merge branch 'current' into add-event-time
mirnawong1 aad3987
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 e92c9db
Update website/docs/reference/source-configs.md
mirnawong1 2b98454
Merge branch 'current' into add-event-time
mirnawong1 3ad1bb6
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3da521f
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 edd1123
add scenarios
mirnawong1 52c0db9
add scenarios
mirnawong1 f461ffa
fold in grace's feedback
mirnawong1 a4f3b23
Merge branch 'current' into add-event-time
mirnawong1 a1c8166
Merge branch 'add-event-time' of github.com:dbt-labs/docs.getdbt.com …
mirnawong1 f1969f4
remove redundant
mirnawong1 c170a3b
Merge branch 'current' into add-event-time
mirnawong1 d6a309b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 3a8dee5
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 5851c2b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 0656327
Update release-notes.md
mirnawong1 57679b2
Merge branch 'current' into add-event-time
mirnawong1 556249a
Update website/docs/docs/dbt-versions/release-notes.md
mirnawong1 0bd8584
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 bd233ad
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 b9e4be0
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 ff3416a
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 613f1ef
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 85f181d
Update website/docs/reference/resource-configs/event-time.md
mirnawong1 4644684
Merge branch 'current' into add-event-time
mirnawong1 8cf073b
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 46763d8
Merge branch 'current' into add-event-time
mirnawong1 d2bf5af
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 6015dee
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 4250c9d
Update website/docs/docs/deploy/advanced-ci.md
mirnawong1 76b12e9
update header adn link
mirnawong1 de8f752
Merge branch 'current' into add-event-time
mirnawong1 4b28bbc
Merge branch 'add-event-time' into update-sources-snapshots
mirnawong1 337248b
add event _time to sources/snapshots/models/seeds (#6384)
mirnawong1 0e16ca6
Update incremental-microbatch.md
mirnawong1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,284 @@ | ||
--- | ||
title: "event_time" | ||
id: "event-time" | ||
sidebar_label: "event_time" | ||
resource_types: [models, seeds, source] | ||
description: "dbt uses event_time to understand when an event occurred. When defined, event_time enables microbatch incremental models and more refined comparison of datasets during Advanced CI." | ||
datatype: string | ||
--- | ||
|
||
Available in dbt Cloud Versionless and dbt Core v1.9 and higher. | ||
Check warning on line 10 in website/docs/reference/resource-configs/event-time.md
|
||
|
||
<Tabs> | ||
<TabItem value="model" label="Models"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
models: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
models: | ||
- name: model_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name="models/modelname.sql"> | ||
|
||
```sql | ||
{{ config( | ||
event_time='my_time_field' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
</TabItem> | ||
|
||
<TabItem value="seeds" label="Seeds"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
seeds: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name='seeds/properties.yml'> | ||
|
||
```yml | ||
seeds: | ||
- name: seed_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
|
||
</File> | ||
</TabItem> | ||
|
||
<TabItem value="snapshot" label="Snapshots"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
snapshots: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<VersionBlock firstVersion="1.9"> | ||
<File name='snapshots/properties.yml'> | ||
|
||
```yml | ||
snapshots: | ||
- name: snapshot_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
</File> | ||
</VersionBlock> | ||
|
||
<VersionBlock lastVersion="1.8"> | ||
|
||
<File name="models/modlename.sql"> | ||
|
||
```sql | ||
|
||
{{ config( | ||
event_time: 'my_time_field' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
|
||
import SnapshotYaml from '/snippets/_snapshot-yaml-spec.md'; | ||
|
||
<SnapshotYaml/> | ||
</VersionBlock> | ||
|
||
|
||
|
||
</TabItem> | ||
|
||
<TabItem value="sources" label="Sources"> | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
sources: | ||
[resource-path:](/reference/resource-configs/resource-path) | ||
+event_time: my_time_field | ||
``` | ||
</File> | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
sources: | ||
- name: source_name | ||
[config](/reference/resource-properties/config): | ||
event_time: my_time_field | ||
``` | ||
|
||
</File> | ||
</TabItem> | ||
</Tabs> | ||
|
||
## Definition | ||
|
||
Set the `event_time` to the name of the field that represents the timestamp of the event -- "at what time did the row occur" -- as opposed to an event ingestion date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block. | ||
|
||
Here are some examples of good and bad `event_time` columns: | ||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- ✅ Good: | ||
- `account_created_at` — This represents the specific time when an account was created, making it a fixed event in time. | ||
- `session_began_at` — This captures the exact timestamp when a user session started, which won’t change and directly ties to the event. | ||
|
||
- ❌ Bad: | ||
|
||
- `_fivetran_synced` — This isn't the time that the event happened, it's the time that the event was ingested. | ||
- `last_updated_at` — This isn't a good use case as this will keep changing over time. | ||
|
||
`event_time` is required for [Incremental microbatch](/docs/build/incremental-microbatch) and highly recommended for [Advanced CI's compare changes](/docs/deploy/advanced-ci#speeding-up-comparisons) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. | ||
|
||
## Examples | ||
|
||
<Tabs> | ||
|
||
<TabItem value="model" label="Models"> | ||
|
||
Here's an example in the `dbt_project.yml` file: | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
models: | ||
my_project: | ||
user_sessions: | ||
+event_time: session_start_time | ||
``` | ||
</File> | ||
|
||
Example in a properties YAML file: | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
models: | ||
- name: user_sessions | ||
config: | ||
event_time: session_start_time | ||
``` | ||
|
||
</File> | ||
|
||
Example in sql model config block: | ||
|
||
<File name="models/user_sessions.sql"> | ||
|
||
```sql | ||
{{ config( | ||
event_time='session_start_time' | ||
) }} | ||
``` | ||
|
||
</File> | ||
|
||
This setup sets `session_start_time` as the `event_time` for the `user_sessions` model. | ||
</TabItem> | ||
|
||
<TabItem value="seeds" label="Seeds"> | ||
|
||
Here's an example in the `dbt_project.yml` file: | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
seeds: | ||
my_project: | ||
my_seed: | ||
+event_time: record_timestamp | ||
``` | ||
|
||
</File> | ||
|
||
Example in a seed properties YAML: | ||
|
||
<File name='seeds/properties.yml'> | ||
|
||
```yml | ||
seeds: | ||
- name: my_seed | ||
config: | ||
event_time: record_timestamp | ||
``` | ||
</File> | ||
|
||
This setup sets `record_timestamp` as the `event_time` for `my_seed`. | ||
|
||
</TabItem> | ||
|
||
<TabItem value="snapshot" label="Snapshots"> | ||
|
||
Here's an example in the `dbt_project.yml` file: | ||
|
||
<File name='dbt_project.yml'> | ||
|
||
```yml | ||
snapshots: | ||
my_project: | ||
my_snapshot: | ||
+event_time: record_timestamp | ||
``` | ||
|
||
</File> | ||
|
||
Example in a snapshot properties YAML: | ||
|
||
<File name='my_project/properties.yml'> | ||
|
||
```yml | ||
snapshots: | ||
- name: my_snapshot | ||
config: | ||
event_time: record_timestamp | ||
``` | ||
</File> | ||
|
||
This setup sets `record_timestamp` as the `event_time` for `my_snapshot`. | ||
|
||
</TabItem> | ||
|
||
<TabItem value="sources" label="Sources"> | ||
|
||
Here's an example of source properties YAML file: | ||
|
||
<File name='models/properties.yml'> | ||
|
||
```yml | ||
sources: | ||
- name: source_name | ||
tables: | ||
- name: table_name | ||
config: | ||
event_time: event_timestamp | ||
``` | ||
</File> | ||
|
||
This setup sets `event_timestamp` as the `event_time` for the specified source table. | ||
|
||
</TabItem> | ||
</Tabs> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.