Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare active_users_aggregates for a backfill with shredder mitigation. #6349

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Prepare active_users_aggregates for a backfill with shredder mitigation. #6349

wants to merge 14 commits into from

Conversation

lucia-vargas-a
Copy link
Contributor

@lucia-vargas-a lucia-vargas-a commented Oct 15, 2024 •
edited
Loading

Description

This PR prepares active_users_aggregates for a backfill with shredder mitigation, which includes renaming columns to cascade upstream changes.

The full backfill from 2018-01-01 until 2024-11-10 is available in moz-fx-data-shared-prod.backfills_staging_derived.firefox_desktop_derived__active_users_aggregates_v4_2024_11_15.

Related Tickets & Documents

Reviewer, please follow this checklist

┆Issue is synchronized with this Jira Task

@lucia-vargas-a
Prepare active_users_aggregates for a backfill with shredder mitigati…
10c0ddc 
…on. Rename columns [first_seen_date, os_version, segment] to cascade upstream changes during the backfill for stable numbers.
@lucia-vargas-a
Update CODEOWNERS with the new version.
1120069
@lucia-vargas-a lucia-vargas-a requested a review from a team as a code owner October 15, 2024 14:29
@lucia-vargas-a
Merge branch 'main' into DENG-4060_prepare_desktop_aua_for_backfill_w…
f4889ca 
…ith_shredder_mitigation
@lucia-vargas-a
Avoid modifying the active_users_aggregates view until the new versio…
bc10f22 
…n is backfilled. Remove the view generation from the new version which conflicts with previous version.
@lucia-vargas-a
Generate DAG with version.
fef2279
@dataops-ci-bot

This comment has been minimized.

Sign in to view
@dataops-ci-bot

This comment has been minimized.

Sign in to view
@dataops-ci-bot

This comment has been minimized.

Sign in to view
bochocki
bochocki reviewed Oct 18, 2024
View reviewed changes
Copy link
Contributor

@bochocki bochocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some comments that I think are worth addressing, but it's possible that they're not very relevant. If we can address the comments this looks good to go.

sql_generators/active_users_aggregates_v3/templates/metadata.yaml Show resolved Hide resolved
sql_generators/active_users_aggregates_v4/__init__.py Show resolved Hide resolved
sql_generators/active_users_aggregates_v4/__init__.py Show resolved Hide resolved
sql_generators/active_users_aggregates_v4/__init__.py Show resolved Hide resolved
sql_generators/active_users_aggregates_v3/templates/mobile_schema.yaml Show resolved Hide resolved
sql_generators/active_users_aggregates_v4/templates/mobile_schema.yaml Show resolved Hide resolved
@dataops-ci-bot

This comment has been minimized.

Sign in to view
@dataops-ci-bot

This comment has been minimized.

Sign in to view
bochocki
bochocki approved these changes Nov 15, 2024
View reviewed changes
Copy link
Contributor

@bochocki bochocki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some very minor nits. Looks great though, thanks Lucia!

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql
@@ -7,7 +7,7 @@ WITH todays_metrics AS (
app_version AS app_version,
normalized_channel AS channel,
IFNULL(country, '??') country,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpicky, but for consistency:

Suggested change
IFNULL(country, '??') country,
IFNULL(country, '??') AS country,

sql_generators/active_users_aggregates_v3/templates/desktop_query.sql
@@ -7,7 +7,7 @@ WITH todays_metrics AS (
app_version AS app_version,
normalized_channel AS channel,
IFNULL(country, '??') country,
city,
IFNULL(city, '??') city,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
IFNULL(city, '??') city,
IFNULL(city, '??') AS city,

@dataops-ci-bot
Copy link

Integration report for "Merge branch 'main' into DENG-4060_prepare_desktop_aua_for_backfill_with_shredder_mitigation"

sql.diff

Click to expand! 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py	2024-11-15 17:17:36.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py	2024-11-15 17:31:40.000000000 +0000
@@ -455,6 +455,23 @@
             checks__fail_fenix_derived__active_users_aggregates__v3
         )
 
+    checks__fail_fenix_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_fenix_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__fail_firefox_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_firefox_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -486,6 +503,23 @@
             checks__fail_firefox_ios_derived__active_users_aggregates__v3
         )
 
+    checks__fail_firefox_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_firefox_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="firefox_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__fail_focus_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_focus_android_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -517,6 +551,23 @@
             checks__fail_focus_android_derived__active_users_aggregates__v3
         )
 
+    checks__fail_focus_android_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_focus_android_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__fail_focus_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_focus_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -548,6 +599,23 @@
             checks__fail_focus_ios_derived__active_users_aggregates__v3
         )
 
+    checks__fail_focus_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_focus_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__fail_klar_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_klar_android_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -579,6 +647,23 @@
             checks__fail_klar_android_derived__active_users_aggregates__v3
         )
 
+    checks__fail_klar_android_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_klar_android_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__fail_klar_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__fail_klar_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -610,6 +695,23 @@
             checks__fail_klar_ios_derived__active_users_aggregates__v3
         )
 
+    checks__fail_klar_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__fail_klar_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_fenix_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_fenix_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -627,6 +729,23 @@
         retries=0,
     )
 
+    checks__warn_fenix_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_fenix_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_firefox_desktop_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -644,6 +763,23 @@
         retries=0,
     )
 
+    checks__warn_firefox_desktop_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_firefox_desktop_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="firefox_desktop_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_firefox_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_firefox_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -661,6 +797,23 @@
         retries=0,
     )
 
+    checks__warn_firefox_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_firefox_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="firefox_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_focus_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_focus_android_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -678,6 +831,23 @@
         retries=0,
     )
 
+    checks__warn_focus_android_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_focus_android_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_focus_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_focus_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -695,6 +865,23 @@
         retries=0,
     )
 
+    checks__warn_focus_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_focus_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_klar_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_klar_android_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -712,6 +899,23 @@
         retries=0,
     )
 
+    checks__warn_klar_android_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_klar_android_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
     checks__warn_klar_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
         task_id="checks__warn_klar_ios_derived__active_users_aggregates__v3",
         source_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
@@ -729,8 +933,25 @@
         retries=0,
     )
 
-    fenix_active_users_aggregates = bigquery_etl_query(
-        task_id="fenix_active_users_aggregates",
+    checks__warn_klar_ios_derived__active_users_aggregates__v4 = bigquery_dq_check(
+        task_id="checks__warn_klar_ios_derived__active_users_aggregates__v4",
+        source_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=False,
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+        retries=0,
+    )
+
+    fenix_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="fenix_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="fenix_derived",
         project_id="moz-fx-data-shared-prod",
@@ -745,8 +966,24 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
-    firefox_desktop_active_users_aggregates = bigquery_etl_query(
-        task_id="firefox_desktop_active_users_aggregates",
+    fenix_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="fenix_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    firefox_desktop_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="firefox_desktop_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_desktop_derived",
         project_id="moz-fx-data-shared-prod",
@@ -762,21 +999,37 @@
     )
 
     with TaskGroup(
-        "firefox_desktop_active_users_aggregates_external",
-    ) as firefox_desktop_active_users_aggregates_external:
+        "firefox_desktop_active_users_aggregates_v3_external",
+    ) as firefox_desktop_active_users_aggregates_v3_external:
         ExternalTaskMarker(
-            task_id="bqetl_dynamic_dau__wait_for_firefox_desktop_active_users_aggregates",
+            task_id="bqetl_dynamic_dau__wait_for_firefox_desktop_active_users_aggregates_v3",
             external_dag_id="bqetl_dynamic_dau",
-            external_task_id="wait_for_firefox_desktop_active_users_aggregates",
+            external_task_id="wait_for_firefox_desktop_active_users_aggregates_v3",
             execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=51300)).isoformat() }}",
         )
 
-        firefox_desktop_active_users_aggregates_external.set_upstream(
-            firefox_desktop_active_users_aggregates
+        firefox_desktop_active_users_aggregates_v3_external.set_upstream(
+            firefox_desktop_active_users_aggregates_v3
         )
 
-    firefox_ios_active_users_aggregates = bigquery_etl_query(
-        task_id="firefox_ios_active_users_aggregates",
+    firefox_desktop_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="firefox_desktop_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="firefox_desktop_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    firefox_ios_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="firefox_ios_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="firefox_ios_derived",
         project_id="moz-fx-data-shared-prod",
@@ -791,8 +1044,24 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
-    focus_android_active_users_aggregates = bigquery_etl_query(
-        task_id="focus_android_active_users_aggregates",
+    firefox_ios_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="firefox_ios_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="firefox_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    focus_android_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="focus_android_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_android_derived",
         project_id="moz-fx-data-shared-prod",
@@ -807,8 +1076,24 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
-    focus_ios_active_users_aggregates = bigquery_etl_query(
-        task_id="focus_ios_active_users_aggregates",
+    focus_android_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="focus_android_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    focus_ios_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="focus_ios_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="focus_ios_derived",
         project_id="moz-fx-data-shared-prod",
@@ -823,8 +1108,24 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
-    klar_android_active_users_aggregates = bigquery_etl_query(
-        task_id="klar_android_active_users_aggregates",
+    focus_ios_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="focus_ios_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="focus_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    klar_android_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="klar_android_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="klar_android_derived",
         project_id="moz-fx-data-shared-prod",
@@ -839,8 +1140,24 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
-    klar_ios_active_users_aggregates = bigquery_etl_query(
-        task_id="klar_ios_active_users_aggregates",
+    klar_android_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="klar_android_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_android_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
+    klar_ios_active_users_aggregates_v3 = bigquery_etl_query(
+        task_id="klar_ios_active_users_aggregates_v3",
         destination_table='active_users_aggregates_v3${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
         dataset_id="klar_ios_derived",
         project_id="moz-fx-data-shared-prod",
@@ -855,6 +1172,22 @@
         parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
     )
 
+    klar_ios_active_users_aggregates_v4 = bigquery_etl_query(
+        task_id="klar_ios_active_users_aggregates_v4",
+        destination_table='active_users_aggregates_v4${{ macros.ds_format(macros.ds_add(ds, -1), "%Y-%m-%d", "%Y%m%d") }}',
+        dataset_id="klar_ios_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=[
+            "[email protected]",
+            "[email protected]",
+            "[email protected]",
+        ],
+        date_partition_parameter=None,
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{macros.ds_add(ds, -1)}}"],
+    )
+
     telemetry_derived__cohort_daily_statistics__v1 = bigquery_etl_query(
         task_id="telemetry_derived__cohort_daily_statistics__v1",
         destination_table="cohort_daily_statistics_v1",
@@ -912,27 +1245,71 @@
     )
 
     checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        fenix_active_users_aggregates
+        fenix_active_users_aggregates_v3
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__fail_fenix_derived__active_users_aggregates__v4.set_upstream(
+        fenix_active_users_aggregates_v4
     )
 
     checks__fail_firefox_ios_derived__active_users_aggregates__v3.set_upstream(
-        firefox_ios_active_users_aggregates
+        firefox_ios_active_users_aggregates_v3
+    )
+
+    checks__fail_firefox_ios_derived__active_users_aggregates__v4.set_upstream(
+        firefox_ios_active_users_aggregates_v4
     )
 
     checks__fail_focus_android_derived__active_users_aggregates__v3.set_upstream(
-        focus_android_active_users_aggregates
+        focus_android_active_users_aggregates_v3
+    )
+
+    checks__fail_focus_android_derived__active_users_aggregates__v4.set_upstream(
+        focus_android_active_users_aggregates_v4
     )
 
     checks__fail_focus_ios_derived__active_users_aggregates__v3.set_upstream(
-        focus_ios_active_users_aggregates
+        focus_ios_active_users_aggregates_v3
+    )
+
+    checks__fail_focus_ios_derived__active_users_aggregates__v4.set_upstream(
+        focus_ios_active_users_aggregates_v4
     )
 
     checks__fail_klar_android_derived__active_users_aggregates__v3.set_upstream(
-        klar_android_active_users_aggregates
+        klar_android_active_users_aggregates_v3
+    )
+
+    checks__fail_klar_android_derived__active_users_aggregates__v4.set_upstream(
+        klar_android_active_users_aggregates_v4
     )
 
     checks__fail_klar_ios_derived__active_users_aggregates__v3.set_upstream(
-        klar_ios_active_users_aggregates
+        klar_ios_active_users_aggregates_v3
+    )
+
+    checks__fail_klar_ios_derived__active_users_aggregates__v4.set_upstream(
+        klar_ios_active_users_aggregates_v4
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
@@ -956,7 +1333,31 @@
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        fenix_active_users_aggregates
+        fenix_active_users_aggregates_v3
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+    )
+
+    checks__warn_fenix_derived__active_users_aggregates__v4.set_upstream(
+        fenix_active_users_aggregates_v4
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
@@ -964,126 +1365,254 @@
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
-        firefox_desktop_active_users_aggregates
+        firefox_desktop_active_users_aggregates_v3
+    )
+
+    checks__warn_firefox_desktop_derived__active_users_aggregates__v4.set_upstream(
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+    )
+
+    checks__warn_firefox_desktop_derived__active_users_aggregates__v4.set_upstream(
+        firefox_desktop_active_users_aggregates_v4
     )
 
     checks__warn_firefox_ios_derived__active_users_aggregates__v3.set_upstream(
-        firefox_ios_active_users_aggregates
+        firefox_ios_active_users_aggregates_v3
+    )
+
+    checks__warn_firefox_ios_derived__active_users_aggregates__v4.set_upstream(
+        firefox_ios_active_users_aggregates_v4
     )
 
     checks__warn_focus_android_derived__active_users_aggregates__v3.set_upstream(
-        focus_android_active_users_aggregates
+        focus_android_active_users_aggregates_v3
+    )
+
+    checks__warn_focus_android_derived__active_users_aggregates__v4.set_upstream(
+        focus_android_active_users_aggregates_v4
     )
 
     checks__warn_focus_ios_derived__active_users_aggregates__v3.set_upstream(
-        focus_ios_active_users_aggregates
+        focus_ios_active_users_aggregates_v3
+    )
+
+    checks__warn_focus_ios_derived__active_users_aggregates__v4.set_upstream(
+        focus_ios_active_users_aggregates_v4
     )
 
     checks__warn_klar_android_derived__active_users_aggregates__v3.set_upstream(
-        klar_android_active_users_aggregates
+        klar_android_active_users_aggregates_v3
+    )
+
+    checks__warn_klar_android_derived__active_users_aggregates__v4.set_upstream(
+        klar_android_active_users_aggregates_v4
     )
 
     checks__warn_klar_ios_derived__active_users_aggregates__v3.set_upstream(
-        klar_ios_active_users_aggregates
+        klar_ios_active_users_aggregates_v3
+    )
+
+    checks__warn_klar_ios_derived__active_users_aggregates__v4.set_upstream(
+        klar_ios_active_users_aggregates_v4
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
     )
 
-    fenix_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v3.set_upstream(
         wait_for_fenix_derived__metrics_clients_last_seen__v1
     )
 
-    firefox_desktop_active_users_aggregates.set_upstream(
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+    )
+
+    fenix_active_users_aggregates_v4.set_upstream(
+        wait_for_fenix_derived__metrics_clients_last_seen__v1
+    )
+
+    firefox_desktop_active_users_aggregates_v3.set_upstream(
+        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+    )
+
+    firefox_desktop_active_users_aggregates_v4.set_upstream(
         wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1
     )
 
-    firefox_ios_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v3.set_upstream(
         wait_for_firefox_ios_derived__metrics_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_firefox_ios_derived__clients_activation__v1
+    )
+
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+    )
+
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_ios_fennec_derived__baseline_clients_last_seen__v1
+    )
+
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_ios_firefox_derived__baseline_clients_last_seen__v1
+    )
+
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_ios_firefoxbeta_derived__baseline_clients_last_seen__v1
+    )
+
+    firefox_ios_active_users_aggregates_v4.set_upstream(
+        wait_for_firefox_ios_derived__metrics_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v3.set_upstream(
+        wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v3.set_upstream(
+        wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v3.set_upstream(
+        wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v3.set_upstream(
+        wait_for_focus_android_derived__metrics_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v3.set_upstream(
+        wait_for_telemetry_derived__core_clients_last_seen__v1
+    )
+
+    focus_android_active_users_aggregates_v4.set_upstream(
         wait_for_checks__fail_org_mozilla_focus_beta_derived__baseline_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
+    focus_android_active_users_aggregates_v4.set_upstream(
         wait_for_checks__fail_org_mozilla_focus_derived__baseline_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
+    focus_android_active_users_aggregates_v4.set_upstream(
         wait_for_checks__fail_org_mozilla_focus_nightly_derived__baseline_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
+    focus_android_active_users_aggregates_v4.set_upstream(
         wait_for_focus_android_derived__metrics_clients_last_seen__v1
     )
 
-    focus_android_active_users_aggregates.set_upstream(
+    focus_android_active_users_aggregates_v4.set_upstream(
         wait_for_telemetry_derived__core_clients_last_seen__v1
     )
 
-    focus_ios_active_users_aggregates.set_upstream(
+    focus_ios_active_users_aggregates_v3.set_upstream(
+        wait_for_bigeye__org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1
+    )
+
+    focus_ios_active_users_aggregates_v3.set_upstream(
+        wait_for_focus_ios_derived__metrics_clients_last_seen__v1
+    )
+
+    focus_ios_active_users_aggregates_v4.set_upstream(
         wait_for_bigeye__org_mozilla_ios_focus_derived__baseline_clients_last_seen__v1
     )
 
-    focus_ios_active_users_aggregates.set_upstream(
+    focus_ios_active_users_aggregates_v4.set_upstream(
         wait_for_focus_ios_derived__metrics_clients_last_seen__v1
     )
 
-    klar_android_active_users_aggregates.set_upstream(
+    klar_android_active_users_aggregates_v3.set_upstream(
         wait_for_checks__fail_org_mozilla_klar_derived__baseline_clients_last_seen__v1
     )
 
-    klar_android_active_users_aggregates.set_upstream(
+    klar_android_active_users_aggregates_v3.set_upstream(
         wait_for_klar_android_derived__metrics_clients_last_seen__v1
     )
 
-    klar_ios_active_users_aggregates.set_upstream(
+    klar_android_active_users_aggregates_v4.set_upstream(
+        wait_for_checks__fail_org_mozilla_klar_derived__baseline_clients_last_seen__v1
+    )
+
+    klar_android_active_users_aggregates_v4.set_upstream(
+        wait_for_klar_android_derived__metrics_clients_last_seen__v1
+    )
+
+    klar_ios_active_users_aggregates_v3.set_upstream(
+        wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1
+    )
+
+    klar_ios_active_users_aggregates_v3.set_upstream(
+        wait_for_klar_ios_derived__metrics_clients_last_seen__v1
+    )
+
+    klar_ios_active_users_aggregates_v4.set_upstream(
         wait_for_checks__fail_org_mozilla_ios_klar_derived__baseline_clients_last_seen__v1
     )
 
-    klar_ios_active_users_aggregates.set_upstream(
+    klar_ios_active_users_aggregates_v4.set_upstream(
         wait_for_klar_ios_derived__metrics_clients_last_seen__v1
     )
 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_dynamic_dau.py /tmp/workspace/generated-sql/dags/bqetl_dynamic_dau.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_dynamic_dau.py	2024-11-15 17:17:36.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_dynamic_dau.py	2024-11-15 17:31:42.000000000 +0000
@@ -126,10 +126,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_firefox_desktop_active_users_aggregates = ExternalTaskSensor(
-        task_id="wait_for_firefox_desktop_active_users_aggregates",
+    wait_for_firefox_desktop_active_users_aggregates_v3 = ExternalTaskSensor(
+        task_id="wait_for_firefox_desktop_active_users_aggregates_v3",
         external_dag_id="bqetl_analytics_aggregations",
-        external_task_id="firefox_desktop_active_users_aggregates",
+        external_task_id="firefox_desktop_active_users_aggregates_v3",
         execution_delta=datetime.timedelta(seconds=35100),
         check_existence=True,
         mode="reschedule",
@@ -194,5 +194,5 @@
     )
 
     telemetry_derived__segmented_dau_28_day_rolling__v1.set_upstream(
-        wait_for_firefox_desktop_active_users_aggregates
+        wait_for_firefox_desktop_active_users_aggregates_v3
     )
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_android_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_ios_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_android_derived: active_users_aggregates_v4
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_ios_derived: active_users_aggregates_v4
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-11-15 17:12:01.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-11-15 17:21:25.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: source
+  mode: NULLABLE
+- name: source
   type: STRING
-- mode: NULLABLE
-  name: event_type
+  mode: NULLABLE
+- name: event_type
   type: STRING
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: subdivision1
+  mode: NULLABLE
+- name: subdivision1
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: event_count
+  mode: NULLABLE
+- name: event_count
   type: INTEGER
-- mode: NULLABLE
-  name: user_count
+  mode: NULLABLE
+- name: user_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-11-15 17:12:01.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-11-15 17:21:19.000000000 +0000
@@ -1,40 +1,40 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: impression_count
+  mode: NULLABLE
+- name: impression_count
   type: INTEGER
-- mode: NULLABLE
-  name: click_count
+  mode: NULLABLE
+- name: click_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/active_users_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/active_users_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/active_users_aggregates/schema.yaml	2024-11-15 17:12:43.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/active_users_aggregates/schema.yaml	2024-11-15 17:21:54.000000000 +0000
@@ -2,81 +2,115 @@
 - name: segment
   type: STRING
   mode: NULLABLE
+  description: Classification of client_ids based on usage and active state.
 - name: attribution_medium
   type: STRING
   mode: NULLABLE
+  description: The utm_medium this install is attributed to. Reported by the install
+    referrer service, not Adjust.
 - name: attribution_source
   type: STRING
   mode: NULLABLE
+  description: The utm_term this install is attributed to. Reported by the install
+    referrer service, not Adjust.
 - name: attributed
   type: BOOLEAN
   mode: NULLABLE
+  description: True if the attribution source and medium are present.
 - name: city
   type: STRING
   mode: NULLABLE
+  description: City reported by the client.
 - name: country
   type: STRING
   mode: NULLABLE
+  description: Country reported by the client.
 - name: distribution_id
   type: STRING
   mode: NULLABLE
+  description: A string containing the distribution identifier. This was used to identify
+    installs from Mozilla Online, but now also identifies partnership deal distributions.
 - name: first_seen_year
   type: INTEGER
   mode: NULLABLE
+  description: Year extracted from the first_seen_date, that corresponds to the date
+    when the first ping was received.
 - name: is_default_browser
   type: BOOLEAN
   mode: NULLABLE
+  description: Whether the browser is set as the default browser on the client side.
 - name: locale
   type: STRING
   mode: NULLABLE
+  description: Locale reported by the client, which is a combination of language and
+    regional settings.
 - name: channel
   type: STRING
   mode: NULLABLE
+  description: Browser installation channel installed on the client.
 - name: os
   type: STRING
   mode: NULLABLE
+  description: Operating system reported by the client.
 - name: os_version
   type: STRING
   mode: NULLABLE
+  description: OS version reported by the client.
 - name: os_version_major
   type: INTEGER
   mode: NULLABLE
+  description: Major or first part of the OS version reported by the client.
 - name: os_version_minor
   type: INTEGER
   mode: NULLABLE
+  description: Minor or second part of the OS version reported by the client.
 - name: submission_date
   type: DATE
   mode: NULLABLE
+  description: Date when ping is received on the server side.
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+  description: The source of a client installation.
 - name: install_source
   type: STRING
   mode: NULLABLE
+  description: The id of the browser distribution made available in installation sources.
 - name: daily_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who report a ping in a day.
 - name: weekly_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 7 days.
 - name: monthly_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 28 days.
 - name: dau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who reported a ping on the submission_date that qualify
+    as active.
 - name: wau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 7 days and qualify
+    as active.
 - name: mau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 28 days and qualify
+    as active.
 - name: app_name
   type: STRING
   mode: NULLABLE
+  description: Browser name.
 - name: app_version
   type: STRING
   mode: NULLABLE
+  description: Browser version installed on the client.
 - name: app_version_major
   type: NUMERIC
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-11-15 17:12:01.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-11-15 17:21:49.000000000 +0000
@@ -26,6 +26,9 @@
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: retained_week_2
   type: BOOLEAN
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-11-15 17:12:01.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-11-15 17:21:56.000000000 +0000
@@ -48,6 +48,10 @@
   description: 'The type of source of a client installation.
 
     '
+- name: install_source
+  type: STRING
+  mode: NULLABLE
+  description: null
 - name: new_profiles
   type: INTEGER
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml	2024-11-15 17:12:44.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/metadata.yaml	2024-11-15 17:26:33.000000000 +0000
@@ -14,6 +14,10 @@
   The table is labeled as "change_controlled", which implies
   that changes require the approval of at least one owner.
 
+  The label "shredder mitigation" indicates that this table is set up for
+  managed backfill with shredder mitigation, as described in
+  https://mozilla.github.io/bigquery-etl/cookbooks/creating_a_derived_dataset/#initiating-the-backfill.
+
   Proposal:
   https://docs.google.com/document/d/1qvWO49Lr_Z_WErh3I3058A3B1YuiuURx19K3aTdmejM/edit?usp=sharing
 owners:
@@ -22,11 +26,12 @@
 labels:
   incremental: true
   change_controlled: true
+  shredder_mitigation: true
   dag: bqetl_analytics_aggregations
   owner1: lvargas
 scheduling:
   dag_name: bqetl_analytics_aggregations
-  task_name: fenix_active_users_aggregates
+  task_name: fenix_active_users_aggregates_v3
   date_partition_offset: -1
 bigquery:
   time_partitioning:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql	2024-11-15 17:12:44.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/query.sql	2024-11-15 17:27:54.000000000 +0000
@@ -180,4 +180,23 @@
 FROM
   todays_metrics
 GROUP BY
-  ALL
+  segment,
+  app_version,
+  attribution_medium,
+  attribution_source,
+  attributed,
+  city,
+  country,
+  distribution_id,
+  first_seen_year,
+  is_default_browser,
+  locale,
+  app_name,
+  channel,
+  os,
+  os_version,
+  os_version_major,
+  os_version_minor,
+  submission_date,
+  adjust_network,
+  install_source
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/schema.yaml	2024-11-15 17:12:44.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v3/schema.yaml	2024-11-15 17:20:59.000000000 +0000
@@ -2,84 +2,112 @@
 - name: segment
   type: STRING
   mode: NULLABLE
-- name: app_version
-  type: STRING
-  mode: NULLABLE
-- name: attribution_medium
+  description: Classification of client_ids based on usage and active state.
+- name: app_name
   type: STRING
   mode: NULLABLE
-- name: attribution_source
+  description: Browser name.
+- name: app_version
   type: STRING
   mode: NULLABLE
-- name: attributed
-  type: BOOLEAN
-  mode: NULLABLE
-- name: city
+  description: Browser version installed on the client.
+- name: channel
   type: STRING
   mode: NULLABLE
+  description: Browser installation channel installed on the client.
 - name: country
   type: STRING
   mode: NULLABLE
-- name: distribution_id
+  description: Country reported by the client.
+- name: city
   type: STRING
   mode: NULLABLE
-- name: first_seen_year
-  type: INTEGER
-  mode: NULLABLE
-- name: is_default_browser
-  type: BOOLEAN
-  mode: NULLABLE
+  description: City reported by the client.
 - name: locale
   type: STRING
   mode: NULLABLE
-- name: app_name
-  type: STRING
-  mode: NULLABLE
-- name: channel
-  type: STRING
+  description: Locale reported by the client, which is a combination of language and regional settings.
+- name: first_seen_year
+  type: INTEGER
   mode: NULLABLE
+  description: Year extracted from the first_seen_date, that corresponds to the date when the first ping was received.
 - name: os
   type: STRING
   mode: NULLABLE
+  description: Operating system reported by the client.
 - name: os_version
   type: STRING
   mode: NULLABLE
+  description: OS version reported by the client.
 - name: os_version_major
   type: INTEGER
   mode: NULLABLE
+  description: Major or first part of the OS version reported by the client.
 - name: os_version_minor
   type: INTEGER
   mode: NULLABLE
+  description: Minor or second part of the OS version reported by the client.
 - name: submission_date
   type: DATE
   mode: NULLABLE
+  description: Date when ping is received on the server side.
+- name: is_default_browser
+  type: BOOLEAN
+  mode: NULLABLE
+  description: Whether the browser is set as the default browser on the client side.
+- name: distribution_id
+  type: STRING
+  mode: NULLABLE
+  description: A string containing the distribution identifier. This was used to identify installs from Mozilla Online, but now also identifies partnership deal distributions.
+- name: attribution_source
+  type: STRING
+  mode: NULLABLE
+  description: The utm_term this install is attributed to. Reported by the install referrer service, not Adjust.
+- name: attribution_medium
+  type: STRING
+  mode: NULLABLE
+  description: The utm_medium this install is attributed to. Reported by the install referrer service, not Adjust.
+- name: attributed
+  type: BOOLEAN
+  mode: NULLABLE
+  description: True if the attribution source and medium are present.
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+  description: The source of a client installation.
 - name: install_source
   type: STRING
   mode: NULLABLE
+  description: The id of the browser distribution made available in installation sources.
 - name: daily_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who report a ping in a day.
 - name: weekly_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 7 days.
 - name: monthly_users
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 28 days.
 - name: dau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who reported a ping on the submission_date that qualify as active.
 - name: wau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 7 days and qualify as active.
 - name: mau
   type: INTEGER
   mode: NULLABLE
+  description: Count of users who have reported a ping over the last 28 days and qualify as active.
 - name: uri_count
   type: INTEGER
   mode: NULLABLE
+  description: Count of uri.
 - name: active_hours
   type: FLOAT64
   mode: NULLABLE
+  description: Count of active hours.
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/checks.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/checks.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/checks.sql	2024-11-15 17:27:55.000000000 +0000
@@ -0,0 +1,152 @@
+
+#warn
+WITH daily_users_sum AS (
+  SELECT
+    SUM(daily_users),
+  FROM
+    `{{ project_id }}.{{ dataset_id }}.{{ table_name }}`
+  WHERE
+    submission_date = @submission_date
+),
+distinct_client_count_nightly_base AS (
+  SELECT
+    client_info.client_id,
+    "nightly" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_fenix_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info("org_mozilla_fenix", client_info.app_build).channel = "nightly"
+    -- NOTE: The next two tables `org_mozilla_fenix_nightly_live.baseline_v1` and `org_mozilla_fennec_aurora_live.baseline_v1`
+    -- are not used as application IDs in Glean, but are also not yet marked as deprecated because they still count for KPIs:
+    -- Related PR https://github.com/mozilla/probe-scraper/pull/640.
+  UNION ALL
+  SELECT
+    client_info.client_id,
+    "nightly" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_fenix_nightly_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info(
+      "org_mozilla_fenix_nightly",
+      client_info.app_build
+    ).channel = "nightly"
+  UNION ALL
+  SELECT
+    client_info.client_id,
+    "nightly" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_fennec_aurora_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info(
+      "org_mozilla_fennec_aurora",
+      client_info.app_build
+    ).channel = "nightly"
+),
+distinct_client_count_base AS (
+    -- release channel
+  SELECT
+    client_info.client_id,
+    "release" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_firefox_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info("org_mozilla_firefox", client_info.app_build).channel = "release"
+    -- beta channel
+  UNION ALL
+  SELECT
+    client_info.client_id,
+    "beta" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_firefox_beta_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info(
+      "org_mozilla_firefox_beta",
+      client_info.app_build
+    ).channel = "beta"
+    -- NOTE: nightly table also contains some entries considered to be "beta" channel by our ETL
+    -- this is why the below entries are included here.
+  UNION ALL
+  SELECT
+    client_info.client_id,
+    "beta" AS channel,
+  FROM
+    `moz-fx-data-shared-prod.org_mozilla_fenix_live.baseline_v1`
+  WHERE
+    DATE(submission_timestamp) = @submission_date
+    AND mozfun.norm.fenix_app_info("org_mozilla_fenix", client_info.app_build).channel = "beta"
+    -- nightly channel
+  UNION ALL
+  SELECT
+    client_id,
+    channel
+  FROM
+    distinct_client_count_nightly_base
+  LEFT JOIN
+    `moz-fx-data-shared-prod.fenix.baseline_clients_last_seen` AS baseline_clients_last_seen
+    USING (client_id)
+  WHERE
+    baseline_clients_last_seen.submission_date = @submission_date
+    AND baseline_clients_last_seen.days_since_seen = 0
+),
+distinct_client_counts_per_channel AS (
+  SELECT
+    channel,
+    COUNT(DISTINCT client_id) AS distinct_client_count,
+  FROM
+    distinct_client_count_base
+  GROUP BY
+    channel
+),
+distinct_client_count AS (
+  SELECT
+    SUM(distinct_client_count),
+  FROM
+    distinct_client_counts_per_channel
+)
+SELECT
+  IF(
+    ABS((SELECT * FROM daily_users_sum) - (SELECT * FROM distinct_client_count)) > 10,
+    ERROR(
+      CONCAT(
+        "Daily_users mismatch between the firefox_ios live (`org_mozilla_firefox_live`, `org_mozilla_fenix_live.baseline_v1`,`org_mozilla_firefox_beta_live.baseline_v1`,`org_mozilla_fenix_nightly_live.baseline_v1`, `org_mozilla_fennec_aurora_live.baseline_v1`) and active_users_aggregates (`{{ dataset_id }}.{{ table_name }}`) tables is greater than 10.",
+        " Live table count: ",
+        (SELECT * FROM distinct_client_count),
+        " | active_users_aggregates (DAU): ",
+        (SELECT * FROM daily_users_sum),
+        " | Delta detected: ",
+        ABS((SELECT * FROM daily_users_sum) - (SELECT * FROM distinct_client_count))
+      )
+    ),
+    NULL
+  );
+
+#fail
+WITH dau_current AS (
+  SELECT
+    SUM(dau) AS dau
+  FROM
+    `{{ project_id }}.{{ dataset_id }}.{{ table_name }}`
+  WHERE
+    submission_date = @submission_date
+),
+dau_previous AS (
+  SELECT
+    SUM(dau) AS dau
+  FROM
+    `{{ project_id }}.{{ dataset_id }}.{{ table_name }}`
+  WHERE
+    submission_date = DATE_SUB(@submission_date, INTERVAL 1 DAY)
+)
+SELECT
+  IF(
+    ABS((SELECT SUM(dau) FROM dau_current) / (SELECT SUM(dau) FROM dau_previous)) > 1.5,
+    ERROR(
+      "Current date's DAU is 50% higher than in previous date. See source table (`{{ project_id }}.{{ dataset_id }}.{{ table_name }}`)!"
+    ),
+    NULL
+  );
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/active_users_aggregates_v4/metadata.yaml	2024-11-15 17:26:33.000000000 +0000
@@ -0,0 +1,65 @@
+friendly_name: Fenix Active Users Aggregates
+description: |-
+  This table contains dau, wau, mau, daily users,
+  weekly users and monthly users for Fenix,
+  aggregated by submission_date, attribution, channel,
+  country, city, device model, distribution_id, os details
+  and activity segment.
+
+  - dau is counting the users who reported a ping on the date and
+  are qualified as active users.
+  - daily_users counts all the users who reported a ping on the date.
+  Only dau is exposed in the view telemetry.active_users_aggregates.
+
+  The table is labeled as "change_controlled", which implies
+  that changes require the approval of at least one owner.
+
+  The label "shredder mitigation" indicates that this table is set up for
+  managed backfill with shredder mitigation, as described in
+  https://mozilla.github.io/bigquery-e

⚠️ Only part of the diff is displayed.

Link to full diff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bochocki bochocki bochocki approved these changes

@gkabbz gkabbz Awaiting requested review from gkabbz gkabbz is a code owner automatically assigned from mozilla/kpi_table_reviewers

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lucia-vargas-a @dataops-ci-bot @bochocki