initial add of query.sql WIP for ga downloads_with_attr_v3 #5397

Marlene-M-Hirose · 2024-04-19T00:27:48Z

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Follow the change control procedure

┆Issue is synchronized with this Jira Task

kwindau

would simplify the group by and exclude empty strings/ (not set) but besides that LGTM

…E after calculation CTE

…rsion, normalized_country_code

SuYoungHong · 2024-05-07T17:41:18Z

I reviewed, left some notes. Supplementary materials here: Downloads with Attribution V3 Review Doc

Actually, I would step back and think about what you want this table to be. I think we discussed it a bit last time. To me, there’s two choices:
1) Replication of downloads_with_attribution_v2
- Will need to be all records from the DL token logs (so 1 record per day per dl token) with the following conditions
- Matched via stub_session_id and GA client id
- Matched via GA client id only with only one session found
- Matched via GA client id only with multiple sessions found (one arbitrarily chosen for match)
- Has GA client id and/or stubsession_id but not match in GA data
- No GA info found so cannot match
2) A table of GA sessions per day
(use case, for marketing to use)
append the following metrics
- Has DL (from the GA side)
- Has DL token logs match
- Is there one, is there multiple?
- Has new profile
- Matched via DL token
- In case of multiple DL tokens or profiles, log that info
- Has returning user profile
- Ditto as above
Has wk4 retained
- Ditto as above

…4 sources. Note:date is set to '2024-02-14' for testing purposes. This will get changed when code is put into production

…4 sources. Note:date is set to '2024-05-24' for testing purposes. This will get changed when code is put into production

dataops-ci-bot · 2024-05-31T17:57:31Z

Integration report for "refactor query.sql to match downloads_with_attribution_v2, but use GA4 sources. Note:date is set to '2024-05-24' for testing purposes. This will get changed when code is put into production"

`sql.diff`

Click to expand!

Only in /tmp/workspace/main-generated-sql/dags/: bqetl_acoustic_suppression_list.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_braze.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_braze_currents.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_data_observability_test_data_copy.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_desktop_engagement_model.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_desktop_retention_model.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_ios_campaign_reporting.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_marketing_suppression_list.py
Only in /tmp/workspace/main-generated-sql/dags/: bqetl_pageload_v1.py
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_contact_export.py /tmp/workspace/generated-sql/dags/bqetl_acoustic_contact_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_contact_export.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_acoustic_contact_export.py	2024-05-31 17:56:38.000000000 +0000
@@ -22,7 +22,7 @@
 
 #### Owner
 
-leli@mozilla.com
+kik@mozilla.com
 
 #### Tags
 
@@ -32,10 +32,10 @@
 
 
 default_args = {
-    "owner": "leli@mozilla.com",
-    "start_date": datetime.datetime(2024, 4, 3, 0, 0),
+    "owner": "kik@mozilla.com",
+    "start_date": datetime.datetime(2021, 3, 1, 0, 0),
     "end_date": None,
-    "email": ["telemetry-alerts@mozilla.com", "leli@mozilla.com"],
+    "email": ["telemetry-alerts@mozilla.com", "kik@mozilla.com"],
     "depends_on_past": False,
     "retry_delay": datetime.timedelta(seconds=300),
     "email_on_failure": True,
@@ -65,44 +65,42 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    acoustic_derived__contact__v1 = bigquery_etl_query(
-        task_id="acoustic_derived__contact__v1",
+    acoustic__contact__v1 = bigquery_etl_query(
+        task_id="acoustic__contact__v1",
         destination_table="contact_v1",
-        dataset_id="acoustic_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com", "telemetry-alerts@mozilla.com"],
+        dataset_id="acoustic",
+        project_id="moz-fx-data-marketing-prod",
+        owner="kik@mozilla.com",
+        email=["kik@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter="submission_date",
         depends_on_past=False,
     )
 
-    acoustic_derived__contact_current_snapshot__v1 = bigquery_etl_query(
-        task_id="acoustic_derived__contact_current_snapshot__v1",
+    acoustic__contact_current_snapshot__v1 = bigquery_etl_query(
+        task_id="acoustic__contact_current_snapshot__v1",
         destination_table="contact_current_snapshot_v1",
-        dataset_id="acoustic_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com", "telemetry-alerts@mozilla.com"],
+        dataset_id="acoustic",
+        project_id="moz-fx-data-marketing-prod",
+        owner="kik@mozilla.com",
+        email=["kik@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter=None,
         depends_on_past=False,
         task_concurrency=1,
     )
 
-    acoustic_external__contact_raw__v1 = bigquery_etl_query(
-        task_id="acoustic_external__contact_raw__v1",
+    acoustic__contact_raw__v1 = bigquery_etl_query(
+        task_id="acoustic__contact_raw__v1",
         destination_table="contact_raw_v1",
-        dataset_id="acoustic_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com", "telemetry-alerts@mozilla.com"],
+        dataset_id="acoustic",
+        project_id="moz-fx-data-marketing-prod",
+        owner="kik@mozilla.com",
+        email=["kik@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter="submission_date",
         depends_on_past=False,
     )
 
-    acoustic_derived__contact__v1.set_upstream(acoustic_external__contact_raw__v1)
+    acoustic__contact__v1.set_upstream(acoustic__contact_raw__v1)
 
-    acoustic_derived__contact_current_snapshot__v1.set_upstream(
-        acoustic_derived__contact__v1
-    )
+    acoustic__contact_current_snapshot__v1.set_upstream(acoustic__contact__v1)
 
-    acoustic_external__contact_raw__v1.set_upstream(wait_for_fivetran_load_completed)
+    acoustic__contact_raw__v1.set_upstream(wait_for_fivetran_load_completed)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_raw_recipient_export.py /tmp/workspace/generated-sql/dags/bqetl_acoustic_raw_recipient_export.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_raw_recipient_export.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_acoustic_raw_recipient_export.py	2024-05-31 17:56:38.000000000 +0000
@@ -22,7 +22,7 @@
 
 #### Owner
 
-leli@mozilla.com
+kik@mozilla.com
 
 #### Tags
 
@@ -32,10 +32,10 @@
 
 
 default_args = {
-    "owner": "leli@mozilla.com",
-    "start_date": datetime.datetime(2024, 4, 3, 0, 0),
+    "owner": "kik@mozilla.com",
+    "start_date": datetime.datetime(2022, 3, 1, 0, 0),
     "end_date": None,
-    "email": ["telemetry-alerts@mozilla.com", "leli@mozilla.com"],
+    "email": ["telemetry-alerts@mozilla.com", "kik@mozilla.com"],
     "depends_on_past": False,
     "retry_delay": datetime.timedelta(seconds=300),
     "email_on_failure": True,
@@ -65,32 +65,28 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    acoustic_derived__raw_recipient__v1 = bigquery_etl_query(
-        task_id="acoustic_derived__raw_recipient__v1",
+    acoustic__raw_recipient__v1 = bigquery_etl_query(
+        task_id="acoustic__raw_recipient__v1",
         destination_table="raw_recipient_v1",
-        dataset_id="acoustic_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com", "telemetry-alerts@mozilla.com"],
+        dataset_id="acoustic",
+        project_id="moz-fx-data-marketing-prod",
+        owner="kik@mozilla.com",
+        email=["kik@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter="submission_date",
         depends_on_past=False,
     )
 
-    acoustic_external__raw_recipient_raw__v1 = bigquery_etl_query(
-        task_id="acoustic_external__raw_recipient_raw__v1",
+    acoustic__raw_recipient_raw__v1 = bigquery_etl_query(
+        task_id="acoustic__raw_recipient_raw__v1",
         destination_table="raw_recipient_raw_v1",
-        dataset_id="acoustic_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com", "telemetry-alerts@mozilla.com"],
+        dataset_id="acoustic",
+        project_id="moz-fx-data-marketing-prod",
+        owner="kik@mozilla.com",
+        email=["kik@mozilla.com", "telemetry-alerts@mozilla.com"],
         date_partition_parameter="submission_date",
         depends_on_past=False,
     )
 
-    acoustic_derived__raw_recipient__v1.set_upstream(
-        acoustic_external__raw_recipient_raw__v1
-    )
+    acoustic__raw_recipient__v1.set_upstream(acoustic__raw_recipient_raw__v1)
 
-    acoustic_external__raw_recipient_raw__v1.set_upstream(
-        wait_for_fivetran_load_completed
-    )
+    acoustic__raw_recipient_raw__v1.set_upstream(wait_for_fivetran_load_completed)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_suppression_list.py /tmp/workspace/generated-sql/dags/bqetl_acoustic_suppression_list.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_acoustic_suppression_list.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_acoustic_suppression_list.py	1970-01-01 00:00:00.000000000 +0000
@@ -1,78 +0,0 @@
-# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
-
-from airflow import DAG
-from airflow.sensors.external_task import ExternalTaskMarker
-from airflow.sensors.external_task import ExternalTaskSensor
-from airflow.utils.task_group import TaskGroup
-import datetime
-from operators.gcp_container_operator import GKEPodOperator
-from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, bigquery_dq_check
-
-docs = """
-### bqetl_acoustic_suppression_list
-
-Built from bigquery-etl repo, [`dags/bqetl_acoustic_suppression_list.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_acoustic_suppression_list.py)
-
-#### Description
-
-ETL for Acoustic suppression list.
-
-#### Owner
-
-leli@mozilla.com
-
-#### Tags
-
-* impact/tier_3
-* repo/bigquery-etl
-"""
-
-
-default_args = {
-    "owner": "leli@mozilla.com",
-    "start_date": datetime.datetime(2024, 4, 3, 0, 0),
-    "end_date": None,
-    "email": ["leli@mozilla.com"],
-    "depends_on_past": False,
-    "retry_delay": datetime.timedelta(seconds=1800),
-    "email_on_failure": True,
-    "email_on_retry": True,
-    "retries": 1,
-}
-
-tags = ["impact/tier_3", "repo/bigquery-etl"]
-
-with DAG(
-    "bqetl_acoustic_suppression_list",
-    default_args=default_args,
-    schedule_interval="0 9 * * *",
-    doc_md=docs,
-    tags=tags,
-) as dag:
-
-    acoustic_external__suppression_list__v1 = bigquery_etl_query(
-        task_id="acoustic_external__suppression_list__v1",
-        destination_table="suppression_list_v1",
-        dataset_id="acoustic_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="leli@mozilla.com",
-        email=["leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    with TaskGroup(
-        "acoustic_external__suppression_list__v1_external",
-    ) as acoustic_external__suppression_list__v1_external:
-        ExternalTaskMarker(
-            task_id="bqetl_marketing_suppression_list__wait_for_acoustic_external__suppression_list__v1",
-            external_dag_id="bqetl_marketing_suppression_list",
-            external_task_id="wait_for_acoustic_external__suppression_list__v1",
-            execution_date="{{ (execution_date - macros.timedelta(seconds=21600)).isoformat() }}",
-        )
-
-        acoustic_external__suppression_list__v1_external.set_upstream(
-            acoustic_external__suppression_list__v1
-        )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_aggregations.py	2024-05-31 17:56:38.000000000 +0000
@@ -80,10 +80,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -92,10 +92,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -104,10 +104,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -116,10 +116,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -128,10 +128,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -140,10 +140,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
         external_dag_id="bqetl_main_summary",
-        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
+        external_task_id="telemetry_derived__clients_last_seen__v1",
         execution_delta=datetime.timedelta(seconds=8100),
         check_existence=True,
         mode="reschedule",
@@ -288,18 +288,6 @@
         )
     )
 
-    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
-        external_dag_id="bqetl_main_summary",
-        external_task_id="telemetry_derived__clients_last_seen__v1",
-        execution_delta=datetime.timedelta(seconds=8100),
-        check_existence=True,
-        mode="reschedule",
-        allowed_states=ALLOWED_STATES,
-        failed_states=FAILED_STATES,
-        pool="DATA_ENG_EXTERNALTASKSENSOR",
-    )
-
     active_users_aggregates_attribution_v1 = bigquery_etl_query(
         task_id="active_users_aggregates_attribution_v1",
         destination_table="active_users_aggregates_attribution_v1",
@@ -330,174 +318,33 @@
         depends_on_past=False,
     )
 
-    active_users_aggregates_v1 = bigquery_etl_query(
-        task_id="active_users_aggregates_v1",
-        destination_table="active_users_aggregates_v1",
-        dataset_id="telemetry_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="lvargas@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        date_partition_parameter="submission_date",
-        depends_on_past=False,
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3 = bigquery_dq_check(
-        task_id="checks__fail_fenix_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
-        dataset_id="fenix_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="lvargas@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
     with TaskGroup(
-        "checks__fail_fenix_derived__active_users_aggregates__v3_external",
-    ) as checks__fail_fenix_derived__active_users_aggregates__v3_external:
+        "active_users_aggregates_device_v1_external",
+    ) as active_users_aggregates_device_v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_checks__fail_fenix_derived__active_users_aggregates__v3",
+            task_id="bqetl_search_dashboard__wait_for_active_users_aggregates_device_v1",
             external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_checks__fail_fenix_derived__active_users_aggregates__v3",
+            external_task_id="wait_for_active_users_aggregates_device_v1",
             execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
         )
 
-        checks__fail_fenix_derived__active_users_aggregates__v3_external.set_upstream(
-            checks__fail_fenix_derived__active_users_aggregates__v3
+        active_users_aggregates_device_v1_external.set_upstream(
+            active_users_aggregates_device_v1
         )
 
-    checks__fail_firefox_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
-        task_id="checks__fail_firefox_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
-        dataset_id="firefox_ios_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="lvargas@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
-    with TaskGroup(
-        "checks__fail_firefox_ios_derived__active_users_aggregates__v3_external",
-    ) as checks__fail_firefox_ios_derived__active_users_aggregates__v3_external:
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_checks__fail_firefox_ios_derived__active_users_aggregates__v3",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_checks__fail_firefox_ios_derived__active_users_aggregates__v3",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
-        )
-
-        checks__fail_firefox_ios_derived__active_users_aggregates__v3_external.set_upstream(
-            checks__fail_firefox_ios_derived__active_users_aggregates__v3
-        )
-
-    checks__fail_focus_android_derived__active_users_aggregates__v3 = bigquery_dq_check(
-        task_id="checks__fail_focus_android_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
-        dataset_id="focus_android_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="lvargas@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
-    with TaskGroup(
-        "checks__fail_focus_android_derived__active_users_aggregates__v3_external",
-    ) as checks__fail_focus_android_derived__active_users_aggregates__v3_external:
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_checks__fail_focus_android_derived__active_users_aggregates__v3",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_checks__fail_focus_android_derived__active_users_aggregates__v3",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
-        )
-
-        checks__fail_focus_android_derived__active_users_aggregates__v3_external.set_upstream(
-            checks__fail_focus_android_derived__active_users_aggregates__v3
-        )
-
-    checks__fail_focus_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
-        task_id="checks__fail_focus_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
-        dataset_id="focus_ios_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="lvargas@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
-    with TaskGroup(
-        "checks__fail_focus_ios_derived__active_users_aggregates__v3_external",
-    ) as checks__fail_focus_ios_derived__active_users_aggregates__v3_external:
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_checks__fail_focus_ios_derived__active_users_aggregates__v3",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_checks__fail_focus_ios_derived__active_users_aggregates__v3",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
-        )
-
-        checks__fail_focus_ios_derived__active_users_aggregates__v3_external.set_upstream(
-            checks__fail_focus_ios_derived__active_users_aggregates__v3
-        )
-
-    checks__fail_klar_ios_derived__active_users_aggregates__v3 = bigquery_dq_check(
-        task_id="checks__fail_klar_ios_derived__active_users_aggregates__v3",
-        source_table="active_users_aggregates_v3",
-        dataset_id="klar_ios_derived",
+    active_users_aggregates_v1 = bigquery_etl_query(
+        task_id="active_users_aggregates_v1",
+        destination_table="active_users_aggregates_v1",
+        dataset_id="telemetry_derived",
         project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
         owner="lvargas@mozilla.com",
         email=[
             "gkaberere@mozilla.com",
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
+        date_partition_parameter="submission_date",
         depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
-    with TaskGroup(
-        "checks__fail_klar_ios_derived__active_users_aggregates__v3_external",
-    ) as checks__fail_klar_ios_derived__active_users_aggregates__v3_external:
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_checks__fail_klar_ios_derived__active_users_aggregates__v3",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_checks__fail_klar_ios_derived__active_users_aggregates__v3",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
-        )
-
-        checks__fail_klar_ios_derived__active_users_aggregates__v3_external.set_upstream(
-            checks__fail_klar_ios_derived__active_users_aggregates__v3
         )
 
     checks__warn_fenix_derived__active_users_aggregates__v3 = bigquery_dq_check(
@@ -634,20 +481,6 @@
         depends_on_past=False,
     )
 
-    with TaskGroup(
-        "firefox_desktop_active_users_aggregates_external",
-    ) as firefox_desktop_active_users_aggregates_external:
-        ExternalTaskMarker(
-            task_id="bqetl_search_dashboard__wait_for_firefox_desktop_active_users_aggregates",
-            external_dag_id="bqetl_search_dashboard",
-            external_task_id="wait_for_firefox_desktop_active_users_aggregates",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=85500)).isoformat() }}",
-        )
-
-        firefox_desktop_active_users_aggregates_external.set_upstream(
-            firefox_desktop_active_users_aggregates
-        )
-
     firefox_ios_active_users_aggregates = bigquery_etl_query(
         task_id="firefox_ios_active_users_aggregates",
         destination_table="active_users_aggregates_v3",
@@ -756,76 +589,36 @@
         wait_for_checks__fail_telemetry_derived__unified_metrics__v1
     )
 
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
-    )
-
-    checks__fail_fenix_derived__active_users_aggregates__v3.set_upstream(
-        fenix_active_users_aggregates
-    )
-
-    checks__fail_firefox_ios_derived__active_users_aggregates__v3.set_upstream(
-        firefox_ios_active_users_aggregates
-    )
-
-    checks__fail_focus_android_derived__active_users_aggregates__v3.set_upstream(
-        focus_android_active_users_aggregates
-    )
-
-    checks__fail_focus_ios_derived__active_users_aggregates__v3.set_upstream(
-        focus_ios_active_users_aggregates
-    )
-
-    checks__fail_klar_ios_derived__active_users_aggregates__v3.set_upstream(
-        klar_ios_active_users_aggregates
-    )
-
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+        fenix_active_users_aggregates
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
     )
 
     checks__warn_fenix_derived__active_users_aggregates__v3.set_upstream(
-        fenix_active_users_aggregates
+        wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
-        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+        firefox_desktop_active_users_aggregates
     )
 
     checks__warn_firefox_desktop_derived__active_users_aggregates__v3.set_upstream(
-        firefox_desktop_active_users_aggregates
+        wait_for_telemetry_derived__clients_last_seen__v1
     )
 
     checks__warn_firefox_ios_derived__active_users_aggregates__v3.set_upstream(
@@ -865,7 +658,7 @@
     )
 
     firefox_desktop_active_users_aggregates.set_upstream(
-        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+        wait_for_telemetry_derived__clients_last_seen__v1
     )
 
     firefox_ios_active_users_aggregates.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py	2024-05-31 17:56:39.000000000 +0000
@@ -101,10 +101,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fenix_derived__baseline_clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -112,10 +112,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -123,10 +123,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -134,10 +134,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -145,10 +145,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
+    wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
         external_dag_id="bqetl_glean_usage",
-        external_task_id="fenix.checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
+        external_task_id="fenix.org_mozilla_firefox_derived__baseline_clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -211,10 +211,10 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_telemetry_derived__clients_last_seen__v2 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_telemetry_derived__clients_last_seen__v2",
+    wait_for_telemetry_derived__clients_last_seen__v1 = ExternalTaskSensor(
+        task_id="wait_for_telemetry_derived__clients_last_seen__v1",
         external_dag_id="bqetl_main_summary",
-        external_task_id="checks__fail_telemetry_derived__clients_last_seen__v2",
+        external_task_id="telemetry_derived__clients_last_seen__v1",
         check_existence=True,
         mode="reschedule",
         allowed_states=ALLOWED_STATES,
@@ -231,7 +231,6 @@
         owner="lvargas@mozilla.com",
         email=[
             "gkaberere@mozilla.com",
-            "kik@mozilla.com",
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
@@ -271,21 +270,76 @@
             execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=72000)).isoformat() }}",
         )
 
-        ExternalTaskMarker(
-            task_id="bqetl_data_observability_test_data_copy__wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-            external_dag_id="bqetl_data_observability_test_data_copy",
-            external_task_id="wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=64800)).isoformat() }}",
+        checks__fail_fenix_derived__firefox_android_clients__v1_external.set_upstream(
+            checks__fail_fenix_derived__firefox_android_clients__v1
         )
 
+    checks__fail_fenix_derived__funnel_retention_clients_week_2__v1 = bigquery_dq_check(
+        task_id="checks__fail_fenix_derived__funnel_retention_clients_week_2__v1",
+        source_table="funnel_retention_clients_week_2_v1",
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="kik@mozilla.com",
+        email=[
+            "gkaberere@mozilla.com",
+            "kik@mozilla.com",
+            "lvargas@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{ds}}"],
+        retries=0,
+    )
+
+    checks__fail_fenix_derived__funnel_retention_clients_week_4__v1 = bigquery_dq_check(
+        task_id="checks__fail_fenix_derived__funnel_retention_clients_week_4__v1",
+        source_table="funnel_retention_clients_week_4_v1",
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="kik@mozilla.com",
+        email=[
+            "gkaberere@mozilla.com",
+            "kik@mozilla.com",
+            "lvargas@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{ds}}"],
+        retries=0,
+    )
+
+    with TaskGroup(
+        "checks__fail_fenix_derived__funnel_retention_clients_week_4__v1_external",
+    ) as checks__fail_fenix_derived__funnel_retention_clients_week_4__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_fivetran_google_ads__wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-            external_dag_id="bqetl_fivetran_google_ads",
-            external_task_id="wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
+            task_id="bqetl_generated_funnels__wait_for_checks__fail_fenix_derived__funnel_retention_clients_week_4__v1",
+            external_dag_id="bqetl_generated_funnels",
+            external_task_id="wait_for_checks__fail_fenix_derived__funnel_retention_clients_week_4__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=75600)).isoformat() }}",
         )
 
-        checks__fail_fenix_derived__firefox_android_clients__v1_external.set_upstream(
-            checks__fail_fenix_derived__firefox_android_clients__v1
+        checks__fail_fenix_derived__funnel_retention_clients_week_4__v1_external.set_upstream(
+            checks__fail_fenix_derived__funnel_retention_clients_week_4__v1
+        )
+
+    checks__fail_fenix_derived__funnel_retention_week_4__v1 = bigquery_dq_check(
+        task_id="checks__fail_fenix_derived__funnel_retention_week_4__v1",
+        source_table="funnel_retention_week_4_v1",
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="kik@mozilla.com",
+        email=[
+            "gkaberere@mozilla.com",
+            "kik@mozilla.com",
+            "lvargas@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{ds}}"],
+        retries=0,
         )
 
     checks__fail_telemetry_derived__clients_first_seen__v2 = bigquery_dq_check(
@@ -349,20 +403,6 @@
             execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=78300)).isoformat() }}",
         )
 
-        ExternalTaskMarker(
-            task_id="bqetl_desktop_engagement_model__wait_for_checks__fail_telemetry_derived__clients_first_seen__v2",
-            external_dag_id="bqetl_desktop_engagement_model",
-            external_task_id="wait_for_checks__fail_telemetry_derived__clients_first_seen__v2",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=50400)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
-            task_id="bqetl_desktop_retention_model__wait_for_checks__fail_telemetry_derived__clients_first_seen__v2",
-            external_dag_id="bqetl_desktop_retention_model",
-            external_task_id="wait_for_checks__fail_telemetry_derived__clients_first_seen__v2",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=50400)).isoformat() }}",
-        )
-
         checks__fail_telemetry_derived__clients_first_seen__v2_external.set_upstream(
             checks__fail_telemetry_derived__clients_first_seen__v2
         )
@@ -376,7 +416,6 @@
         owner="lvargas@mozilla.com",
         email=[
             "gkaberere@mozilla.com",
-            "kik@mozilla.com",
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
@@ -386,42 +425,6 @@
         retries=0,
     )
 
-    checks__warn_fenix_derived__funnel_retention_clients_week_2__v1 = bigquery_dq_check(
-        task_id="checks__warn_fenix_derived__funnel_retention_clients_week_2__v1",
-        source_table="funnel_retention_clients_week_2_v1",
-        dataset_id="fenix_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="kik@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "kik@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
-    checks__warn_fenix_derived__funnel_retention_clients_week_4__v1 = bigquery_dq_check(
-        task_id="checks__warn_fenix_derived__funnel_retention_clients_week_4__v1",
-        source_table="funnel_retention_clients_week_4_v1",
-        dataset_id="fenix_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="kik@mozilla.com",
-        email=[
-            "gkaberere@mozilla.com",
-            "kik@mozilla.com",
-            "lvargas@mozilla.com",
-            "telemetry-alerts@mozilla.com",
-        ],
-        depends_on_past=False,
-        parameters=["submission_date:DATE:{{ds}}"],
-        retries=0,
-    )
-
     checks__warn_fenix_derived__funnel_retention_week_4__v1 = bigquery_dq_check(
         task_id="checks__warn_fenix_derived__funnel_retention_week_4__v1",
         source_table="funnel_retention_week_4_v1",
@@ -488,20 +491,6 @@
         depends_on_past=False,
     )
 
-    with TaskGroup(
-        "fenix_derived__funnel_retention_clients_week_4__v1_external",
-    ) as fenix_derived__funnel_retention_clients_week_4__v1_external:
-        ExternalTaskMarker(
-            task_id="bqetl_generated_funnels__wait_for_fenix_derived__funnel_retention_clients_week_4__v1",
-            external_dag_id="bqetl_generated_funnels",
-            external_task_id="wait_for_fenix_derived__funnel_retention_clients_week_4__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=75600)).isoformat() }}",
-        )
-
-        fenix_derived__funnel_retention_clients_week_4__v1_external.set_upstream(
-            fenix_derived__funnel_retention_clients_week_4__v1
-        )
-
     fenix_derived__funnel_retention_week_4__v1 = bigquery_etl_query(
         task_id="fenix_derived__funnel_retention_week_4__v1",
         destination_table="funnel_retention_week_4_v1",
@@ -518,19 +507,6 @@
         depends_on_past=False,
     )
 
-    with TaskGroup(
-        "fenix_derived__funnel_retention_week_4__v1_external",
-    ) as fenix_derived__funnel_retention_week_4__v1_external:
-        ExternalTaskMarker(
-            task_id="bqetl_fivetran_google_ads__wait_for_fenix_derived__funnel_retention_week_4__v1",
-            external_dag_id="bqetl_fivetran_google_ads",
-            external_task_id="wait_for_fenix_derived__funnel_retention_week_4__v1",
-        )
-
-        fenix_derived__funnel_retention_week_4__v1_external.set_upstream(
-            fenix_derived__funnel_retention_week_4__v1
-        )
-
     firefox_android_clients = bigquery_etl_query(
         task_id="firefox_android_clients",
         destination_table="firefox_android_clients_v1",
@@ -539,7 +515,6 @@
         owner="lvargas@mozilla.com",
         email=[
             "gkaberere@mozilla.com",
-            "kik@mozilla.com",
             "lvargas@mozilla.com",
             "telemetry-alerts@mozilla.com",
         ],
@@ -574,6 +549,22 @@
         firefox_android_clients
     )
 
+    checks__fail_fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
+        fenix_derived__funnel_retention_clients_week_2__v1
+    )
+
+    checks__fail_fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
+        fenix_derived__funnel_retention_clients_week_4__v1
+    )
+
+    checks__fail_fenix_derived__funnel_retention_week_4__v1.set_upstream(
+        checks__fail_fenix_derived__funnel_retention_clients_week_4__v1
+    )
+
+    checks__fail_fenix_derived__funnel_retention_week_4__v1.set_upstream(
+        fenix_derived__funnel_retention_week_4__v1
+    )
+
     checks__fail_telemetry_derived__clients_first_seen__v2.set_upstream(
         clients_first_seen_v2
     )
@@ -594,16 +585,8 @@
         firefox_android_clients
     )
 
-    checks__warn_fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        fenix_derived__funnel_retention_clients_week_2__v1
-    )
-
-    checks__warn_fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        fenix_derived__funnel_retention_clients_week_4__v1
-    )
-
     checks__warn_fenix_derived__funnel_retention_week_4__v1.set_upstream(
-        fenix_derived__funnel_retention_clients_week_4__v1
+        checks__fail_fenix_derived__funnel_retention_clients_week_4__v1
     )
 
     checks__warn_fenix_derived__funnel_retention_week_4__v1.set_upstream(
@@ -621,23 +604,23 @@
     )
 
     fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_2__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
@@ -645,27 +628,27 @@
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_fennec_aurora_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_firefox_beta_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_clients_week_4__v1.set_upstream(
-        wait_for_checks__fail_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
+        wait_for_org_mozilla_firefox_derived__baseline_clients_last_seen__v1
     )
 
     fenix_derived__funnel_retention_week_4__v1.set_upstream(
-        fenix_derived__funnel_retention_clients_week_4__v1
+        checks__fail_fenix_derived__funnel_retention_clients_week_4__v1
     )
 
     firefox_android_clients.set_upstream(wait_for_copy_deduplicate_all)
@@ -699,5 +682,5 @@
     )
 
     telemetry_derived__clients_first_seen_28_days_later__v1.set_upstream(
-        wait_for_checks__fail_telemetry_derived__clients_last_seen__v2
+        wait_for_telemetry_derived__clients_last_seen__v1
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_braze.py /tmp/workspace/generated-sql/dags/bqetl_braze.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_braze.py	2024-05-31 17:46:08.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_braze.py	1970-01-01 00:00:00.000000000 +0000
@@ -1,596 +0,0 @@
-# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
-
-from airflow import DAG
-from airflow.sensors.external_task import ExternalTaskMarker
-from airflow.sensors.external_task import ExternalTaskSensor
-from airflow.utils.task_group import TaskGroup
-import datetime
-from operators.gcp_container_operator import GKEPodOperator
-from utils.constants import ALLOWED_STATES, FAILED_STATES
-from utils.gcp import bigquery_etl_query, bigquery_dq_check
-
-docs = """
-### bqetl_braze
-
-Built from bigquery-etl repo, [`dags/bqetl_braze.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_braze.py)
-
-#### Description
-
-ETL for Braze workflows.
-## Triage notes:
-Don't rerun this DAG!
-Ping Chelsey Beck or Leli Schiestl.
-If both are out, follow the [workbook](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/730234942/bqetl+braze+DAG+workbook) in confluence.
-
-#### Owner
-
-cbeck@mozilla.com
-
-#### Tags
-
-* impact/tier_2
-* repo/bigquery-etl
-"""
-
-
-default_args = {
-    "owner": "cbeck@mozilla.com",
-    "start_date": datetime.datetime(2024, 4, 15, 0, 0),
-    "end_date": None,
-    "email": ["cbeck@mozilla.com", "leli@mozilla.com"],
-    "depends_on_past": False,
-    "retry_delay": datetime.timedelta(seconds=1800),
-    "email_on_failure": True,
-    "email_on_retry": True,
-    "retries": 1,
-}
-
-tags = ["impact/tier_2", "repo/bigquery-etl"]
-
-with DAG(
-    "bqetl_braze",
-    default_args=default_args,
-    schedule_interval="30 4,12,20 * * *",
-    doc_md=docs,
-    tags=tags,
-) as dag:
-
-    braze_derived__newsletters__v1 = bigquery_etl_query(
-        task_id="braze_derived__newsletters__v1",
-        destination_table="newsletters_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_derived__products__v1 = bigquery_etl_query(
-        task_id="braze_derived__products__v1",
-        destination_table="products_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_derived__subscriptions__v1 = bigquery_etl_query(
-        task_id="braze_derived__subscriptions__v1",
-        destination_table="subscriptions_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_derived__subscriptions_map__v1 = bigquery_etl_query(
-        task_id="braze_derived__subscriptions_map__v1",
-        destination_table=None,
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        sql_file_path="sql/moz-fx-data-shared-prod/braze_derived/subscriptions_map_v1/script.sql",
-    )
-
-    braze_derived__user_profiles__v1 = bigquery_etl_query(
-        task_id="braze_derived__user_profiles__v1",
-        destination_table="user_profiles_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_derived__users__v1 = bigquery_etl_query(
-        task_id="braze_derived__users__v1",
-        destination_table="users_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_derived__waitlists__v1 = bigquery_etl_query(
-        task_id="braze_derived__waitlists__v1",
-        destination_table="waitlists_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_firefox_subscriptions_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_firefox_subscriptions_sync__v1",
-        destination_table="changed_firefox_subscriptions_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_newsletters_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_newsletters_sync__v1",
-        destination_table="changed_newsletters_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_products_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_products_sync__v1",
-        destination_table="changed_products_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_subscriptions__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_subscriptions__v1",
-        destination_table="changed_subscriptions_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_users__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_users__v1",
-        destination_table="changed_users_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_users_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_users_sync__v1",
-        destination_table="changed_users_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__changed_waitlists_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__changed_waitlists_sync__v1",
-        destination_table="changed_waitlists_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__delete_users_sync__v1 = bigquery_etl_query(
-        task_id="braze_external__delete_users_sync__v1",
-        destination_table="delete_users_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    braze_external__users_previous_day_snapshot__v1 = bigquery_etl_query(
-        task_id="braze_external__users_previous_day_snapshot__v1",
-        destination_table=None,
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        sql_file_path="sql/moz-fx-data-shared-prod/braze_external/users_previous_day_snapshot_v1/script.sql",
-    )
-
-    braze_external__users_previous_day_snapshot__v2 = bigquery_etl_query(
-        task_id="braze_external__users_previous_day_snapshot__v2",
-        destination_table=None,
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        sql_file_path="sql/moz-fx-data-shared-prod/braze_external/users_previous_day_snapshot_v2/script.sql",
-    )
-
-    checks__fail_braze_derived__newsletters__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__newsletters__v1",
-        source_table="newsletters_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__products__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__products__v1",
-        source_table="products_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__subscriptions__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__subscriptions__v1",
-        source_table="subscriptions_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__subscriptions_map__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__subscriptions_map__v1",
-        source_table="subscriptions_map_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__user_profiles__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__user_profiles__v1",
-        source_table="user_profiles_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__users__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__users__v1",
-        source_table="users_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_derived__waitlists__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_derived__waitlists__v1",
-        source_table="waitlists_v1",
-        dataset_id="braze_derived",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__fail_braze_external__changed_subscriptions__v1 = bigquery_dq_check(
-        task_id="checks__fail_braze_external__changed_subscriptions__v1",
-        source_table="changed_subscriptions_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=True,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__warn_braze_external__changed_firefox_subscriptions_sync__v1 = bigquery_dq_check(
-        task_id="checks__warn_braze_external__changed_firefox_subscriptions_sync__v1",
-        source_table="changed_firefox_subscriptions_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__warn_braze_external__changed_newsletters_sync__v1 = bigquery_dq_check(
-        task_id="checks__warn_braze_external__changed_newsletters_sync__v1",
-        source_table="changed_newsletters_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__warn_braze_external__changed_products_sync__v1 = bigquery_dq_check(
-        task_id="checks__warn_braze_external__changed_products_sync__v1",
-        source_table="changed_products_sync_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__warn_braze_external__changed_subscriptions__v1 = bigquery_dq_check(
-        task_id="checks__warn_braze_external__changed_subscriptions__v1",
-        source_table="changed_subscriptions_v1",
-        dataset_id="braze_external",
-        project_id="moz-fx-data-shared-prod",
-        is_dq_check_fail=False,
-        owner="cbeck@mozilla.com",
-        email=["cbeck@mozilla.com", "leli@mozilla.com"],
-        depends_on_past=False,
-        task_concurrency=1,
-        retries=0,
-    )
-
-    checks__war

⚠️ Only part of the diff is displayed.

Link to full diff

initial add of query.sql WIP for ga downloads_with_attr_v3

2dd0e0b

Marlene-M-Hirose requested a review from kwindau April 19, 2024 00:27

Marlene-M-Hirose added 2 commits April 18, 2024 14:30

take out filter from test table

34e2344

add in has_ga_download_event column

cba4bab