Changes to active_users_aggregates view #5395

lucia-vargas-a · 2024-04-19T00:16:22Z

Update the column names in the view active_users_aggregates. Merge after PR-5396.

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Follow the change control procedure

┆Issue is synchronized with this Jira Task

* Adding a new table (moz-fx-data-shared-prod.firefox_ios_derived.ltv_states_v1) & corresponding view (moz-fx-data-shared-prod.firefox_ios.ltv_states)

#5406) * DENG-3462 create view on top of clients_last_seen_v2 with new boolean columns is_dau, is_wau, etc * DENG-3462 remove is_mobile column

* Backfill new table * update end_date to incorporate new fields

)

* Add UDFs for extracting campaign info * Add campaigns table * Add ad_groups table * Add schemas * Uncomment UDFs * Format sql files * Reformat UDFs * Use common most_recent_rec terminology * Replace 3 UDFs with one * reformat & fix tests * Reformat again

* Extract campaign_id and ad_group_id in fac * Add Android App Campaign Stats table * Dont rely on fac view * Reformat

* Added nightly, experiments and 1pct tables

…5386) * Added deletion date field --------- Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>

* DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

* updating create statement * joining on users table to filter for active and ensuring there is at least one subscription * joining on users to filter for active * adding dev subscription group * removing fxa_id in favor of has_fxa * bringing in update timestamp for downstream use * updating formatting and adding filter for active users * adding filter for one active newsletter * updating tests * adding fxa id back to users table to join to products * updating query * updating values * updating tests * fix test for subscriptions * changing schema to array * updating format * updating to pull in all subscriptions with statuses * removing create statement * updating subscriptions query to make it an array and updating associated tests * updating formatting and comment --------- Co-authored-by: Leli Schiestl <lschiestl@mozilla.com>

* DENG-3471 suppression list from campaign monitor * Update sql/moz-fx-data-shared-prod/marketing_suppression_list_external/campaign_monitor_suppression_list_v1/metadata.yaml Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com> * after seans coder review --------- Co-authored-by: Ben Wu <12437227+BenWu@users.noreply.github.com>

…dated private bqetl (#5419)

* DENG-3511 remove channel from input since no longer required for updated UDF * DENG-3511 remove channel from input since no longer required for updated UDF

* DENG-3186 new desktop engagement model * DENG-3186 add distribution ID to eng client V1 query * DENG-3186 - add distribution ID to the schema.yaml for eng client * DENG-3186 add distribution ID to the agg table * DENG-3186 adding locale to both client & agg tables * DENG-3186 add app_version to client & agg tables * DENG-3168 move lifecycle stage from engagement agg table to engagement agg view * DENG-3186 - add new desktop_engagement_client_v1 table to shredder * DENG-3186 update formatting of config.py * DENG-3186 switch to get country, os, os version, and build # from clients last seen and remove join to clients daily

* Update Glean generators checks. * Remove new_profiles check, not required. * warn or fail depending on range * Set check to fail if clients are missing, which is an ETL issue. * Checks fail when difference is higher than 1%. * Checks fail when difference is higher than 1%. * Make checks fail on a smaller difference.

* DENG-3471 suppression list from campaign monitor * status quo * DENG-3410 Main Suppression List * adding metadata

dataops-ci-bot · 2024-05-29T18:32:55Z

Integration report for "Merge branch 'main' into DENG-2989_update_active_users_view"

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates/view.sql	2024-05-29 18:27:43.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/active_users_aggregates/view.sql	2024-05-29 18:27:44.000000000 +0000
@@ -19,11 +19,13 @@
   os_version_major,
   os_version_minor,
   submission_date,
-  language_name,
+  locale,
   dau,
-  weekly_users AS wau,
-  monthly_users AS mau,
-  new_profiles,
+  wau,
+  mau,
+  daily_users,
+  weekly_users,
+  monthly_users,
   uri_count,
   active_hours,
   app_name,
@@ -54,11 +56,13 @@
   os_version_major,
   os_version_minor,
   submission_date,
-  language_name,
+  locale,
   dau,
-  weekly_users AS wau,
-  monthly_users AS mau,
-  new_profiles,
+  wau,
+  mau,
+  daily_users,
+  weekly_users,
+  monthly_users,
   uri_count,
   active_hours,
   app_name,

Link to full diff

Changes to active_users_aggregates.

5acb0d3

lucia-vargas-a marked this pull request as draft April 19, 2024 00:17

lucia-vargas-a changed the title ~~Changes to active_users_aggregates.~~ Changes to active_users_aggregates view Apr 22, 2024

lucia-vargas-a requested a review from bochocki April 22, 2024 14:31

Merge branch 'main' into DENG-2989_update_active_users_view

53c9f0e

This comment has been minimized.

Sign in to view

bochocki approved these changes Apr 23, 2024

View reviewed changes

kwindau and others added 21 commits April 24, 2024 21:26

GROWTH-147 iOS LTV (#5400)

d250cce

* Adding a new table (moz-fx-data-shared-prod.firefox_ios_derived.ltv_states_v1) & corresponding view (moz-fx-data-shared-prod.firefox_ios.ltv_states)

GROWTH-147 backfill (#5407)

9ba0ee9

DENG-3462 create view on top of clients_last_seen_v2 with new boolean… (

6817bd9

#5406) * DENG-3462 create view on top of clients_last_seen_v2 with new boolean columns is_dau, is_wau, etc * DENG-3462 remove is_mobile column

Backfill new newtab_clients_daily_v1 table (#5408)

43ebcbc

* Backfill new table * update end_date to incorporate new fields

GROWTH-147 complete backfill of firefox_ios_derived.ltv_states_v1 (#5409

e5dc2ec

)

Google ads etl (#5393)

bb86daa

* Add UDFs for extracting campaign info * Add campaigns table * Add ad_groups table * Add schemas * Uncomment UDFs * Format sql files * Reformat UDFs * Use common most_recent_rec terminology * Replace 3 UDFs with one * reformat & fix tests * Reformat again

App campaign stats (#5413)

0b5f36d

* Extract campaign_id and ad_group_id in fac * Add Android App Campaign Stats table * Dont rely on fac view * Reformat

DENG-2621: Add pageload derived tables (#5359)

19d6b41

* Added nightly, experiments and 1pct tables

DENG-3235: Added deletion date field to bigquery_tables_inventory_v1 (#…

86009ec

…5386) * Added deletion date field --------- Co-authored-by: Chelsey Beck <64881557+chelseybeck@users.noreply.github.com>

Complete backfill (#5414)

3f8240c

fixing the readme (#5418)

f3d1fae

DENG-3511 update funnel_derived_clients to remove channel to match up…

7d82586

…dated private bqetl (#5419)

DENG-3511 update views to use updated udfs (#5420)

0726e84

* DENG-3511 remove channel from input since no longer required for updated UDF * DENG-3511 remove channel from input since no longer required for updated UDF

DENG-3168 backfill new desktop_engagement_client_v1 table (#5422)

9c762b5

Deng 3410 main suppression list (#5423)

84d678b

* DENG-3471 suppression list from campaign monitor * status quo * DENG-3410 Main Suppression List * adding metadata

Merge branch 'main' into DENG-2989_update_active_users_view

93b5e7b

This comment has been minimized.

Sign in to view

lucia-vargas-a marked this pull request as ready for review May 14, 2024 14:59

Merge branch 'main' into DENG-2989_update_active_users_view

6cbafb0

This comment has been minimized.

Sign in to view

Merge branch 'main' into DENG-2989_update_active_users_view

905c028

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to active_users_aggregates view #5395

Changes to active_users_aggregates view #5395

lucia-vargas-a commented Apr 19, 2024 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented May 29, 2024

Changes to active_users_aggregates view #5395

Are you sure you want to change the base?

Changes to active_users_aggregates view #5395

Conversation

lucia-vargas-a commented Apr 19, 2024 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented May 29, 2024

Integration report for "Merge branch 'main' into DENG-2989_update_active_users_view"

sql.diff

lucia-vargas-a commented Apr 19, 2024 •

edited

`sql.diff`