chore(dags): recategorize Airflow impact tiers - #9849
LiamMcFall wants to merge 4 commits into
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR retags impact/tier_* on 57 DAGs in dags.yaml following the Airflow triage workweek (DENG-11496): mostly tier_1 → tier_2 and tier_2 → tier_3 downgrades, a handful of upgrades (bqetl_accounts_db*, bqetl_search_dashboard, bqetl_releases, bqetl_ctxsvc_derived, bqetl_desktop_engagement_model, bqetl_desktop_retention_model, bqetl_cohort_retention), and the repo's first impact/tier_0 on bqetl_glean_usage. No other files change, and nothing in the tag validation constrains tier values, so tier_0 is accepted as-is.
Tiering is a judgment call the triage group owns, so my comments are limited to places where a new tag conflicts with something checkable in the repo: a DAG description that promises a different response time, a dependency whose upstream is now slower-SLA than its tier_1 consumer, a hardcoded tier in the task-markers generator, and a few tags whose definitions in docs/reference/airflow_tags.md don't match what the DAG does.
| schedule_interval: "1h" | ||
| tags: | ||
| - impact/tier_1 | ||
| - impact/tier_2 |
There was a problem hiding this comment.
issue: This tier contradicts the DAG's own triage notes a few lines above: "None of these tasks should fail. If they do it is very likely that other/all ETL tasks will subsequently fail as well. Any failures should be communicated to the Data Infra Working Group as soon as possible." impact/tier_2 per airflow_tags.md means "not time-sensitive ... expected to be resolved within 2–3 business days", which is the opposite instruction for the pipeline canary (it also runs hourly with retries: 0). Keep this at impact/tier_1 (or amend the description so triage isn't given two conflicting response times).
| schedule_interval: 0 3 * * * | ||
| tags: | ||
| - impact/tier_2 | ||
| - impact/tier_1 |
There was a problem hiding this comment.
issue: This creates a tier inversion with bqetl_unified, which this same PR drops to impact/tier_2 (line 803). contextual_services_derived/adm_forecasting_v1 and contextual_services_derived/suggest_revenue_levers_daily_v1 (both dag_name: bqetl_ctxsvc_derived) select from moz-fx-data-shared-prod.telemetry.unified_metrics, i.e. telemetry_derived.unified_metrics_v1 in bqetl_unified. As tagged, a bqetl_unified failure gets a 2–3 business day response while its tier_1 consumer is expected to recover the same business day, so triage can't meet the tier_1 promise. Either keep bqetl_unified at tier_1 or leave bqetl_ctxsvc_derived at tier_2.
| tags: | ||
| - repo/bigquery-etl | ||
| - impact/tier_1 | ||
| - impact/tier_2 |
There was a problem hiding this comment.
suggestion: This DAG runs every 10 minutes and lands data in a GCS bucket that Merino reads for live Newtab recommendations, which matches tier_1's "time-sensitive processes such as data sent to external systems" rather than tier_2's "not time-sensitive ... resolved within 2–3 business days". Serving would run on stale recommendations for days under the tier_2 SLA. Keeping this one at tier_1 (the daily bqetl_merino_newtab_priors_to_gcs downgrade on line 1761 is more defensible) would keep the tag aligned with the cadence.
| schedule_interval: 0 4 * * * | ||
| tags: | ||
| - impact/tier_1 | ||
| - impact/tier_3 |
There was a problem hiding this comment.
suggestion: Two-tier drop to the "no impact on other processes and is not used to generate any metrics used by business users" bucket doesn't fit what this DAG builds: sql_generators/usage_reporting generates usage_reporting_clients_first_seen_v1, usage_reporting_clients_daily_v1, usage_reporting_clients_last_seen_v1 and usage_reporting_active_users_aggregates_v1 — DAU/WAU/MAU aggregates, and the tables are also registered in bigquery_etl/shredder/config.py. tier_2 looks like the right landing spot unless these aggregates are genuinely unused now.
| start_date: "2023-11-20" | ||
| tags: | ||
| - impact/tier_1 | ||
| - impact/tier_0 |
There was a problem hiding this comment.
suggestion: This is the first tier_0 tag in the repo, and it leaves the other tier_0 example from the docs ("copy dedupe, Glean usage") behind: the generated copy-deduplicate task-markers DAG hardcodes "impact/tier_1" in bigquery_etl/query_scheduling/copy_deduplicate_task_markers.py:106. Since that DAG mirrors copy_deduplicate 1:1, bumping it to impact/tier_0 in the same PR would keep the two foundational pipelines tagged consistently.
| retry_delay: 5m | ||
| tags: | ||
| - impact/tier_2 | ||
| - impact/tier_3 |
There was a problem hiding this comment.
nitpick: bqetl_braze ("ETL for Braze workflows", runs 3×/day) pushes data into an external marketing platform, which tier_3's "no impact on other processes" doesn't describe — the docs put data sent to external/marketing platforms at tier_1, and the related bqetl_marketing_suppression_list stays at tier_2 in this PR. tier_2 would keep the Braze DAGs consistent with the rest of that pipeline.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Integration report
|
This branch has not been deployed
2 participants
Description
Recategorization of DAG tiers based on airflow triage workweek.
Related Tickets & Documents
Reviewer, please follow this checklist