DENG-8178: initiate backfill for serp_events_v2

[kbammarito]

Description

Initiates a managed backfill for firefox_desktop_derived.serp_events_v2

Extends SERP history to the retention floor so search_derived.search_clients_daily_glean_v1 (still to be created) can be backfilled over the same range — glean_v1 cannot start earlier than its sources, and this is the binding one.

Full re-run from raw with no shredder mitigation, so every column of every partition in range is rewritten and historical values move with deletions and late-arriving data. glean_v1's own backfill will follow this one.

Related Tickets & Documents

• DENG-8178

Reviewer, please follow this checklist

[alekhyamoz]

lgtm

[github-actions]

This PR adds a new backfill.yaml for firefox_desktop_derived.serp_events_v2 with a single Initiate entry covering 2024-08-07 through 2026-09-21, a full re-run from raw with no shredder mitigation and no overrides, to extend SERP history back to the retention floor ahead of a search_clients_daily_glean_v1 backfill.

The entry itself is well-formed YAML and the reason/watchers are filled in, but I found three problems that would stop it from producing the intended data. The most consequential is that this table's generated metadata uses date_partition_parameter: null and reaches its partition through the Airflow destination_table decorator, which the managed backfill path doesn't honor — that needs resolving before the date-range details matter. The retention-boundary and date-offset findings are independent of that and are also worth fixing.

bqetl backfill validate against the generated SQL tree should surface the retention rejection on its own; the other two won't fail validation and only show up at run time.

[github-actions]

issue: this start_date is one day past the retention check, so backfill initiate will reject the entry.

validate_retention_range uses the smaller of NBR_DAYS_RETAINED (775) and the table's expiration_days; the generated metadata sets expiration_days: 775 (sql_generators/serp_events_v2/templates/metadata.yaml), so the effective limit is 775. The check is:

if backfill_entry.start_date < backfill_entry.entry_date - datetime.timedelta(days=retention_days - 1):

2026-09-21 - 774 days is 2024-08-08, and 2024-08-07 < 2024-08-08, so with override_retention_limit: false this raises Cannot backfill more than 775 days prior to entry date.

Move start_date to 2024-08-08 or later. Sitting exactly on the boundary is also fragile in practice — a partition dated at the edge of the 775-day expiration_days window expires within a day of being swapped into production, so a few days of headroom is worth taking.

[alekhyamoz]

@kbammarito good flag. Please change the start date

[kbammarito]

I updated the start and end dates

[github-actions]

issue: serp_events_v2 isn't backfillable with a plain entry like this one — the managed backfill will overwrite the whole staging table on every date and then fail at complete.

The generated metadata sets scheduling.date_partition_parameter: null and no date_partition_offset (sql_generators/serp_events_v2/templates/metadata.yaml:22); the daily Airflow task hits a single partition via the templated destination_table decorator (serp_events_v2${{ execution_date - 24h }}) instead. The backfill machinery never reads destination_table:

• get_backfill_partition() returns None when the parameter is null and the offset is 0 (bigquery_etl/backfill/date_range.py:95), so _backfill_query leaves the staging destination undecorated (bigquery_etl/cli/query.py:516-525) and runs bq query --replace against the full staging table. All 776 dates run that way, in parallel, so each one truncates the table and only whichever job lands last survives. The same reasoning is already spelled out in the initiate code comment at bigquery_etl/cli/backfill.py:1067-1070.
• complete then raises Null partition found completing backfill ... (bigquery_etl/cli/backfill.py:1468-1471), since partitioning_type is day but the partition still resolves to None.

To run this you need the entry (or the table) to resolve a real partition per date. The cleanest fix is to give the table date_partition_parameter: submission_date with date_partition_offset: -1 in the generator's metadata template, which matches what the query actually emits and what the Airflow decorator already does — then the normal day-by-day path works for both initiate and complete. The alternative documented path is a custom_query_path with override_depends_on_past_null_partition: true (docs/cookbooks/backfilling_a_table.md, "Per-partition custom-query backfill"), but note that path forces date_partition_parameter: submission_date with offset 0, which would land each day's rows in the wrong partition here.

[github-actions]

issue: the date range is off by one relative to the partitions it produces, and the tail end has no upstream data yet.

The backfill binds each date in [start_date, end_date] to @submission_date (the table's parameters: submission_date:DATE:{{ds}}), but the query selects DATE(submission_timestamp) = DATE_SUB(@submission_date, INTERVAL 1 DAY) and emits DATE(submission_timestamp) AS submission_date (sql_generators/serp_events_v2/templates/desktop_query.sql:115,219). So this range yields rows dated 2024-08-06 through 2026-09-20, one day earlier than the entry reads — the leading day is already outside the 775-day window and the intended final day is never produced.

Separately, end_date equals entry_date, which docs/cookbooks/backfilling_a_table.md calls out: the upstream ETL for the most recent days hasn't run when the backfill starts, so those partitions get written empty and the empty partitions are copied into production on complete. This table also documents a 1-day lag, so the newest reliably-populated submission_date is roughly two days back.

Shift both endpoints up by one and pull end_date back to a date whose upstream data is settled (e.g. start_date: 2024-08-09, end_date: 2026-09-19).

[kbammarito]

I updated the start and end dates

[scholtzan]

Integration report

• View full diff, if available (SQL + DAGs) (requires private-bigquery-etl access)
• Private CI run
• Bigeye monitoring dry-run: ✅ plan clean — view plan (requires private-bigquery-etl access)