Skip to content

RMST-514: Add derived dataset for Remote Settings uptake telemetry - #9906

Open
leplatrem wants to merge 8 commits into
mainfrom
rmst-514-clients-uptake
Open

leplatrem wants to merge 8 commits into
mainfrom
rmst-514-clients-uptake

Conversation

@leplatrem

Copy link
Copy Markdown
Contributor

@leplatrem
leplatrem requested a review from a team as a code owner September 22, 2026 11:18

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a new remote_settings_derived dataset containing clients_uptake_events_v1, a daily incremental table that unions uptake_remotesettings Glean events from firefox_desktop, fenix, and firefox_ios events_unnested views, tagging each row with an implementation (gecko/rust) and platform, and flattening the event extras into typed extra_* columns. It is scheduled on bqetl_default, partitioned on submission_timestamp with a 365-day expiration, and clustered on channel/implementation/source/status.

I verified the mechanics that tend to break this kind of query: the client_info struct is normalized identically by the generated cross_channel_events_unnested template for all three apps, so the UNION ALL is type-compatible; all seven !include-field-description targets exist in bigquery_etl/schema/global.yaml; the 17 schema fields match the final SELECT in name, order, and type (DECIMAL is a NUMERIC alias and has precedent in monitoring.remote_settings_uptake_live); and bqetl_default exists in dags.yaml. The main thing I'd want resolved before this lands is the shredder registration for client_id — the auto-discovery path doesn't cover this dataset. The rest are convention and observability points.

Per the reviewer checklist, I checked for pre-existing coverage: monitoring.remote_settings_uptake_live reads the same events but only from firefox_desktop_live for real-time Grafana monitoring, so a historical multi-platform table is a genuine addition rather than a duplicate. Labels used below are issue / suggestion / nitpick; blocking is the human reviewer's call.

Comment on lines +82 to +88
mozfun.map.get_key(event_extra, 'value') AS extra_status,
mozfun.map.get_key(event_extra, 'trigger') AS extra_trigger,
mozfun.map.get_key(event_extra, 'source') AS extra_source,
mozfun.map.get_key(event_extra, 'errorName') AS extra_errorname,
mozfun.map.get_key(event_extra, 'timestamp') AS extra_timestamp,
SAFE_CAST(mozfun.map.get_key(event_extra, 'age') AS DECIMAL) AS extra_age,
SAFE_CAST(mozfun.map.get_key(event_extra, 'duration') AS INT64) AS extra_duration

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Every extra_* column depends on an extra key looked up by literal string (value, trigger, source, errorName, age, duration). If any client renames a key — or if the Rust implementation uses different names than Gecko's GIFFT-mirrored valuemozfun.map.get_key returns NULL and the column silently fills with NULLs for those rows, with no failure anywhere in the pipeline.

Adding a checks.sql (see docs/reference/data_checks.md) that asserts a non-null rate per implementation/platform for at least extra_status and extra_source would surface that instead of producing a quietly-empty column. This matters especially for the rust rows, whose keys aren't exercised by the existing monitoring.remote_settings_uptake_live view.

@scholtzan

This comment has been minimized.

@leplatrem

leplatrem commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

I have a few questions:

  • will this dataset be accessible from Yardstick since it's not in the monitoring folder? Should this be in the monitoring folder along the live data sql/moz-fx-data-shared-prod/monitoring/remote_settings_uptake_live/?
  • I don't care about client_id values, but I may want to group by on them. Should I hash them or something to avoid the shredder code? Is it ok like this?
  • Is my folder layout correct? I now regret not having put sql/moz-fx-data-shared-prod/remote_settings_logs_aggregates all in the same remote_settings folder to keep everything tidy. I assume it's too late or we'll break consumers, right?

Thank you!

@leplatrem
leplatrem force-pushed the rmst-514-clients-uptake branch from 0f70f63 to d68265c Compare September 22, 2026 14:00
@scholtzan

This comment has been minimized.

@scholtzan

This comment has been minimized.

@scholtzan

Copy link
Copy Markdown
Contributor

Integration report

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants