[DENG-11197] Migrate orphaning_dashboard job from spark#538
Draft
BenWu wants to merge 2 commits into
Draft
Conversation
| python -m update_orphaning_dashboard.main --run-date 2026-06-08 --dry-run | ||
| ``` | ||
|
|
||
| Authenticate with `gcloud auth application-default login`. `--dry-run` writes |
Member
There was a problem hiding this comment.
I guess we have to install the Google CLI to use this command: https://cloud.google.com/cli
@BenWu Could you please confirm?
Contributor
Author
There was a problem hiding this comment.
Yes some instructions on setup and accessing bigquery can be found here https://docs.telemetry.mozilla.org/cookbooks/bigquery/access.html#bigquery-access-request. You should already have the necessary access by default
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://mozilla-hub.atlassian.net/browse/DENG-11197
Migration of the spark job in https://github.com/mozilla/telemetry-airflow/blob/main/jobs/update_orphaning_dashboard_etl.py. This pushes the computations into bigquery so spark isn't needed. The output is the same as the existing job and testable locally by running the frontend with:
# run date should line up with what's in https://telemetry.mozilla.org/update-orphaning/ python -m update_orphaning_dashboard.main --run-date 2026-05-31 --dry-run python serve_frontend.pySwitching to glean was not part of this.
Checklist for reviewer:
Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
Scan the PR and verify that no changes (particularly to
.circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logsEnsure the container image will be using permissions granted to telemetry-airflow responsibly.