[HWORKS-2810] Add Airflow 3.0.6 docs (upgrade + security + operator notes)#588
Open
jimdowling wants to merge 6 commits into
Open
[HWORKS-2810] Add Airflow 3.0.6 docs (upgrade + security + operator notes)#588jimdowling wants to merge 6 commits into
jimdowling wants to merge 6 commits into
Conversation
Three new pages and one rewrite, covering the v3 upgrade landing on the `airflow3-upgrade` branch across hopsworks-front, hopsworks-ee, docker-images, and hopsworks-helm. - `user_guides/projects/airflow/airflow3_upgrade.md` — porting checklist from Airflow 1.10 → 3.0.6 (schedule_interval → schedule, the FAB-era operator imports → standard provider package, `dag_id` naming scheme), the new Hopsworks operators / sensors shipped in `apache-airflow-providers-hopsworks`, the per-DAG API key written into an Airflow Variable (so the DAG file never carries the secret), the Hopsworks UI's Last-runs column and row-click behavior, the metadata-DB-on-upgrade caveat, and the task-logs-across-pod-restarts caveat (the `hostname_callable = airflow.utils.net.get_host_ip_address` scheduler config is what makes serve_logs reachable cross-pod). - `user_guides/projects/airflow/security_model.md` — surface-by-surface authorization matrix. Notes that Audit Log is intentionally visible to all users but row-filtered server-side to `dag_id IN (user's project dag_ids)` via the auth-manager's `do_orm_execute` hook, that Airflow 3.0.x does not expose a synchronous refresh_user hook so membership is stable for the cookie TTL, and that the per-DAG API key Variable is the one platform-managed exception to the "don't put project-private secrets in Airflow Variables" rule. - `setup_installation/admin/airflow3.md` — operator notes. Pins to `apache/airflow:3.0.6-python3.12`. Calls out that `db-reset` is install-only (`hopsworkslib.isInstall` gate, set `global._hopsworks.mode=install` on first install). Documents the liveness-probe choice (TCP on 8793 for scheduler, `/proc/1/comm` read for dag-processor) because `pgrep` is unreliable under `yama.ptrace_scope >= 1` / `hidepid` hardening. - `user_guides/projects/airflow/airflow.md` — rewrite the operator invocation examples for the v3 import paths, drop the obsolete `access_control` parameter (the new auth manager doesn't honor it; authz comes from the dag_id-to-project sidecar table), and link forward to the two new pages plus the upgrade page. Nav entries added under `User Guides → Projects → Airflow` and under `Setup → Administration`. Reviewed-by: OpenAI Codex (GPT-5 via codex-plugin-cc 0.1.0) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds and updates Hopsworks documentation to cover the Airflow 3.0.6 upgrade, including DAG porting guidance, the updated security/authorization model, and operator-facing deployment notes, and wires the new pages into the MkDocs navigation.
Changes:
- Added new Airflow 3 upgrade guide and a dedicated security model page under
User Guides → Projects → Airflow. - Added new operator notes page under
Setup and Administration. - Updated the existing Airflow overview page for Airflow 3 import paths/behavior and to link to the new docs; updated
mkdocs.ymlnav accordingly.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| mkdocs.yml | Adds nav entries for the new Airflow pages and operator notes. |
| docs/user_guides/projects/airflow/security_model.md | New page defining the Airflow 3 authorization surface and non-isolation caveats. |
| docs/user_guides/projects/airflow/airflow3_upgrade.md | New upgrade/porting checklist and behavior notes for Airflow 3.0.6 in Hopsworks. |
| docs/user_guides/projects/airflow/airflow.md | Updates the Airflow overview to reflect Airflow 3, new provider imports, and links to the new pages. |
| docs/setup_installation/admin/airflow3.md | New admin/operator notes for chart behavior, key rotation, probes, and upgrade caveats. |
Docs were silent on the migration path for DAGs composed before the no-embed change. The `HOPSWORKS_API_KEY` env-var line still works as the fourth tier of the hook's fallback, so existing DAG files keep running; users who want the key out of the file regenerate the DAG. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ences `md-snakeoil` runs in CI and strips F401 unused imports inside Python code fences. Two doc blocks were demonstration imports of the new Hopsworks providers — restore them with `# noqa: F401` so snakeoil keeps them, and pre-apply the trailing-blank-line trim the CI runs after snakeoil so `git diff --exit-code` passes on the next run. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR review pointed out that the per-DAG `hopsworks_api_key_<sha256(dag_id)[:16]>` Airflow Variable is not actually a per-DAG security boundary at task runtime: `dag_id` is non-secret and the hash is reproducible, so any running DAG can `Variable.get` another DAG's hashed name. Make this explicit in the security model so users understand the fallback is a shared credential surface within the cluster, not an isolated per-DAG vault. Point at task-token-exchange as the path that actually stays per-task. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…gicalclocks#588 https://hopsworks.atlassian.net/browse/HWORKS-2810 Reconcile the inconsistencies the Copilot review flagged and fix the snippets that lint with F821: security_model.md: - Reflowed the multi-sentence-on-one-line lines per the repo's one-sentence-per-line style (intro, "What is not isolated"). - The `POST /auth/token` reference now spells out the full proxied path (`POST /hopsworks-api/airflow/auth/token`) so it matches the upgrade page and operators don't have to guess what's under the `[api] base_url` prefix. setup_installation/admin/airflow3.md: - Reflowed the multi-line intro. - Reverse-proxy section was contradictory with security_model.md: it claimed membership is refreshed "on every forwarded request" via an `X-Hopsworks-JWT` header. The auth manager doesn't refresh per-request in 3.0.x — it caches `project_ids` / `project_roles` for the cookie TTL (1 hour) and the Hopsworks backend evicts the cache via `POST /auth/internal/invalidate` on membership changes. Rewritten to match what the code does and cross-link to the security model page. user_guides/projects/airflow/airflow.md: - Three Python snippets reference `HopsworksLaunchOperator`, `HopsworksJobSuccessSensor`, and `HopsworksHdfsSensor` without importing them. `snakeoil` lints every fenced Python block via ruff and F821s on undefined names. Added the `from hopsworks.airflow.{operators,sensors} import ...` lines so the blocks are self-contained. - The page described the runs column as "colored dots"; the v3 upgrade page (and the actual UI) renders **colored squares**. Aligned the wording and reused the same colour legend. user_guides/projects/airflow/airflow3_upgrade.md: - Reflowed the "must rewrite" paragraph that mixed three sentences on one line. Reference-style heading-ID linking (Copilot suggestion) was left as a docs-team follow-up: mkdocs + mike already resolves the `.md` relative links correctly across versioned builds, and switching to autorefs heading IDs would touch every airflow page in lockstep without a corresponding repo-wide policy decision. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/HWORKS-2810 Snakeoil's ruff rules expect two blank lines between top-level import statements and the next definition / statement (E302). The three operator + sensor examples on the Airflow user-guide page had only one blank line between the `from … import …` and the following call, so the CI snakeoil step rewrote them and then failed on the resulting `git diff --exit-code`. Add the second blank line in each of the three code blocks so the rendered docs match what snakeoil would produce. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documentation pages for the Airflow 3.0.6 upgrade landing on the
airflow3-upgradebranch inhopsworks-front,hopsworks-ee,docker-images, andhopsworks-helm.user_guides/projects/airflow/airflow3_upgrade.md— Airflow 1.10 → 3.0.6 porting checklist, new Hopsworks operators / sensors (HopsworksLaunchOperator,HopsworksJobSuccessSensor,HopsworksHdfsSensor), the per-DAG API key written into an Airflow Variable (so the DAG file never carries the secret), the Hopsworks UI's Last-runs column behavior, the metadata-DB-on-upgrade caveat, and the task-logs-across-pod-restarts caveat (hostname_callable = get_host_ip_addressmakes serve_logs reachable cross-pod).user_guides/projects/airflow/security_model.md— surface-by-surface authorization matrix. Audit Log is visible to all authenticated users with rows filtered server-side todag_id IN (user's project dag_ids)via the auth-manager'sdo_orm_executehook; membership refresh model under Airflow 3.0.x; per-DAG API key Variable as the one platform-managed exception to the "no Airflow Variables for project secrets" rule.setup_installation/admin/airflow3.md— operator notes. Pins toapache/airflow:3.0.6-python3.12.db-resetis install-only (hopsworkslib.isInstall). Liveness probes use TCP //proc/1/commbecausepgrepis unreliable underyama.ptrace_scope >= 1/hidepid.user_guides/projects/airflow/airflow.md— v3 import paths, drop the obsoleteaccess_controlparameter (the new auth manager doesn't honor it), link forward to the new pages.mkdocs.ymlnav entries underUser Guides → Projects → AirflowandSetup → Administration.Test plan
npx markdownlint-cli2 "**/*.md"passes on the diff (verified locally: 0 errors).mkdocs build -ssucceeds with the new nav entries.security_model.md,airflow3_upgrade.md) resolve.🤖 Generated with Claude Code