feat(bi/superset): adopt image.source/image.tag config - #584
Open
RonaldHensbergen wants to merge 5 commits into
Open
feat(bi/superset): adopt image.source/image.tag config#584RonaldHensbergen wants to merge 5 commits into
RonaldHensbergen wants to merge 5 commits into
Conversation
Extends the image.source (build|registry) / image.tag config pattern introduced for Dagster (#532/#533) to modules/bi/superset/module.yaml, so cds up/render can pull the published, scanned, signed docker.io/ronaldsoeverein/superset image instead of always building locally. No renderer/planner/validator changes needed: _apply_image_source and validate_image_source_config are already module-agnostic, and Superset's module id ('superset') matches publish-images.yml's matrix image name and docker.io ref exactly. - modules/bi/superset/module.yaml: add image.source (enum build/registry, default build) and image.tag to configSchema, mirroring Dagster's field shape (no image.variant, since Superset has no base/hardened split). - tests/test_renderer.py: new SupersetImageSourceRenderingTest covering default build behavior, registry+tag rewrite for both superset-init and superset services, and registry-without-tag fallback. Resolves #534 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-tracked default tags Addresses feedback on PR #584: adopting image.source/image.tag in module configSchema alone left no way to actually opt into it without hand-editing profile YAML, and no way to know what tag to pull without a hardcoded, drifting literal. - cli/main.py: new --image-source {build,registry} flag on render/up/test (mirrors --hardened), plus a new cds config image.source project default (mirrors environment/security.strict): cds config set/get/unset image.source, applied whenever --image-source is omitted. - cli/planner.py: build_plan() gains an image_source parameter and _apply_image_source_override(), which only touches modules whose configSchema exposes image.source (schema-driven, like _supports_image_variant). Switching to "registry" without an explicit config.image.tag re-derives the tag from the module's schema-pinned default, correctly re-prefixing it with "<variant>-" when --hardened (or any other non-base image.variant) is combined with --image-source registry -- matching publish-images.yml's own variant-prefixing scheme. An explicitly profile-pinned tag is never overwritten. - modules/orchestration/dagster/module.yaml, modules/bi/superset/module.yaml: image.tag now has a hardcoded "# cds-registry-tag-source: ..." marked default (1.13.20 / 6.1.0) instead of no default, so --image-source registry / cds config set image.source registry has a valid tag to pull without profile changes. - renovate.json: two new customManagers keep those defaults in sync -- pypi/dagster for Dagster's default (mirrors the dagster== pin in images/dagster/requirements.txt) and docker/apache/superset for Superset's (mirrors the apache/superset base pin in images/superset/base/Dockerfile). Validated with renovate-config-validator --strict. - tests/test_planner.py: new coverage for the image_source override (supported/unsupported modules, explicit-tag preservation, and the hardened+registry variant-prefix interaction). - tests/test_main.py: new cds config set/get/unset image.source coverage, mirroring the existing security.strict tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…work Superset's metadata database is always postgres, so it always needs psycopg2. Previously that dependency was only added via a build-time IMAGE_PACKAGES arg passed from module.yaml, which .github/workflows/publish-images.yml never sets when it builds the published image. As a result, `cds up --image-source registry` pulled an image missing psycopg2, and superset-init failed with ModuleNotFoundError: No module named 'psycopg2'. Move psycopg2-binary into images/superset/requirements.txt (installed unconditionally in the Dockerfile) instead of relying on an ARG that's only populated for local builds, mirroring how Dagster's DB_BACKEND ARG default matches its always-needed postgres case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #534 (part of #407) and adds the CLI-level layer needed to actually use it: adopting
image.source/image.taginmodules/bi/superset/module.yaml'sconfigSchemaalone left no way to opt intoregistrymode without hand-editing profile YAML, and no way to know what tag to pull without a hardcoded, drifting literal. This PR closes both gaps.Module schema (#534)
modules/bi/superset/module.yaml: addsimage.source(enumbuild/registry, defaultbuild) andimage.tagtoconfigSchema, mirroring Dagster's field shape (noimage.variant, since Superset has no base/hardened split).cli/renderer.py/cli/validator.pychanges needed for this part:_apply_image_source()andvalidate_image_source_config()are already module-agnostic, and Superset's module id (superset) matchespublish-images.yml's matrix image name anddocker.io/ronaldsoeverein/<name>ref exactly.CLI:
--image-sourceflag +cds config image.sourcecli/main.py: new--image-source {build,registry}flag onrender/up/test(mirrors--hardened), plus a newcds configkeyimage.source(mirrorsenvironment/security.strict) —cds config set/get/unset image.source, applied automatically whenever--image-sourceis omitted.image.sourcefits a project-wide default well (one team-wide "deploy from registry" toggle); a single globalimage.tagdoes not, since each module tracks an independently-versioned tag, so nocds configkey was added forimage.tag.cli/planner.py:build_plan()gains animage_sourceparameter and_apply_image_source_override(), schema-driven like the existing_supports_image_variant()— only touches modules whoseconfigSchemaexposesimage.source. Switching to"registry"without an explicitconfig.image.tagre-derives the tag from the module's schema-pinned default, correctly re-prefixing it with"<variant>-"when--hardened(or any other non-baseimage.variant) is combined with--image-source registry, matchingpublish-images.yml's own variant-prefixing scheme. An explicitly profile-pinned tag is never overwritten.Hardcoded, Renovate-tracked default tags
modules/orchestration/dagster/module.yaml/modules/bi/superset/module.yaml:image.tagnow has a# cds-registry-tag-source: ...-markeddefault(1.13.20/6.1.0) instead of no default, so--image-source registry/cds config set image.source registryhas a valid tag to pull without any profile changes.renovate.json: two newcustomManagerskeep those defaults in sync automatically —pypi/dagsterfor Dagster's default (mirrors thedagster==pin inimages/dagster/requirements.txt) anddocker/apache/supersetfor Superset's (mirrors theapache/supersetbase pin inimages/superset/base/Dockerfile). Validated withrenovate-config-validator --strict.Manually verified
cds render --image-source registry --hardenedonprofiles/local-dagster-postgres-superset: Dagster rendersdocker.io/ronaldsoeverein/dagster:hardened-1.13.20, Superset (no variant) rendersdocker.io/ronaldsoeverein/superset:6.1.0.cds render(no flags): both services keep theirbuild:blocks, unaffected.cds config set/get/unset image.source: round-trips correctly through.cds/config.json.Out of scope (tracked separately)
image.source/image.taginto a shared JSON Schema fragment reused by both modules — optional per Adopt image.source in modules/bi/superset/module.yaml #534, left as-is sinceconfigSchemablocks aren't composed via$refacross module.yaml files today.docs/modules.md, README error-code/flag reference) — tracked in Document image.source (build|registry) in docs/modules.md and README #535, which explicitly depends on this issue.Testing
python -m unittest discover -s tests -p "test_*.py" -v— 618 tests, all pass (1 skipped)make lint— passesnpx --yes --package renovate -- renovate-config-validator --strict renovate.json— passescds validate profiles/local-dagster-postgres-superset/profile.yaml— still validResolves #534