Skip to content

feat(bi/superset): adopt image.source/image.tag config - #584

Open
RonaldHensbergen wants to merge 5 commits into
mainfrom
feat/407-superset-image-source
Open

feat(bi/superset): adopt image.source/image.tag config#584
RonaldHensbergen wants to merge 5 commits into
mainfrom
feat/407-superset-image-source

Conversation

@RonaldHensbergen

@RonaldHensbergen RonaldHensbergen commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Summary

Implements #534 (part of #407) and adds the CLI-level layer needed to actually use it: adopting image.source/image.tag in modules/bi/superset/module.yaml's configSchema alone left no way to opt into registry mode without hand-editing profile YAML, and no way to know what tag to pull without a hardcoded, drifting literal. This PR closes both gaps.

Module schema (#534)

  • modules/bi/superset/module.yaml: adds image.source (enum build/registry, default build) and image.tag to configSchema, mirroring Dagster's field shape (no image.variant, since Superset has no base/hardened split).
  • No cli/renderer.py/cli/validator.py changes needed for this part: _apply_image_source() and validate_image_source_config() are already module-agnostic, and Superset's module id (superset) matches publish-images.yml's matrix image name and docker.io/ronaldsoeverein/<name> ref exactly.

CLI: --image-source flag + cds config image.source

  • cli/main.py: new --image-source {build,registry} flag on render/up/test (mirrors --hardened), plus a new cds config key image.source (mirrors environment/security.strict) — cds config set/get/unset image.source, applied automatically whenever --image-source is omitted. image.source fits a project-wide default well (one team-wide "deploy from registry" toggle); a single global image.tag does not, since each module tracks an independently-versioned tag, so no cds config key was added for image.tag.
  • cli/planner.py: build_plan() gains an image_source parameter and _apply_image_source_override(), schema-driven like the existing _supports_image_variant() — only touches modules whose configSchema exposes image.source. Switching to "registry" without an explicit config.image.tag re-derives the tag from the module's schema-pinned default, correctly re-prefixing it with "<variant>-" when --hardened (or any other non-base image.variant) is combined with --image-source registry, matching publish-images.yml's own variant-prefixing scheme. An explicitly profile-pinned tag is never overwritten.

Hardcoded, Renovate-tracked default tags

  • modules/orchestration/dagster/module.yaml / modules/bi/superset/module.yaml: image.tag now has a # cds-registry-tag-source: ...-marked default (1.13.20 / 6.1.0) instead of no default, so --image-source registry / cds config set image.source registry has a valid tag to pull without any profile changes.
  • renovate.json: two new customManagers keep those defaults in sync automatically — pypi/dagster for Dagster's default (mirrors the dagster== pin in images/dagster/requirements.txt) and docker/apache/superset for Superset's (mirrors the apache/superset base pin in images/superset/base/Dockerfile). Validated with renovate-config-validator --strict.

Manually verified

  • cds render --image-source registry --hardened on profiles/local-dagster-postgres-superset: Dagster renders docker.io/ronaldsoeverein/dagster:hardened-1.13.20, Superset (no variant) renders docker.io/ronaldsoeverein/superset:6.1.0.
  • cds render (no flags): both services keep their build: blocks, unaffected.
  • cds config set/get/unset image.source: round-trips correctly through .cds/config.json.

Out of scope (tracked separately)

Testing

  • python -m unittest discover -s tests -p "test_*.py" -v — 618 tests, all pass (1 skipped)
  • make lint — passes
  • npx --yes --package renovate -- renovate-config-validator --strict renovate.json — passes
  • cds validate profiles/local-dagster-postgres-superset/profile.yaml — still valid

Resolves #534

Extends the image.source (build|registry) / image.tag config pattern
introduced for Dagster (#532/#533) to modules/bi/superset/module.yaml,
so cds up/render can pull the published, scanned, signed
docker.io/ronaldsoeverein/superset image instead of always building
locally. No renderer/planner/validator changes needed: _apply_image_source
and validate_image_source_config are already module-agnostic, and
Superset's module id ('superset') matches publish-images.yml's matrix
image name and docker.io ref exactly.

- modules/bi/superset/module.yaml: add image.source (enum build/registry,
  default build) and image.tag to configSchema, mirroring Dagster's field
  shape (no image.variant, since Superset has no base/hardened split).
- tests/test_renderer.py: new SupersetImageSourceRenderingTest covering
  default build behavior, registry+tag rewrite for both superset-init and
  superset services, and registry-without-tag fallback.

Resolves #534

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RonaldHensbergen and others added 4 commits September 1, 2026 23:43
…-tracked default tags

Addresses feedback on PR #584: adopting image.source/image.tag in module
configSchema alone left no way to actually opt into it without hand-editing
profile YAML, and no way to know what tag to pull without a hardcoded,
drifting literal.

- cli/main.py: new --image-source {build,registry} flag on render/up/test
  (mirrors --hardened), plus a new cds config image.source project default
  (mirrors environment/security.strict): cds config set/get/unset
  image.source, applied whenever --image-source is omitted.
- cli/planner.py: build_plan() gains an image_source parameter and
  _apply_image_source_override(), which only touches modules whose
  configSchema exposes image.source (schema-driven, like
  _supports_image_variant). Switching to "registry" without an explicit
  config.image.tag re-derives the tag from the module's schema-pinned
  default, correctly re-prefixing it with "<variant>-" when --hardened
  (or any other non-base image.variant) is combined with --image-source
  registry -- matching publish-images.yml's own variant-prefixing scheme.
  An explicitly profile-pinned tag is never overwritten.
- modules/orchestration/dagster/module.yaml,
  modules/bi/superset/module.yaml: image.tag now has a hardcoded
  "# cds-registry-tag-source: ..." marked default (1.13.20 / 6.1.0)
  instead of no default, so --image-source registry / cds config set
  image.source registry has a valid tag to pull without profile changes.
- renovate.json: two new customManagers keep those defaults in sync --
  pypi/dagster for Dagster's default (mirrors the dagster== pin in
  images/dagster/requirements.txt) and docker/apache/superset for
  Superset's (mirrors the apache/superset base pin in
  images/superset/base/Dockerfile). Validated with
  renovate-config-validator --strict.
- tests/test_planner.py: new coverage for the image_source override
  (supported/unsupported modules, explicit-tag preservation, and the
  hardened+registry variant-prefix interaction).
- tests/test_main.py: new cds config set/get/unset image.source coverage,
  mirroring the existing security.strict tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…work

Superset's metadata database is always postgres, so it always needs
psycopg2. Previously that dependency was only added via a build-time
IMAGE_PACKAGES arg passed from module.yaml, which
.github/workflows/publish-images.yml never sets when it builds the
published image. As a result, `cds up --image-source registry` pulled
an image missing psycopg2, and superset-init failed with
ModuleNotFoundError: No module named 'psycopg2'.

Move psycopg2-binary into images/superset/requirements.txt (installed
unconditionally in the Dockerfile) instead of relying on an ARG that's
only populated for local builds, mirroring how Dagster's DB_BACKEND ARG
default matches its always-needed postgres case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt image.source in modules/bi/superset/module.yaml

1 participant