diff --git a/.gitignore b/.gitignore index 5615ac24..75a9381f 100644 --- a/.gitignore +++ b/.gitignore @@ -53,3 +53,4 @@ tmp/ # Built Go binaries (build artifacts, never commit) /duckgres-controlplane /duckgres-worker +/devserver diff --git a/CLAUDE.md b/CLAUDE.md index 640e87a6..d647f985 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,7 +46,7 @@ In topologies 2 and 3, the control plane also exposes an Arrow Flight SQL ingres - Flight SQL ingress adapter: `flight_ingress.go` - Runtime loops: `janitor.go`, `leader_loop.go`, `memory_rebalancer.go`, `runtime_tracker.go` - K8s / multitenant under build tag `kubernetes` (including: `multitenant.go`, `k8s_pool.go`, `k8s_pool_acquire.go`, `k8s_pool_spawn.go`, `k8s_pool_lifecycle.go`, `k8s_pool_reconcile.go`, `k8s_pool_helpers.go`, `k8s_factory.go`, `org_router.go`, `org_reserved_pool.go`, `sts_broker.go`, `shared_worker_activator.go`, `worker_rpc_security.go`, `janitor_leader_k8s.go`) - - Subpackages: `admin/` (HTTP admin API, `kubernetes` tag), `provisioner/` (k8s controller, `kubernetes` tag), `provisioning/` (HTTP API), `configstore/` (Postgres-backed config) + - Subpackages: `admin/` (HTTP admin API + dashboard, `kubernetes` tag; includes the models explorer UI `static/models.html` + `models_api.go`, and `devserver/` for local UI dev against a port-forwarded CP — see `admin/README.md`), `provisioner/` (k8s controller, `kubernetes` tag), `provisioning/` (HTTP API), `configstore/` (Postgres-backed config) - **duckdbservice/** — DuckDB Arrow Flight SQL service - Core: `service.go`, `flight_handler.go`, `arrow_helpers.go`, `auth.go`, `config.go` - Lifecycle, caching, profiling, metrics: `activation.go`, `transient.go`, `cache_proxy.go`, `profiling.go`, `progress.go`, `metrics.go` diff --git a/controlplane/admin/README.md b/controlplane/admin/README.md new file mode 100644 index 00000000..51654bd8 --- /dev/null +++ b/controlplane/admin/README.md @@ -0,0 +1,74 @@ +# Control-plane admin dashboard + +The control plane (multi-tenant / `kubernetes` build tag) serves a small admin +dashboard on `:8080`. It is gated behind the internal secret — there is no +public exposure; reach it via `kubectl port-forward`. + +## Pages + +| Route | Page | Purpose | +|-------|------|---------| +| `/`, `/models` | `static/models.html` | **Models explorer** — the primary surface. Sidebar of every config-store model grouped Tenants / Config / Runtime, a table of the selected model's rows, and a click-through detail panel. Nested warehouse sub-configs render as expandable sections. | +| `/orgs` | `static/orgs.html` | Org list + create/delete. | +| `/workers` | `static/workers.html` | Live worker/duckling status. | +| `/sessions` | `static/sessions.html` | Active sessions. | +| `/settings` | `static/settings.html` | Singleton config editing. | +| `POST /login` | `static/login.html` | Token login (sets the `duckgres_admin_token` cookie). | + +The pages are plain HTML + vanilla JS served via `//go:embed static/*` and +`html/template`. The models explorer and login page follow the dark "mission +control" design language (Chakra Petch + IBM Plex Mono, cyan/amber accents). +Because the templates are parsed with `html/template`, **do not write `{{` in +their inline JS** — only `login.html` uses template directives (`{{.Next}}` / +`{{.Error}}`). + +## Auth + +- Service-to-service: `X-Duckgres-Internal-Secret` header. +- Dashboard UI: the `POST /login` form mints an `HttpOnly` cookie. +- The internal secret plus rotation fallbacks are accepted (see `TokenSet`). +- `?token=` URL auth is deliberately rejected (#721). + +## Models explorer API (read-only) + +Backed by `models_api.go`. The registry in `modelDescriptors()` is the single +source of truth — add a config-store model there and it appears in the sidebar, +listing, and column derivation. + +- `GET /api/v1/models` → `{ "models": [ { key, label, group, count } ] }` — sidebar with live row counts. +- `GET /api/v1/models/:model` → `{ key, label, group, table, columns, count, truncated, rows }` — one model's rows. + +Rows are scanned into the typed model and marshaled via its `json` tags, so +columns tagged `json:"-"` (`OrgUser.Password`, `OrgUserSecret.Ciphertext`, +`DuckLakeConfig.S3SecretKey`, the singleton IDs) are dropped from the response — +**never swap the typed scan for a raw `map` scan**, that would leak them. +Runtime-schema tables (`worker_records`, `cp_instances`, …) are schema-qualified +with `ConfigStore.RuntimeSchema()`. Listings are capped at `modelsRowLimit` +(`truncated: true` when hit). + +Covered by `models_api_test.go` (redaction + real query path) and the +`models_explorer_api` assertion in `tests/e2e-mw-dev/harness.sh`. + +## Local UI development (no redeploy) + +Iterate on the UI against a *deployed* control plane without rebuilding the +image. A tiny dev proxy (`devserver/`) serves `static/` off disk and proxies +`/api`, `/login`, `/health` to the port-forwarded CP, injecting the internal +secret server-side. The browser sees one origin → no CORS, and the secret never +reaches page JS. The embedded UI uses relative `/api/v1` paths, so the same HTML +runs byte-for-byte embedded or under the dev proxy. + +```sh +# 1. port-forward the deployed control plane's admin API (own terminal): +just ui-port-forward # defaults: posthog-mw-dev / duckgres / duckgres-control-plane +# or: kubectl --context -n port-forward deploy/ 8080:8080 + +# 2. serve the UI locally, pointed at the port-forward: +DUCKGRES_INTERNAL_SECRET= just ui-dev +# → http://127.0.0.1:5173 + +# 3. edit controlplane/admin/static/*.html and refresh the browser. No rebuild. +``` + +Once the markup is right, it is already what `//go:embed` bakes into the CP — no +separate build step. diff --git a/controlplane/admin/api.go b/controlplane/admin/api.go index 6cb06997..66d7fd5b 100644 --- a/controlplane/admin/api.go +++ b/controlplane/admin/api.go @@ -72,6 +72,10 @@ type OrgStackInfo interface { // RegisterAPI registers all admin REST endpoints on the given router group. func RegisterAPI(r *gin.RouterGroup, store *configstore.ConfigStore, info OrgStackInfo) { registerAPIWithStore(r, newGormAPIStore(store), info) + // Generic read-only models explorer (sidebar + table + detail UI). Reads + // the concrete store directly because it needs the runtime schema name and + // raw DB for tables the typed apiStore interface doesn't surface. + registerModelsAPI(r, store) } func registerAPIWithStore(r *gin.RouterGroup, store apiStore, info OrgStackInfo) { diff --git a/controlplane/admin/dashboard.go b/controlplane/admin/dashboard.go index a9907cd6..ee9a2cdc 100644 --- a/controlplane/admin/dashboard.go +++ b/controlplane/admin/dashboard.go @@ -85,7 +85,11 @@ func APIAuthMiddleware(tokens TokenSet) gin.HandlerFunc { // RegisterDashboard serves the admin dashboard on the Gin engine. func RegisterDashboard(r *gin.Engine, tokens TokenSet) { - r.GET("/", dashboardPageHandler("index.html", tokens)) + // The models explorer is the primary dashboard surface: a sidebar of every + // config-store model, a table of its rows, and a detail view per row. "/" + // and "/models" both land here. The legacy per-entity pages stay reachable. + r.GET("/", dashboardPageHandler("models.html", tokens)) + r.GET("/models", dashboardPageHandler("models.html", tokens)) r.GET("/orgs", dashboardPageHandler("orgs.html", tokens)) r.GET("/workers", dashboardPageHandler("workers.html", tokens)) r.GET("/sessions", dashboardPageHandler("sessions.html", tokens)) diff --git a/controlplane/admin/devserver/main.go b/controlplane/admin/devserver/main.go new file mode 100644 index 00000000..06947d41 --- /dev/null +++ b/controlplane/admin/devserver/main.go @@ -0,0 +1,131 @@ +// Command devserver serves the admin dashboard's static assets off disk and +// reverse-proxies the API to a port-forwarded control plane, so the UI can be +// iterated on locally without rebuilding/redeploying the control-plane image. +// +// It mirrors the embedded serving contract exactly: the browser talks to a +// single origin (this server), static files are served from ./static, and any +// /api/, /login, or /health request is proxied to the target CP with the +// internal-secret header injected server-side. Because it's same-origin from +// the browser's point of view there is no CORS to configure and the secret +// never reaches the page's JavaScript — the UI uses relative /api/v1 paths and +// works byte-for-byte the same whether embedded or served here. +// +// Usage: +// +// # 1. port-forward the control plane's admin API (separate terminal): +// kubectl --context posthog-mw-dev -n port-forward deploy/duckgres-control-plane 8080:8080 +// +// # 2. run the dev server pointing at it: +// DUCKGRES_INTERNAL_SECRET= go run ./controlplane/admin/devserver +// # then open http://127.0.0.1:5173 +// +// Edit controlplane/admin/static/*.html and just refresh the browser — no +// rebuild. Once happy, the same files are what //go:embed bakes into the CP. +package main + +import ( + "flag" + "log" + "net/http" + "net/http/httputil" + "net/url" + "os" + "path/filepath" + "strings" +) + +func main() { + listen := flag.String("listen", "127.0.0.1:5173", "address to serve the UI on") + target := flag.String("target", "http://127.0.0.1:8080", "base URL of the port-forwarded control plane admin API") + staticDir := flag.String("static", defaultStaticDir(), "directory of static UI assets to serve off disk") + secret := flag.String("secret", os.Getenv("DUCKGRES_INTERNAL_SECRET"), "internal secret injected as X-Duckgres-Internal-Secret on proxied requests (defaults to $DUCKGRES_INTERNAL_SECRET)") + flag.Parse() + + if *secret == "" { + log.Println("WARNING: no internal secret set — proxied API requests will be rejected with 401. Set --secret or $DUCKGRES_INTERNAL_SECRET.") + } + + targetURL, err := url.Parse(*target) + if err != nil { + log.Fatalf("invalid --target %q: %v", *target, err) + } + + absStatic, err := filepath.Abs(*staticDir) + if err != nil { + log.Fatalf("resolve --static %q: %v", *staticDir, err) + } + if _, err := os.Stat(filepath.Join(absStatic, "models.html")); err != nil { + log.Fatalf("static dir %q has no models.html (run from repo root, or pass --static): %v", absStatic, err) + } + + proxy := httputil.NewSingleHostReverseProxy(targetURL) + baseDirector := proxy.Director + proxy.Director = func(req *http.Request) { + baseDirector(req) + req.Host = targetURL.Host + // Inject the service-to-service secret server-side; never trust or + // forward a browser cookie/credential through the dev proxy. + req.Header.Del("Cookie") + if *secret != "" { + req.Header.Set("X-Duckgres-Internal-Secret", *secret) + } + } + + mux := http.NewServeMux() + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + if isProxiedPath(r.URL.Path) { + proxy.ServeHTTP(w, r) + return + } + serveStatic(w, r, absStatic) + }) + + log.Printf("duckgres admin UI dev server: http://%s → proxying /api,/login,/health to %s", *listen, targetURL) + log.Printf("serving static assets from %s (edit + refresh, no rebuild)", absStatic) + if err := http.ListenAndServe(*listen, mux); err != nil { + log.Fatal(err) + } +} + +// isProxiedPath reports whether a request should be forwarded to the control +// plane rather than served from the local static dir. +func isProxiedPath(p string) bool { + return strings.HasPrefix(p, "/api/") || p == "/login" || p == "/health" +} + +// dashboardPages maps a request path (route or bare filename) to the static +// asset that serves it. The values are literals, so the filename handed to +// http.ServeFile never derives from the request — no path traversal is possible +// regardless of what the client sends. Routes mirror the CP's +// RegisterDashboard table; "/" and "/models" both land on the models explorer. +var dashboardPages = map[string]string{ + "": "models.html", + "models": "models.html", + "orgs": "orgs.html", + "workers": "workers.html", + "sessions": "sessions.html", + "settings": "settings.html", + "login": "login.html", + "models.html": "models.html", + "orgs.html": "orgs.html", + "workers.html": "workers.html", + "sessions.html": "sessions.html", + "settings.html": "settings.html", + "login.html": "login.html", +} + +// serveStatic serves the dashboard's static assets. Any unrecognized path falls +// back to the models explorer (single-page-style entry). +func serveStatic(w http.ResponseWriter, r *http.Request, dir string) { + file, ok := dashboardPages[strings.TrimPrefix(r.URL.Path, "/")] + if !ok { + file = "models.html" + } + http.ServeFile(w, r, filepath.Join(dir, file)) +} + +// defaultStaticDir resolves the static dir relative to the repo layout so the +// server can be run from the repo root with no flags. +func defaultStaticDir() string { + return filepath.Join("controlplane", "admin", "static") +} diff --git a/controlplane/admin/models_api.go b/controlplane/admin/models_api.go new file mode 100644 index 00000000..1c9bfe0e --- /dev/null +++ b/controlplane/admin/models_api.go @@ -0,0 +1,241 @@ +//go:build kubernetes + +package admin + +import ( + "net/http" + "reflect" + "strings" + + "github.com/gin-gonic/gin" + "github.com/posthog/duckgres/controlplane/configstore" +) + +// modelsRowLimit caps how many rows a single model listing returns. The +// config-schema tables are tiny, but runtime tables (worker_records, +// org_connection_queue) can grow; this keeps a stray listing from streaming an +// unbounded result into the browser. A listing at the cap sets `truncated`. +const modelsRowLimit = 2000 + +// modelGroup buckets descriptors into the UI sidebar sections. +type modelGroup string + +const ( + modelGroupTenants modelGroup = "Tenants" + modelGroupConfig modelGroup = "Config" + modelGroupRuntime modelGroup = "Runtime" +) + +// modelDescriptor describes one browsable config-store table for the generic +// models explorer. The descriptors are the single source of truth the sidebar, +// the listing endpoint, and the column derivation all read from — adding a new +// config-store model is one entry here. +type modelDescriptor struct { + // Key is the URL slug and stable identifier (e.g. "orgs"). + Key string + // Label is the human-facing sidebar name. + Label string + // Group is the sidebar section this model lives under. + Group modelGroup + // Table is the bare table name (the model's TableName()). Runtime models + // are schema-qualified at query time with the CP runtime schema. + Table string + // Runtime marks a table that lives in the control-plane runtime schema + // (cp_instances, worker_records, …) rather than the snapshot-backed config + // schema, so reads must be schema-qualified. + Runtime bool + // newSlice returns a pointer to a fresh, empty typed slice to scan into. + // Scanning into the typed model (not a map[string]any) means the struct's + // json tags apply on marshal, so json:"-" fields — OrgUser.Password, + // OrgUserSecret.Ciphertext, DuckLakeConfig.S3SecretKey, the singleton IDs — + // are dropped from the API response for free. Never swap this for a raw + // map scan: that would leak those columns. + newSlice func() any + // elem is the element type behind newSlice, used to derive column order. + elem reflect.Type +} + +// modelDescriptors returns the ordered registry of browsable models. Order is +// the sidebar order. Every persisted configstore model that an operator might +// want to inspect belongs here; if you add a model in configstore/models.go, +// add it here too (and assert it in models_api_test.go). +func modelDescriptors() []modelDescriptor { + mk := func(key, label string, group modelGroup, runtime bool, sample any) modelDescriptor { + // table name from the model's TableName() via the gorm Tabler contract. + table := "" + if t, ok := sample.(interface{ TableName() string }); ok { + table = t.TableName() + } + elem := reflect.TypeOf(sample) + return modelDescriptor{ + Key: key, + Label: label, + Group: group, + Table: table, + Runtime: runtime, + elem: elem, + newSlice: func() any { + return reflect.New(reflect.SliceOf(elem)).Interface() + }, + } + } + return []modelDescriptor{ + mk("orgs", "Orgs", modelGroupTenants, false, configstore.Org{}), + mk("org-users", "Org Users", modelGroupTenants, false, configstore.OrgUser{}), + mk("org-user-secrets", "Org User Secrets", modelGroupTenants, false, configstore.OrgUserSecret{}), + mk("managed-warehouses", "Managed Warehouses", modelGroupTenants, false, configstore.ManagedWarehouse{}), + + mk("global-config", "Global Config", modelGroupConfig, false, configstore.GlobalConfig{}), + mk("ducklake-config", "DuckLake Config", modelGroupConfig, false, configstore.DuckLakeConfig{}), + mk("ratelimit-config", "Rate Limit Config", modelGroupConfig, false, configstore.RateLimitConfig{}), + mk("querylog-config", "Query Log Config", modelGroupConfig, false, configstore.QueryLogConfig{}), + + mk("cp-instances", "Control Plane Instances", modelGroupRuntime, true, configstore.ControlPlaneInstance{}), + mk("worker-records", "Worker Records", modelGroupRuntime, true, configstore.WorkerRecord{}), + mk("flight-session-records", "Flight Session Records", modelGroupRuntime, true, configstore.FlightSessionRecord{}), + mk("org-connection-queue", "Org Connection Queue", modelGroupRuntime, true, configstore.OrgConnectionQueueEntry{}), + mk("org-connection-leases", "Org Connection Leases", modelGroupRuntime, true, configstore.OrgConnectionLease{}), + } +} + +type modelsHandler struct { + store *configstore.ConfigStore +} + +// registerModelsAPI registers the read-only generic models explorer used by the +// admin dashboard's models browser. +func registerModelsAPI(r *gin.RouterGroup, store *configstore.ConfigStore) { + h := &modelsHandler{store: store} + r.GET("/models", h.listModels) + r.GET("/models/:model", h.getModel) +} + +// qualifiedTable returns the table name to read for a descriptor, schema- +// qualifying runtime tables with the CP runtime schema. +func (h *modelsHandler) qualifiedTable(d modelDescriptor) string { + if d.Runtime { + return h.store.RuntimeSchema() + "." + d.Table + } + return d.Table +} + +// modelSummary is a sidebar entry: identity plus a live row count. +type modelSummary struct { + Key string `json:"key"` + Label string `json:"label"` + Group string `json:"group"` + Count int64 `json:"count"` +} + +func (h *modelsHandler) listModels(c *gin.Context) { + db := h.store.DB() + descriptors := modelDescriptors() + out := make([]modelSummary, 0, len(descriptors)) + for _, d := range descriptors { + var count int64 + // A count failure on one table must not blank the whole sidebar; report + // -1 so the UI can show "?" while the rest of the list still renders. + if err := db.Table(h.qualifiedTable(d)).Count(&count).Error; err != nil { + count = -1 + } + out = append(out, modelSummary{ + Key: d.Key, + Label: d.Label, + Group: string(d.Group), + Count: count, + }) + } + c.JSON(http.StatusOK, gin.H{"models": out}) +} + +// modelListing is the response for one model's rows. +type modelListing struct { + Key string `json:"key"` + Label string `json:"label"` + Group string `json:"group"` + Table string `json:"table"` + Columns []string `json:"columns"` + Count int64 `json:"count"` + Truncated bool `json:"truncated"` + Rows any `json:"rows"` +} + +func (h *modelsHandler) getModel(c *gin.Context) { + key := c.Param("model") + var desc *modelDescriptor + for _, d := range modelDescriptors() { + if d.Key == key { + d := d + desc = &d + break + } + } + if desc == nil { + c.JSON(http.StatusNotFound, gin.H{"error": "unknown model: " + key}) + return + } + + db := h.store.DB() + table := h.qualifiedTable(*desc) + + var total int64 + if err := db.Table(table).Count(&total).Error; err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + + rows := desc.newSlice() + // Limit + 1 detects truncation without a second query. + if err := db.Table(table).Limit(modelsRowLimit + 1).Find(rows).Error; err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()}) + return + } + truncated := false + rv := reflect.ValueOf(rows).Elem() + if rv.Len() > modelsRowLimit { + truncated = true + rv.Set(rv.Slice(0, modelsRowLimit)) + } + + c.JSON(http.StatusOK, modelListing{ + Key: desc.Key, + Label: desc.Label, + Group: string(desc.Group), + Table: table, + Columns: jsonFieldOrder(desc.elem), + Count: total, + Truncated: truncated, + Rows: rows, + }) +} + +// jsonFieldOrder returns the ordered top-level JSON keys for a struct type, +// skipping fields tagged json:"-". Embedded warehouse sub-structs are named +// fields (json:"s3", …) so they surface as a single nested-object column, which +// is what the detail view renders. This gives the table stable column headers +// even for an empty result set (the common case for runtime tables). +func jsonFieldOrder(t reflect.Type) []string { + if t == nil || t.Kind() != reflect.Struct { + return nil + } + cols := make([]string, 0, t.NumField()) + for i := 0; i < t.NumField(); i++ { + f := t.Field(i) + if f.PkgPath != "" { + continue // unexported + } + tag := f.Tag.Get("json") + name := f.Name + if tag != "" { + parts := strings.Split(tag, ",") + if parts[0] == "-" { + continue + } + if parts[0] != "" { + name = parts[0] + } + } + cols = append(cols, name) + } + return cols +} diff --git a/controlplane/admin/models_api_test.go b/controlplane/admin/models_api_test.go new file mode 100644 index 00000000..3f362bf6 --- /dev/null +++ b/controlplane/admin/models_api_test.go @@ -0,0 +1,155 @@ +//go:build kubernetes + +package admin + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/gin-gonic/gin" + "github.com/posthog/duckgres/controlplane/configstore" +) + +// TestModelDescriptorsRedaction is the no-DB guard for the load-bearing +// security invariant of the models explorer: secret-bearing columns must never +// appear as browsable columns. Because the listing scans into the typed model +// and marshals via json tags, a field tagged json:"-" is dropped — this test +// pins that the sensitive fields stay tagged and that the column derivation +// (jsonFieldOrder) honors it. If someone retags Password/Ciphertext/S3SecretKey +// to be serialized, this fails before the secret can leak through the UI. +func TestModelDescriptorsRedaction(t *testing.T) { + descs := modelDescriptors() + + // Registry must cover exactly the persisted models we expect. + wantKeys := map[string]bool{ + "orgs": true, "org-users": true, "org-user-secrets": true, "managed-warehouses": true, + "global-config": true, "ducklake-config": true, "ratelimit-config": true, "querylog-config": true, + "cp-instances": true, "worker-records": true, "flight-session-records": true, + "org-connection-queue": true, "org-connection-leases": true, + } + seen := map[string]bool{} + for _, d := range descs { + if seen[d.Key] { + t.Fatalf("duplicate descriptor key %q", d.Key) + } + seen[d.Key] = true + if d.Table == "" { + t.Errorf("descriptor %q has empty Table (missing TableName?)", d.Key) + } + if d.newSlice == nil || d.elem == nil { + t.Errorf("descriptor %q missing newSlice/elem", d.Key) + } + } + for k := range wantKeys { + if !seen[k] { + t.Errorf("registry missing expected model %q", k) + } + } + for k := range seen { + if !wantKeys[k] { + t.Errorf("registry has unexpected model %q (update test if intentional)", k) + } + } + + // Sensitive columns must not be derivable for the UI. + mustHide := map[string][]string{ + "org-users": {"password"}, + "org-user-secrets": {"ciphertext"}, + "ducklake-config": {"s3_secret_key", "s3secretkey"}, + "global-config": {"id"}, + } + for _, d := range descs { + cols := jsonFieldOrder(d.elem) + lower := map[string]bool{} + for _, c := range cols { + lower[strings.ToLower(c)] = true + } + for _, bad := range mustHide[d.Key] { + if lower[strings.ToLower(bad)] { + t.Errorf("model %q exposes column %q — must be json:\"-\"", d.Key, bad) + } + } + } +} + +// TestModelsAPIPostgres exercises the real query path against the config store: +// sidebar counts, a config-schema listing, a runtime-schema listing (schema +// qualification), and the end-to-end redaction guarantee — a seeded password +// hash must never appear in the API response bytes. +func TestModelsAPIPostgres(t *testing.T) { + store := newPostgresConfigStore(t) + + const secretHash = "$2a$10$DEADBEEFdeadbeefDEADBEEFdeadbeefDEADBEEFdeadbeefDEADBE" + if err := store.DB().Create(&configstore.Org{Name: "acme", DatabaseName: "acme_db"}).Error; err != nil { + t.Fatalf("seed org: %v", err) + } + if err := store.DB().Create(&configstore.OrgUser{OrgID: "acme", Username: "reader", Password: secretHash}).Error; err != nil { + t.Fatalf("seed user: %v", err) + } + + gin.SetMode(gin.TestMode) + r := gin.New() + registerModelsAPI(r.Group("/api/v1"), store) + + get := func(path string) (*httptest.ResponseRecorder, map[string]json.RawMessage) { + req := httptest.NewRequest(http.MethodGet, path, nil) + rec := httptest.NewRecorder() + r.ServeHTTP(rec, req) + var body map[string]json.RawMessage + if rec.Code == http.StatusOK { + if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil { + t.Fatalf("decode %s: %v (%s)", path, err, rec.Body.String()) + } + } + return rec, body + } + + // Sidebar listing. + rec, body := get("/api/v1/models") + if rec.Code != http.StatusOK { + t.Fatalf("GET /models = %d", rec.Code) + } + var summaries []modelSummary + if err := json.Unmarshal(body["models"], &summaries); err != nil { + t.Fatalf("decode summaries: %v", err) + } + gotOrgUsers := false + for _, s := range summaries { + if s.Key == "org-users" { + gotOrgUsers = true + if s.Count != 1 { + t.Errorf("org-users count = %d, want 1", s.Count) + } + } + } + if !gotOrgUsers { + t.Fatalf("sidebar missing org-users") + } + + // Config-schema listing must not leak the password hash. + rec, _ = get("/api/v1/models/org-users") + if rec.Code != http.StatusOK { + t.Fatalf("GET /models/org-users = %d", rec.Code) + } + if strings.Contains(rec.Body.String(), secretHash) { + t.Fatalf("org-users listing leaked password hash:\n%s", rec.Body.String()) + } + if !strings.Contains(rec.Body.String(), "reader") { + t.Fatalf("org-users listing missing seeded user") + } + + // Runtime-schema listing (schema qualification) must succeed even when empty. + rec, _ = get("/api/v1/models/worker-records") + if rec.Code != http.StatusOK { + t.Fatalf("GET /models/worker-records = %d (runtime schema qualification broken)", rec.Code) + } + + // Unknown model → 404. + rec, _ = get("/api/v1/models/nope") + if rec.Code != http.StatusNotFound { + t.Errorf("GET /models/nope = %d, want 404", rec.Code) + } +} diff --git a/controlplane/admin/static/login.html b/controlplane/admin/static/login.html index 08456f80..473614cd 100644 --- a/controlplane/admin/static/login.html +++ b/controlplane/admin/static/login.html @@ -3,33 +3,72 @@ - Duckgres - Admin Login - + Duckgres — Admin Login + + + + - -
-
-

Duckgres Admin

-

Enter the admin token printed in the control-plane logs.

-
+ +
+
DUCKGRES
+
Control Plane Admin
{{if .Error}} -
- {{.Error}} -
+
{{.Error}}
{{end}} -
+ -
- - -
- + + +
diff --git a/controlplane/admin/static/models.html b/controlplane/admin/static/models.html new file mode 100644 index 00000000..e61558cb --- /dev/null +++ b/controlplane/admin/static/models.html @@ -0,0 +1,489 @@ + + + + + + Duckgres — Control Plane + + + + + + +
+
+ DUCKGRES + Control Plane +
+
+ +
connecting
+
+ +
+ +
+
+ — + +
+ + + +
+
Select a model from the sidebar.
+
+ +
+ +
+
+

Session Expired

+

The admin token is no longer valid.

+ Reload to sign in +
+
+ + + + diff --git a/justfile b/justfile index 63eddb55..0a99e67a 100644 --- a/justfile +++ b/justfile @@ -46,6 +46,18 @@ run-control-plane: build build-k8s-image tag="duckgres:test": docker build --build-arg BUILD_TAGS=kubernetes -t {{tag}} . +# Port-forward a deployed control plane's admin API to localhost:8080 (run in its own terminal) +[group('dev')] +ui-port-forward context="posthog-mw-dev" namespace="duckgres" deploy="duckgres-control-plane": + kubectl --context {{context}} -n {{namespace}} port-forward deploy/{{deploy}} 8080:8080 + +# Serve the admin UI locally, proxying /api to the port-forwarded control plane. +# Run `just ui-port-forward` first; set DUCKGRES_INTERNAL_SECRET to the CP secret. +# Edit controlplane/admin/static/*.html and refresh the browser — no rebuild/redeploy. +[group('dev')] +ui-dev target="http://127.0.0.1:8080" listen="127.0.0.1:5173": + go run ./controlplane/admin/devserver --target {{target}} --listen {{listen}} + # Recreate the local kind cluster used by the portable K8s integration flow [group('dev')] kind-cluster-reset: diff --git a/tests/e2e-mw-dev/harness.sh b/tests/e2e-mw-dev/harness.sh index 89a3be9d..6afa90f8 100755 --- a/tests/e2e-mw-dev/harness.sh +++ b/tests/e2e-mw-dev/harness.sh @@ -52,6 +52,10 @@ # isolation : two tenants see distinct catalogs (cross-tenant read denied). # admin auth : the dashboard rejects ?token= query-param auth (#721); only # the internal-secret header / POST login form authenticate. +# models api : the dashboard models explorer lists every config-store model +# with row counts (GET /api/v1/models) and one model's rows + +# columns (GET /api/v1/models/:model), redacting secret-bearing +# columns (org_users.password must never appear in the UI). # lifecycle : deprovision → warehouse deleted → Duckling CR fully gone # (finalizer cascade that drops the cnpg role+db completed). # Same-id re-provision is covered across runs by run.sh, not @@ -1307,6 +1311,47 @@ internal_secret_fallback_auth() { [ "$code" = "401" ] || fail "GET /api/v1/orgs with garbage secret returned $code, want 401 (accept-list must stay closed)" } +# ---- admin models explorer API --------------------------------------------- +# The dashboard's generic models explorer (sidebar + table + detail UI) is +# backed by GET /api/v1/models (sidebar: every config-store model + a live row +# count) and GET /api/v1/models/:model (one model's rows + derived columns). +# This asserts the surface is wired and — load-bearing — that the listing never +# leaks secret-bearing columns: org_users carry a bcrypt password hash in the +# DB, but Password is json:"-", so it must NOT appear in the API response. A +# regression here (someone retagging Password) would surface a credential hash +# in the operator UI. +models_explorer_api() { + log "admin models explorer API (sidebar + listing + redaction)" + # Sidebar: must enumerate the known models across all three groups. + idx="$(curl -fsS -H "$H" "$API/api/v1/models")" + for key in orgs org-users managed-warehouses global-config worker-records org-connection-leases; do + echo "$idx" | jq -e --arg k "$key" '.models[] | select(.key == $k)' >/dev/null \ + || fail "models index missing model '$key'" + done + # The seeded cnpg org must show up with a >=1 count on the orgs model. + orgcount="$(echo "$idx" | jq -r '.models[] | select(.key=="orgs") | .count')" + [ "$orgcount" -ge 1 ] 2>/dev/null || fail "orgs model count '$orgcount' < 1 (expected seeded orgs)" + + # org-users listing: present users, derive columns, and NEVER a password. + users="$(curl -fsS -H "$H" "$API/api/v1/models/org-users")" + echo "$users" | jq -e '.columns | index("username")' >/dev/null \ + || fail "org-users listing missing 'username' column" + if echo "$users" | jq -e '.columns | index("password")' >/dev/null 2>&1; then + fail "org-users listing exposes a 'password' column (must be redacted)" + fi + if echo "$users" | jq -e '[.rows[] | has("password")] | any' >/dev/null 2>&1; then + fail "org-users row carries a password field (credential hash leak via UI)" + fi + # A runtime-schema model must list (schema qualification works), returning + # columns even when the table is empty. + curl -fsS -H "$H" "$API/api/v1/models/worker-records" \ + | jq -e '.columns | index("worker_id")' >/dev/null \ + || fail "worker-records listing missing 'worker_id' column (runtime schema qualification broken)" + # Unknown model → 404. + code="$(curl -s -o /dev/null -w '%{http_code}' -H "$H" "$API/api/v1/models/not-a-model")" + [ "$code" = "404" ] || fail "GET /api/v1/models/not-a-model returned $code, want 404" +} + # ---- tenant isolation ----------------------------------------------------- # Two tenants (cnpg + ext) back onto distinct DuckLake metadata stores, so a # table created by one is invisible to the other. Ported (logical half) from @@ -1556,6 +1601,11 @@ main() { log "settling ${CONFIG_POLL_SETTLE:-12}s for CP auth cache…" sleep "${CONFIG_POLL_SETTLE:-12}" + # Admin models explorer: orgs + their users now exist in the config store, so + # the sidebar counts are non-trivial and the org-users redaction check bites + # against a real bcrypt-hash-bearing row. + models_explorer_api + # Join the kubectl background download started at script load — first k use # is inside the lanes, so the fetch overlapped the whole provisioning phase. bootstrap_kubectl @@ -1579,7 +1629,7 @@ main() { # mid-run image bump); it stays covered by the controlplane/ unit tests. log "SKIP version-reaper (needs an in-run image bump; see README)" - log "PASS: admin-no-query-token + wire + malformed-startup-resilience + jsonb-concat + cold-burst-absorption + pipeline-error-recovery + cancel-reuse + activation(DuckLake/Iceberg) + iceberg-cold-first-connect + ext-forks + worker-pod + concurrency + durability + crash-recovery + busy-only-do-not-disrupt + graceful-drain + one-session-per-worker + parallel-cold-burst-ramp + worker-sizing(cnpg DuckLake+Iceberg) + org-default-profile(ext) + persistent-user-secrets(cnpg+ext, cross-user isolation) + isolation + lifecycle-teardown, on cnpg & ext (4 parallel lanes)" + log "PASS: admin-no-query-token + models-explorer-api(redaction) + wire + malformed-startup-resilience + jsonb-concat + cold-burst-absorption + pipeline-error-recovery + cancel-reuse + activation(DuckLake/Iceberg) + iceberg-cold-first-connect + ext-forks + worker-pod + concurrency + durability + crash-recovery + busy-only-do-not-disrupt + graceful-drain + one-session-per-worker + parallel-cold-burst-ramp + worker-sizing(cnpg DuckLake+Iceberg) + org-default-profile(ext) + persistent-user-secrets(cnpg+ext, cross-user isolation) + isolation + lifecycle-teardown, on cnpg & ext (4 parallel lanes)" } main "$@"