AET-DevOps26 · f-s-h · Jul 10, 2026 · Jul 9, 2026 · Jul 9, 2026 · Jul 9, 2026
diff --git a/README.md b/README.md
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -0,0 +1,58 @@
+# Architecture
+
+This is the detailed companion to the [README's Architecture section](../README.md#architecture): per-service responsibilities, who calls whom, and how a request is authenticated end to end. There is deliberately no UML diagram here — see [`docs/outdated/`](outdated/) for why the previous ones were retired.
+
+## Subsystems
+
+### Client — `web-client`
+
+A React 19 + TypeScript SPA (Vite). It is the only subsystem with a UI, and it talks to **every** backend service directly over REST through the reverse proxy — there is no backend-for-frontend or API gateway aggregation layer. Each feature area (members, events, feedback, finance, letters, organization, the GenAI helper) has its own typed API client generated from `api/openapi.yaml`.
+
+Auth: `keycloak-js` (PKCE S256) obtains a JWT from Keycloak; an Axios request interceptor attaches it as `Authorization: Bearer <token>` and refreshes it proactively when it's within 30s of expiry.
+
+### Server — six Spring Boot 3 microservices
+
+| Service | Owns | Notes |
+|---|---|---|
+| Organization | sports, teams, role assignment | Also calls the Keycloak Admin REST API directly (`KeycloakRoleService`) to keep realm role assignments in sync with team/role changes made in-app |
+| Member | member profiles | Also calls the Keycloak Admin REST API (`KeycloakService`) for the same reason |
+| Event | training sessions, enrollment, attendance | |
+| Feedback | trainer feedback, progress notes | Read by the GenAI service (see below) |
+| Finance | one-time/recurring billing | |
+| Letter | PDF/email generation from templates | The only Spring service with no database — templates and generated artifacts are not persisted as domain rows |
+
+Each service is a stateless OAuth2 resource server: it validates the incoming Bearer JWT against Keycloak's JWK set and maps `realm_access.roles` claims to Spring `ROLE_*` authorities. None of the six call each other directly — the only cross-service traffic on the server side is organization/member → Keycloak (role sync) and GenAI → feedback (below).
+
+Organization-service and member-service both authenticate to the Keycloak Admin REST API as the same confidential client, `org-role-sync` (service-account/client-credentials grant — see [Proxy & Auth](#proxy--auth-traefik--nginx-ingress--keycloak) below). Neither hardcodes its secret; both read it from an env var (`KEYCLOAK_ADMIN_CLIENT_SECRET` / `KEYCLOAK_SERVICE_ACCOUNT_CLIENT_SECRET` respectively) sourced from the same GitHub secret in every deploy target — see [docs/cicd.md](cicd.md).
+
+### GenAI — `py-genai-helper`
+
+A Python 3.12 / Flask service using LangChain. Unlike the Spring services, it is a REST **client** as well as a server:
+
+- It calls **feedback-service** (`GET /feedback`, Bearer-forwarded) to pull the data it summarizes when generating a member/team report.
+- It calls **either OpenAI or a local Ollama instance** to run inference, selected per-request via the `uselocal` field on the report-generation endpoints (not a global config flag) — the web client exposes this as the "Use local LLM" toggle on the helper page.
+- RAG question-answering persists uploaded documents in a **Chroma** vector store (a PVC-backed directory in Kubernetes, a bind-mounted volume elsewhere) rather than PostgreSQL.
+
+### Database — PostgreSQL
+
+Single instance, schema-per-service, documented in the [README's Database section](../README.md#database). No service reads another service's schema directly; cross-schema foreign keys exist (e.g. `event.events.creator_id → member.members.id`) but are enforced by the database, not queried across services.
+
+### Proxy & Auth — Traefik / nginx ingress + Keycloak
+
+- **Docker Compose / Azure VM**: Traefik terminates TLS (Let's Encrypt on the VM), applies a `forward-auth` middleware backed by its own confidential Keycloak client (session-cookie based, separate from the app-level Bearer tokens below), and strips path prefixes before forwarding to each service.
+- **Kubernetes**: the cluster's own nginx ingress does the prefix-stripping and routing instead of Traefik; TLS is handled at the cluster edge.
+- **Keycloak** (realm `devops`) is the single OIDC provider in every environment. Four confidential/public clients exist: `devops-client` (public, PKCE — the React app), `traefik-forward-auth` (confidential — gates the browser session at the proxy), `grafana` (confidential — Grafana's own admin-only OAuth login), and `org-role-sync` (confidential, service account only, no browser flow — organization-service and member-service's Admin REST API client, see above). None of the three confidential clients' secrets is hardcoded: each is templated as a `__PLACEHOLDER__` in `infra/keycloak/realm-config.json`, substituted at container start (Compose/VM) or chart render (Helm) from the matching GitHub secret — see [docs/cicd.md](cicd.md).
+
+## Request lifecycle (example: loading the members page)
+
+1. Browser requests `/` → proxy's forward-auth middleware checks for a session cookie; none yet → redirected through Keycloak's login page → cookie set on success.
+2. `web-client` (now loaded) separately obtains its own JWT via `keycloak-js`, independent of the forward-auth session cookie above — these are two different auth mechanisms layered on the same Keycloak realm, not one shared session.
+3. `web-client` calls `GET /api/v1/members` with `Authorization: Bearer <jwt>`.
+4. Proxy strips `/api/v1` (member-service's own routes start at `/members`), forwards to `member-service`.
+5. `member-service` validates the JWT against Keycloak's JWK set, maps roles, authorizes, queries its own `member` schema, returns JSON.
+
+The GenAI report-generation flow adds one more hop: `web-client → py-genai-helper → feedback-service` (step 3 becomes two REST calls instead of one, with the same Bearer token forwarded along), plus an out-of-band call from `py-genai-helper` to OpenAI or Ollama that never passes through the proxy.
+
+## Environment differences
+
+The only things that differ between local Docker Compose, the Azure VM, and Kubernetes are: the proxy implementation (Traefik vs. nginx ingress), the JWT issuer URI each service is configured with (must match the `iss` claim of tokens actually issued by that environment's Keycloak — internal Docker hostname locally, public HTTPS URL on the VM, internal ClusterIP DNS on Kubernetes), and TLS termination. The application code and the Prometheus scrape config are identical across all three — see [docs/deployment.md](deployment.md).
diff --git a/docs/cicd.md b/docs/cicd.md
@@ -0,0 +1,64 @@
+# CI/CD
+
+Detailed companion to the [README's CI/CD section](../README.md#cicd): every job, what gates a merge, and the full secrets/variables inventory.
+
+## CI — `.github/workflows/ci.yml`
+
+Runs on every pull request (plus manual `workflow_dispatch`). All jobs below feed into a single required `ci-success` aggregate job — branch protection only needs to require that one check.
+
+| Job | What it does |
+|---|---|
+| `organization-service`, `member-service`, `event-service`, `feedback-service`, `finance-service`, `letter-service` | Per Spring service: Checkstyle lint, then `./gradlew build` (compiles + runs JUnit tests); test reports uploaded as artifacts |
+| `py-genai-helper` | `ruff check .` then `pytest -q` |
+| `web-client` | Only runs if `web-client/**` changed (path filter via `dorny/paths-filter`) — `pnpm typecheck`, `pnpm lint`, `pnpm test:coverage`, `pnpm build`; coverage report uploaded as an artifact |
+| `docker-build` | Whole-system `docker compose -f infra/docker-compose.yml build`, then verifies every expected image tag actually exists — catches Dockerfile/compose drift that per-service builds wouldn't |
+| `codeql` | CodeQL SAST across `java-kotlin`, `python`, and `javascript-typescript` (matrix), `security-extended` query pack |
+| `openapi-lint` | Spectral lint of `api/openapi.yaml` |
+| `helm-validate` | `helm lint`, `helm template`, then schema-validates the rendered manifests with `kubeconform -strict` |
+
+## CD — `.github/workflows/cd.yml`
+
+Runs on every push to `main` (plus manual `workflow_dispatch`). Two deploy targets run in parallel:
+
+| Job | What it does |
+|---|---|
+| `deploy` (Ansible) | Writes the SSH key and three `.env` files from secrets — `infra/.env` (Postgres + Keycloak passwords and client secrets, for `docker-compose.yml`), `services/py-genai-helper/.env`, `services/spring-letter/.env` — builds a one-host inventory from the `VM_HOST` secret, runs `infra/ansible/playbook.yml` against the Azure VM |
+| `docker-push` | Matrix build: all 6 Spring services + `py-genai-helper` + `web-client` + `api-docs`, pushed to `ghcr.io/<repo>/<service>:{sha,latest}` with BuildKit's GitHub Actions cache backend (`type=gha`, scoped per service) |
+| `deploy-k8s` | Writes the kubeconfig from a secret, refreshes several ConfigMaps/Secrets that the Helm chart references but doesn't own (monitoring config, Keycloak theme, per-service env files — see [`infra/helm/README.md`](../infra/helm/README.md)), clears any stuck Helm release lock, then `helm upgrade --install --rollback-on-failure --timeout 15m` passing every database/Keycloak/client secret individually via `--set` (see table below) |
+
+`deploy-k8s` depends on `docker-push` (Kubernetes pulls prebuilt images from ghcr, unlike the VM which builds locally via Ansible/Compose).
+
+## Required GitHub secrets / variables
+
+| Kind | Name | Used by | Purpose |
+|---|---|---|---|
+| Variable | `AZURE_CLIENT_ID` | `infra.yml` | OIDC app registration (Service Principal) |
+| Variable | `AZURE_TENANT_ID` | `infra.yml` | Azure AD tenant |
+| Variable | `AZURE_SUBSCRIPTION_ID` | `infra.yml` | Target subscription |
+| Secret | `VM_SSH_PUBLIC_KEY` | `infra.yml` | Public key planted on the VM by Terraform |
+| Secret | `SSH_PRIVATE_KEY` | `cd.yml` (`deploy`) | Matching private key for Ansible to SSH in |
+| Secret | `VM_HOST` | `cd.yml` (`deploy`) | Ansible inventory host — the VM's FQDN |
+| Secret | `GENAI_ENV_CONTENT` | `cd.yml` (`deploy`, `deploy-k8s`) | Contents of `services/py-genai-helper/.env` |
+| Secret | `LETTER_ENV_CONTENT` | `cd.yml` (`deploy`, `deploy-k8s`) | Contents of `services/spring-letter/.env` (mail credentials) |
+| Secret | `KUBECONFIG` | `cd.yml` (`deploy-k8s`) | Kubeconfig for the RKE2 cluster |
+| Secret | `DB_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Shared Postgres admin (`app_admin`) password. VM: written into `infra/.env` as `POSTGRES_PASSWORD`. K8s: Helm `--set database.password` |
+| Secret | `DB_ORGANIZATION_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `organization_user` DB password. VM: `ORGANIZATION_DB_PASSWORD`. K8s: `--set database.users.organization.password` |
+| Secret | `DB_MEMBER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `member_user` DB password. VM: `MEMBER_DB_PASSWORD`. K8s: `--set database.users.member.password` |
+| Secret | `DB_EVENT_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `event_user` DB password. VM: `EVENT_DB_PASSWORD`. K8s: `--set database.users.event.password` |
+| Secret | `DB_FEEDBACK_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `feedback_user` DB password. VM: `FEEDBACK_DB_PASSWORD`. K8s: `--set database.users.feedback.password` |
+| Secret | `DB_FINANCE_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `finance_user` DB password. VM: `FINANCE_DB_PASSWORD`. K8s: `--set database.users.finance.password` |
+| Secret | `DB_LETTER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `letter_user` DB password. VM: `LETTER_DB_PASSWORD`. K8s: `--set database.users.letter.password` |
+| Secret | `DB_REPORTS_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `reports_user` DB password. VM: `REPORTS_DB_PASSWORD`. K8s: `--set database.users.reports.password` |
+| Secret | `KEYCLOAK_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Keycloak's own bootstrap console admin password. VM: `KEYCLOAK_ADMIN_PASSWORD`. K8s: `--set keycloak.adminPassword` |
+| Secret | `KEYCLOAK_DB_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Keycloak's own Postgres password. VM: `KEYCLOAK_DB_PASSWORD`. K8s: `--set keycloak.db.password` |
+| Secret | `KEYCLOAK_REALM_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Password for the seeded realm user `admin` (an application-level account, distinct from the Keycloak console admin above). VM: `KEYCLOAK_REALM_ADMIN_PASSWORD`. K8s: `--set keycloak.users.admin.password` |
+| Secret | `KEYCLOAK_REALM_USER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Password for the seeded realm user `user`. VM: `KEYCLOAK_REALM_USER_PASSWORD`. K8s: `--set keycloak.users.user.password` |
+| Secret | `KEYCLOAK_ADMIN_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `org-role-sync` Keycloak service-account client, used by **both** organization-service and member-service to call the Keycloak Admin REST API. VM: `KEYCLOAK_ADMIN_CLIENT_SECRET`. K8s: `--set` on both `services.organization-service.env.KEYCLOAK_ADMIN_CLIENT_SECRET` and `services.member-service.env.KEYCLOAK_SERVICE_ACCOUNT_CLIENT_SECRET` (same value, two consumers) |
+| Secret | `FORWARD_AUTH_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `traefik-forward-auth` Keycloak client. VM: `FORWARD_AUTH_CLIENT_SECRET`. K8s: `--set forwardAuth.clientSecret` |
+| Secret | `FORWARD_AUTH_COOKIE_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Session-cookie signing secret for the forward-auth middleware. VM: `FORWARD_AUTH_COOKIE_SECRET`. K8s: `--set forwardAuth.cookieSecret` |
+| Secret | `GRAFANA_OAUTH_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `grafana` Keycloak client (Grafana's own generic-OAuth login). VM: `GRAFANA_OAUTH_CLIENT_SECRET`. K8s: `--set monitoring.grafana.oauthClientSecret` |
+| Built-in | `GITHUB_TOKEN` | `cd.yml` (`docker-push`) | ghcr push auth (`packages: write` permission) |
+
+Each of the 16 password/secret entries above (`DB_ADMIN_PASSWORD` through `GRAFANA_OAUTH_CLIENT_SECRET`) is the single source of truth for that value across **both** deploy targets — nothing is hardcoded in `docker-compose.yml`, `infra/keycloak/realm-config.json` (templated with `__PLACEHOLDER__` tokens, substituted at container start), or the Helm chart's `values.yaml` (which ships `CHANGE_ME_*` placeholders, meant to be overridden, never deployed as-is). For local development, copy [`infra/.env.example`](../infra/.env.example) to `infra/.env` — it documents which local value maps to which GitHub secret.
+
+The Azure OIDC service principal needs `Contributor` on the subscription and `Storage Blob Data Contributor` on the `stdevoops26tfstate` state account. No long-lived Azure client secret is stored anywhere — auth is federated via OIDC.
diff --git a/docs/deployment.md b/docs/deployment.md
@@ -0,0 +1,67 @@
+# Deployment
+
+Detailed companion to the [README's Deployment environments section](../README.md#deployment-environments). Three environments run the system; this page covers the two continuously-live ones. For local Docker Compose, the [README's Running Locally section](../README.md#running-locally) is already the complete reference.
+
+## Azure VM
+
+A single Ubuntu 24.04 VM in Azure's **Poland Central** region runs the exact same `infra/docker-compose.yml` stack as local dev, minus the override file — so it terminates real TLS via Let's Encrypt instead of serving plain HTTP.
+
+**Live URL:** <https://team-devoops.polandcentral.cloudapp.azure.com>
+
+### Provisioning — Terraform (`infra/terraform/`)
+
+| Resource | Purpose |
+|---|---|
+| `azurerm_resource_group` | `rg-team-devoops` |
+| `azurerm_virtual_network` + `azurerm_subnet` | `10.0.0.0/16` / `10.0.1.0/24` |
+| `azurerm_network_security_group` | Allows inbound 22 (SSH), 80 (HTTP), 443 (HTTPS) only |
+| `azurerm_public_ip` | Static, with a free Azure-assigned FQDN (`domain_name_label`) |
+| `azurerm_linux_virtual_machine` | `Standard_D2as_v4`, Ubuntu 24.04 LTS, SSH-key auth only (`disable_password_authentication = true`) |
+
+Remote state lives in Azure Blob Storage (`stdevoops26tfstate/tfstate`), shared and locked between local runs and CI so two applies can't race. Auth is OIDC (`ARM_USE_OIDC=true`) — no long-lived Azure client secret is stored anywhere.
+
+Run it via the **`infra` workflow** (`.github/workflows/infra.yml`, manual `workflow_dispatch` with `plan` / `apply` / `destroy`), or locally:
+
+```bash
+az login
+az account set --subscription <AZURE_SUBSCRIPTION_ID>
+export ARM_SUBSCRIPTION_ID=<AZURE_SUBSCRIPTION_ID>
+export ARM_USE_AZUREAD=true
+cd infra/terraform
+echo "admin_ssh_public_key = \"$(cat ~/.ssh/team-devoops-azure.pub)\"" > terraform.tfvars
+terraform init
+terraform plan
+```
+
+### Configuration — Ansible (`infra/ansible/playbook.yml`)
+
+Runs after Terraform, targeting the VM's public IP:
+
+1. Install Docker (apt repo + `docker-ce`, `docker-compose-plugin`).
+2. Clone/update the repo at the deployed commit (`git` module, `force: true` so the working tree always matches `main`).
+3. Write `infra/.env` (Postgres and Keycloak passwords, OIDC client secrets), `services/py-genai-helper/.env`, and `services/spring-letter/.env` from content passed in as extra-vars (sourced from GitHub Secrets — never committed). Nothing in `docker-compose.yml` is hardcoded; every credential is a `${VAR}` resolved from `infra/.env` at container start.
+4. `docker compose -f infra/docker-compose.yml up -d --build --remove-orphans`.
+
+Triggered automatically by the **`cd` workflow** on every push to `main` — see [docs/cicd.md](cicd.md) for the exact secrets involved.
+
+### Typical workflow
+
+1. Infra change (new resource, VM size, etc.) → push → manually trigger `infra` workflow with `apply`.
+2. App code change → merge to `main` → `cd` workflow redeploys automatically (both this VM and Kubernetes, in parallel).
+3. Teardown → trigger `infra` workflow with `destroy`.
+
+## Kubernetes (TUM RKE2 / Rancher)
+
+The same services are also deployed via a Helm umbrella chart to the course's RKE2 cluster, with autoscaling and rolling-update self-healing configured per service.
+
+**Live URL:** <https://ge83mom-devops26.stud.k8s.aet.cit.tum.de> · **Namespace:** `ge83mom-devops26` · **Chart:** [`infra/helm/team-devoops`](../infra/helm/team-devoops)
+
+This path has enough moving parts (one-time Secret/ConfigMap bootstrapping, the Keycloak theme's projected-volume workaround, resource-quota tuning to fit Ollama into a fixed namespace quota, autoscaling scope) that it has its own dedicated, actively-maintained reference — **[`infra/helm/README.md`](../infra/helm/README.md)** — rather than being duplicated here. Highlights:
+
+- Images are built and pushed to `ghcr.io/aet-devops26/team-devoops/<service>` by the `cd` workflow's `docker-push` job, then `deploy-k8s` runs `helm upgrade --install --rollback-on-failure`.
+- PostgreSQL, Prometheus, Grafana, Loki, and Ollama each run in-cluster with their own PVC on the cluster's default StorageClass (`csi-rbd-sc`, ReadWriteOnce).
+- On every pull request, `ci.yml`'s `helm-validate` job lints the chart, renders it, and schema-validates the output with `kubeconform` — a broken chart can't merge.
+
+## Why not Kubernetes on Azure too
+
+The spec's cloud-environment requirement is satisfied by the Azure VM (Docker Compose) rather than a managed Kubernetes service like AKS; the Kubernetes requirement is satisfied by the course's Rancher-managed RKE2 cluster instead. Both requirements are met, just not by the same deployment target — provisioning a second Kubernetes cluster on Azure was judged not to add anything beyond what the VM and the RKE2 cluster already each demonstrate independently.