diff --git a/README.md b/README.md index 17f53d2e..2e6751be 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,13 @@ A centralized sports club management platform that combines member administratio Club organizers get an all-in-one tool for managing members, automating billing, and overseeing events. Members and trainers benefit from structured training overviews and AI-generated progress reports based on attendance records, trainer notes, and member profiles. +**Live deployments:** + +| Environment | URL | +|---|---| +| Kubernetes (TUM RKE2 / Rancher) | | +| Azure VM | | + ## Features - **Organization service** — CRUD for sports, teams and roles (e.g. member, trainer, admin) @@ -12,7 +19,7 @@ Club organizers get an all-in-one tool for managing members, automating billing, - **Feedback service** — personalized feedback and progress reports - **Finance service** — one-time and recurring billing linked to members - **Letter service** — PDF/email generation from templates with dynamic member data -- **GenAI helper** — analyzes member data and trainer notes to generate personalized feedback and progress reports (supports OpenAI and local LLMs) +- **GenAI helper** — analyzes member data and trainer notes to generate personalized feedback and progress reports; supports both a cloud provider (OpenAI) and a local model (Ollama), selectable per request, and answers questions over uploaded documents via RAG (Chroma vector store) ## Repository Structure @@ -21,7 +28,7 @@ repo/ ├── api/ # Single source of truth for API contracts │ ├── openapi.yaml # Versioned OpenAPI spec (OpenAPI 3.0.3) │ └── scripts/ # Code-gen scripts (gen-all.sh, gen-spring.sh, …) -├── docs/ # Project documentation +├── docs/ # Project documentation (see "Docs" below) ├── services/ │ ├── spring-*/ # Java 21, Spring Boot 3 microservices │ │ └── src/generated/ # ⚠ Generated — do not edit by hand @@ -35,11 +42,12 @@ repo/ ## Architecture -All services run in Docker and are exposed through a single **Traefik** reverse -proxy on port **80**. Traefik routes requests by path prefix and strips the full -prefix before forwarding, so each service receives only the resource path (e.g. -`GET /api/v1/organization/sports` → organization-service receives `GET /sports`). -The Spring Boot services and the GenAI service share a **PostgreSQL** database. +The system is a set of independently deployable services behind a single reverse proxy — Traefik in Docker Compose / on the Azure VM, the cluster's own nginx ingress on Kubernetes. The proxy routes by path prefix and strips it before forwarding, so e.g. `GET /api/v1/organization/sports` reaches organization-service as `GET /sports`. + +- **Client** — a React SPA that talks to every backend service directly over REST through the proxy (including the GenAI service — there's no server-side fan-out on its behalf). +- **Server** — six Spring Boot 3 microservices (organization, member, event, feedback, finance, letter), each owning one schema in a shared PostgreSQL instance and validating requests as a stateless OAuth2 resource server against Keycloak. +- **GenAI** — a Python/Flask + LangChain service (`py-genai-helper`) that is itself a REST client of feedback-service (to pull the data it summarizes) and of either OpenAI or a local Ollama instance (to run inference). +- **Auth** — Keycloak issues JWTs; Traefik's forward-auth middleware gates browser sessions, and each Spring/Flask service independently validates the Bearer token against Keycloak's JWK set. | Service | External route | Internal port | Stack | |---|---|---|---| @@ -55,372 +63,137 @@ The Spring Boot services and the GenAI service share a **PostgreSQL** database. | Keycloak | `/auth` | 8080 | Keycloak 26 | | Grafana | `/dashboard` (admin only) | 3000 | Grafana 11 | | Prometheus | internal only | 9090 | Prometheus v2 | -| Traefik dashboard | `http://localhost:8080` (local only) | — | Traefik v3 | | PostgreSQL | internal only | 5432 | postgres:15 | -## Code Generation - -`api/openapi.yaml` is the single source of truth. Three generators derive code -from it that **must never be edited by hand**: +Full per-service responsibilities, interface contracts, and the request/auth lifecycle: **[docs/architecture.md](docs/architecture.md)**. -| Generator | Tool | Output | -|---|---|---| -| Spring Boot API interfaces + models | `openapitools/openapi-generator-cli:v7.14.0` (Docker) | `services/spring-*/src/generated/java/` | -| Pydantic v2 models | `datamodel-code-generator` (pip) | `services/py-genai-helper/generated/models.py` | -| TypeScript types | `openapi-typescript` (pnpm devDep) | `web-client/src/api.ts` | +## API -Run all generators at once: +`api/openapi.yaml` (OpenAPI 3.0.3) is the single source of truth for every REST contract. Three generators derive code from it that must never be edited by hand: Spring interfaces/models (`openapitools/openapi-generator-cli`), Pydantic v2 models (`datamodel-code-generator`), and TypeScript types (`openapi-typescript`). Run all three with: ```bash ./api/scripts/gen-all.sh ``` -The `openapi-codegen` pre-commit hook runs this automatically whenever -`api/openapi.yaml` is staged. If any generated file changes, the hook re-stages -the output and aborts so you can review the diff before re-committing. +The `openapi-codegen` pre-commit hook re-runs this automatically whenever `api/openapi.yaml` changes and aborts the commit so the diff can be reviewed. A live Swagger UI is served at `/docs` in every environment. -Prerequisites: **Docker** (Spring generator), **`datamodel-code-generator`** -(`pip install datamodel-code-generator`), **`pnpm`** (already a devDependency -in `web-client/`). +## Database -## Developer Setup +All five schema-owning Spring services share a single **PostgreSQL 15** instance (`app_db`), each with a dedicated schema and a least-privilege user: -This repo uses [`pre-commit`](https://pre-commit.com) to run the same fast lint -checks locally that CI gates on (ruff, eslint, end-of-file fixer, pnpm lockfile -sync, etc.). One-time setup per developer: +| Service | Schema | User | +|---|---|---| +| Organization | `organization` | `organization_user` | +| Member | `member` | `member_user` | +| Event | `event` | `event_user` | +| Feedback | `feedback` | `feedback_user` | +| Finance | `finance` | `finance_user` | -```bash -pip install pre-commit datamodel-code-generator # or: pipx install pre-commit -pre-commit install # installs the pre-commit git hook -pre-commit install --hook-type pre-push # installs the pre-push hook -pre-commit run --all-files # optional one-time clean-up pass -``` +Schemas and users are created at DB init time by [`infra/postgres/init-db.sh`](infra/postgres/init-db.sh). Each service runs its own **Flyway** migrations on startup (`V1__create_tables.sql`, `V2__add_foreign_keys.sql` for cross-schema references, granted via `ALTER DEFAULT PRIVILEGES`). The letter service has no database; the GenAI service persists RAG documents in a Chroma vector store instead of PostgreSQL. + +## Authentication (Keycloak) -What runs when: +All services are protected by [Keycloak 26](https://www.keycloak.org) via OIDC/JWT, included in both the Docker Compose stack and the Helm chart. -| Stage | Hooks | +| Realm | `devops` | |---|---| -| `pre-commit` (every commit) | end-of-file-fixer, trailing-whitespace, check-yaml/json, merge-conflict guard, large-file guard, **ruff** (lint + format, py-genai-helper), **eslint --fix** (web-client), **pnpm-lock-sync** (regenerates `web-client/pnpm-lock.yaml` when `package.json` changes), **openapi-codegen** (regenerates all generated sources when `api/openapi.yaml` changes) | -| `pre-push` (only on push) | **Spectral** lint of `api/openapi.yaml` (if changed), **Checkstyle** for all Spring services (if Java sources changed) | +| Admin user | `admin` / `admin123` locally (roles: `admin`, `member`) | +| Regular user | `user` / `user123` locally (role: `member`) | -Auto-fixing hooks (ruff, eslint, pnpm-lock-sync, openapi-codegen, -end-of-file-fixer, etc.) will modify files and **abort the commit** so you can -re-stage and re-commit. +Passwords shown are the local-dev defaults from [`infra/.env.example`](infra/.env.example); on the VM and Kubernetes they come from GitHub Secrets instead — nothing is hardcoded in `docker-compose.yml`, `infra/keycloak/realm-config.json`, or the Helm chart. The web client (`devops-client`, public/PKCE S256) redirects to Keycloak automatically (`login-required`). Traefik's forward-auth middleware, Grafana's own OAuth login, and the organization/member services' Keycloak Admin API client (`org-role-sync`) each use their own confidential client. Locally, Keycloak's admin console is at ; in production it's behind the proxy at `/auth/admin`. Full JWT-validation, client, and per-environment issuer-URI details: [docs/architecture.md](docs/architecture.md). + +## Developer Setup -Bypass (emergencies only -- CI will still gate): +This repo uses [`pre-commit`](https://pre-commit.com) to run the same fast checks locally that CI gates on (ruff, eslint, checkstyle, Spectral, end-of-file fixer, pnpm lockfile sync, OpenAPI codegen, …): ```bash -git commit --no-verify -git push --no-verify +pip install pre-commit datamodel-code-generator +pre-commit install +pre-commit install --hook-type pre-push ``` -The full hook configuration lives in [`.pre-commit-config.yaml`](.pre-commit-config.yaml) -and the helper scripts under [`scripts/hooks/`](scripts/hooks/). +Auto-fixing hooks modify files and abort the commit so you can re-stage. Bypass only in emergencies (`git commit --no-verify` / `git push --no-verify`) — CI still gates. Full hook config: [`.pre-commit-config.yaml`](.pre-commit-config.yaml). ## Running Locally -Spin up the full stack on your machine with Docker Compose: - ```bash cd infra +cp .env.example .env # first time only — local-dev secrets, gitignored docker compose up -d --build ``` -This auto-merges [`infra/docker-compose.override.yml`](infra/docker-compose.override.yml), -which strips TLS / Let's Encrypt / Host-based routing from the base file so -everything is reachable on plain HTTP: +This auto-merges [`infra/docker-compose.override.yml`](infra/docker-compose.override.yml), which strips TLS/Let's-Encrypt/Host-routing so everything is reachable on plain HTTP: | URL | Service | |---|---| | | Web client | | | Swagger UI | -| | Grafana (monitoring, admin only) | -| | APIs (organization, members, events, feedback, finance, letters, helper) | -| | Keycloak (via Traefik) | -| | Keycloak (direct, for admin console) | +| | Grafana (admin only) | +| | APIs | +| | Keycloak | | | Traefik dashboard | -> **Do not run** `docker compose -f infra/docker-compose.yml up` locally — that -> skips the override, causing Traefik to request a real Let's Encrypt cert for -> the production hostname from your laptop. Failed challenges count toward the -> production rate limit. - -Tear down: -```bash -cd infra && docker compose down # keeps the postgres volume -cd infra && docker compose down -v # wipes the postgres volume too -``` - -## Production Deployment - -The stack runs on a single Azure VM in **Poland Central**, fronted by Traefik with a -real TLS certificate from Let's Encrypt (production CA). Everything is -automated; no manual VM access is required for normal deploys. - -**Live URL:** - -### Infrastructure stack - -| Layer | Tool | What it does | -|---|---|---| -| Provisioning | **Terraform** (AzureRM ~> 4.0) | Resource group, VNet, NSG (22/80/443), static public IP + free Azure FQDN, Ubuntu 24.04 VM | -| Configuration | **Ansible** | Installs Docker, clones repo, writes `.env`, runs `docker compose up` | -| CI/CD | **GitHub Actions** (OIDC, no client secrets) | `infra.yml` (manual: plan/apply/destroy) and `cd.yml` (auto on push to `main`) | -| Remote state | **Azure Blob Storage** (`stdevoops26tfstate/tfstate`) | Shared, locked Terraform state — survives between CI runs | -| TLS | **Let's Encrypt** (HTTP-01 via Traefik) | Cert persisted in a Docker volume; auto-renewed | - -### GitHub Actions workflows - -- **`infra` workflow** — manual (`workflow_dispatch`). Choose `plan`, `apply`, or `destroy`. -- **`cd` workflow** — runs automatically on every push to `main` (and is also `workflow_dispatch`-able). Deploys the current `main` to the VM via Ansible **and** to the Kubernetes cluster via Helm (see [Kubernetes deployment](#kubernetes-deployment-helm)). - -### Required GitHub secrets / variables +> **Do not run** `docker compose -f infra/docker-compose.yml up` directly — that skips the override and Traefik will try to request a real Let's Encrypt cert for the production hostname from your laptop. -| Kind | Name | Purpose | -|---|---|---| -| Variable | `AZURE_CLIENT_ID` | OIDC app registration (Service Principal) | -| Variable | `AZURE_TENANT_ID` | Azure AD tenant | -| Variable | `AZURE_SUBSCRIPTION_ID` | Target subscription | -| Secret | `VM_SSH_PUBLIC_KEY` | Public key planted on the VM by Terraform | -| Secret | `SSH_PRIVATE_KEY` | Matching private key for Ansible to SSH in | -| Secret | `VM_HOST` | Host Ansible connects to — use the FQDN above | -| Secret | `GENAI_ENV_CONTENT` | Contents of `services/py-genai-helper/.env` | -| Secret | `LETTER_ENV_CONTENT` | Contents of `services/spring-letter/.env` (mail credentials) | -| Secret | `KUBECONFIG` | Kubeconfig for the RKE2 cluster (used by the `deploy-k8s` job) | -| Secret | `DB_ADMIN_PASSWORD` | Shared Postgres admin password. VM: `POSTGRES_PASSWORD` in `infra/.env`. K8s: Helm `database.password` | -| Secret | `DB_ORGANIZATION_PASSWORD` | `organization_user` DB password. VM: `ORGANIZATION_DB_PASSWORD`. K8s: Helm `database.users.organization.password` | -| Secret | `DB_MEMBER_PASSWORD` | `member_user` DB password. VM: `MEMBER_DB_PASSWORD`. K8s: Helm `database.users.member.password` | -| Secret | `DB_EVENT_PASSWORD` | `event_user` DB password. VM: `EVENT_DB_PASSWORD`. K8s: Helm `database.users.event.password` | -| Secret | `DB_FEEDBACK_PASSWORD` | `feedback_user` DB password. VM: `FEEDBACK_DB_PASSWORD`. K8s: Helm `database.users.feedback.password` | -| Secret | `DB_FINANCE_PASSWORD` | `finance_user` DB password. VM: `FINANCE_DB_PASSWORD`. K8s: Helm `database.users.finance.password` | -| Secret | `DB_LETTER_PASSWORD` | `letter_user` DB password. VM: `LETTER_DB_PASSWORD`. K8s: Helm `database.users.letter.password` | -| Secret | `DB_REPORTS_PASSWORD` | `reports_user` DB password. VM: `REPORTS_DB_PASSWORD`. K8s: Helm `database.users.reports.password` | -| Secret | `KEYCLOAK_ADMIN_PASSWORD` | Keycloak bootstrap admin console password. VM: `KEYCLOAK_ADMIN_PASSWORD`. K8s: Helm `keycloak.adminPassword` | -| Secret | `KEYCLOAK_DB_PASSWORD` | Keycloak's own Postgres password. VM: `KEYCLOAK_DB_PASSWORD`. K8s: Helm `keycloak.db.password` | -| Secret | `KEYCLOAK_REALM_ADMIN_PASSWORD` | Password for the seeded `admin` realm user. VM: `KEYCLOAK_REALM_ADMIN_PASSWORD`. K8s: Helm `keycloak.users.admin.password` | -| Secret | `KEYCLOAK_REALM_USER_PASSWORD` | Password for the seeded `user` realm user. VM: `KEYCLOAK_REALM_USER_PASSWORD`. K8s: Helm `keycloak.users.user.password` | -| Secret | `FORWARD_AUTH_COOKIE_SECRET` | 32+ char random string signing forward-auth session cookies. VM: `FORWARD_AUTH_COOKIE_SECRET`. K8s: Helm `forwardAuth.cookieSecret` | -| Secret | `KEYCLOAK_ADMIN_CLIENT_SECRET` | Secret for the `org-role-sync` Keycloak service-account client (`manage-users`/`view-clients` — used by organization-service and member-service to call the Keycloak Admin REST API). VM: `KEYCLOAK_ADMIN_CLIENT_SECRET`. K8s: Helm `services.organization-service.env.KEYCLOAK_ADMIN_CLIENT_SECRET` and `services.member-service.env.KEYCLOAK_SERVICE_ACCOUNT_CLIENT_SECRET` (same value, two consumers) | -| Secret | `FORWARD_AUTH_CLIENT_SECRET` | Secret for the `traefik-forward-auth` Keycloak client (Traefik/oauth2-proxy's OIDC client, gates every proxied route). VM: `FORWARD_AUTH_CLIENT_SECRET`. K8s: Helm `forwardAuth.clientSecret` | -| Secret | `GRAFANA_OAUTH_CLIENT_SECRET` | Secret for the `grafana` Keycloak client (Grafana's own generic-OAuth login). VM: `GRAFANA_OAUTH_CLIENT_SECRET`. K8s: Helm `monitoring.grafana.oauthClientSecret` | - -Each of these 16 secrets is the single source of truth for that value across **both** deploy targets — the `deploy` job's "Write compose .env file" step assembles `infra/.env` from them for the VM, and the `deploy-k8s` job passes them individually via `--set` for Helm. No GitHub secret's value is duplicated by a *different* GitHub secret (`KEYCLOAK_ADMIN_CLIENT_SECRET` is intentionally passed to two `--set` targets, since organization-service and member-service both authenticate to Keycloak as the same `org-role-sync` client). - -The OIDC service principal needs `Contributor` on the subscription (to manage -resources in `rg-team-devoops`) and `Storage Blob Data Contributor` on the -state account `stdevoops26tfstate` (to read/write tfstate). - -### Typical workflow - -1. **Change infra** → push to `main` (or any branch), trigger `infra` workflow with `apply`. -2. **Change app code** → merge to `main`; `cd` runs automatically and redeploys. -3. **Tear down** → trigger `infra` workflow with `destroy`. - -### Running Terraform locally +Tear down with `docker compose down` (keeps the Postgres volume) or `docker compose down -v` (wipes it too). -```bash -az login -az account set --subscription -export ARM_SUBSCRIPTION_ID= -export ARM_USE_AZUREAD=true -cd infra/terraform -echo "admin_ssh_public_key = \"$(cat ~/.ssh/team-devoops-azure.pub)\"" > terraform.tfvars -terraform init -terraform plan -``` +## CI/CD -Local and CI share the same remote state, so do not run `apply` in both at the -same time (the backend's blob lease will block one, but coordinate anyway). +**CI** (`.github/workflows/ci.yml`) runs on every pull request: build + test + lint (Checkstyle/ruff/ESLint) for each of the six Spring services, `py-genai-helper`, and `web-client`; a whole-system `docker compose build`; CodeQL SAST across Java/Python/TypeScript; OpenAPI linting (Spectral); and Helm chart linting + `kubeconform` schema validation. A single `ci-success` job aggregates all of these into one required check. -### Kubernetes deployment (Helm) +**CD** (`.github/workflows/cd.yml`) runs on every push to `main` and deploys to both live environments in parallel: the Azure VM via Ansible, and the Kubernetes cluster via `helm upgrade --install` (after building and pushing every image to GHCR). All credentials are injected from GitHub Secrets — nothing is hardcoded in the workflows. -In addition to the Azure VM, the stack is also deployed to a **Kubernetes -cluster** (TUM RKE2) via a Helm umbrella chart. Both deploys run in parallel on -push to `main`; the VM path is unchanged. - -| Aspect | Value | -|---|---| -| Chart | [`infra/helm/team-devoops`](infra/helm/team-devoops) | -| Namespace | `ge83mom-devops26` | -| Host | | -| Ingress | cluster `nginx` ingress (path-prefix routing, prefix stripped per service) | -| Images | built and pushed to `ghcr.io/aet-devops26/team-devoops/` | -| Database | in-cluster PostgreSQL `StatefulSet` + PVC (cluster default StorageClass) | -| Monitoring | in-cluster Prometheus + Grafana, each with its own PVC (see [Monitoring](#monitoring)) | -| Local LLM | in-cluster Ollama + PVC, reachable only by `py-genai-helper` (see [`services/py-genai-helper/README.md`](services/py-genai-helper/README.md)) | - -The `cd` workflow's `docker-push` job builds and pushes all service images to -ghcr (tagged with the commit SHA), then `deploy-k8s` runs `helm upgrade ---install` against the cluster. On pull requests, the `ci` workflow's -`helm-validate` job lints and schema-validates the chart with `kubeconform`. - -See [`infra/helm/README.md`](infra/helm/README.md) for the chart layout, required -one-time secrets (`genai-env`, `ghcr-pull`), the Prometheus/Grafana ConfigMaps, -and manual deploy instructions. +Full job-by-job breakdown and the required secrets table: **[docs/cicd.md](docs/cicd.md)**. -## Database +## Deployment environments -All five Spring services share a single **PostgreSQL 15** instance (`app_db`) but each owns a dedicated schema and a least-privilege user: +Besides local Docker Compose (above), the system runs continuously in two more places: -| Service | Schema | User | -|---|---|---| -| Organization | `organization` | `organization_user` | -| Member | `member` | `member_user` | -| Event | `event` | `event_user` | -| Feedback | `feedback` | `feedback_user` | -| Finance | `finance` | `finance_user` | - -Schemas and users are created at DB init time by [`infra/postgres/init-db.sh`](infra/postgres/init-db.sh). Each service runs its own **Flyway** migrations on startup: - -- `V1__create_tables.sql` — creates all tables for that schema -- `V2__add_foreign_keys.sql` — adds cross-schema foreign keys (e.g. `event.events.creator_id → member.members.id`) - -Cross-schema `REFERENCES` privileges are granted via `ALTER DEFAULT PRIVILEGES` in `init-db.sh`, so foreign-key constraints across schemas work on fresh deploys without manual intervention. - -The letter service has no database (`spring.flyway.enabled=false`). The GenAI service uses file-based storage for RAG documents. - -## Authentication (Keycloak) +- **Azure VM** — a single Ubuntu VM provisioned by Terraform, configured by Ansible, running the same `docker-compose.yml` stack with a real Let's Encrypt certificate. +- **Kubernetes (TUM RKE2 / Rancher)** — a Helm umbrella chart ([`infra/helm/team-devoops`](infra/helm/team-devoops)) deploying every service, PostgreSQL, Ollama, and the monitoring stack into namespace `ge83mom-devops26`, with autoscaling (HPA) and rolling-update self-healing configured per service. -All services are protected by [Keycloak 26](https://www.keycloak.org) via OIDC/JWT. Keycloak is included in both the Docker Compose stack and the Helm chart — no separate installation is needed. +Provisioning details, Ansible/Terraform/Helm internals, and per-environment secrets: **[docs/deployment.md](docs/deployment.md)**. -### Realm & users +## Monitoring -| Realm | `devops` | -|---|---| -| Admin user | `admin` / `admin123` locally (roles: `admin`, `member`) | -| Regular user | `user` / `user123` locally (role: `member`) | +**Prometheus** tracks request count, latency, and error rate for every Spring service, the GenAI service, and Keycloak, plus a handful of business-level custom metrics (letters sent/generated, RAG query rate, report-generation rate — each split by LLM provider). **Grafana** (admin-only, OAuth via Keycloak) visualizes them across three dashboards provisioned as code, and two alert rules (service down, high p95 latency) are wired into Grafana's unified alerting. **Loki + Grafana Alloy** centralize logs from every service/pod so they're searchable in Grafana instead of `docker logs`/`kubectl logs`. -Passwords shown are the local-dev defaults from [`infra/.env.example`](infra/.env.example). On the VM and Kubernetes they come from the `KEYCLOAK_REALM_ADMIN_PASSWORD`/`KEYCLOAK_REALM_USER_PASSWORD` GitHub secrets instead — see [Required GitHub secrets / variables](#required-github-secrets--variables). ("Realm" distinguishes these two application-level accounts from the Keycloak server's own bootstrap console admin, `KEYCLOAK_ADMIN_PASSWORD`.) `infra/keycloak/realm-config.json` stores these as `__KEYCLOAK_REALM_ADMIN_PASSWORD__`/`__KEYCLOAK_REALM_USER_PASSWORD__` placeholders, substituted at container start (Compose) or chart render (Helm). +Runs identically in all three environments from the same config. Full metrics table, dashboard breakdown, and alert definitions: **[docs/monitoring.md](docs/monitoring.md)**. -### Clients +## Testing -| Client | Type | Used by | +| Layer | Tool | Scope | |---|---|---| -| `devops-client` | public, PKCE S256 | React frontend | -| `traefik-forward-auth` | confidential | Traefik forward-auth middleware | -| `grafana` | confidential | Grafana's own generic-OAuth login (admin-only, see [Monitoring](#monitoring)) | -| `org-role-sync` | confidential, service account (`manage-users`/`view-clients`) | organization-service and member-service, to call the Keycloak Admin REST API | - -The three confidential clients' secrets are local-dev defaults from `infra/.env.example` (matching what's committed in `infra/keycloak/realm-config.json`'s `__KEYCLOAK_ADMIN_CLIENT_SECRET__`/`__FORWARD_AUTH_CLIENT_SECRET__`/`__GRAFANA_OAUTH_CLIENT_SECRET__` placeholders); on the VM and Kubernetes they come from the `KEYCLOAK_ADMIN_CLIENT_SECRET`/`FORWARD_AUTH_CLIENT_SECRET`/`GRAFANA_OAUTH_CLIENT_SECRET` GitHub secrets — see [Required GitHub secrets / variables](#required-github-secrets--variables). - -### Local login - -When running with Docker Compose, Keycloak is available at . The realm is auto-imported on first start from [`infra/keycloak/realm-config.json`](infra/keycloak/realm-config.json). - -The web client redirects to Keycloak automatically (`login-required` strategy). Log in with any of the test users above. +| Spring services (×6) | JUnit (via `./gradlew build`) | Service/controller logic per domain | +| GenAI (`py-genai-helper`) | pytest | RAG pipeline, report generation, LLM provider selection | +| Web client | Vitest + jsdom | Core user workflows (auth, CRUD flows per feature area) | -### Production admin console +All of the above run automatically in CI on every pull request (see [CI/CD](#cicd)); a PR cannot merge if any of them fail. -Keycloak is publicly accessible via Traefik at . Admin console: `/auth/admin`. - -### Spring services — JWT validation - -Each Spring service is a stateless OAuth2 resource server. It validates Bearer JWTs against Keycloak's JWK set and extracts roles from the `realm_access.roles` claim, mapping them to Spring `ROLE_*` authorities (e.g. `"admin"` → `ROLE_admin`). - -| Property | Purpose | -|---|---| -| `spring.security.oauth2.resourceserver.jwt.issuer-uri` | Validates the `iss` claim in incoming JWTs | -| `spring.security.oauth2.resourceserver.jwt.jwk-set-uri` | URL to fetch Keycloak's public signing keys | +```bash +# Spring service, from services/spring-/ +./gradlew build -These are set in each service's `src/main/resources/application.properties` as defaults (pointing at the local Keycloak on `localhost:8081/auth`). On the Azure VM, `docker-compose.yml` overrides `SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI` with the public HTTPS issuer so it matches the `iss` claim in tokens issued by production Keycloak. The JWK set URI always uses the internal Docker hostname `http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs`. On Kubernetes they are injected via the `env:` block in `infra/helm/team-devoops/values.yaml` using the internal `keycloak` ClusterIP DNS name. +# GenAI service, from services/py-genai-helper/ +pytest -q -## Monitoring +# Web client, from web-client/ +pnpm test +``` -Monitoring runs in **all three environments** (local, Azure VM, and the -Kubernetes cluster) from the same underlying config. - -**Prometheus** scrapes request count, latency, and error-rate metrics from -every Spring service (`/actuator/prometheus`, via Micrometer), the GenAI -service (`/metrics`, via `prometheus-flask-exporter`), Keycloak (`/metrics` on -its management port, `KC_METRICS_ENABLED=true`), and Traefik itself -(edge-level metrics; Kubernetes uses the cluster's own nginx ingress instead of -Traefik, so this one is compose-only). Scrape targets are statically -configured in [`infra/prometheus/prometheus.yml`](infra/prometheus/prometheus.yml) -— the same file is used unchanged across all three environments, since -Docker Compose service names and Kubernetes Service names are identical -strings and both resolve the same way. Prometheus itself is never exposed -outside the internal network/cluster in any environment. - -**Grafana** visualizes these metrics and is reachable at `/dashboard` -(`http://localhost/dashboard` locally, `https://team-devoops.polandcentral.cloudapp.azure.com/dashboard` -on the Azure VM, `https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/dashboard` -on Kubernetes). Dashboards and datasources are provisioned as code from -[`infra/grafana/`](infra/grafana/) — nothing is configured by hand in the -running instance. - -Grafana is **admin-only**: it authenticates exclusively through its own -Keycloak generic-OAuth login (client `grafana`, no local username/password -form). `GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH` maps the realm `admin` -role to Grafana's `Admin` org role and everything else to an empty string; -combined with `GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_STRICT=true`, logging in -with any account that doesn't hold the `admin` realm role is rejected -outright. (Mapping the non-admin branch to the literal string `"None"` -instead does *not* reject the login — verified locally: Grafana treats -`"None"` as a real, if content-less, org role and still creates a session; -only an empty string actually triggers `role_attribute_strict`'s rejection.) -Prometheus itself is never routed through Traefik/ingress at all, so it has -no login of its own to configure. - -Three dashboards ship out of the box ([`infra/grafana/dashboards/`](infra/grafana/dashboards/)): - -| Dashboard | Covers | -|---|---| -| `service-overview.json` | Request rate, p95 latency, and error rate per Spring service (dropdown to filter), plus up/down status, Keycloak login rate, and letters sent/generated | -| `genai-service.json` | Request rate, p95 latency, and error rate for the GenAI service, plus RAG query rate/latency and report-generation rate by kind/status, each broken out by LLM provider (OpenAI vs. local Ollama) | -| `logs.json` | Centralized logs from every service (Loki), filterable by container, plus a log-volume graph | +## Team & Responsibilities -Beyond the generic per-request metrics above, a few **business-level custom -metrics** are tracked so the dashboards reflect what the system is actually -doing, not just HTTP noise: +Each team member owns a primary subsystem, with cross-cutting collaboration on integration and deployment: -| Metric | Service | What it shows | -|---|---|---| -| `letters_sent_total{status}` | letter-service | Actual mail delivery outcomes — ties directly to the mail credentials/health-check work above | -| `letters_generated_total` | letter-service | PDF letters generated | -| `genai_rag_queries_total{status,provider}`, `genai_rag_query_duration_seconds{provider}` | py-genai-helper | RAG question-answering usage and latency, split by OpenAI vs. local Ollama | -| `genai_report_generation_total{kind,status,provider}` | py-genai-helper | Member/team AI report generation attempts, split by OpenAI vs. local Ollama | -| `http_server_requests_seconds_count{job="keycloak", uri=".../protocol/openid-connect/token"}` | Keycloak | Login rate — Keycloak already exposes Micrometer-style HTTP metrics on its management port (`KC_METRICS_ENABLED=true`), reused as-is rather than building a separate login-tracking mechanism | - -Two alert rules are provisioned in Grafana's unified alerting -([`infra/grafana/provisioning/alerting/rules.yaml`](infra/grafana/provisioning/alerting/rules.yaml)) -and surface directly in the Grafana UI/dashboards — no notification -channel is configured, by design: - -- **Service down** — any scraped target reporting `up == 0` for 1 minute -- **High p95 latency** — a Spring service's p95 request latency above 1s for 5 minutes - -### Log aggregation - -**Loki** centralizes logs from every container/pod, browsable and searchable -in Grafana's `logs.json` dashboard (or Grafana's Explore view) instead of -`docker logs`/`kubectl logs` one container at a time. Same three-environment -scope as everything else above, with **Grafana Alloy** as the log shipper — -but the shipping mechanism deliberately differs by environment: - -- **Local/VM**: Alloy uses `discovery.docker` + `loki.source.docker` - ([`infra/alloy/config.alloy`](infra/alloy/config.alloy)), reading every - container's logs via the Docker socket. -- **Kubernetes**: Alloy uses `loki.source.kubernetes` - ([`infra/helm/team-devoops/files/alloy-config.alloy`](infra/helm/team-devoops/files/alloy-config.alloy)), - which fetches pod logs **through the Kubernetes API** (the `pods/log` - subresource) rather than reading node-local log files. This is a deliberate - choice, not the more common Promtail-as-DaemonSet setup: a DaemonSet reading - `/var/log/pods` via hostPath needs a `ClusterRole` and would also be able to - read *every other team's* pod logs on the same shared node — neither is - appropriate (or, for the `ClusterRole` part, even possible: verified against - the actual cluster that this identity can create namespaced `Role`/ - `RoleBinding` but not `ClusterRole`/`ClusterRoleBinding`). Alloy instead runs - as a plain namespace-scoped `Deployment` with a `Role` granting only - `list pods` / `get pods/log` inside `ge83mom-devops26`. - -Both variants need explicit relabeling (`discovery.relabel`) to turn -`__meta_docker_container_name` / `__meta_kubernetes_pod_name` into real -`container`/`pod` log labels — without it, every container's logs land in one -indistinguishable `service_name="unknown_service"` stream, which was caught -by actually querying Loki's label set while testing, not by inspection. +- **Raphael Frank** — infrastructure and CI/CD pipeline (Docker Compose, Traefik, Terraform/Ansible, Kubernetes/Helm, GitHub Actions, monitoring stack), plus contributions across the Spring services and the GenAI service. +- **Fady Samman** — web client (React SPA: routing, auth integration, all feature pages and API wiring). +- **Fabian Heinrich** — GenAI service (LangChain/RAG/LLM provider integration) and Spring service implementation. ## Docs +- [Architecture](docs/architecture.md) — subsystem responsibilities, interface contracts, request/auth lifecycle +- [Deployment](docs/deployment.md) — Terraform, Ansible, Helm chart reference, per-environment secrets +- [CI/CD](docs/cicd.md) — workflow job breakdown, required GitHub secrets +- [Monitoring](docs/monitoring.md) — metrics table, dashboard panels, alert rule definitions - [Problem Statement](docs/problem-statement.md) -- [System Architecture](docs/system-architecture.md) -- [Backlog](docs/backlog.md) +- [Outdated](docs/outdated/) — early architecture doc, UML diagrams, and backlog, kept for history but superseded by the above diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..3b0bef86 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,58 @@ +# Architecture + +This is the detailed companion to the [README's Architecture section](../README.md#architecture): per-service responsibilities, who calls whom, and how a request is authenticated end to end. There is deliberately no UML diagram here — see [`docs/outdated/`](outdated/) for why the previous ones were retired. + +## Subsystems + +### Client — `web-client` + +A React 19 + TypeScript SPA (Vite). It is the only subsystem with a UI, and it talks to **every** backend service directly over REST through the reverse proxy — there is no backend-for-frontend or API gateway aggregation layer. Each feature area (members, events, feedback, finance, letters, organization, the GenAI helper) has its own typed API client generated from `api/openapi.yaml`. + +Auth: `keycloak-js` (PKCE S256) obtains a JWT from Keycloak; an Axios request interceptor attaches it as `Authorization: Bearer ` and refreshes it proactively when it's within 30s of expiry. + +### Server — six Spring Boot 3 microservices + +| Service | Owns | Notes | +|---|---|---| +| Organization | sports, teams, role assignment | Also calls the Keycloak Admin REST API directly (`KeycloakRoleService`) to keep realm role assignments in sync with team/role changes made in-app | +| Member | member profiles | Also calls the Keycloak Admin REST API (`KeycloakService`) for the same reason | +| Event | training sessions, enrollment, attendance | | +| Feedback | trainer feedback, progress notes | Read by the GenAI service (see below) | +| Finance | one-time/recurring billing | | +| Letter | PDF/email generation from templates | The only Spring service with no database — templates and generated artifacts are not persisted as domain rows | + +Each service is a stateless OAuth2 resource server: it validates the incoming Bearer JWT against Keycloak's JWK set and maps `realm_access.roles` claims to Spring `ROLE_*` authorities. None of the six call each other directly — the only cross-service traffic on the server side is organization/member → Keycloak (role sync) and GenAI → feedback (below). + +Organization-service and member-service both authenticate to the Keycloak Admin REST API as the same confidential client, `org-role-sync` (service-account/client-credentials grant — see [Proxy & Auth](#proxy--auth-traefik--nginx-ingress--keycloak) below). Neither hardcodes its secret; both read it from an env var (`KEYCLOAK_ADMIN_CLIENT_SECRET` / `KEYCLOAK_SERVICE_ACCOUNT_CLIENT_SECRET` respectively) sourced from the same GitHub secret in every deploy target — see [docs/cicd.md](cicd.md). + +### GenAI — `py-genai-helper` + +A Python 3.12 / Flask service using LangChain. Unlike the Spring services, it is a REST **client** as well as a server: + +- It calls **feedback-service** (`GET /feedback`, Bearer-forwarded) to pull the data it summarizes when generating a member/team report. +- It calls **either OpenAI or a local Ollama instance** to run inference, selected per-request via the `uselocal` field on the report-generation endpoints (not a global config flag) — the web client exposes this as the "Use local LLM" toggle on the helper page. +- RAG question-answering persists uploaded documents in a **Chroma** vector store (a PVC-backed directory in Kubernetes, a bind-mounted volume elsewhere) rather than PostgreSQL. + +### Database — PostgreSQL + +Single instance, schema-per-service, documented in the [README's Database section](../README.md#database). No service reads another service's schema directly; cross-schema foreign keys exist (e.g. `event.events.creator_id → member.members.id`) but are enforced by the database, not queried across services. + +### Proxy & Auth — Traefik / nginx ingress + Keycloak + +- **Docker Compose / Azure VM**: Traefik terminates TLS (Let's Encrypt on the VM), applies a `forward-auth` middleware backed by its own confidential Keycloak client (session-cookie based, separate from the app-level Bearer tokens below), and strips path prefixes before forwarding to each service. +- **Kubernetes**: the cluster's own nginx ingress does the prefix-stripping and routing instead of Traefik; TLS is handled at the cluster edge. +- **Keycloak** (realm `devops`) is the single OIDC provider in every environment. Four confidential/public clients exist: `devops-client` (public, PKCE — the React app), `traefik-forward-auth` (confidential — gates the browser session at the proxy), `grafana` (confidential — Grafana's own admin-only OAuth login), and `org-role-sync` (confidential, service account only, no browser flow — organization-service and member-service's Admin REST API client, see above). None of the three confidential clients' secrets is hardcoded: each is templated as a `__PLACEHOLDER__` in `infra/keycloak/realm-config.json`, substituted at container start (Compose/VM) or chart render (Helm) from the matching GitHub secret — see [docs/cicd.md](cicd.md). + +## Request lifecycle (example: loading the members page) + +1. Browser requests `/` → proxy's forward-auth middleware checks for a session cookie; none yet → redirected through Keycloak's login page → cookie set on success. +2. `web-client` (now loaded) separately obtains its own JWT via `keycloak-js`, independent of the forward-auth session cookie above — these are two different auth mechanisms layered on the same Keycloak realm, not one shared session. +3. `web-client` calls `GET /api/v1/members` with `Authorization: Bearer `. +4. Proxy strips `/api/v1` (member-service's own routes start at `/members`), forwards to `member-service`. +5. `member-service` validates the JWT against Keycloak's JWK set, maps roles, authorizes, queries its own `member` schema, returns JSON. + +The GenAI report-generation flow adds one more hop: `web-client → py-genai-helper → feedback-service` (step 3 becomes two REST calls instead of one, with the same Bearer token forwarded along), plus an out-of-band call from `py-genai-helper` to OpenAI or Ollama that never passes through the proxy. + +## Environment differences + +The only things that differ between local Docker Compose, the Azure VM, and Kubernetes are: the proxy implementation (Traefik vs. nginx ingress), the JWT issuer URI each service is configured with (must match the `iss` claim of tokens actually issued by that environment's Keycloak — internal Docker hostname locally, public HTTPS URL on the VM, internal ClusterIP DNS on Kubernetes), and TLS termination. The application code and the Prometheus scrape config are identical across all three — see [docs/deployment.md](deployment.md). diff --git a/docs/cicd.md b/docs/cicd.md new file mode 100644 index 00000000..4ae7a58e --- /dev/null +++ b/docs/cicd.md @@ -0,0 +1,64 @@ +# CI/CD + +Detailed companion to the [README's CI/CD section](../README.md#cicd): every job, what gates a merge, and the full secrets/variables inventory. + +## CI — `.github/workflows/ci.yml` + +Runs on every pull request (plus manual `workflow_dispatch`). All jobs below feed into a single required `ci-success` aggregate job — branch protection only needs to require that one check. + +| Job | What it does | +|---|---| +| `organization-service`, `member-service`, `event-service`, `feedback-service`, `finance-service`, `letter-service` | Per Spring service: Checkstyle lint, then `./gradlew build` (compiles + runs JUnit tests); test reports uploaded as artifacts | +| `py-genai-helper` | `ruff check .` then `pytest -q` | +| `web-client` | Only runs if `web-client/**` changed (path filter via `dorny/paths-filter`) — `pnpm typecheck`, `pnpm lint`, `pnpm test:coverage`, `pnpm build`; coverage report uploaded as an artifact | +| `docker-build` | Whole-system `docker compose -f infra/docker-compose.yml build`, then verifies every expected image tag actually exists — catches Dockerfile/compose drift that per-service builds wouldn't | +| `codeql` | CodeQL SAST across `java-kotlin`, `python`, and `javascript-typescript` (matrix), `security-extended` query pack | +| `openapi-lint` | Spectral lint of `api/openapi.yaml` | +| `helm-validate` | `helm lint`, `helm template`, then schema-validates the rendered manifests with `kubeconform -strict` | + +## CD — `.github/workflows/cd.yml` + +Runs on every push to `main` (plus manual `workflow_dispatch`). Two deploy targets run in parallel: + +| Job | What it does | +|---|---| +| `deploy` (Ansible) | Writes the SSH key and three `.env` files from secrets — `infra/.env` (Postgres + Keycloak passwords and client secrets, for `docker-compose.yml`), `services/py-genai-helper/.env`, `services/spring-letter/.env` — builds a one-host inventory from the `VM_HOST` secret, runs `infra/ansible/playbook.yml` against the Azure VM | +| `docker-push` | Matrix build: all 6 Spring services + `py-genai-helper` + `web-client` + `api-docs`, pushed to `ghcr.io//:{sha,latest}` with BuildKit's GitHub Actions cache backend (`type=gha`, scoped per service) | +| `deploy-k8s` | Writes the kubeconfig from a secret, refreshes several ConfigMaps/Secrets that the Helm chart references but doesn't own (monitoring config, Keycloak theme, per-service env files — see [`infra/helm/README.md`](../infra/helm/README.md)), clears any stuck Helm release lock, then `helm upgrade --install --rollback-on-failure --timeout 15m` passing every database/Keycloak/client secret individually via `--set` (see table below) | + +`deploy-k8s` depends on `docker-push` (Kubernetes pulls prebuilt images from ghcr, unlike the VM which builds locally via Ansible/Compose). + +## Required GitHub secrets / variables + +| Kind | Name | Used by | Purpose | +|---|---|---|---| +| Variable | `AZURE_CLIENT_ID` | `infra.yml` | OIDC app registration (Service Principal) | +| Variable | `AZURE_TENANT_ID` | `infra.yml` | Azure AD tenant | +| Variable | `AZURE_SUBSCRIPTION_ID` | `infra.yml` | Target subscription | +| Secret | `VM_SSH_PUBLIC_KEY` | `infra.yml` | Public key planted on the VM by Terraform | +| Secret | `SSH_PRIVATE_KEY` | `cd.yml` (`deploy`) | Matching private key for Ansible to SSH in | +| Secret | `VM_HOST` | `cd.yml` (`deploy`) | Ansible inventory host — the VM's FQDN | +| Secret | `GENAI_ENV_CONTENT` | `cd.yml` (`deploy`, `deploy-k8s`) | Contents of `services/py-genai-helper/.env` | +| Secret | `LETTER_ENV_CONTENT` | `cd.yml` (`deploy`, `deploy-k8s`) | Contents of `services/spring-letter/.env` (mail credentials) | +| Secret | `KUBECONFIG` | `cd.yml` (`deploy-k8s`) | Kubeconfig for the RKE2 cluster | +| Secret | `DB_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Shared Postgres admin (`app_admin`) password. VM: written into `infra/.env` as `POSTGRES_PASSWORD`. K8s: Helm `--set database.password` | +| Secret | `DB_ORGANIZATION_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `organization_user` DB password. VM: `ORGANIZATION_DB_PASSWORD`. K8s: `--set database.users.organization.password` | +| Secret | `DB_MEMBER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `member_user` DB password. VM: `MEMBER_DB_PASSWORD`. K8s: `--set database.users.member.password` | +| Secret | `DB_EVENT_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `event_user` DB password. VM: `EVENT_DB_PASSWORD`. K8s: `--set database.users.event.password` | +| Secret | `DB_FEEDBACK_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `feedback_user` DB password. VM: `FEEDBACK_DB_PASSWORD`. K8s: `--set database.users.feedback.password` | +| Secret | `DB_FINANCE_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `finance_user` DB password. VM: `FINANCE_DB_PASSWORD`. K8s: `--set database.users.finance.password` | +| Secret | `DB_LETTER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `letter_user` DB password. VM: `LETTER_DB_PASSWORD`. K8s: `--set database.users.letter.password` | +| Secret | `DB_REPORTS_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | `reports_user` DB password. VM: `REPORTS_DB_PASSWORD`. K8s: `--set database.users.reports.password` | +| Secret | `KEYCLOAK_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Keycloak's own bootstrap console admin password. VM: `KEYCLOAK_ADMIN_PASSWORD`. K8s: `--set keycloak.adminPassword` | +| Secret | `KEYCLOAK_DB_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Keycloak's own Postgres password. VM: `KEYCLOAK_DB_PASSWORD`. K8s: `--set keycloak.db.password` | +| Secret | `KEYCLOAK_REALM_ADMIN_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Password for the seeded realm user `admin` (an application-level account, distinct from the Keycloak console admin above). VM: `KEYCLOAK_REALM_ADMIN_PASSWORD`. K8s: `--set keycloak.users.admin.password` | +| Secret | `KEYCLOAK_REALM_USER_PASSWORD` | `cd.yml` (`deploy`, `deploy-k8s`) | Password for the seeded realm user `user`. VM: `KEYCLOAK_REALM_USER_PASSWORD`. K8s: `--set keycloak.users.user.password` | +| Secret | `KEYCLOAK_ADMIN_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `org-role-sync` Keycloak service-account client, used by **both** organization-service and member-service to call the Keycloak Admin REST API. VM: `KEYCLOAK_ADMIN_CLIENT_SECRET`. K8s: `--set` on both `services.organization-service.env.KEYCLOAK_ADMIN_CLIENT_SECRET` and `services.member-service.env.KEYCLOAK_SERVICE_ACCOUNT_CLIENT_SECRET` (same value, two consumers) | +| Secret | `FORWARD_AUTH_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `traefik-forward-auth` Keycloak client. VM: `FORWARD_AUTH_CLIENT_SECRET`. K8s: `--set forwardAuth.clientSecret` | +| Secret | `FORWARD_AUTH_COOKIE_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Session-cookie signing secret for the forward-auth middleware. VM: `FORWARD_AUTH_COOKIE_SECRET`. K8s: `--set forwardAuth.cookieSecret` | +| Secret | `GRAFANA_OAUTH_CLIENT_SECRET` | `cd.yml` (`deploy`, `deploy-k8s`) | Secret for the `grafana` Keycloak client (Grafana's own generic-OAuth login). VM: `GRAFANA_OAUTH_CLIENT_SECRET`. K8s: `--set monitoring.grafana.oauthClientSecret` | +| Built-in | `GITHUB_TOKEN` | `cd.yml` (`docker-push`) | ghcr push auth (`packages: write` permission) | + +Each of the 16 password/secret entries above (`DB_ADMIN_PASSWORD` through `GRAFANA_OAUTH_CLIENT_SECRET`) is the single source of truth for that value across **both** deploy targets — nothing is hardcoded in `docker-compose.yml`, `infra/keycloak/realm-config.json` (templated with `__PLACEHOLDER__` tokens, substituted at container start), or the Helm chart's `values.yaml` (which ships `CHANGE_ME_*` placeholders, meant to be overridden, never deployed as-is). For local development, copy [`infra/.env.example`](../infra/.env.example) to `infra/.env` — it documents which local value maps to which GitHub secret. + +The Azure OIDC service principal needs `Contributor` on the subscription and `Storage Blob Data Contributor` on the `stdevoops26tfstate` state account. No long-lived Azure client secret is stored anywhere — auth is federated via OIDC. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 00000000..abe3f730 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,67 @@ +# Deployment + +Detailed companion to the [README's Deployment environments section](../README.md#deployment-environments). Three environments run the system; this page covers the two continuously-live ones. For local Docker Compose, the [README's Running Locally section](../README.md#running-locally) is already the complete reference. + +## Azure VM + +A single Ubuntu 24.04 VM in Azure's **Poland Central** region runs the exact same `infra/docker-compose.yml` stack as local dev, minus the override file — so it terminates real TLS via Let's Encrypt instead of serving plain HTTP. + +**Live URL:** + +### Provisioning — Terraform (`infra/terraform/`) + +| Resource | Purpose | +|---|---| +| `azurerm_resource_group` | `rg-team-devoops` | +| `azurerm_virtual_network` + `azurerm_subnet` | `10.0.0.0/16` / `10.0.1.0/24` | +| `azurerm_network_security_group` | Allows inbound 22 (SSH), 80 (HTTP), 443 (HTTPS) only | +| `azurerm_public_ip` | Static, with a free Azure-assigned FQDN (`domain_name_label`) | +| `azurerm_linux_virtual_machine` | `Standard_D2as_v4`, Ubuntu 24.04 LTS, SSH-key auth only (`disable_password_authentication = true`) | + +Remote state lives in Azure Blob Storage (`stdevoops26tfstate/tfstate`), shared and locked between local runs and CI so two applies can't race. Auth is OIDC (`ARM_USE_OIDC=true`) — no long-lived Azure client secret is stored anywhere. + +Run it via the **`infra` workflow** (`.github/workflows/infra.yml`, manual `workflow_dispatch` with `plan` / `apply` / `destroy`), or locally: + +```bash +az login +az account set --subscription +export ARM_SUBSCRIPTION_ID= +export ARM_USE_AZUREAD=true +cd infra/terraform +echo "admin_ssh_public_key = \"$(cat ~/.ssh/team-devoops-azure.pub)\"" > terraform.tfvars +terraform init +terraform plan +``` + +### Configuration — Ansible (`infra/ansible/playbook.yml`) + +Runs after Terraform, targeting the VM's public IP: + +1. Install Docker (apt repo + `docker-ce`, `docker-compose-plugin`). +2. Clone/update the repo at the deployed commit (`git` module, `force: true` so the working tree always matches `main`). +3. Write `infra/.env` (Postgres and Keycloak passwords, OIDC client secrets), `services/py-genai-helper/.env`, and `services/spring-letter/.env` from content passed in as extra-vars (sourced from GitHub Secrets — never committed). Nothing in `docker-compose.yml` is hardcoded; every credential is a `${VAR}` resolved from `infra/.env` at container start. +4. `docker compose -f infra/docker-compose.yml up -d --build --remove-orphans`. + +Triggered automatically by the **`cd` workflow** on every push to `main` — see [docs/cicd.md](cicd.md) for the exact secrets involved. + +### Typical workflow + +1. Infra change (new resource, VM size, etc.) → push → manually trigger `infra` workflow with `apply`. +2. App code change → merge to `main` → `cd` workflow redeploys automatically (both this VM and Kubernetes, in parallel). +3. Teardown → trigger `infra` workflow with `destroy`. + +## Kubernetes (TUM RKE2 / Rancher) + +The same services are also deployed via a Helm umbrella chart to the course's RKE2 cluster, with autoscaling and rolling-update self-healing configured per service. + +**Live URL:** · **Namespace:** `ge83mom-devops26` · **Chart:** [`infra/helm/team-devoops`](../infra/helm/team-devoops) + +This path has enough moving parts (one-time Secret/ConfigMap bootstrapping, the Keycloak theme's projected-volume workaround, resource-quota tuning to fit Ollama into a fixed namespace quota, autoscaling scope) that it has its own dedicated, actively-maintained reference — **[`infra/helm/README.md`](../infra/helm/README.md)** — rather than being duplicated here. Highlights: + +- Images are built and pushed to `ghcr.io/aet-devops26/team-devoops/` by the `cd` workflow's `docker-push` job, then `deploy-k8s` runs `helm upgrade --install --rollback-on-failure`. +- PostgreSQL, Prometheus, Grafana, Loki, and Ollama each run in-cluster with their own PVC on the cluster's default StorageClass (`csi-rbd-sc`, ReadWriteOnce). +- On every pull request, `ci.yml`'s `helm-validate` job lints the chart, renders it, and schema-validates the output with `kubeconform` — a broken chart can't merge. + +## Why not Kubernetes on Azure too + +The spec's cloud-environment requirement is satisfied by the Azure VM (Docker Compose) rather than a managed Kubernetes service like AKS; the Kubernetes requirement is satisfied by the course's Rancher-managed RKE2 cluster instead. Both requirements are met, just not by the same deployment target — provisioning a second Kubernetes cluster on Azure was judged not to add anything beyond what the VM and the RKE2 cluster already each demonstrate independently. diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 00000000..36c33faa --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,52 @@ +# Monitoring + +Detailed companion to the [README's Monitoring section](../README.md#monitoring). Runs identically in all three environments (local, Azure VM, Kubernetes) from the same config — Docker Compose service names and Kubernetes Service names are the same strings and resolve the same way, so [`infra/prometheus/prometheus.yml`](../infra/prometheus/prometheus.yml) is used unchanged everywhere. + +## Metrics — Prometheus + +| Job | Path | Exposed via | +|---|---|---| +| `organization-service`, `member-service`, `event-service`, `feedback-service`, `finance-service`, `letter-service` | `/actuator/prometheus` | Micrometer | +| `py-genai-helper` | `/metrics` | `prometheus-flask-exporter` | +| `keycloak` | `/metrics` (management port) | `KC_METRICS_ENABLED=true` | +| `traefik` | `/metrics` | Built-in Traefik metrics (compose/VM only — Kubernetes uses the cluster's nginx ingress instead) | + +Every job above yields request count, latency, and status-code histograms for free. Prometheus itself is never exposed outside the internal network/cluster in any environment. + +### Business-level custom metrics + +Generic per-request metrics tell you the system is *up*; these tell you it's doing its actual job: + +| Metric | Service | What it shows | +|---|---|---| +| `letters_sent_total{status}` | letter-service | Actual mail delivery outcomes | +| `letters_generated_total` | letter-service | PDF letters generated | +| `genai_rag_queries_total{status,provider}`, `genai_rag_query_duration_seconds{provider}` | py-genai-helper | RAG question-answering usage/latency, split OpenAI vs. local Ollama | +| `genai_report_generation_total{kind,status,provider}` | py-genai-helper | Member/team AI report generation attempts, split by provider | +| `http_server_requests_seconds_count{job="keycloak", uri=".../protocol/openid-connect/token"}` | Keycloak | Login rate (reused from Keycloak's own Micrometer-style metrics, not a separate mechanism) | + +## Dashboards — Grafana + +Reachable at `/dashboard` in every environment, admin-only: authentication is exclusively via Keycloak generic-OAuth (client `grafana`), with `GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_STRICT=true` rejecting any login whose realm role doesn't map to a non-empty Grafana org role — mapping the non-admin branch to an empty string is what actually rejects the login (mapping it to the literal string `"None"` does not; verified locally). Dashboards and datasources are provisioned as code from [`infra/grafana/`](../infra/grafana/), nothing configured by hand. + +| Dashboard | Covers | +|---|---| +| `service-overview.json` | Request rate, p95 latency, error rate per Spring service (dropdown filter), up/down status, Keycloak login rate, letters sent/generated | +| `genai-service.json` | Request rate, p95 latency, error rate for the GenAI service, RAG query rate/latency, report-generation rate by kind/status — each split by LLM provider | +| `logs.json` | Centralized logs from every service (via Loki), filterable by container, plus a log-volume graph | + +## Alerts + +Provisioned in Grafana's unified alerting ([`infra/grafana/provisioning/alerting/rules.yaml`](../infra/grafana/provisioning/alerting/rules.yaml)), surfaced directly in the Grafana UI — no external notification channel configured, by design: + +- **Service down** (`service-down`) — any scraped target matching the core-services job regex reports `up == 0`, for ≥1 minute, severity `critical`. +- **High p95 latency** (`high-latency`) — `histogram_quantile(0.95, ...http_server_requests_seconds_bucket...)` exceeds 1s, for ≥5 minutes, severity `warning`. + +## Log aggregation — Loki + Grafana Alloy + +Centralizes logs from every container/pod so they're searchable in Grafana instead of `docker logs` / `kubectl logs` one at a time. The shipping mechanism deliberately differs by environment: + +- **Local/VM**: Alloy uses `discovery.docker` + `loki.source.docker` ([`infra/alloy/config.alloy`](../infra/alloy/config.alloy)), reading every container's logs via the Docker socket. +- **Kubernetes**: Alloy uses `loki.source.kubernetes` ([`infra/helm/team-devoops/files/alloy-config.alloy`](../infra/helm/team-devoops/files/alloy-config.alloy)), fetching pod logs through the Kubernetes API (`pods/log` subresource) rather than reading node-local log files. This is deliberate, not the more common Promtail-as-DaemonSet pattern: a DaemonSet reading `/var/log/pods` via hostPath needs a `ClusterRole` and would also read every other team's pods on the same shared node — neither is appropriate (the `ClusterRole` part isn't even possible: this identity can create namespaced `Role`/`RoleBinding` but not cluster-scoped equivalents, verified against the actual cluster). Alloy instead runs as a plain namespace-scoped `Deployment` with a `Role` granting only `list pods` / `get pods/log` inside `ge83mom-devops26`. + +Both variants need explicit relabeling (`discovery.relabel`) to turn `__meta_docker_container_name` / `__meta_kubernetes_pod_name` into real `container`/`pod` log labels — without it, every container's logs land in one indistinguishable `service_name="unknown_service"` stream, caught by querying Loki's actual label set while testing, not by inspection. diff --git a/docs/SportsClub_Class_Diagram.png b/docs/outdated/SportsClub_Class_Diagram.png similarity index 100% rename from docs/SportsClub_Class_Diagram.png rename to docs/outdated/SportsClub_Class_Diagram.png diff --git a/docs/SportsClub_Component_Diagram.png b/docs/outdated/SportsClub_Component_Diagram.png similarity index 100% rename from docs/SportsClub_Component_Diagram.png rename to docs/outdated/SportsClub_Component_Diagram.png diff --git a/docs/SportsClub_Use_Case_Diagram.png b/docs/outdated/SportsClub_Use_Case_Diagram.png similarity index 100% rename from docs/SportsClub_Use_Case_Diagram.png rename to docs/outdated/SportsClub_Use_Case_Diagram.png diff --git a/docs/backlog.md b/docs/outdated/backlog.md similarity index 100% rename from docs/backlog.md rename to docs/outdated/backlog.md diff --git a/docs/system-architecture.md b/docs/outdated/system-architecture.md similarity index 100% rename from docs/system-architecture.md rename to docs/outdated/system-architecture.md