Harden the Zava SRE Agent deployment and correlation demo

.gitignore

@@ -9,3 +9,6 @@
 Thumbs.db
 Desktop.ini
 samples/deployment-compliance/skills/.DS_Store
+
+# Browser automation artifacts
+.playwright-mcp/

labs/zava-aks-postgres/.github/skills/deploying-demo/SKILL.md

@@ -10,10 +10,14 @@ Run these and install anything missing:
 - `az version` — need 2.60+
 - `azd version` — need 1.9+
 - `pwsh -v` — need 7.4+
+- Azure permission: Owner, User Access Administrator, or equivalent
+  `Microsoft.Authorization/roleAssignments/write` at subscription scope. The
+  template grants the agent runtime identity subscription Reader for correlation
+  context and removes it during `azd down`.
 
 > Note: `kubectl` is **not** required on your local workstation. The cluster is private. Operator
 > scripts in this repo go through `az aks command invoke` (wrapped by `scripts/_aks-helpers.ps1`).
-> The SRE Agent reaches the cluster the same way through its `az` CLI tools — no kubeconfig either side.
+> The SRE Agent uses its built-in `RunKubectlReadCommand` and `RunKubectlWriteCommand` tools instead.
 
 ## Phase 1: Azure Deployment
 1. Check if user has a subscription: `az account show`
@@ -25,7 +29,7 @@ Run these and install anything missing:
 
 ## Phase 2: Verify Deployment
 The AKS API server is private (`enablePrivateCluster: true`) — local kubectl
-will not work. Use the same path the SRE Agent uses:
+will not work. Human operators use the Azure-proxied command-invoke path:
 
 ```powershell
 . .\scripts\_aks-helpers.ps1
@@ -46,12 +50,19 @@ $ip = ($r.logs -replace '[^\d\.]','').Trim()
        -Command "kubectl exec -n zava-demo deploy/zava-api -- wget -qO- http://localhost:3001/api/health"
    ```
 
+> If you are manually testing from an SRE Agent chat and absolutely need terminal-native kubectl,
+> first run a read-only command against this cluster with `RunKubectlReadCommand`. That warms the
+> process-local AKS CA path; terminal kubectl can then use an already-valid kubeconfig in the same
+> runtime. A runtime restart clears the warm-up. Since `RunKubectl*` accepts the same kubectl
+> commands, prefer it directly. See `docs/aks-access-and-auth.md` for the kubeconfig, TLS,
+> authentication, authorization, and private-network mechanics behind this behavior.
+
 ## Phase 3: Sync knowledge + verify SRE Agent
 
 The agent itself — connectors, custom skills, response plans, autonomous mode,
 Azure Monitor binding — is already provisioned by Bicep during `azd up`. This
-script just uploads knowledge files (the one data-plane piece with no ARM API)
-and prints a verification readout of the Bicep-deployed assets.
+script uploads knowledge files, syncs the agent-global custom instructions,
+enables the Microsoft Learn MCP tools, and verifies the complete configuration.
 
 1. Get azd values: `$env:SRE_AGENT_ENDPOINT = azd env get-value SRE_AGENT_ENDPOINT` (and RESOURCE_GROUP, SRE_AGENT_NAME)
 2. Run: `.\scripts\setup-sre-agent.ps1` (auto-detects ResourceGroup and AgentName from `azd env`)

labs/zava-aks-postgres/.github/skills/managing-sre-agent/SKILL.md

@@ -10,17 +10,23 @@ For **this demo**, agent configuration is declared in Bicep
 `Microsoft.App/agents/*` ARM resources:
 
 - **Agent settings** — autonomous mode, High access level, Azure Monitor incident binding
-- **Connectors** — `app-insights`, `log-analytics`, `azure-monitor` (MonitorClient), `microsoft-learn` (MCP)
-- **Custom skills** — `database-incidents`, `performance-incidents`, `application-incidents`, `general-triage` (the unknown bucket), `proactive-health-check` (auto-selected by description; max 5 concurrent)
-- **Response plans / incident filters** — `zava-database`, `zava-performance`, `zava-application` (known-good, autonomous) + `zava-unknown` (catch-all, Review mode)
-  (routed by `titleContains` / `titleNotContains`)
+- **Connectors** — `app-insights`, `log-analytics`, `azure-monitor` (MonitorClient), `learn-docs` (Microsoft Learn no-auth MCP)
+- **Custom skills** — `database-incidents`, `performance-incidents`, `application-incidents`, `general-triage` (the unknown bucket), `proactive-health-check`, `incident-correlation` (auto-selected by description; max 5 concurrent)
+- **Response plans / incident filters** — `zava-database`, `zava-performance`, `zava-application` (purpose-built, autonomous) + `zava-unknown` (bounded fallback, Review mode), routed by `titleContains` / `titleNotContains`
 - **RBAC** — system-assigned managed identity granted Reader, Monitoring Reader,
-  Contributor, and AKS RBAC Cluster Admin on the resource group
+  Contributor, and AKS RBAC Cluster Admin on the resource group; the runtime
+  user-assigned identity also has subscription-level Reader so the
+  correlation skill can read Alerts Management and Resource Health event feeds
 
 To change any of these, **edit the Bicep and run `azd provision`**. There is no
 data-plane CLI tool for them in this repo.
 
-## Knowledge base (the one data-plane piece)
+Do not design overlapping response plans around an assumed priority or
+specificity rule. Treat multiple matches as undefined, keep purpose-built
+filters mutually exclusive where routing matters, and make any fallback both
+positively scoped and explicitly exclude every known route.
+
+## Data-plane configuration
 
 ARM does not yet surface SRE Agent knowledge files, so they're uploaded by
 `scripts/setup-sre-agent.ps1`:
@@ -42,6 +48,11 @@ To remove a knowledge file: delete the local `.md`, then delete the correspondin
 `<name>.md` from the agent's Builder UI > Knowledge sources view (the
 script does not delete remote files that are no longer present locally).
 
+The same script also syncs the singleton agent-global custom instructions from
+`sre-config/custom-instructions.md` and enables the Microsoft Learn MCP tools.
+Keep global instructions short; detailed procedures belong in a skill so they
+load only when relevant.
+
 ## When helping users
 
 1. **"Add a skill / response plan / connector"** — edit `infra/modules/sre-agent.bicep`
@@ -52,10 +63,15 @@ script does not delete remote files that are no longer present locally).
    output reports `[OK]` or `[MISSING]` for every Bicep-deployed asset.
 4. **Activity-log alerts gotcha** — they fire as Sev4 regardless of the configured
    severity, so response plan filters must match all severities (Bicep already does).
-5. **Runbook philosophy** — the five skills (`database-incidents`, `performance-incidents`,
-   `application-incidents`, `general-triage`, `proactive-health-check`) in `sre-agent.bicep`
+5. **Runbook philosophy** — the six skills (`database-incidents`, `performance-incidents`,
+   `application-incidents`, `general-triage`, `proactive-health-check`, `incident-correlation`) in `sre-agent.bicep`
    state the facts the agent can't infer (the RBAC it holds, what each alert means, which
    table to look at) — e.g. the `database-incidents` runbook's `postgres-unreachable` triage
    table maps alert → ARM-state check → action TYPE — while keeping the actual remediation at
    the action-type level, NOT copy-paste SQL/kubectl recipes. Preserve both halves when
    adding/modifying skills. See AGENTS.md "Non-Obvious Things" for the full rationale.
+6. **Kubernetes tool guidance** — wire `RunKubectlReadCommand` / `RunKubectlWriteCommand`
+   into runtime skills and use them directly. If an ad-hoc chat must use terminal-native
+   kubectl, first issue a built-in read against the same cluster to warm the process-local
+   AKS CA path; the terminal command still needs an already-valid kubeconfig, and the warm-up
+   is lost on runtime restart. This is a side note, not a reason to add `RunInTerminal` to skills.

labs/zava-aks-postgres/.github/skills/running-demo/SKILL.md

@@ -12,7 +12,7 @@ This skill drives the full demo using Playwright MCP for browser control. Execut
 ```powershell
 # AKS is a private cluster — kubectl from your local workstation won't work without VPN/jumpbox.
 # Use `Invoke-AksCommand` (wraps `az aks command invoke` for human-operator polling/diagnostics).
-# The SRE Agent uses native kubectl; this helper is for human operators without the agent's VNet/DNS/proxy setup.
+# The SRE Agent uses the built-in RunKubectl* system tools; this helper is for human operators.
 . .\scripts\_aks-helpers.ps1
 $rg  = (azd env get-value RESOURCE_GROUP)
 $aks = (azd env get-value AKS_CLUSTER_NAME)
@@ -24,6 +24,12 @@ $storeUrl = "http://$ip"
 $agentUrl = (azd env get-value AGENT_PORTAL_URL)  # deep-links to this agent's blade — sign in if prompted
 ```
 
+When observing or prompting the SRE Agent, prefer its built-in `RunKubectlReadCommand` and
+`RunKubectlWriteCommand`; they accept the same kubectl commands used in a terminal. If a manual
+test absolutely requires terminal-native kubectl, run a built-in read against the cluster first
+to warm the process-local AKS CA path, then use the existing valid terminal kubeconfig. The warm-up
+does not survive a runtime restart.
+
 ## Scenario 1: Database Outage
 
 ### Step 1: Show healthy state
@@ -81,7 +87,7 @@ Wait 30 seconds.
 
 ### Step 4: Watch the agent
 1. Check SRE Agent portal for investigation
-2. Agent needs to find the K8s NetworkPolicy via native `kubectl get networkpolicy -n zava-demo -o yaml` and remove it via `kubectl delete networkpolicy database-tier-isolation -n zava-demo` (run in its sandbox terminal) — this is harder than Scenario 1 and may take longer
+2. Agent needs to find the K8s NetworkPolicy with `RunKubectlReadCommand` using `kubectl get networkpolicy -n zava-demo -o yaml`, then remove it with `RunKubectlWriteCommand` using `kubectl delete networkpolicy database-tier-isolation -n zava-demo` - this is harder than Scenario 1 and may take longer
 3. Poll for NetworkPolicy removal (the AKS API server is private — go through ARM):
    ```powershell
    Invoke-AksCommand -ResourceGroup $rg -ClusterName $aks -Command "kubectl get networkpolicy -n zava-demo"
@@ -117,7 +123,7 @@ If the script aborts with "Telemetry pipeline is dead", the api pods stopped sen
 4. (`break-db-perf.ps1` already launched a 15-min in-cluster Kubernetes Job (`zava-cat-load` in the `zava-demo` namespace) that hammers `/api/products/category/<X>` over the cluster-internal Service DNS. This pushes real traffic past the alert's 30ms threshold — the 1Hz `__probe` is excluded by the alert KQL. The Job auto-cleans 60s after completion via `ttlSecondsAfterFinished`; `fix-db-perf.ps1` also deletes it explicitly. Run with `-NoLoad` to skip.)
 
 ### Step 4: Watch agent
-1. Monitor SRE Agent portal — it should detect slow response times via App Insights, identify the missing index, and run `CREATE INDEX CONCURRENTLY` in-cluster via `bin/run-sql.js` (the agent runs native `kubectl exec -n zava-demo deploy/zava-api -- node bin/run-sql.js "<SQL>"` from its sandbox terminal — the helper reuses the pod's workload identity)
+1. Monitor SRE Agent portal - it should detect slow response times via App Insights, identify the missing index, and run `CREATE INDEX CONCURRENTLY` in-cluster via `bin/run-sql.js` (`RunKubectlWriteCommand` executes `kubectl exec -n zava-demo deploy/zava-api -- node bin/run-sql.js "<SQL>"`; the helper reuses the pod's workload identity)
 2. Do not run `fix-db-perf.ps1` as part of the demo — same rule as the other scenarios: the script is post-demo cleanup, not an agent-failure fallback.
 
 ### Step 5: Show recovery
@@ -170,6 +176,45 @@ If the script aborts with "Telemetry pipeline is dead", the api pods stopped sen
 1. Navigate to `$storeUrl/api/products` — returns 200 again
 2. Navigate to `$storeUrl` — products load; take screenshot
 
+## Scenario 5: Compound Independent Faults
+
+This scenario overlaps Scenario 3 and Scenario 4 by 90 seconds. It should
+produce two separate alerts and two independent causes, not one causal story.
+
+### Step 1: Confirm healthy state
+1. Navigate to `$storeUrl` and `$storeUrl/api/health`
+2. Confirm products load and the database is connected
+
+### Step 2: Break both paths
+```powershell
+.\.github\skills\running-demo\scripts\break-compound.ps1
+```
+The script drops both category indexes, starts the sustained category load,
+waits 90 seconds, then deploys `FAULT_INJECT=500`.
+
+### Step 3: Verify the overlap
+1. `$storeUrl/api/products` returns HTTP 500 while `/api/health` remains healthy.
+2. Query `/api/diagnostics`; both category indexes are absent and product scans are sequential.
+3. Confirm the `zava-cat-load` Job is active through `Invoke-AksCommand`.
+4. Expect both `Zava-products-query-slow` and `Zava-http-5xx-errors` within 5-10 minutes.
+
+### Step 4: Grade the investigation
+A correct investigation enumerates the nearby alerts and disabled rule inventory,
+then proves the mechanisms are independent:
+- 5xx failures are app-local (`localhost:3001`) and correlate with the rollout.
+- PostgreSQL CPU/latency rises, but its slow queries succeed and create no failed PG dependencies.
+- `Zava-db-cpu-saturation` is present but disabled.
+
+Do not accept alert timestamps alone as causality; every dispatching rule uses
+PT5M evaluation and the 90-second injection order can be reversed at alert time.
+
+### Step 5: Cleanup
+Let the SRE Agent remediate during a demo. For post-demo cleanup or test teardown:
+```powershell
+.\.github\skills\running-demo\scripts\fix-compound.ps1
+```
+Verify `/api/products` returns 200, both indexes exist, and the load Job is gone.
+
 ## Chat demo: interrogate the hub firewall (network device)
 
 No break needed — this shows the agent treating the **hub Azure Firewall** as a queryable "network device" in the hub-and-spoke topology.

labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-bad-deploy.ps1

@@ -35,6 +35,11 @@ $ErrorActionPreference = "Stop"
 . "$PSScriptRoot\..\..\..\..\scripts\_aks-helpers.ps1"
 $ctx = Resolve-AksContext -ResourceGroup $ResourceGroup -ClusterName $ClusterName
 
+# Azure Monitor's stateful per-rule instance is separate from agent-side merge.
+# Refuse to inject a new fault while the prior condition is still Fired, and
+# close a resolved prior instance so this run dispatches as a fresh alert.
+Reset-DemoAlertRule -ResourceGroup $ctx.ResourceGroup -AlertRuleName 'Zava-http-5xx-errors'
+
 # Telemetry precheck. Zava-http-5xx-errors evaluates the requests/failed metric,
 # which is derived from AppRequests telemetry. If the api isn't currently sending
 # telemetry to the workspace, the alert can never fire no matter how many 500s the