From 32aa0d654aabba2d4f10e34f61e6f56f506fd16a Mon Sep 17 00:00:00 2001 From: Roland Krummenacher Date: Tue, 4 Aug 2026 09:35:44 +0200 Subject: [PATCH 1/5] docs(agents): add KQL language rules where query-writing agents load them The KQL rules from the coding guidelines (#2223) live in docs-wiki and AGENTS.md, which contributors and repo coding agents read but shipped plugin agents never load. The ftk-database-query agent and the finops-toolkit skill generate ad-hoc KQL at runtime and would reproduce the exact pitfalls fixed repo-wide in #2189/#2220/#2225 (bare joins defaulting to innerunique, tolower() comparisons, contains vs has, fullouter without key coalescing). Distill the runtime-relevant rules into the KQL language rules section of finops-hub-database-guide.md, which the skill task routing loads before any query is written, and reference it from the agent's operational guidelines. Both agent-skill structures (agent-skills today, agent-plugin after #2167) link to src/queries, so the rules survive the plugin restructuring. Co-Authored-By: Claude Fable 5 --- src/queries/finops-hub-database-guide.md | 23 +++++++++++++++++++ .../agents/ftk-database-query.md | 13 ++++++----- 2 files changed, 30 insertions(+), 6 deletions(-) diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index 1ea575922..ad647b29d 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -14,6 +14,7 @@ This document provides a comprehensive overview of how to query and analyze data - [Prerequisites](#prerequisites) - [Overview](#overview) - [Query best practices](#query-best-practices) + - [KQL language rules](#kql-language-rules) - [Key enrichment columns](#key-enrichment-columns) - [Example queries](#example-queries) - [Example query: Cost by billing profile, invoice section, team, product, application](#example-query-cost-by-billing-profile-invoice-section-team-product-application) @@ -100,6 +101,27 @@ The FinOps hubs database is designed to support advanced cost and usage analytic - **Leverage Enrichment Columns:** Columns prefixed with `x_` provide additional context and enrichment for FinOps analysis. +### KQL language rules + +Apply these rules to every query you write or modify. They mirror the project coding guidelines and come from real correctness bugs fixed across the toolkit ([#2189](https://github.com/microsoft/finops-toolkit/pull/2189), [#2220](https://github.com/microsoft/finops-toolkit/pull/2220), [#2225](https://github.com/microsoft/finops-toolkit/pull/2225)). + +**String matching** + +- Never wrap a column in `tolower()`/`toupper()` to compare it. Every plain KQL string operator is already case-insensitive (`=~`, `has`, `contains`, `startswith`, `in~`); the `_cs` variants and `==`/`in` are the case-sensitive forms. Write `Col =~ 'value'`, not `tolower(Col) == 'value'`. +- Prefer `has` when the needle is a whole term or a separator-bounded phrase (`ResourceId has '/microsoft.capacity/reservationorders/'`) — it uses the term index. Use `contains` only when the needle can be fused inside a larger token (`ConsumedUnit contains 'MB'` also matches `Mbps`). +- Collapse operator chains: `Col in~ ('a', 'b')` instead of repeated `=~` with `or`; `has_any`/`has_all` instead of `has` chains. + +**Joins and lookups** + +- Never write a bare `| join` — always state `kind=` explicitly. The default flavor is `innerunique`, which deduplicates the *left* side on the join key and silently drops rows. +- To enrich cost rows from a small reference table, use `lookup kind=leftouter` instead of `join kind=leftouter`. It broadcasts the small side and emits no duplicated key columns. +- Neither `join` nor `lookup` deduplicates the right side — a reference table with more than one row per key multiplies your fact rows. Guarantee one row per key with `summarize take_any(Col1), take_any(Col2) by Key` (`distinct` over multiple columns does *not* guarantee this). +- For exclusions ("rows with no match"), use `join kind=leftanti` — not `kind=leftouter` + `where isempty(...)`, which inflates counts when the right side has duplicate keys. +- For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. +- For grand totals and percent-of-total, use `let Total = toscalar(...)` — never a cross join (`on 1 == 1` is not valid KQL and fails at runtime). + +**Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and allows at most 3 joins per query. Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. + --- ## Key Enrichment Columns @@ -694,6 +716,7 @@ The following table lists the columns produced in the `All available columns` qu | 2025-05-16 | 1.0 | FinOps Toolkit Team | Initial documentation | | 2025-05-16 | 1.1 | FinOps Toolkit Team | Expanded schema, glossary, references | | 2026-05-28 | 1.2 | Sprint 3000 UAT | Live-Hub schema audit: `Costs()`, `Prices()`, `Recommendations()` numeric columns retyped from `decimal` to `real` to match deployed Hub schema (cause of SEM0019 errors). `Recommendations()` table expanded from 12 to 20 columns to add the 8 columns present in the live schema. `x_RecommendationDate` documented as commonly-null in live Hubs (root cause of T-3000.13). | +| 2026-08-04 | 1.3 | FinOps Toolkit Team | Added KQL language rules (case-insensitive operators, explicit join kinds, `lookup` for dimension enrichment) distilled from the project coding guidelines so query-writing agents load them alongside the schema. | --- diff --git a/src/templates/claude-plugin/agents/ftk-database-query.md b/src/templates/claude-plugin/agents/ftk-database-query.md index 42e20002b..d20bdca89 100644 --- a/src/templates/claude-plugin/agents/ftk-database-query.md +++ b/src/templates/claude-plugin/agents/ftk-database-query.md @@ -129,12 +129,13 @@ The plugin provides an `azure-mcp-server` with the Kusto namespace for executing 1. **Check the query catalog first**: Before writing custom KQL, check if `skills/finops-toolkit/references/queries/catalog/` has a query that matches the user's scenario. 2. **Start with costs-enriched-base**: For custom analysis not covered by the catalog, begin with `costs-enriched-base.kql` as your foundation. -3. **Use precise column names**: Reference exact field names from the schema. Columns prefixed with `x_` are toolkit enrichments. -4. **Filter early**: Always scope queries to relevant time periods using `ChargePeriodStart` before aggregation. -5. **Prefer EffectiveCost**: Use `EffectiveCost` (after discounts) as the default cost metric unless the user specifically asks for `BilledCost` (billed), `ContractedCost` (negotiated), or `ListCost` (retail). -6. **Handle tags carefully**: Tags is a dynamic column. Extract values with `tostring(Tags['key-name'])`. -7. **Format results**: Present query output in markdown tables with clear column headers. Include the source query and any parameter values used. -8. **Explain the query**: When constructing KQL, explain what data you're accessing, which table function, and why. +3. **Follow the KQL language rules**: Apply the "KQL language rules" section of `skills/finops-toolkit/references/queries/finops-hub-database-guide.md` to every query you write. In particular: never write a bare `| join` (the `innerunique` default silently drops rows — always state `kind=`), prefer `lookup` for enriching from small reference tables, and never use `tolower()` in comparisons (KQL string operators are case-insensitive — use `=~`, `has`, `in~`). +4. **Use precise column names**: Reference exact field names from the schema. Columns prefixed with `x_` are toolkit enrichments. +5. **Filter early**: Always scope queries to relevant time periods using `ChargePeriodStart` before aggregation. +6. **Prefer EffectiveCost**: Use `EffectiveCost` (after discounts) as the default cost metric unless the user specifically asks for `BilledCost` (billed), `ContractedCost` (negotiated), or `ListCost` (retail). +7. **Handle tags carefully**: Tags is a dynamic column. Extract values with `tostring(Tags['key-name'])`. +8. **Format results**: Present query output in markdown tables with clear column headers. Include the source query and any parameter values used. +9. **Explain the query**: When constructing KQL, explain what data you're accessing, which table function, and why. ## FinOps Domain Context From 636db16f3998958f6d6b176c9760fbdca227d820 Mon Sep 17 00:00:00 2001 From: Roland Krummenacher Date: Tue, 4 Aug 2026 09:42:42 +0200 Subject: [PATCH 2/5] =?UTF-8?q?docs:=20address=20Copilot=20review=20?= =?UTF-8?q?=E2=80=94=20precise=20case-sensitivity=20wording,=20legacy=20ex?= =?UTF-8?q?ample=20note,=20markdownlint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Reword the case-insensitivity rule so it no longer claims every plain operator is case-insensitive (== and in are not); scope the claim to matching operators with the _cs/equality forms as the opt-in. - Add an explicit note that the legacy 'join ... on 1 == 1' example further down is replaced by toscalar() in #2225 and must not be copied. - Fix MD036 (bold-as-heading) and align the new change log row's table pipes with the header (MD060). Co-Authored-By: Claude Fable 5 --- src/queries/finops-hub-database-guide.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index ad647b29d..944ad4781 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -105,13 +105,13 @@ The FinOps hubs database is designed to support advanced cost and usage analytic Apply these rules to every query you write or modify. They mirror the project coding guidelines and come from real correctness bugs fixed across the toolkit ([#2189](https://github.com/microsoft/finops-toolkit/pull/2189), [#2220](https://github.com/microsoft/finops-toolkit/pull/2220), [#2225](https://github.com/microsoft/finops-toolkit/pull/2225)). -**String matching** +#### String matching -- Never wrap a column in `tolower()`/`toupper()` to compare it. Every plain KQL string operator is already case-insensitive (`=~`, `has`, `contains`, `startswith`, `in~`); the `_cs` variants and `==`/`in` are the case-sensitive forms. Write `Col =~ 'value'`, not `tolower(Col) == 'value'`. +- Never wrap a column in `tolower()`/`toupper()` to compare it. KQL's string *matching* operators are case-insensitive by default (`=~`, `has`, `contains`, `startswith`, `in~`); case-sensitive matching is what you opt into via the `_cs` variants and the equality operators `==`/`in`. Write `Col =~ 'value'`, not `tolower(Col) == 'value'`. - Prefer `has` when the needle is a whole term or a separator-bounded phrase (`ResourceId has '/microsoft.capacity/reservationorders/'`) — it uses the term index. Use `contains` only when the needle can be fused inside a larger token (`ConsumedUnit contains 'MB'` also matches `Mbps`). - Collapse operator chains: `Col in~ ('a', 'b')` instead of repeated `=~` with `or`; `has_any`/`has_all` instead of `has` chains. -**Joins and lookups** +#### Joins and lookups - Never write a bare `| join` — always state `kind=` explicitly. The default flavor is `innerunique`, which deduplicates the *left* side on the join key and silently drops rows. - To enrich cost rows from a small reference table, use `lookup kind=leftouter` instead of `join kind=leftouter`. It broadcasts the small side and emits no duplicated key columns. @@ -120,6 +120,8 @@ Apply these rules to every query you write or modify. They mirror the project co - For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. - For grand totals and percent-of-total, use `let Total = toscalar(...)` — never a cross join (`on 1 == 1` is not valid KQL and fails at runtime). +> **Note:** One legacy example later in this guide still uses the `join ... on 1 == 1` pattern. It is replaced with `toscalar()` in [#2225](https://github.com/microsoft/finops-toolkit/pull/2225) — do not copy it. + **Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and allows at most 3 joins per query. Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. --- @@ -716,7 +718,7 @@ The following table lists the columns produced in the `All available columns` qu | 2025-05-16 | 1.0 | FinOps Toolkit Team | Initial documentation | | 2025-05-16 | 1.1 | FinOps Toolkit Team | Expanded schema, glossary, references | | 2026-05-28 | 1.2 | Sprint 3000 UAT | Live-Hub schema audit: `Costs()`, `Prices()`, `Recommendations()` numeric columns retyped from `decimal` to `real` to match deployed Hub schema (cause of SEM0019 errors). `Recommendations()` table expanded from 12 to 20 columns to add the 8 columns present in the live schema. `x_RecommendationDate` documented as commonly-null in live Hubs (root cause of T-3000.13). | -| 2026-08-04 | 1.3 | FinOps Toolkit Team | Added KQL language rules (case-insensitive operators, explicit join kinds, `lookup` for dimension enrichment) distilled from the project coding guidelines so query-writing agents load them alongside the schema. | +| 2026-08-04 | 1.3 | FinOps Toolkit Team | Added KQL language rules (case-insensitive operators, explicit join kinds, `lookup` for dimension enrichment) distilled from the project coding guidelines so query-writing agents load them alongside the schema. | --- From ad35f43f9da55dc6c5877c9f7ca8c323530a5ebf Mon Sep 17 00:00:00 2001 From: Roland Krummenacher Date: Tue, 4 Aug 2026 10:30:16 +0200 Subject: [PATCH 3/5] docs: drop transient legacy-example note The rule itself already states the pattern is invalid; the note would go stale the moment #2225 replaces the example it points at. Co-Authored-By: Claude Fable 5 --- src/queries/finops-hub-database-guide.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index 944ad4781..8c577a0cd 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -119,9 +119,6 @@ Apply these rules to every query you write or modify. They mirror the project co - For exclusions ("rows with no match"), use `join kind=leftanti` — not `kind=leftouter` + `where isempty(...)`, which inflates counts when the right side has duplicate keys. - For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. - For grand totals and percent-of-total, use `let Total = toscalar(...)` — never a cross join (`on 1 == 1` is not valid KQL and fails at runtime). - -> **Note:** One legacy example later in this guide still uses the `join ... on 1 == 1` pattern. It is replaced with `toscalar()` in [#2225](https://github.com/microsoft/finops-toolkit/pull/2225) — do not copy it. - **Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and allows at most 3 joins per query. Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. --- From cd1ae51426e9eb9fff03b2cb785512efab6b5ac7 Mon Sep 17 00:00:00 2001 From: Roland Krummenacher Date: Tue, 4 Aug 2026 10:35:01 +0200 Subject: [PATCH 4/5] docs: trim grand-total rule to the toscalar() recommendation The 'on 1 == 1 is invalid KQL' clause was trivia about an edge case, not a widespread bad practice; the actionable part is computing totals once with toscalar() instead of joining an aggregate subquery. Co-Authored-By: Claude Fable 5 --- src/queries/finops-hub-database-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index 8c577a0cd..eead9eccf 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -118,7 +118,7 @@ Apply these rules to every query you write or modify. They mirror the project co - Neither `join` nor `lookup` deduplicates the right side — a reference table with more than one row per key multiplies your fact rows. Guarantee one row per key with `summarize take_any(Col1), take_any(Col2) by Key` (`distinct` over multiple columns does *not* guarantee this). - For exclusions ("rows with no match"), use `join kind=leftanti` — not `kind=leftouter` + `where isempty(...)`, which inflates counts when the right side has duplicate keys. - For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. -- For grand totals and percent-of-total, use `let Total = toscalar(...)` — never a cross join (`on 1 == 1` is not valid KQL and fails at runtime). +- For grand totals and percent-of-total, compute the total once with `let Total = toscalar(...)` instead of joining an aggregate subquery onto every row. **Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and allows at most 3 joins per query. Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. --- From de3dfa189ba9d6a66b6067f012fecb9b64882aa3 Mon Sep 17 00:00:00 2001 From: Roland Krummenacher Date: Thu, 6 Aug 2026 10:15:07 +0200 Subject: [PATCH 5/5] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20ARG?= =?UTF-8?q?=20join-limit=20precision,=20has=20semantics,=20missing=20blank?= =?UTF-8?q?=20line?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- src/queries/finops-hub-database-guide.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index eead9eccf..cc97f9eaf 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -108,7 +108,7 @@ Apply these rules to every query you write or modify. They mirror the project co #### String matching - Never wrap a column in `tolower()`/`toupper()` to compare it. KQL's string *matching* operators are case-insensitive by default (`=~`, `has`, `contains`, `startswith`, `in~`); case-sensitive matching is what you opt into via the `_cs` variants and the equality operators `==`/`in`. Write `Col =~ 'value'`, not `tolower(Col) == 'value'`. -- Prefer `has` when the needle is a whole term or a separator-bounded phrase (`ResourceId has '/microsoft.capacity/reservationorders/'`) — it uses the term index. Use `contains` only when the needle can be fused inside a larger token (`ConsumedUnit contains 'MB'` also matches `Mbps`). +- Prefer `has` when the needle is a whole term or a separator-bounded phrase (`ResourceId has '/microsoft.capacity/reservationorders/'`) — it uses the term index and matches an exact substring whose edges fall on term boundaries (so separators inside the needle must match the data exactly). Use `contains` only when the needle can be fused inside a larger token (`ConsumedUnit contains 'MB'` also matches `Mbps`). - Collapse operator chains: `Col in~ ('a', 'b')` instead of repeated `=~` with `or`; `has_any`/`has_all` instead of `has` chains. #### Joins and lookups @@ -119,7 +119,8 @@ Apply these rules to every query you write or modify. They mirror the project co - For exclusions ("rows with no match"), use `join kind=leftanti` — not `kind=leftouter` + `where isempty(...)`, which inflates counts when the right side has duplicate keys. - For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. - For grand totals and percent-of-total, compute the total once with `let Total = toscalar(...)` instead of joining an aggregate subquery onto every row. -**Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and allows at most 3 joins per query. Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. + +**Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and documents a limit of 3 `join`/`union` operations combined per query (enforcement has been observed to be lax, but don't design queries that rely on more). Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. ---