diff --git a/src/queries/finops-hub-database-guide.md b/src/queries/finops-hub-database-guide.md index 1ea575922..cc97f9eaf 100644 --- a/src/queries/finops-hub-database-guide.md +++ b/src/queries/finops-hub-database-guide.md @@ -14,6 +14,7 @@ This document provides a comprehensive overview of how to query and analyze data - [Prerequisites](#prerequisites) - [Overview](#overview) - [Query best practices](#query-best-practices) + - [KQL language rules](#kql-language-rules) - [Key enrichment columns](#key-enrichment-columns) - [Example queries](#example-queries) - [Example query: Cost by billing profile, invoice section, team, product, application](#example-query-cost-by-billing-profile-invoice-section-team-product-application) @@ -100,6 +101,27 @@ The FinOps hubs database is designed to support advanced cost and usage analytic - **Leverage Enrichment Columns:** Columns prefixed with `x_` provide additional context and enrichment for FinOps analysis. +### KQL language rules + +Apply these rules to every query you write or modify. They mirror the project coding guidelines and come from real correctness bugs fixed across the toolkit ([#2189](https://github.com/microsoft/finops-toolkit/pull/2189), [#2220](https://github.com/microsoft/finops-toolkit/pull/2220), [#2225](https://github.com/microsoft/finops-toolkit/pull/2225)). + +#### String matching + +- Never wrap a column in `tolower()`/`toupper()` to compare it. KQL's string *matching* operators are case-insensitive by default (`=~`, `has`, `contains`, `startswith`, `in~`); case-sensitive matching is what you opt into via the `_cs` variants and the equality operators `==`/`in`. Write `Col =~ 'value'`, not `tolower(Col) == 'value'`. +- Prefer `has` when the needle is a whole term or a separator-bounded phrase (`ResourceId has '/microsoft.capacity/reservationorders/'`) — it uses the term index and matches an exact substring whose edges fall on term boundaries (so separators inside the needle must match the data exactly). Use `contains` only when the needle can be fused inside a larger token (`ConsumedUnit contains 'MB'` also matches `Mbps`). +- Collapse operator chains: `Col in~ ('a', 'b')` instead of repeated `=~` with `or`; `has_any`/`has_all` instead of `has` chains. + +#### Joins and lookups + +- Never write a bare `| join` — always state `kind=` explicitly. The default flavor is `innerunique`, which deduplicates the *left* side on the join key and silently drops rows. +- To enrich cost rows from a small reference table, use `lookup kind=leftouter` instead of `join kind=leftouter`. It broadcasts the small side and emits no duplicated key columns. +- Neither `join` nor `lookup` deduplicates the right side — a reference table with more than one row per key multiplies your fact rows. Guarantee one row per key with `summarize take_any(Col1), take_any(Col2) by Key` (`distinct` over multiple columns does *not* guarantee this). +- For exclusions ("rows with no match"), use `join kind=leftanti` — not `kind=leftouter` + `where isempty(...)`, which inflates counts when the right side has duplicate keys. +- For two-period comparisons, `join kind=fullouter` is correct, but coalesce the key columns afterwards (`| extend Key = coalesce(Key, Key1) | project-away Key1`) or right-only rows render with empty keys. +- For grand totals and percent-of-total, compute the total once with `let Total = toscalar(...)` instead of joining an aggregate subquery onto every row. + +**Azure Resource Graph is different.** If you are writing ARG queries (resource inventory via `az graph query` — not the hub database), the bare-join `innerunique` trap is the same, but ARG supports *no* `lookup` and no semi/anti join flavors, and documents a limit of 3 `join`/`union` operations combined per query (enforcement has been observed to be lax, but don't design queries that rely on more). Exclusions in ARG must use the `join kind=leftouter` + `where isempty(...)` emulation with a key-unique right side. + --- ## Key Enrichment Columns @@ -694,6 +716,7 @@ The following table lists the columns produced in the `All available columns` qu | 2025-05-16 | 1.0 | FinOps Toolkit Team | Initial documentation | | 2025-05-16 | 1.1 | FinOps Toolkit Team | Expanded schema, glossary, references | | 2026-05-28 | 1.2 | Sprint 3000 UAT | Live-Hub schema audit: `Costs()`, `Prices()`, `Recommendations()` numeric columns retyped from `decimal` to `real` to match deployed Hub schema (cause of SEM0019 errors). `Recommendations()` table expanded from 12 to 20 columns to add the 8 columns present in the live schema. `x_RecommendationDate` documented as commonly-null in live Hubs (root cause of T-3000.13). | +| 2026-08-04 | 1.3 | FinOps Toolkit Team | Added KQL language rules (case-insensitive operators, explicit join kinds, `lookup` for dimension enrichment) distilled from the project coding guidelines so query-writing agents load them alongside the schema. | --- diff --git a/src/templates/claude-plugin/agents/ftk-database-query.md b/src/templates/claude-plugin/agents/ftk-database-query.md index 42e20002b..d20bdca89 100644 --- a/src/templates/claude-plugin/agents/ftk-database-query.md +++ b/src/templates/claude-plugin/agents/ftk-database-query.md @@ -129,12 +129,13 @@ The plugin provides an `azure-mcp-server` with the Kusto namespace for executing 1. **Check the query catalog first**: Before writing custom KQL, check if `skills/finops-toolkit/references/queries/catalog/` has a query that matches the user's scenario. 2. **Start with costs-enriched-base**: For custom analysis not covered by the catalog, begin with `costs-enriched-base.kql` as your foundation. -3. **Use precise column names**: Reference exact field names from the schema. Columns prefixed with `x_` are toolkit enrichments. -4. **Filter early**: Always scope queries to relevant time periods using `ChargePeriodStart` before aggregation. -5. **Prefer EffectiveCost**: Use `EffectiveCost` (after discounts) as the default cost metric unless the user specifically asks for `BilledCost` (billed), `ContractedCost` (negotiated), or `ListCost` (retail). -6. **Handle tags carefully**: Tags is a dynamic column. Extract values with `tostring(Tags['key-name'])`. -7. **Format results**: Present query output in markdown tables with clear column headers. Include the source query and any parameter values used. -8. **Explain the query**: When constructing KQL, explain what data you're accessing, which table function, and why. +3. **Follow the KQL language rules**: Apply the "KQL language rules" section of `skills/finops-toolkit/references/queries/finops-hub-database-guide.md` to every query you write. In particular: never write a bare `| join` (the `innerunique` default silently drops rows — always state `kind=`), prefer `lookup` for enriching from small reference tables, and never use `tolower()` in comparisons (KQL string operators are case-insensitive — use `=~`, `has`, `in~`). +4. **Use precise column names**: Reference exact field names from the schema. Columns prefixed with `x_` are toolkit enrichments. +5. **Filter early**: Always scope queries to relevant time periods using `ChargePeriodStart` before aggregation. +6. **Prefer EffectiveCost**: Use `EffectiveCost` (after discounts) as the default cost metric unless the user specifically asks for `BilledCost` (billed), `ContractedCost` (negotiated), or `ListCost` (retail). +7. **Handle tags carefully**: Tags is a dynamic column. Extract values with `tostring(Tags['key-name'])`. +8. **Format results**: Present query output in markdown tables with clear column headers. Include the source query and any parameter values used. +9. **Explain the query**: When constructing KQL, explain what data you're accessing, which table function, and why. ## FinOps Domain Context