adk-telemetry: attribute Connect + FlightCheck by connector (ADO 7943641) [main-ca] by amilandi · Pull Request #345 · microsoft/Employee-Self-Service-Agent-Developer-Kit

amilandi · 2026-09-25T16:37:10Z

Summary

main-ca companion to #344. Cherry-picks the same connector-attribution change onto main-ca so CA-mode kits carry the new connector dimension too and Aria dashboards can split adoption / reliability by Workday vs ServiceNow across BOTH the DA and CA forks.

Closes ADO 7943641 on the CA branch.

Cherry-pick delta

Straight cherry-pick of 85a05c46 from #344 with one 3-line merge resolution:

src/skills/connect/step1.md §1.3 Workday branch: main-ca replaced the body with a redirect to the setup orchestrator; adk-telemetry: attribute Connect + FlightCheck by connector (ADO 7943641) #344 added the emit_capability.py connect --connector workday line at the top of the (DA) branch. Resolution: prepend the emit line to main-ca's orchestrator-redirect body so the Workday attribution still fires before setup runs.

Everything else applied cleanly.

Taxonomy / wiring / schema

Identical to #344 — see that PR for the full write-up:

connector ∈ { workday, servicenow, legacy, unknown, "" }
--connector flag on emit_capability.py, derived on-emit for FlightCheck from scope + category
Schema bumps: ADK 1.4.0, FlightCheck 1.2
Docs: telemetry_queries.kql §9 (connector-split queries)

Tests

223 passed (tests/test_adk_telemetry.py + tests/flightcheck/test_telemetry.py). One fewer than #344 (224) because main-ca predates one unrelated test that's on main; not connector-related.

Merge ordering

Independent of #344 — both target the same ADO item but different branches. Whichever merges first, the other stays green.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864

…641) Adds a bounded `connector` dimension to every Connect capability and FlightCheck event so PMs can report Workday vs ServiceNow adoption and reliability separately instead of a single generic `connect` wedge. Taxonomy: workday | servicenow | legacy | unknown | ` (empty = not connector-scoped). CLI: `python scripts/emit_capability.py connect --connector <workday|servicenow>`. FlightCheck derives it on-emit from scope (run) and category prefix (check); `full` scope + cross-cutting categories emit empty and per-check attribution provides the split. Schema bumps: adk_telemetry 1.3.0 -> 1.4.0, flightcheck 1.1 -> 1.2. Docs: adds telemetry_queries.kql section 9 with connector-split adoption / reliability queries. No dashboard work in this PR (per PM ask; can wire tiles once the dim lands in Kusto). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864

The shim now always passes `connector=<value>` alongside `block=True` when it invokes `emit_capability_use` (ADO 7943641). The pre-existing test's monkey-patched lambda signature `(capability, block)` was missing the `connector` param, so the kwarg raised TypeError inside the shim's blanket `except Exception` -> the emit was silently swallowed and `emitted` stayed empty. Update the lambda to accept `connector` and assert the empty-connector round-trip (`('setup', '', True)`). Add a companion test that exercises the flag-forwarding case (`--worker connect --connector workday`) -> `('connect', 'workday', True)` to lock down the parent-shim -> subprocess argv attribution round-trip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864

…tion-ca

nkemms · 2026-09-25T17:45:38Z

Local re-review found no new main-CA-specific issues. The Workday route correctly emits connect --connector workday after selection and then dispatches to the setup skill without changing the existing orchestration.

PR 345 inherits the shared telemetry and KQL findings already documented on PR 344: #344 (comment). All 227 targeted tests passed locally; the PR 337 conflict appears to be merge-order coordination.

1. CLI runtime forwards connector to ADK emitters Both --scope and --checkpoint code paths in flightcheck/cli.py now derive the connector (from scope / from first result category) and pass it to emit_flightcheck_run/emit_flightcheck_result. Previously the ADK adk.flightcheck.* event family emitted empty connector for real runs even though standalone helper tests exercised the kwarg. 2. _WORKDAY_SCOPES covers workdayda and topics SCOPE_MAP defines both as Workday-only (workdayda -> Workday DA, topics -> Workday Topics). Left out of the frozenset earlier, so runs on those scopes emitted connector=''. 3. KQL 9a uses tenant_id (not tenantId) for adk_capability_use ADK snake_case dim; tenantId belongs to the legacy ESSMakerKit event shape and would silently return 0 tenants for ADK cubes. 4. Rollout query 9d is null-safe and separates 'unknown' Handles pre-1.4.0 events where the column is absent (isnull) plus 1.4.0+ events with connector='' (isempty). 'unknown' now reports separately instead of folding into 'attributed'. 5. Parent-path connector forwarding is directly asserted Two new tests in tests/scripts/test_emit_capability.py assert the Popen argv includes '--connector <value>' for both '--connector v' and '--connector=v' input forms, and omits it entirely when no flag is supplied (so worker parsing sees the correct default). New tests: * test_capability_emit_parent_forwards_connector_argv * test_capability_emit_parent_omits_connector_when_not_supplied * TestCheckpointAdkConnector (3 tests: workday / servicenow / cross-cutting) * test_derive_connector_from_scope_known: added workdayda + topics + case/whitespace variants Full run: 1408 passed (test_adk_telemetry + flightcheck/). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864

nkemms · 2026-09-25T23:52:24Z

Re-reviewed the latest main-CA update. The inherited PR 344 findings are addressed, and no new main-CA-specific issue was found.

The branch currently conflicts with updated main-ca in the shared telemetry files. When resolving, preserve connector attribution alongside the existing agent_type, toolkit_git_sha, and toolkit_git_branch dimensions.

The same single-checkpoint prerequisite-failure attribution edge remains as follow-up work and does not need to hold this PR.

249 targeted tests passed locally.

nkemms

Reviewed the latest main-CA update; inherited findings are addressed and no new port-specific issue was found. Remaining target-branch reconciliation and attribution edge are documented as follow-up.

…metry-connector-attribution-ca-tmp # Conflicts: # solutions/ess-maker-skills/scripts/adk_telemetry.py # solutions/ess-maker-skills/scripts/flightcheck/telemetry.py

amilandi · 2026-09-25T23:59:45Z

Thanks — merge conflicts against main-ca resolved in 864f5de (pushed). The merge preserves all three feature families in the shared telemetry files:

this PR's connector attribution (derived connector on adk.flightcheck.run / .result / .capability.use + on the legacy ESSMakerKit.FlightCheck.Run / .Check events),
the agent_type common dimension from main-ca, and
the toolkit_git_sha / toolkit_git_branch common dimensions from main-ca.

Schema versions bumped:

adk_telemetry.SCHEMA_VERSION → 1.6.0 (kept 1.4.0 / 1.5.0 notes for toolkit_git_* and agent_type; added 1.6.0 note for connector).
flightcheck.telemetry.TELEMETRY_SCHEMA_VERSION → 1.4 (kept 1.2 / 1.3 notes for toolkitGitSha / toolkitGitBranch and agentType; added 1.4 note for connector).

Version-gate tests updated. 250 targeted tests pass locally (tests/test_adk_telemetry.py + tests/flightcheck/test_telemetry.py).

Single-checkpoint prerequisite-failure attribution edge acknowledged as follow-up, kept out of this PR per your note.

nkemms

Re-approved after the latest main-ca merge. The conflict resolution preserves connector attribution together with agent type and toolkit Git dimensions; focused validation passed.

Avery Milandin and others added 3 commits September 25, 2026 09:36

Merge branch 'main-ca' into amilandin/adk-telemetry-connector-attribu…

8ffea81

…tion-ca

nkemms previously approved these changes Sep 25, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main-ca' into amilandin/adk-tele…

864f5de

…metry-connector-attribution-ca-tmp # Conflicts: # solutions/ess-maker-skills/scripts/adk_telemetry.py # solutions/ess-maker-skills/scripts/flightcheck/telemetry.py

amilandi dismissed nkemms’s stale review via 864f5de September 25, 2026 23:58

nkemms approved these changes Sep 26, 2026

View reviewed changes

amilandi merged commit 867d1b4 into main-ca Sep 26, 2026
7 checks passed

Navigation Menu

adk-telemetry: attribute Connect + FlightCheck by connector (ADO 7943641) [main-ca] - #345

adk-telemetry: attribute Connect + FlightCheck by connector (ADO 7943641) [main-ca]#345
amilandi merged 5 commits into
main-cafrom
amilandin/adk-telemetry-connector-attribution-ca

amilandi commented Sep 25, 2026

Uh oh!

nkemms commented Sep 25, 2026

Uh oh!

nkemms commented Sep 25, 2026

Uh oh!

nkemms left a comment

Uh oh!

amilandi commented Sep 25, 2026

Uh oh!

nkemms left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

amilandi commented Sep 25, 2026

Summary

Cherry-pick delta

Taxonomy / wiring / schema

Tests

Merge ordering

Uh oh!

nkemms commented Sep 25, 2026

Uh oh!

nkemms commented Sep 25, 2026

Uh oh!

nkemms left a comment

Choose a reason for hiding this comment

Uh oh!

amilandi commented Sep 25, 2026

Uh oh!

nkemms left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants