Improve extract and publish log readability: tier grouping, timing, and per-resource retry reporting

[Copilot]

Extract and publish output was a flat, hard-to-scan stream. Extract silently dropped APIs that had no spec or operations. Neither command reported elapsed time. Retry warnings showed raw millisecond delays with no resource or attempt number, and they appeared between PUT lines.

Publish output (before → after)

PUT namedvalue/src-nv-plain
Server error 503, retrying after 1175.1730686888397ms
PUT tag/tag1
--- Summary ---
2 creates/updates, 0 patches, 0 deletes, 0 skipped

── Tier 1: Independent resources (2) ──
PUT namedvalue/src-nv-plain (2 retries)
PUT tag/tag1
Tier 1 completed in 3.2s

--- Summary ---
2 creates/updates, 0 patches, 0 deletes, 0 skipped
2 retries across 1 resource
Completed in 12.3s

Extract (src/cli/extract-command.ts)

• The Extracted N <Type>(s) / Failed … lines are grouped under Tier N: <label> headings.
• Every API is listed. APIs without a spec, operations or revisions show as definition only.
• The total line includes duration: Total: 96 resources extracted, 0 errors in 12.3s. The time covers extraction plus moving the files into the output directory.
• JSON summary.elapsedMs added.

Publish (src/services/publish-service.ts, src/cli/publish-command.ts)

• Each tier gets a header and a Tier N completed in … line; delete tiers get the same, labelled Delete tier N.
  • The header count is the number of descriptors scheduled in the tier, after skipping auto-generated named values and children already published by their parent API or product.
  • It can differ from the number of PUT lines, because publishing an API or product also emits lines for its children.
• PublishResult gains optional elapsedMs, totalRetries and retriedResources; PublishActionResult gains retries. The existing function body moved into runPublishPipeline so these fields are filled on every return path.
• Text summary adds the retry line (only when retries occurred) and Completed in ….
• JSON summary adds totalRetries, retriedResources and elapsedMs; each action adds retries.

Retries (src/clients/apim-client.ts)

• New message format: Server error 503 on PUT namedValues/nv-1 (attempt 1/4), retrying in 1.2s. The same format applies to 429s, network failures, and delete conflicts.
• describeRequestTarget(url) shortens ARM URLs to the path below the APIM service and drops query strings, so storage signatures never appear in logs.
• New src/lib/retry-tracker.ts counts retries per resource using AsyncLocalStorage, which keeps counts correct while resources publish in parallel.
  • Publish wraps each resource in trackRetries().
  • The client calls recordRetry() on every retry.
  • Inside a tracked scope the retry message is logged at debug level; outside one it stays a warning.

Shared helpers

• src/lib/format-duration.ts: formats times as 1.2s or 2m 5.0s.
• TIER_LABELS in src/lib/dependency-graph.ts, used by both extract and publish.

Review notes

• Behaviour change: during publish, the individual retry messages only appear at --log-level debug. Extract and other callers still show them as warnings.
• JSON mode: publish already wrote PUT lines to stdout in --format json mode. The tier headers follow the same path; that existing problem is not fixed here.
• Per the repo's code-review requirement, this PR still needs the CodeReviewer step from .squad/ceremonies.md.

• Fixes Improve Extract and Publish log readability #162

[Copilot]

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

JSON output is corrupted by progress text, retry attribution can target the wrong resource, and some APIs remain omitted.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Suppress tier headers in JSON output · New
• List APIs with failed sub-resource extraction · New
• Avoid retrying or delaying after the final 429 attempt · New
• Attribute retries to the correct resource actions · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Retry attribution and terminal 429 counting are inaccurate, and tier output further corrupts JSON-mode stdout.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Suppress tier headers in JSON output
• Attribute retries to the correct resource actions
• Avoid retrying or delaying after the final 429 attempt
• List APIs with failed sub-resource extraction

[Copilot]

Copilot review overview

🔵 Needs a closer look

Retry totals are inflated for exhausted 429 responses and misattributed when parent publishers emit multiple resource actions.

Review effort: Balanced
Findings: 3

Open (3)

• Attribute retries to the correct resource actions
• Avoid retrying or delaying after the final 429 attempt
• List APIs with failed sub-resource extraction

Resolved since last review (1)

• Suppress tier headers in JSON output

[Copilot]

Copilot review overview

🔵 Needs a closer look

A terminal 429 response is incorrectly counted and delayed as an additional retry.

Review effort: Balanced
Findings: 3

Open (3)

• Attribute retries to the correct resource actions
• Avoid retrying or delaying after the final 429 attempt
• List APIs with failed sub-resource extraction

[Copilot]

Copilot review overview

🔵 Needs a closer look

Environment-mapped child retries are attributed to the wrong publish action because canonical and deployed descriptor keys differ.

Review effort: Balanced
Findings: 1

Open (1)

• List APIs with failed sub-resource extraction

Resolved since last review (2)

• Attribute retries to the correct resource actions
• Avoid retrying or delaying after the final 429 attempt

Previously missed (1)

In code that hasn't changed since last review

Align retry tracking identities for mapped API and product resources

src/​services/​publish-service.ts:1137

When environment mapping changes resource names, this lookup compares different identities: ApimClient records retries under the deployed descriptor (for example dev-api1/get-items), while publisher results retain the canonical descriptor (api1/get-items; see api-publisher.ts:788-797 and resource-publisher.ts:524-536). The child key therefore never matches, and remainder assigns all retries to the root action, making per-action retry suffixes and retriedResources incorrect for mapped API/product children. Preserve or translate the canonical descriptor key at the request-tracking boundary so both sides use the same identity.

[Copilot]

Copilot review overview

🟡 Changes recommended

Environment mapping breaks child retry attribution, and two added tests contain correctness issues.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Initialize policies in the ApiExtractionResult fixture · New

Resolved since last review (1)

• List APIs with failed sub-resource extraction

Previously missed (2)

In code that hasn't changed since last review

Match retry lookups against environment-mapped descriptors

src/​services/​publish-service.ts:1137

With environment mapping enabled, the client records retries under the deployed descriptor key (for example ApiOperation:dev-api/get), while related action results retain the canonical descriptor (ApiOperation:api/get). This lookup therefore misses child results and assigns their retries to the root action via the remainder fallback, making the per-action suffix and retriedResources incorrect for mapped APIs/products. Match against the env-mapped form of each result descriptor (while preserving canonical output), or carry the canonical key into the retry scope.

Assert stdout calls before restoring the mock

tests/​unit/​services/​publish-service.test.ts:151

The assertion runs after mockRestore(), which resets the spy's mock state and clears its call history, so this test can pass even if runPublish wrote text to stdout. Assert before restoring the spy (inside the try) so the regression test observes the actual calls.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Workspace APIs remain omitted, failed resources can be counted as extracted, and env mapping breaks child retry attribution.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Initialize policies in the ApiExtractionResult fixture

Previously missed (2)

In code that hasn't changed since last review

Workspace APIs are omitted from the every-API listing

src/​cli/​extract-command.ts:267

The new “every API” listing still excludes all workspace-scoped APIs. Those APIs are extracted separately in workspace-extractor.ts:193-219; they are not added to result.typeResults or result.apiResults, and WorkspaceExtractionResult only exposes aggregate counts. Include workspace API summaries in the extraction result and merge them into this detail section so the PR's stated behavior applies to every extracted API.

Failed extraction entries are counted as successful

src/​cli/​extract-command.ts:289

These lengths include entries whose extraction status is error; both extractApiRevisions and extractResourceType retain failed entries in their result arrays. A failed operation/revision is therefore reported as successfully present (for example, 1 ops). Count only entries with status === 'success', as the JSON summary and total extraction count already do.