diff --git a/README.md b/README.md index 2a10bd9..02500dc 100644 --- a/README.md +++ b/README.md @@ -88,8 +88,74 @@ Rule: More context: - [release_bundle/2026-04_newsletter_launch/START_HERE.md](release_bundle/2026-04_newsletter_launch/START_HERE.md) +- [release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md](release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md) +- [release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md](release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md) - [release_bundle/2026-02_newsletter_launch/public/START_HERE.md](release_bundle/2026-02_newsletter_launch/public/START_HERE.md) +## Current Release Bundle: May 2026 + +The May 2026 bundle is the current public companion for the cost-aware newsletter +generation work. It is meant to help a developer answer three practical +questions: + +1. How do I generate a newsletter with fewer wasted tokens? +2. How do I optimize a similar agentic workflow without lowering quality? +3. What changed across the newsletter generation system beyond the May issue + itself? + +Start here: + +- [Start here](release_bundle/2026-05_newsletter_cost_optimization/START_HERE.md) -- thin pointer to the customer companion. +- [Customer companion](release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md) -- canonical landing page with developer and admin guidance for cost-aware Copilot usage. +- [System release notes](release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md) -- technical deep dive with exact before/after token counts, code links, and an illustrative cost translation. +- [Admin and FinOps guide](release_bundle/2026-05_newsletter_cost_optimization/ADMIN_FINOPS_GUIDE.md) -- budgets, reporting, governance, attribution, baseline, and showback guidance. +- [Developer guide](release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_GUIDE.md) -- cost-aware agentic workflows with public-safe worked examples. +- [Product feature quick hits](release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md) -- first-party source inventory for billing, budgets, observability, routing, and token-mechanics references. + +Public wording intentionally uses rounded aggregate/proxy metrics and caveats. +Exact retained-run logs, private paths, raw token tables, and internal source +notes stay in the private source repository. + +## Lower-Token Newsletter Workflow + +The most reliable way to lower token pressure is to reduce repeated work while +keeping validation in the route. Use the admitted prompt-rendered production path +as the oracle for its pinned range, then use the diagnostic harness only when you +need phase-level repair. + +```bash +# 1. Prepare the cycle and clear stale intermediates. +bash tools/prepare_newsletter_cycle.sh 2026-02-14 2026-04-16 --no-reuse + +# 2. Run the prompt-rendered production oracle for the admitted April range. +make newsletter-gen START=2026-02-14 END=2026-04-16 MODE=production + +# 3. Validate the generated newsletter. +make validate-newsletter FILE=output/2026-04_april_newsletter.md +bash tools/validate_pipeline_strict.sh 2026-02-14 2026-04-16 --require-fresh --production-artifacts +``` + +`MODE=production` is intentionally pinned in +[render_product_run_prompt.sh](tools/render_product_run_prompt.sh). For other +date ranges, use the same workflow pattern, but do not swap arbitrary dates into +`MODE=production` unless the helper has been extended and validated for that +range. + +Optimization order: + +1. Bind the date range, source set, and acceptance criteria before generation. +2. Reuse accepted artifacts only when identity, freshness, and scope are clear. +3. Compact expensive curation inputs only after preserving required source + classes and fallback. +4. Suppress broad search or tools only when the relevant files and artifacts are + already known. +5. Promote model, reasoning, or output-shape changes only after route-level + validation passes. + +The May bundle explains the measured workflow signal and the claim boundaries in +more detail. It does not claim Copilot billing savings, durable savings, model +superiority, or universal percentages. + ## System Overview | Component | Count | Key Files | @@ -150,8 +216,17 @@ make newsletter START= END= # Full pipeline orchestration make help # Show all 62 targets ``` +Newsletter-specific validation can also be run directly: + +```bash +bash .github/skills/newsletter-validation/scripts/validate_newsletter.sh output/YYYY-MM_month_newsletter.md +bash tools/validate_pipeline_strict.sh START_DATE END_DATE --require-fresh --production-artifacts +bash tools/score-v2-rubric.sh output/YYYY-MM_month_newsletter.md +``` + ## Documentation - [Public repo guide](reference/public_repo_guide.md) -- publication boundary and review checklist +- [May cost optimization bundle](release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md) -- canonical shipped-newsletter companion, persona paths, playbooks, examples, and system release notes - [April launch bundle](release_bundle/2026-04_newsletter_launch/START_HERE.md) -- production command and validation gates - [February public launch bundle](release_bundle/2026-02_newsletter_launch/public/START_HERE.md) -- public case study and runnable example diff --git a/docs/architecture.md b/docs/architecture.md index 20fd2f9..df1a252 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -33,7 +33,30 @@ flowchart LR - Phase prompts: `.github/prompts/*.prompt.md` - Fresh-cycle prep: `tools/prepare_newsletter_cycle.sh` - Strict validation: `tools/validate_pipeline_strict.sh` +- Production prompt rendering: `tools/render_product_run_prompt.sh` +- Newsletter validation: `.github/skills/newsletter-validation/scripts/validate_newsletter.sh` +- Public snapshot boundary: `tools/public_snapshot_allowlist.txt` and `tools/public_snapshot_prune.txt` - Deterministic event sources: - `kb/EVENT_SOURCES.yaml` - `tools/extract_event_sources.py` - Output sample: `output/2026-02_february_newsletter.md` + +## Cost-Aware Architecture + +The May 2026 update makes cost behavior part of the workflow architecture: + +```mermaid +flowchart LR + A[Accepted source sets] --> B[Artifact reuse] + B --> C[Compact Phase 3 working set] + C --> D[Readiness checks] + D --> E[Tool-bounded synthesis] + D --> F[Fallback to fuller context] + E --> G[Newsletter validation] + F --> G + G --> H[Claim boundaries] +``` + +The important architecture rule is that optimization and quality are coupled. A route is not accepted because it uses fewer tokens; it is accepted only when the final newsletter passes validation and the evidence is labeled correctly. + +For the concrete file map, see the [May system release notes](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md). diff --git a/docs/how-it-works.md b/docs/how-it-works.md index bb44469..956a95f 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -6,6 +6,8 @@ way. If you want to run it, start with: - [Start here (Feb 2026)](launch/2026-02/start-here.md) +- [May 2026 cost optimization companion](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md) +- [May 2026 system release notes](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md) ## Pipeline Phases @@ -31,6 +33,23 @@ Examples: - `workspace/newsletter_phase3_curated_sections_*.md` - `output/YYYY-MM_month_newsletter.md` +## Cost-Aware Run Pattern + +The current production-like route starts with a clean cycle, runs the admitted prompt-rendered production command for its pinned range, and validates the output before any cost or quality claim is made. + +```bash +bash tools/prepare_newsletter_cycle.sh 2026-02-14 2026-04-16 --no-reuse +make newsletter-gen START=2026-02-14 END=2026-04-16 MODE=production +make validate-newsletter FILE=output/2026-04_april_newsletter.md +bash tools/validate_pipeline_strict.sh 2026-02-14 2026-04-16 --require-fresh --production-artifacts +``` + +`MODE=production` is pinned to the April 2026 range by the prompt renderer. For another date range, reuse the workflow pattern and validation gates, but first extend and validate the admitted prompt-rendered mode for that range. + +The token-efficient path is about reducing repeated work, not just shortening prompts. The system now emphasizes accepted source sets, artifact reuse, compact Phase 3 curation inputs, readiness checks, and fallback to fuller context when quality is at risk. + +Use `tools/run_newsletter_orchestrated.sh` for phase-local diagnosis when the prompt-rendered route needs repair. Treat it as diagnostic unless the release notes for a given run say otherwise. + ## Self-Learning Loop Corrections are encoded into skills and references, then validated by scoring and rule @@ -45,3 +64,5 @@ Finding -> Root cause -> Skill/rule update -> Regenerate -> Validate -> Record l ## Trust Disk, Not Self-Reports The system relies on file-based proof and deterministic checks rather than agent claims. + +For cost-related claims, the proof must also say what it does not prove. The May bundle labels workflow movement as aggregate/proxy evidence, not Copilot billing proof, durable savings, model superiority, or fleet readiness. diff --git a/docs/index.md b/docs/index.md index f48bfdb..8a62f5c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,12 +7,33 @@ A public, reusable system for drafting enterprise-focused Copilot newsletters. If you only read one page, start here: - [Start here (Feb 2026)](launch/2026-02/start-here.md) +For the current cost-aware workflow release: +- [May 2026 cost optimization companion](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md) (canonical landing page from the shipped newsletter) +- [May 2026 system release notes](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md) +- [May 2026 product feature quick hits](https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md) + +The companion's `START HERE` section routes to the start-here copy, FinOps playbook, worked examples, admin guide, developer guide, and source references. + Then, if you want the backstory: - [Short case study](launch/2026-02/case-study.md) - [Timeline](launch/2026-02/timeline.md) Want to see a real shipped example? - [Published February issue (Discussion #18)](https://github.com/briancl2/CustomerNewsletter/discussions/18) +- [Published May issue (Discussion #21)](https://github.com/briancl2/CustomerNewsletter/discussions/21) + +## Current Release Focus + +The May 2026 release bundle explains how the newsletter generation system was updated to reduce repeated agent work while keeping validation in the route. It covers: + +- how to generate a newsletter with fewer wasted tokens +- how to apply the same workflow patterns to other agentic systems +- what changed across the source-pruning, artifact-reuse, compact-working-set, validation, and publication surfaces +- which Copilot billing, budget, reporting, model-routing, and provider token-mechanics sources support the guidance +- how FinOps teams can reason about baselines, budget layers, showback, and savings-claim discipline +- worked examples for content generation, docs updates, debugging, budget-block triage, and custom agent workflows + +The bundle uses rounded aggregate/proxy workflow metrics with explicit non-billing caveats. Private run logs, exact token tables, retained evidence paths, and internal source notes are not published. ## Try It @@ -49,3 +70,4 @@ copilot --agent customer_newsletter --model claude-opus-4.7 -i - [How it works](how-it-works.md) - [Architecture](architecture.md) - [Feb 2026 system report](reports/newsletter_system_report_2026-02.md) +- [May 2026 cost optimization bundle](https://github.com/briancl2/CustomerNewsletter/tree/main/release_bundle/2026-05_newsletter_cost_optimization) diff --git a/mkdocs.yml b/mkdocs.yml index 5453b79..896fd47 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -22,6 +22,10 @@ nav: - Start Here: launch/2026-02/start-here.md - Case Study: launch/2026-02/case-study.md - Timeline: launch/2026-02/timeline.md + - May 2026 Cost Bundle: + - Customer Companion: https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md + - System Release Notes: https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md + - Product Quick Hits: https://github.com/briancl2/CustomerNewsletter/blob/main/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md - How It Works: how-it-works.md - Architecture: architecture.md - Reports: diff --git a/release_bundle/2026-05_newsletter_cost_optimization/ADMIN_FINOPS_GUIDE.md b/release_bundle/2026-05_newsletter_cost_optimization/ADMIN_FINOPS_GUIDE.md new file mode 100644 index 0000000..158380a --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/ADMIN_FINOPS_GUIDE.md @@ -0,0 +1,161 @@ +# Admin And FinOps Guide: Budgets, Reporting, And Governance + +> **Bottom line:** Pair usage-based billing controls with reporting, model policy, and developer guidance; budgets limit runaway spend, but efficient work comes from workflow design. **For:** Admin & FinOps owners -- billing managers, platform owners, and FinOps partners. **Read time:** about 8 minutes. + +This guide combines admin readiness and FinOps operating guidance for usage-based billing (UBB) and agentic development at scale. It is customer-safe guidance, not invoice analysis or a savings guarantee. Use the [admin track](#admin-and-platform-track) for budgets, reporting, and governance, and the [FinOps track](#finops-track) for the operating model, evidence typing, and showback. + +## Operating Model + +Cost control works best when finance, platform, and developer enablement move together. Budgets and reports do not replace workflow discipline, but they make it possible to manage it. + +| Layer | Owner | Question | Output | +|---|---|---|---| +| Billing authority | Billing or enterprise owner | Which charges are invoice-grade? | Billing report and budget configuration. | +| Usage visibility | Platform or analytics owner | Who is using which surfaces? | Usage exports, team reports, CLI metrics, code review tracking. | +| Workflow behavior | Engineering lead | Which patterns cause repeated attempts or repairs? | Team guidance, validation gates, scoped prompts, reusable artifacts. | +| Policy | Platform and governance owners | Which controls should apply by default? | User-level budgets, cost-center limits, model policy, tool boundaries. | +| Support path | Admin and support owners | What happens when a user is blocked? | Escalation path and override process. | + +## Admin And Platform Track + +### Review First + +1. **UBB readiness:** AI Credits, included usage, model-specific pricing, budgets, and escalation paths. +2. **Reporting:** April reports, team-level metrics, CLI activity, plan mode, report download URLs, and audit APIs. +3. **Model policy:** Auto model selection, model-cost documentation, and model rules. +4. **Observability:** Copilot SDK OpenTelemetry, BYOK surface-specific visibility, and terminal output compression. +5. **Governance:** terminal sandboxing, local MCP server sandboxing, network-domain policy, command risk assessment, cloud-agent runner controls, and firewall settings. + +### Before UBB Go-Live + +- Confirm budget owners and escalation paths. +- Pull April reports as a baseline. +- Set a universal user-level budget and define power-user overrides. +- Decide how cost-center and enterprise spending limits behave when included pooled credits are exhausted. +- Assign alert recipients and support-path owners for blocked users. +- Decide where model rules are required. +- Explain model-specific AI Credit pricing with GitHub billing docs, not provider API price sheets or hearsay. +- Give teams developer guidance before enforcing blunt usage limits. + +### Weekly Cost Visibility Review + +- Review usage by team and surface. +- Separate plan mode, CLI activity, code review, and cloud-agent activity where possible. +- Ask whether spikes came from valuable work, repeated failed work, or missing workflow boundaries. +- Feed repeated cost patterns back into developer guidance. + +### Billing Caveats To Communicate + +- Code completions and Next Edit Suggestions are not billed in AI Credits for paid Copilot plans. +- Copilot code review should be tracked separately because it can consume AI Credits and GitHub Actions minutes. +- User-facing budget block messages may not identify which budget layer caused the block; support processes should be ready to diagnose user-level, cost-center, and enterprise limits. +- Telemetry is engineering evidence, not billing proof. + +### Product Surfaces + +This is the admin-facing subset. For the full feature inventory with status labels, use [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md). + +| Surface | Admin question | Source | +|---|---|---| +| UBB and AI Credits | Who owns budget thresholds and escalation paths? | https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/ | +| Model pricing | How should teams understand model-cost differences? | https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing | +| Auto model selection | Where can task-aware routing replace manual model choice? | https://github.blog/changelog/2026-05-20-auto-model-selection-now-routes-based-on-your-task-in-vs-code | +| Model rules | Which models are available to which organizations? | https://github.blog/changelog/2026-05-26-target-copilot-models-to-organizations-with-model-rules | +| Plan mode metrics | How much work is planning rather than editing? | https://github.blog/changelog/2026-03-02-copilot-metrics-now-includes-plan-mode | +| CLI activity metrics | Are terminal-native agent workflows visible? | https://github.blog/changelog/2026-04-10-copilot-cli-activity-now-included-in-usage-metrics-totals-and-feature-breakdowns | +| Team-level metrics API | Can teams self-serve usage reporting? | https://github.blog/changelog/2026-05-14-team-level-copilot-usage-metrics-now-available-via-api | +| Report download URLs | Can finance integrations retrieve stable reports? | https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls | +| Cloud-agent audit API | Which repos have cloud-agent settings enabled? | https://github.blog/changelog/2026-05-18-audit-repository-copilot-cloud-agent-configuration-via-the-rest-api | +| Copilot SDK OpenTelemetry | Can custom agents be traced as engineering signal, not billing proof? | https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry | +| Cloud-agent runner controls | Where can agents run? | https://github.blog/changelog/2026-04-03-organization-runner-controls-for-copilot-cloud-agent | +| Cloud-agent firewall settings | What network access do agents have? | https://github.blog/changelog/2026-04-03-organization-firewall-settings-for-copilot-cloud-agent | +| MCP secret scanning | Are MCP-connected tools covered by secret scanning? | https://github.blog/changelog/2026-05-05-secret-scanning-with-github-mcp-server-is-now-generally-available | + +## FinOps Track + +### Evidence Types + +Use the right evidence for the right decision. + +| Evidence type | Good for | Not good for | +|---|---|---| +| Invoice or billing report | Finance-grade spend decisions. | Explaining every workflow cause. | +| GitHub usage export or report | Trends, concentration, rollout monitoring. | Proving exact avoided spend. | +| Product telemetry or traces | Engineering diagnosis and workflow tuning. | Invoice-grade claims by itself. | +| Workflow experiment | Finding repeated work and quality tradeoffs. | Universal model or savings claims. | +| Estimate | Planning and prioritization. | Customer-facing proof without caveats. | + +### Baseline Process + +1. Confirm budget owners, cost-center mapping, and enterprise spending-limit behavior. +2. Pull available usage reports before changing policy. +3. Segment usage by team, surface, and known workflow where reporting allows. +4. Separate chat, agents, CLI activity, code review, and custom-agent workflows where possible. +5. Identify concentrated usage and ask whether it came from valuable work, repeated failed work, or missing guidance. +6. Publish developer guidance before tightening controls. +7. Review weekly during rollout, then monthly once usage stabilizes. + +### Showback Starter + +Use a simple showback table before moving toward chargeback. + +| Field | Example | +|---|---| +| Team or cost center | Platform Engineering | +| Budget owner | Engineering director | +| Usage owner | Platform lead | +| Primary surfaces | VS Code chat, Copilot CLI, code review | +| Review cadence | Weekly during rollout | +| Override policy | Power-user override with owner approval | +| Support path | Admin triage, then billing owner escalation | +| Quality guardrail | Tests and validation remain required even when usage is constrained | + +### Budget Block Triage + +When a developer is blocked, diagnose the layer before changing behavior. + +1. Check the user-level budget. +2. Check cost-center and enterprise spending limits. +3. Check whether the user is a known power user or running a temporary rollout task. +4. Check whether the work is repeated because of unclear task boundaries, missing files, or failed validations. +5. Decide whether the fix is an override, a policy adjustment, or workflow guidance. + +### Savings Claim Discipline + +Use precise wording: + +- Say **usage decreased** only when usage reports show it. +- Say **workflow repeated less work** when telemetry or experiments show fewer retries, tool calls, or repairs. +- Say **invoice savings** only when billing reports support it. +- Say **quality-preserving optimization** only when the output passed the agreed gates. + +Avoid turning a workflow experiment into a universal percentage, a model ranking, or a billing promise. + +## Recommended Practices + +- Pair budget controls with developer workflow guidance. +- Use metrics to ask better questions, not to shame usage. +- Track quality and outcome alongside usage. +- Treat model rules as policy, not as model ranking. +- Make observability a prerequisite for custom agent scale. +- Keep security and cost governance together because tool scope affects both. + +## Common Pitfalls + +- High usage needs context before it is classified as waste. +- Telemetry is useful for engineering diagnosis; invoice savings need billing evidence. +- Cheaper models and lower reasoning settings still need quality validation. +- Broad autonomous tool access should follow audit, sandbox, and network-boundary decisions. +- BYOK, OpenTelemetry, and provider pricing guidance should specify whether the question is Copilot billing, Copilot reporting, direct provider usage, or custom workflow instrumentation. + +## First-Party References + +- [Models and pricing for GitHub Copilot](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) +- [Getting started with budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) +- [Optimizing your budget configuration](https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration) +- [Managing AI credits, GitHub Well-Architected](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) +- [Copilot usage metrics reports now use GitHub-owned download URLs](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls) +- [Team-level Copilot usage metrics API](https://github.blog/changelog/2026-05-14-team-level-copilot-usage-metrics-now-available-via-api) +- [Copilot CLI activity in usage metrics](https://github.blog/changelog/2026-04-10-copilot-cli-activity-now-included-in-usage-metrics-totals-and-feature-breakdowns) + +For the complete bundle link list, use [PUBLIC_SOURCES.md](PUBLIC_SOURCES.md). diff --git a/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md b/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md index 22efddc..3eafc0a 100644 --- a/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md +++ b/release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md @@ -1,30 +1,161 @@ # Customer Companion: Cost-Aware Copilot Usage -This is the customer-safe companion to the May 2026 newsletter. It summarizes the practical UBB and cost-aware workflow guidance without exposing experiment logs or stronger claims than the evidence supports. +> **Bottom line:** Re-engineering this newsletter's own AI workflow cut **roughly 27-45% of the direct provider tokens per run (about $6-12 per run in API-equivalent terms)** while quality gates held. The same moves -- route over prompt, compact context, artifact reuse, quality-gated fallback -- paired with usage-based billing controls are how you hold agentic Copilot costs down. + +This is the customer-safe companion to the May 2026 newsletter. It summarizes practical usage-based billing (UBB), Copilot governance, and cost-aware workflow guidance without exposing experiment logs or stronger claims than the evidence supports. + +Use this file when you want the operating guidance. Use [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md) when you want the implementation map of what changed in the newsletter generation system, including exact before/after token counts and an illustrative cost translation. + +## From Brian + +For May's newsletter, I set out to learn how to optimize my generation workflow to understand what works and what doesn't for cost reduction. I asked Copilot to do most of the heavy lifting, setting up experiments, measuring what works and what doesn't, and then baking in the proven learnings into the newsletter generation process. The most humbling finding is that the cheapest or shortest prompt is rarely the cheapest workflow. Most of the savings came from making the system stop repeating work, not from asking for shorter responses. If you are a developer or otherwise responsible for AI costs, that is the lever I would reach for first -- Brian + +## START HERE + +This is the canonical landing page for the bundle. The shipped May newsletter intentionally links to `CUSTOMER_COMPANION.md`. [START_HERE.md](START_HERE.md) is a thin pointer back to this file for readers browsing the release bundle directory. + +| Time | Path | Use when | +|---|---|---| +| 3 minutes | [Executive Summary](#executive-summary) below | You need the core message and the takeaways. | +| 10 minutes | Keep reading this companion | You want the practical UBB, developer workflow, governance, and source-link guidance in one place. | +| Implementation planning | [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md), [ADMIN_FINOPS_GUIDE.md](ADMIN_FINOPS_GUIDE.md) | You are turning the guidance into team operating practices. | +| Technical deep dive | [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md), [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md), [PUBLIC_SOURCES.md](PUBLIC_SOURCES.md) | You want implementation details, exact numbers, product controls, and public sources. | + +| Role | Recommended path | +|---|---| +| Developer or team lead | [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md) -> [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md) | +| Admin & FinOps owner | [ADMIN_FINOPS_GUIDE.md](ADMIN_FINOPS_GUIDE.md) -> [Measurement Checklist](#measurement-checklist) -> [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md) | + +If you just need the core message and the takeaways in three minutes, read the [Executive Summary](#executive-summary) below; if you are presenting the cost story, the [NEWSLETTER_MAY_COST_SECTION.md](NEWSLETTER_MAY_COST_SECTION.md) excerpt is the verbatim newsletter section. + +This bundle provides workflow guidance, public-source references, and public-safe lessons from the newsletter generation system. It is not billing proof, a durable savings claim, a model recommendation, or a universal benchmark. ## Executive Summary +> **Expected savings, concretely.** In my own newsletter runs, the accepted optimized route moved a full current-cycle generation from about 9.0M to about 5.0M direct provider tokens (~45% fewer), and a fixed-corpus run from about 7.8M to about 5.7M (~27% fewer). At a blended illustrative rate of about $3 per million direct provider tokens (a `gpt-5.5`-class list price -- substitute your own from the [OpenAI pricing](https://developers.openai.com/api/docs/pricing) page), that is roughly **$12 per current-cycle run** and **$6 per fixed-corpus run** of API-equivalent token cost removed. These are illustrative engineering estimates on direct provider tokens, **not** a GitHub Copilot AI Credits bill. For the full before/after tables and the cost math, see [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md). + - **UBB starts June 1 for affected customers.** Metered Copilot interactions use GitHub AI Credits where applicable. - **Developer clarification:** code completions and Next Edit Suggestions are not billed in AI Credits for paid Copilot plans. Developers should use GitHub's model and pricing documentation to understand which interactions can consume AI Credits. - **Code review clarification:** Copilot code review should be tracked separately because it can consume AI Credits and GitHub Actions minutes. - **Routing clarification:** Auto model selection can help route work by task and model health, but it should be treated as workflow guidance, not billing proof. - **Budgets and behavior must be paired.** ULBs, cost-center budgets, enterprise limits, alerts, and usage exports work best when developers also get workflow guidance. - **The useful optimization target is finished work.** Measure the route to a correct, reviewed, tested result, not only one prompt or one phase. -- **A real workflow moved materially.** In this newsletter system, retained aggregate/proxy comparisons showed roughly **25-45% less token mass** after workflow changes such as phase-specific routing, compact working sets, artifact reuse, and quality-gated fallback. Treat that as a measured workflow signal, not billing proof. +- **A real workflow moved materially.** When I applied phase-specific routing, compact working sets, artifact reuse, and quality-gated fallback to my own newsletter system, rounded aggregate/proxy comparisons showed roughly **27-45% less token mass** per run. Treat that as a workflow-specific engineering signal, not billing proof or a transferable savings percentage. + +## Mental Model + +Copilot UBB is token-based, but it is not simply each provider's public API bill passed through unchanged. Copilot uses GitHub AI Credits, Copilot-specific model pricing, cached-token categories where applicable, and feature-specific billing behavior. GitHub Copilot billing docs are authoritative for Copilot charges. Provider API docs are useful for understanding token economics, caching, context windows, reasoning controls, and batch-style optimization patterns. + +Useful finished-work cost is closer to this shape: + +```text +finished-work cost = + model and feature pricing + x input + cached input + output token behavior + x attempts and retries + + tool, search, review, runtime, and validation side costs + + repair work when quality fails +``` + +The practical target is not the shortest prompt. It is the route that reaches a correct, reviewed, tested result with the fewest failed attempts, unnecessary tool calls, repeated context assembly, and repairs. ## What Changed In The Workflow -The cost story behind this newsletter is concrete: the generation system was changed to do less repeated AI work while keeping validation in the loop. +The cost story behind this newsletter is concrete: I changed the generation system to do less repeated AI work while keeping validation in the loop. For the exact before/after token counts and the cost translation, see [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md); for the developer patterns behind each change, see [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md). | Change | Customer-safe takeaway | Metric signal | |---|---|---:| -| Phase-specific routing | Different phases deserve different model/reasoning choices; test the route, not the model in isolation. | Integrated comparisons showed roughly 25-45% lower aggregate/proxy token mass. | +| Phase-specific routing | Different phases deserve different model/reasoning choices; test the route, not the model in isolation. | Integrated comparisons showed roughly 27-45% lower aggregate/proxy token mass. | | Compact working set | Smaller context can help when required source classes and fallback are preserved. | One Phase 3 example reduced the curation token load by roughly nine-tenths. | | Artifact reuse / no-refetch | Reuse accepted artifacts instead of re-fetching, but bind identity, freshness, and scope first. | Helped reduce repeated retrieval and synthesis work inside the accepted route. | | stdout/no-tools route | Suppress unnecessary tools only after readiness checks pass; keep fallback available. | Repaired route passed V2 quality and newsletter validation. | | Failed output-shape policy | Shorter output instructions can increase total route cost by causing compensation elsewhere. | Negative result; not promoted. | -The point is not that these exact percentages transfer to another team. The point is that finished-workflow measurement can reveal where agentic work is being repeated, amplified, or repaired. +The point is not that these exact percentages transfer to another team. The point I took away is that finished-workflow measurement can reveal where agentic work is being repeated, amplified, or repaired. + +## Developer Workflow Guide + +### Generate A Newsletter With Fewer Wasted Tokens + +Use the admitted prompt-rendered production route as the source of truth for its pinned range, then use phase-level tools only when diagnosis is needed. + +```bash +# Prepare a clean cycle so old intermediates do not pollute the route. +bash tools/prepare_newsletter_cycle.sh 2026-02-14 2026-04-16 --no-reuse + +# Run the canonical prompt-rendered production path for the admitted April range. +make newsletter-gen START=2026-02-14 END=2026-04-16 MODE=production + +# Validate the generated issue and retained artifacts. +make validate-newsletter FILE=output/2026-04_april_newsletter.md +bash tools/validate_pipeline_strict.sh 2026-02-14 2026-04-16 --require-fresh --production-artifacts +``` + +`MODE=production` is pinned by the prompt renderer. For another month, copy the +workflow shape and validation discipline, but first add or admit the new date +range instead of assuming the April production mode accepts arbitrary dates. + +Use these rules while running or adapting the pipeline: + +1. Define the date range, audience, scope, and acceptance criteria before generation. +2. Let Phase 1A/1B bind source identity before later phases synthesize. +3. Reuse accepted artifacts only when freshness, identity, and scope are clear. +4. Compact Phase 3 curation inputs only after required source classes are preserved. +5. Keep validation inside the cost gate. A cheaper invalid newsletter is not cheaper. +6. Use `make validate-newsletter`, strict validation, and scoring before promoting a route change. +7. Treat `tools/run_newsletter_orchestrated.sh` as diagnostic unless the prompt-rendered route needs phase-local repair. + +### Optimize A Similar Workflow + +The same pattern applies to other agentic workflows: + +1. **Map the route.** Identify phases, inputs, outputs, tools, validations, and stop conditions. +2. **Find repeated work.** Look for repeated source discovery, re-reading, broad search, and repeated synthesis. +3. **Bind artifacts.** Save accepted source sets, prompt hashes, generated artifacts, validation receipts, and freshness checks. +4. **Compact only after preserving floors.** Define mandatory source classes, API contracts, security constraints, or test coverage before shrinking context. +5. **Constrain tools when reuse is the goal.** If the agent can browse freely, it may reconstruct work that already passed. +6. **Route by task, not habit.** Test model and reasoning choices phase-by-phase against output quality. +7. **Measure finished work.** Include requests, retries, repairs, tool calls, validation, and final acceptance. +8. **Keep a fallback.** Optimized routes should fail closed to fuller context or human review when readiness checks fail. + +## Decision Tree + +```text +Is the agent failing, retrying, or producing weak output? + -> Tighten acceptance criteria, tests, validation, and task boundaries first. + +Is the agent exploring too much code or too many sources? + -> Provide exact files, source bundles, logs, and known constraints. + -> Use read-only research before implementation. + +Is context large or stale? + -> Split research, plan, implementation, and validation. + -> Carry forward only the plan, source list, constraints, and commands. + +Is spend concentrated in a few people, teams, or surfaces? + -> Review usage exports, team metrics, CLI activity, and code review separately. + -> Use model guidance and budget overrides before blanket restrictions. + +Are developers blocked by budgets? + -> Diagnose user-level, cost-center, and enterprise budget layers. + -> Keep validation in place while owners decide whether the fix is an override, budget change, or workflow guidance. + +Is the same context reused frequently? + -> Stabilize instructions, prompt prefixes, source bundles, and tool schemas. + -> Track cache or reuse indicators when available, without treating them as billing proof. +``` + +## Scenario Guidance + +| Scenario | Recommended pattern | Model posture | Cost control | +|---|---|---|---| +| Small refactor or doc update | Provide exact files and expected diff shape. | Lightweight or mid-tier. | Avoid broad search; set a stopping condition. | +| Codebase exploration | Ask for read-only mapping of relevant files and risks. | Mid-tier; reasoning if architecture is unclear. | No code changes; carry only useful context forward. | +| Architecture decision | Ask for options, tradeoffs, constraints, and a recommendation. | Reasoning/frontier where complexity warrants it. | Bound the question and require assumptions. | +| Implementation from a known plan | Start fresh with approved plan and file list. | Mid-tier where quality gates are strong. | Avoid re-research; run tests; stop at planned scope. | +| Unknown failure debugging | Provide logs, stack traces, repro steps, and recent changes. | Reasoning for root cause, then cheaper scoped patching where appropriate. | Use hypothesis -> test -> fix -> validation. | +| Test generation | Provide target behavior, edge cases, and framework. | Lightweight or mid-tier. | Keep production code unchanged unless the task explicitly includes it. | +| Repeated team workflow | Convert into instructions, prompt files, skills, or custom agents. | Depends on workflow complexity. | Reduce repeated discovery and inconsistent tool use. | ## Terms @@ -36,6 +167,8 @@ The point is not that these exact percentages transfer to another team. The poin ## Admin Checklist +For the full admin and FinOps operating model -- budgets, reporting, showback, and governance -- see [ADMIN_FINOPS_GUIDE.md](ADMIN_FINOPS_GUIDE.md). For the product controls behind these actions, see [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md). + | Owner | Action | Why It Matters | Source | |---|---|---|---| | Enterprise owner or billing manager | Set a universal user-level budget. | ULBs prevent a small number of users or sessions from consuming the shared pool early. | [Budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) | @@ -45,6 +178,16 @@ The point is not that these exact percentages transfer to another team. The poin | Platform owner | Publish developer guidance before enforcement. | Budget controls reduce runaway usage, but efficient work depends on task boundaries, context hygiene, model routing, and validation. | [Optimize AI usage](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) | | FinOps or platform owner | Review usage weekly during rollout, then monthly once stable. | Usage reports, team metrics, and CLI metrics help identify concentrated consumption, temporary spikes, and users blocked before finishing useful work. | [Usage reports](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls) | +## Budget And Governance Notes + +- Budget setup is conceptually straightforward, but rollout requires ownership decisions: user-level budget defaults, power-user overrides, cost-center mapping, enterprise spending limits, alert recipients, hard-stop behavior, and a support path for blocked users. +- Use ULBs to prevent a small number of users or agentic sessions from consuming the shared pool early. +- Use cost-center and enterprise budgets to manage additional spend exposure after included credits are exhausted, depending on configuration. +- Make sure support teams can diagnose which budget layer caused a block. +- Review usage weekly during rollout, then monthly once stable. Look for concentrated per-user consumption, premium-model concentration, temporary spikes, and teams blocked before completing productive work. +- Track Copilot code review separately from normal chat and agent usage because it may consume AI Credits and GitHub Actions minutes. +- For custom agentic workflows built with the Copilot SDK, OpenTelemetry can connect agent sessions, tool calls, and application traces. For standard VS Code and Copilot CLI usage, start with GitHub usage reports and available IDE/CLI telemetry, then add SDK/OpenTelemetry where your team owns the workflow harness. + ## Developer Playbook Do: @@ -56,16 +199,34 @@ Do: - Provide exact files, logs, stack traces, issue links, constraints, and acceptance criteria when known. - Use tests, linters, type checks, security scans, build commands, and CI checks. - Measure the full route: requests, tool calls, retries, validations, repairs, and final acceptance. +- Start a fresh session between phases when old context no longer matters. +- Keep repository instructions short, specific, and grounded in repeated agent failure modes. +- Reuse artifacts only when identity, freshness, and scope are clear. + +Common pitfalls: + +- Shorter prompts alone rarely fix an expensive workflow if retries, repairs, or broad search continue. +- Phase-local token movement is diagnostic until the full route and quality gates are measured. +- Context reduction needs a fallback path for required source material. +- Lower reasoning or cheaper models should be promoted only when quality evidence supports the route. +- Workflow telemetry is engineering evidence; invoice-grade billing claims need billing data. +- One workflow experiment should not become a universal model recommendation. +- Output length limits work best as a formatting tool, not the primary cost-control strategy. +- Agents should have bounded search and tool access when the relevant files are already known. +- User-level budget changes should be checked against cost-center and enterprise budgets. +- BYOK, provider pricing, and OpenTelemetry guidance should specify the surface and whether the question is Copilot billing, reporting, or custom workflow instrumentation. -Do not: +## Claim Discipline -- Do not claim savings from shorter prompts alone. -- Do not treat phase-local token movement as full-route savings. -- Do not strip context without a fallback path to restore required source material. -- Do not force lower reasoning or cheaper models without quality evidence. -- Do not treat internal telemetry as billing proof. -- Do not quote private exact token counts in customer material unless the evidence owner approves the metric and caveat. -- Do not turn one workflow experiment into a universal model recommendation. +Label every optimization claim by evidence type: + +- **Official product behavior:** first-party GitHub or Microsoft product documentation, changelog, or release note. +- **GitHub usage report/export:** usage reporting or export evidence. +- **Workflow telemetry:** aggregate/proxy data from a bounded workflow or experiment. +- **Workflow experiment:** bounded experimental result with quality and provenance checks. +- **Recommendation/inference:** guidance derived from evidence, not direct proof. + +Keep evidence types separate. Workflow telemetry is not billing proof, and one workflow experiment is not a universal model benchmark. ## Measurement Checklist @@ -78,17 +239,45 @@ Before claiming savings, confirm: - The evidence says what it does not prove. - Negative results are recorded as guardrails. - The measurement is tied to a real workflow change, not a generic prompt-style preference. +- The requested model and the model or feature actually reported in usage data are recorded where available. +- Similar tasks, repos, source conditions, and validation bars are compared before quoting movement. ## Source Links +### GitHub Copilot And UBB + +- [GitHub Copilot plans](https://docs.github.com/en/copilot/get-started/plans) - [Models and pricing for GitHub Copilot](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) - [Getting started with budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) +- [Optimizing your budget configuration](https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration) - [Improving agent quality to optimize AI usage](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) - [Managing AI credits, GitHub Well-Architected](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) - [Auto model selection now routes based on your task in VS Code](https://github.blog/changelog/2026-05-20-auto-model-selection-now-routes-based-on-your-task-in-vs-code) - [Copilot usage metrics reports now use GitHub-owned download URLs](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls) - [OpenTelemetry instrumentation for Copilot SDK](https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry) +### Provider Token Mechanics + +Use these for general token, context, caching, batch, reasoning, and evaluation concepts. Use GitHub Copilot billing documentation for Copilot charge and AI Credit questions. + +- [OpenAI pricing](https://developers.openai.com/api/docs/pricing) +- [OpenAI cost optimization](https://developers.openai.com/api/docs/guides/cost-optimization) +- [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) +- [OpenAI conversation state](https://developers.openai.com/api/docs/guides/conversation-state) +- [OpenAI reasoning models](https://developers.openai.com/api/docs/guides/reasoning) +- [OpenAI evals](https://developers.openai.com/api/docs/guides/evals) +- [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) +- [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) +- [Anthropic token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting) +- [Anthropic context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows) +- [Anthropic extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) +- [Google Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing) +- [Google Gemini context caching](https://ai.google.dev/gemini-api/docs/caching) +- [Google Gemini token counting](https://ai.google.dev/gemini-api/docs/tokens) +- [Google Gemini thinking](https://ai.google.dev/gemini-api/docs/thinking) +- [Google Gemini Batch API](https://ai.google.dev/gemini-api/docs/batch-api) +- [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) + ## Boundary This companion does not claim billing savings, durable savings, a universal percentage, request-level provider-token proof, model superiority, or fleet readiness. Those boundaries are intentional. diff --git a/release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_GUIDE.md b/release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_GUIDE.md new file mode 100644 index 0000000..c6ebafd --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_GUIDE.md @@ -0,0 +1,131 @@ +# Developer Guide: Cost-Aware Agentic Workflows + +> **Bottom line:** Optimize the route, not the prompt; a shorter prompt can still cost more if it drives retries, broad search, and repairs. **For:** developers and AI workflow owners. **Read time:** about 9 minutes. + +This guide is for developers reducing token and cost pressure without weakening quality. It pairs core rules with worked scenarios. For the system that produced these lessons and its before/after numbers, see [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md). + +## Core Rule + +Optimize the route, not the prompt. A prompt can be shorter while the workflow costs more. A model can be cheaper per token and still create repair work. The useful unit is the finished workflow with quality, validation, and review intact. + +## Recommended Practices + +1. Measure the full workflow: inputs, outputs, requests, tools, retries, validations, and repairs. +2. Put quality gates inside the cost gate. A cheaper invalid answer is not cheaper. +3. Start by removing repeated discovery and repeated context assembly. +4. Use stable prompts, stable source bundles, and explicit acceptance criteria. +5. Preserve required source classes before reducing context. +6. Add readiness checks before using compact or no-tools routes. +7. Keep a fallback to a fuller route when readiness fails. +8. Reuse artifacts only when identity, freshness, and source scope are clear. +9. Restrict tools when reuse is the goal; otherwise the agent may reconstruct the same work. +10. Change model or reasoning only where the task is bounded and quality can be checked. +11. Treat cache and telemetry as diagnostic unless billing-grade controls exist. +12. Keep lightweight records: source links, prompts, validation reports, and known non-claims. + +## Common Pitfalls + +1. Telemetry alone is not invoice-grade billing evidence. +2. One task or phase is not the same as full-route savings. +3. Lower reasoning is task-specific, not globally cheaper by default. +4. Output budgets can help formatting, but they are a weak default cost strategy. In this system, an output-shape policy added millions of tokens instead of saving them. +5. Source pruning needs compensation checks for search, requests, cache, and repairs. +6. A cheaper output that misses the quality threshold is not an accepted optimization. +7. Percentages from different corpora or conditions should not be combined. +8. Routing evidence should not become model-superiority language. +9. Negative results are useful guardrails when they are recorded clearly. + +## Decision Checklists + +**Before changing model or reasoning:** Is the task bounded and source-defined? Is the quality gate independent of the model choice? Do you have a rollback path? Are you avoiding model-superiority language? + +**Before reducing context:** Which source classes are mandatory? Does the compact artifact preserve those classes? Can the model compensate with search or tools? Is there a fallback to the fuller route? Does final validation catch missing coverage? + +**Before reusing artifacts:** Is artifact identity fixed? Is freshness clear? Are tools and refetches bounded? Is output validated after reuse? Does the claim stay local to the workflow? + +**Before claiming savings:** Is the result route-level, not a single step only? Did quality pass? Are corpora comparable enough for the wording? Is this estimate, telemetry, usage export, or invoice-grade evidence? What does the evidence not prove? + +## Worked Examples + +These examples apply the rules above without treating this newsletter system as a universal benchmark. They use public-safe patterns: bind the task, preserve required context, validate output, and avoid claiming billing savings from workflow telemetry alone. + +### Example 1: Newsletter Or Content Generation + +A team produces a recurring technical newsletter. The draft quality is good, but the workflow repeatedly re-reads source material, performs broad searches late, and needs repair passes after assembly. + +| Step | Pattern | Why it helps | +|---|---|---| +| Source binding | Capture accepted URLs, dates, and source classes before synthesis. | Later phases do not rediscover the same material. | +| Artifact reuse | Save accepted interim files and use them as the curation input. | Reduces repeated fetch/search work. | +| Compact working set | Build a smaller curation packet after source coverage is checked. | Keeps expensive synthesis focused. | +| Validation gate | Run structural, content, and link checks before publication. | Prevents a cheaper weak draft from becoming accepted output. | +| Fallback | Re-open fuller context only when readiness checks fail. | Keeps optimization reversible. | + +**Validation:** required source classes are still represented; final output passes newsletter validation and link checks; any token or request movement is described as workflow evidence, not billing proof. + +### Example 2: Scoped Documentation Update + +A developer asks an agent to refresh docs after a feature change. The agent explores the whole repository even though the changed files are known. + +1. Provide the exact files that need edits. +2. Provide the expected outcome and stopping condition. +3. Ask for a read-only check before implementation if the context is uncertain. +4. Disable or discourage broad search after the file list is confirmed. +5. Run Markdown lint, link checks, or project-specific docs validation. + +```text +Update these files only: README.md, docs/how-it-works.md, and docs/architecture.md. +Goal: reflect the new release bundle navigation. +Do a brief read-only pass first, then make minimal edits. +Stop after validation and summarize changed links. +``` + +**Validation:** only intended docs changed; links resolve; new docs match the current implementation; the agent did not re-run unrelated discovery or refactor unrelated files. + +### Example 3: Debugging Unknown Failure + +A test fails intermittently. The team wants root cause, not a trial-and-error patch loop. + +| Phase | Agent instruction | Gate | +|---|---|---| +| Hypothesis | Summarize the failure and propose one testable hypothesis. | Hypothesis names a falsifiable signal. | +| Evidence | Read the failing test, recent logs, and the smallest relevant implementation surface. | No broad repository sweep unless the first hypothesis fails. | +| Patch | Make the smallest root-cause fix. | Diff is scoped to the failing behavior. | +| Validate | Run the failing test and a nearby regression set. | Tests pass or the hypothesis is revised. | +| Record | Capture the lesson if the failure mode is likely to recur. | Future runs avoid the same repair work. | + +**Validation:** the fix is tied to the observed failure; the failing test now passes; any broader test run is proportional to risk; the team records whether the cause was unclear task scope, missing logs, stale context, or a real code defect. + +### Example 4: Budget Block During Rollout + +A developer is blocked during UBB rollout, and the team is unsure whether to raise the user budget, change model policy, or reduce agent use. + +1. Identify which budget layer blocked usage: user-level, cost-center, or enterprise. +2. Check whether the user is doing high-value work that needs a temporary override. +3. Check whether repeated attempts came from missing files, weak task boundaries, or failed validation. +4. Separate code review, CLI activity, and chat or agent usage where reporting allows. +5. Decide whether the response is an override, guidance, model policy, or workflow repair. + +**Good outcome:** the user finishes valid work without weakening tests or validation, and the admin team learns whether the budget was too low, the workflow was wasteful, or the work was correctly high-usage. See the [Admin And FinOps Guide](ADMIN_FINOPS_GUIDE.md) for the triage owners. + +### Example 5: Custom Agent Workflow + +A platform team owns a custom agent or workflow harness and wants to reduce repeated context assembly while making cost claims responsibly. + +- Add trace or log identifiers for phases, tools, and accepted artifacts. +- Record prompt versions and source bundle hashes. +- Keep static instructions stable where caching or reuse may help. +- Validate output quality before comparing routes. +- Track retries, repairs, and tool calls, not just prompt length. +- Use OpenTelemetry or equivalent tracing where the workflow is instrumented. + +**Claim boundary:** the team can say the workflow repeated less work or needed fewer repairs when telemetry shows that. Invoice savings require billing data. + +## Reuse Checklist + +- Is the task bounded? +- Are required sources or files named? +- Is there a validation gate independent of the optimization? +- Is fallback available if compact context fails? +- Are metrics labeled as billing data, usage report, telemetry, experiment, or estimate? +- Is the final result accepted by the same quality bar as the baseline? diff --git a/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_MAY_COST_SECTION.md b/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_MAY_COST_SECTION.md new file mode 100644 index 0000000..ef5a4d5 --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_MAY_COST_SECTION.md @@ -0,0 +1,36 @@ +# May Newsletter UBB And Cost-Aware Usage Section + +> **Bottom line:** The verbatim newsletter section this bundle expands, kept as provenance. **For:** anyone who wants the exact source text. **Read time:** 3 minutes. + +Source: May 2026 GitHub Customer Newsletter. + +## UBB Readiness And Cost-Aware Copilot Usage + +**The theme this month is operating readiness.** Usage-based billing starts June 1, so the work now is budget setup, user guidance, model policy, reporting access, and a support path for teams whose agentic usage grows quickly. + +New definitions: [**UBB**](https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/) means usage-based billing; [**AI Credits**](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) are GitHub's billing unit for metered Copilot usage where applicable; [**ULB**](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) means user-level budget; [pooled credits](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) are included monthly AI Credits shared across licensed users in the billing entity; [additional spend](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) means metered charges after included pooled credits are exhausted. + +## Action Required Now: UBB Readiness + +| Owner | Action | Why it matters | Source | +|---|---|---|---| +| Enterprise owner or billing manager | Set a universal [user-level budget](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls). | [ULBs](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) cap each user's AI Credit usage across pooled credits and additional metered usage, helping prevent a small number of users or sessions from consuming the shared pool early. | [Budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) | +| Billing manager | Identify power users and define [individual overrides](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls). | Developers running frequent agent sessions, large-codebase work, code review, or frontier-model workflows may need a [higher limit](https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration) to avoid blocking productive work. | [Budget optimization](https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration) | +| Billing manager and cost owners | Configure [cost-center](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) and [enterprise spending limits](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls). | These controls manage [additional spend exposure](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) after included pooled credits are exhausted, and the [most restrictive applicable budget](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) can block usage. | [Managing AI credits](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) | +| Billing manager | Decide where [hard stops](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) and alert recipients belong. | Spending limits without stop-usage enforcement may only alert; alert ownership determines who responds when thresholds are reached. | [Budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) | +| Platform owner | Publish [developer usage guidance](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) before enforcement. | Budget controls limit runaway consumption, but efficient work comes from clearer tasks, scoped context, model-to-task matching, and validation gates. | [Optimize AI usage](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) | +| FinOps or platform owner | Review [usage exports and reports](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls) weekly during rollout, then monthly once stable. | Usage exports, [team metrics](https://github.blog/changelog/2026-05-14-team-level-copilot-usage-metrics-now-available-via-api), and [CLI activity metrics](https://github.blog/changelog/2026-04-10-copilot-cli-activity-now-included-in-usage-metrics-totals-and-feature-breakdowns) help identify concentrated consumption, premium-model concentration, temporary spikes, and users blocked before completing useful work. | [April reports](https://github.blog/changelog/2026-05-12-april-reports-are-now-available-to-prepare-for-usage-based-billing) | + +Related webinar recordings: [Understanding Budgets Webinar](https://www.youtube.com/watch?v=F2rJ55VVf44) | [Token Optimization Webinar](https://www.youtube.com/watch?v=0IOcPqubpMc) + +## Developer Playbook: Cost-Aware Copilot Usage + +- **Developer billing clarifications** -- [Code completions and Next Edit Suggestions](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing#code-completions) are not billed in AI Credits for paid Copilot plans. UBB attention should focus on chat, agents, premium models, large-context work, repeated sessions, and [**Copilot code review**](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing#pricing-and-usage-cost-considerations-for-copilot-code-review). Code review is a separate tracking surface because token consumption is billed in AI Credits and the agentic infrastructure consumes GitHub Actions minutes. - [Models and Pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) | [Code Review Docs](https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review) + +- **Feature-specific token and context controls** -- [Auto model selection](https://github.blog/changelog/2026-05-20-auto-model-selection-now-routes-based-on-your-task-in-vs-code), [prompt caching](https://code.visualstudio.com/updates/v1_118), [tool search](https://code.visualstudio.com/updates/v1_119), [agentic search and execution tools](https://code.visualstudio.com/updates/v1_118), [OpenTelemetry tracing for agent sessions](https://code.visualstudio.com/updates/v1_119), [model details in agent responses](https://code.visualstudio.com/updates/v1_119), [terminal output compression](https://code.visualstudio.com/updates/v1_121), [reasoning controls](https://code.visualstudio.com/updates/v1_121), and [Copilot SDK OpenTelemetry](https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry) all help teams understand, manage, or control context and token usage. Treat those signals as engineering input before writing policy. - [Token Efficiency Notes](https://code.visualstudio.com/updates/v1_118) | [Agent Observability Notes](https://code.visualstudio.com/updates/v1_119) | [Terminal Controls Notes](https://code.visualstudio.com/updates/v1_121) + +- **Workflow design beats blunt output limits** -- Useful agent work gets cheaper through better routes, not just shorter prompts. Start with [clear tasks and stopping conditions](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage), split larger work into [research, plan, implementation, and validation phases](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage), use the smallest capable model for the phase, and keep [deterministic guardrails](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) in the loop. - [Optimize AI Usage](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) | [VS Code Harness Blog](https://code.visualstudio.com/blogs/2026/05/15/agent-harnesses-github-copilot-vscode) + +- **Measure before policy** -- [Usage reports](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls), [team metrics](https://github.blog/changelog/2026-05-14-team-level-copilot-usage-metrics-now-available-via-api), [CLI metrics](https://github.blog/changelog/2026-04-10-copilot-cli-activity-now-included-in-usage-metrics-totals-and-feature-breakdowns), and OpenTelemetry can all provide engineering signals. Treat metrics as engineering evidence, not invoices, durable savings proof, or model-superiority evidence. - [Copilot Usage Metrics API](https://docs.github.com/rest/copilot/copilot-usage-metrics) | [OpenTelemetry Docs](https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry) + +- **May cost optimization bundle** -- I packaged the evidence, do's and don'ts, and source notes behind this newsletter into a [Cost Optimization Guide](CUSTOMER_COMPANION.md). It shows the practical learning process behind the guidance, including the quality gates that kept the evidence honest and the things that did not work. Use it as a deeper companion, not as a billing benchmark. diff --git a/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md b/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md index c269bd8..d455745 100644 --- a/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md +++ b/release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md @@ -1,39 +1,101 @@ # Newsletter Generation System Release Notes +> **Bottom line:** I re-engineered this newsletter's generation system to do less repeated AI work; the accepted route cut aggregate token totals by roughly 27-45% per run -- about $6-12 per run in illustrative API-equivalent terms -- while quality gates stayed in force. **For:** developers and Admin & FinOps owners who want the technical map and the before/after evidence. **Read time:** about 12 minutes. + This note summarizes the customer-safe newsletter generation system changes included with the May 2026 public catch-up. It is about the reusable newsletter pipeline itself, not the May newsletter content. +Use this file when you want the technical map: what changed, where it changed, what each change does, what it cost, and how to operate the updated system. For developer/admin cost guidance, start with [CUSTOMER_COMPANION.md](CUSTOMER_COMPANION.md). For product-source inventory, use [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md). + ## Measured Impact -The cost optimization work produced measurable aggregate/proxy movement in retained comparisons. These are not billing claims, but they are meaningful workflow measurements: +The cost optimization work I ran produced measurable aggregate/proxy movement in retained comparison runs. These are workflow-engineering measurements of direct provider token totals, not GitHub Copilot billing claims. + +| Comparison run | Before (tokens) | After (tokens) | Change | Quality status | +|---|---:|---:|---:|---| +| Integrated fixed-corpus | 7,846,260 | 5,699,291 | about -27% | Quality gates passed. | +| Integrated current-cycle | 9,001,070 | 4,957,442 | about -45% | Newsletter validation passed; corpus caveats apply. | +| Phase 3 compact working-set example | 284,862 | 31,333 | about -89% | Source coverage and curation checks passed. | +| Cheaper but blocked route | 5,785,064 | 4,501,184 | about -22% | Rejected: quality fell below threshold (V2 `39/50`). | +| Repaired optimized route | — | 4,388,338 | — | Accepted: V2 `48/50`, newsletter validation passed. | + +On the integrated runs, requests and tool calls dropped alongside tokens: the fixed-corpus run moved from `126` to `102` requests and `212` to `184` tool calls; the current-cycle run moved from `125` to `93` requests and `267` to `149` tool calls. The system got cheaper because I made it do less repeated work, not because it merely asked for shorter answers -- that was the central lesson I took from the May cycle. + +These totals count direct provider tokens across the full route. They are engineering evidence under bounded workflow conditions; they are not a durable percentage, a universal benchmark, or a Copilot AI Credits invoice. + +## Illustrative Cost Translation + +These token movements can be translated into an illustrative dollar figure so the "bottom line" is concrete. The translation is engineering math on direct provider tokens, **not** a GitHub Copilot AI Credits bill. + +The system's own matched-pair experiments recorded both token deltas and the corresponding API-equivalent cost deltas (see the output-shape negative result below). Two of those matched pairs imply a blended rate of about **$3 per million direct provider tokens** (`+2,091,727` tokens for `+$6.18`; `+2,295,172` tokens for `+$7.05`). That blended rate is consistent with current frontier list pricing such as a `gpt-5.5`-class model; substitute your own current rate from the [OpenAI pricing](https://developers.openai.com/api/docs/pricing) page for your own estimates. -| Signal | Rounded public-safe metric | Quality status | -|---|---:|---| -| Integrated fixed-corpus comparison | About 27% lower aggregate token total | Quality gates passed. | -| Integrated current-cycle comparison | About 45% lower aggregate token total | Newsletter validation passed; corpus caveats apply. | -| Phase 3 compact working-set example | About 89% lower Phase 3 curation token load | Source coverage and curation checks passed. | -| Cheaper but blocked route | About 22% lower aggregate token total | Rejected because quality fell below threshold. | -| Repaired optimized route | V2 `48/50` and newsletter validation passed | Accepted as the safer route shape. | +Applying that illustrative `~$3 / million-token` rate: -The headline: the system got cheaper because it did less repeated work, not because it merely asked for shorter answers. The strongest levers were source/candidate discipline, compact synthesis inputs, accepted-artifact reuse, tool suppression when safe, and quality gates that rejected cheap but weak output. +| Comparison run | Tokens saved | Illustrative API-equivalent saving per run | +|---|---:|---:| +| Integrated current-cycle | about 4,043,000 | about **$12** | +| Integrated fixed-corpus | about 2,147,000 | about **$6** | +| Phase 3 compact working-set example | about 253,000 | under **$1** | + +Read this as: with a `gpt-5.5`-class blended rate, the workflow changes removed roughly `$12` of API-equivalent token cost from a full current-cycle newsletter run. The number is illustrative and route-specific. It is not a billing-savings claim, it does not include tool/runtime/validation side costs, and it does not transfer to other workflows or to GitHub Copilot AI Credit pricing, which uses its own model-specific conversion. + +## What The Strongest Levers Were + +The strongest levers were source/candidate discipline, compact synthesis inputs, accepted-artifact reuse, tool suppression when safe, and quality gates that rejected cheap but weak output. Each lever, its before/after evidence, and the file that implements it are mapped in [How The Cost Mechanisms Fit Together](#how-the-cost-mechanisms-fit-together) below. ## What Changed +### Operator Entry Points + +| Task | Command or file | Use it when | +|---|---|---| +| Run a production-like newsletter generation | `make newsletter-gen START=2026-02-14 END=2026-04-16 MODE=production` | You want the admitted prompt-rendered production oracle. | +| Retain proof artifacts for a production run | `make newsletter-proof-run START=2026-02-14 END=2026-04-16 MODE=production` | You need retained evidence for the admitted April range in the source repo. | +| Prepare a clean cycle | [prepare_newsletter_cycle.sh](../../tools/prepare_newsletter_cycle.sh) | You need to clear stale intermediates before a run. | +| Render the exact product prompt | [render_product_run_prompt.sh](../../tools/render_product_run_prompt.sh) | You want to inspect or run the prompt outside Make. | +| Diagnose phase-local behavior | [run_newsletter_orchestrated.sh](../../tools/run_newsletter_orchestrated.sh) | The prompt-rendered route failed and needs phase-local repair. | +| Validate newsletter content | [validate_newsletter.sh](../../.github/skills/newsletter-validation/scripts/validate_newsletter.sh) | You need deterministic newsletter checks. | +| Validate retained artifacts | [validate_pipeline_strict.sh](../../tools/validate_pipeline_strict.sh) | You need source, scope, freshness, and production-artifact gates. | +| Score editorial quality | [score-v2-rubric.sh](../../tools/score-v2-rubric.sh) | You need rubric-level quality evidence. | + +Production-like path for the admitted April range: + +```bash +bash tools/prepare_newsletter_cycle.sh 2026-02-14 2026-04-16 --no-reuse +make newsletter-gen START=2026-02-14 END=2026-04-16 MODE=production +make validate-newsletter FILE=output/2026-04_april_newsletter.md +bash tools/validate_pipeline_strict.sh 2026-02-14 2026-04-16 --require-fresh --production-artifacts +``` + +The prompt renderer currently pins `production` mode to `2026-02-14` through +`2026-04-16` and `benchmark` mode to `2025-12-05` through `2026-02-13`. For other +date ranges, treat the commands above as the workflow pattern, then extend and +validate the admitted prompt-rendered mode before using `MODE=production`. + +Diagnostic path: + +```bash +bash tools/run_newsletter_orchestrated.sh START_DATE END_DATE +bash tools/score-v2-rubric.sh output/YYYY-MM_month_newsletter.md +``` + +The diagnostic path is for localizing failures and measuring candidate route changes. It is not the default production authority. + ### Safer Public Publishing - Added an allowlist-and-prune publication flow so public snapshots copy only approved surfaces and remove stale target-only files before validation. See [publish_public_snapshot.sh](../../tools/publish_public_snapshot.sh), [public_snapshot_allowlist.txt](../../tools/public_snapshot_allowlist.txt), and [public_snapshot_prune.txt](../../tools/public_snapshot_prune.txt). - Made the publisher pull-request safe by default. It now leaves changes uncommitted unless `--commit` is explicitly provided. This helps reviewers inspect the public diff before publishing. - Added target safety checks so the publisher refuses to run against the source repo, refuses nested source/target layouts, validates relative allowlist and prune entries, and treats sensitive-scan errors as hard failures. -- Expanded sensitive-pattern scanning for local paths, private source links, private proof markers, stale release-bundle internals, and other non-public terms. +- Expanded sensitive-pattern scanning for local paths, non-public source links, proof markers, stale release-bundle internals, and other non-public terms. ### Public Snapshot Boundary -- Removed raw public copies of private `planning/`, `workspace/`, `runs/`, internal February source notes, and stale output variants from the public snapshot boundary. -- Replaced broad `config/` publication with the public benchmark-mode config only: [feb2026_consistency.json](../../config/benchmark_modes/feb2026_consistency.json). Internal experiment policies, feature flags, fixture packs, and retained-run benchmark lanes stay private. +- Removed raw public copies of non-public `planning/`, `workspace/`, `runs/`, February source notes, and stale output variants from the public snapshot boundary. +- Replaced broad `config/` publication with the public benchmark-mode config only: [feb2026_consistency.json](../../config/benchmark_modes/feb2026_consistency.json). Experiment policies, feature flags, fixture packs, and retained-run benchmark lanes stay out of the public bundle. - Kept customer-safe release material selective: February public launch assets, April launch command notes, and May customer-facing cost guidance. See [February public launch](../2026-02_newsletter_launch/public/START_HERE.md), [April launch notes](../2026-04_newsletter_launch/START_HERE.md), and [May customer companion](CUSTOMER_COMPANION.md). ### Stronger Validation Gates -- Updated the all-suite test runner to distinguish required public-safe suites from private retained-fixture suites. When private-only fixtures are absent from the public snapshot, those suites are skipped with an explicit reason rather than failing the public build. See [test_all.sh](../../tools/test_all.sh). +- Updated the all-suite test runner to distinguish required public-safe suites from retained-fixture suites. When non-public fixtures are absent from the public snapshot, those suites are skipped with an explicit reason rather than failing the public build. See [test_all.sh](../../tools/test_all.sh). - Added targeted validator coverage for May root-cause drift classes in [validate_newsletter.sh](../../.github/skills/newsletter-validation/scripts/validate_newsletter.sh) and its self-test harness [test_validator.sh](../../tools/test_validator.sh). - Added or synced regression coverage for phase contracts, product prompt rendering, scope alignment, public newsletter validation, external critique fixtures, source-pruning receipts, output-shape receipts, and route-lock telemetry under [tools/](../../tools/) and [tests/](../../tests/). @@ -43,7 +105,44 @@ The headline: the system got cheaper because it did less repeated work, not beca - Added artifact reuse and no-refetch receipts so accepted intermediate artifacts can be consumed without repeated retrieval. See [build_artifact_reuse_admission_receipt.py](../../tools/build_artifact_reuse_admission_receipt.py), [build_artifact_reuse_no_refetch_receipt.py](../../tools/build_artifact_reuse_no_refetch_receipt.py), and [build_artifact_reuse_phase3_proof_receipt.py](../../tools/build_artifact_reuse_phase3_proof_receipt.py). - Added compact working-set generation and Phase 3 readiness checks to reduce synthesis load while protecting source coverage. See [build_phase3_working_set.py](../../tools/build_phase3_working_set.py), [validate_phase3_curated.py](../../tools/validate_phase3_curated.py), and [validate_phase3_v2_readiness.py](../../tools/validate_phase3_v2_readiness.py). - Added route telemetry and fail-closed proof helpers so token movement is tied to route identity and quality state. See [run_copilot_phase.py](../../tools/run_copilot_phase.py), [build_phase_token_telemetry_receipt.py](../../tools/build_phase_token_telemetry_receipt.py), and [test_phase_route_lock_telemetry.sh](../../tools/test_phase_route_lock_telemetry.sh). -- Kept failed output-shape work visible as a negative result. The tested shape reduced the wrong thing and amplified total route cost, so it stayed out of the accepted path. See [apply_newsletter_output_shape_policy.py](../../tools/apply_newsletter_output_shape_policy.py) and [build_output_shape_experiment_receipt.py](../../tools/build_output_shape_experiment_receipt.py). +- Kept failed output-shape work visible as a negative result. The tested shape reduced the wrong thing and amplified total route cost, so it stayed out of the accepted path. Across three matched pairs it increased total direct provider tokens by `2,091,727`, `2,295,172`, and `166,553`, which mapped to API-equivalent cost increases of `+$6.18`, `+$7.05`, and `+$1.09`. See [apply_newsletter_output_shape_policy.py](../../tools/apply_newsletter_output_shape_policy.py) and [build_output_shape_experiment_receipt.py](../../tools/build_output_shape_experiment_receipt.py). + +### How The Cost Mechanisms Fit Together + +The accepted route is not one trick. It is a set of controls that reduce repeated work while keeping fallback and validation intact. + +| Lever | Where it changed | What it does | How to use it safely | +|---|---|---|---| +| Source/candidate pruning | [apply_newsletter_source_pruning_policy.py](../../tools/apply_newsletter_source_pruning_policy.py), source-pruning receipts | Narrows downstream material to accepted source sets. | Preserve source floors and measure search/request compensation. | +| Artifact reuse | [build_artifact_reuse_admission_receipt.py](../../tools/build_artifact_reuse_admission_receipt.py), [build_artifact_reuse_no_refetch_receipt.py](../../tools/build_artifact_reuse_no_refetch_receipt.py) | Lets later phases consume accepted artifacts instead of re-fetching. | Bind identity, freshness, scope, and validation before reuse. | +| Compact Phase 3 working set | [build_phase3_working_set.py](../../tools/build_phase3_working_set.py), [validate_phase3_curated.py](../../tools/validate_phase3_curated.py) | Reduces expensive curation context while preserving required coverage. One Phase 3 example moved `284,862 -> 31,333` tokens (about -89%). | Keep required source classes and fallback to fuller context. | +| Tool-suppressed Phase 3 route | [run_phase3_stdout_no_tools_artifact_reuse.py](../../tools/run_phase3_stdout_no_tools_artifact_reuse.py), [validate_phase3_v2_readiness.py](../../tools/validate_phase3_v2_readiness.py) | Prevents tool-heavy reconstruction when accepted artifacts are ready. | Run readiness checks first; fail closed to fallback. | +| Route telemetry | [run_copilot_phase.py](../../tools/run_copilot_phase.py), [build_phase_token_telemetry_receipt.py](../../tools/build_phase_token_telemetry_receipt.py) | Connects route identity, prompts, token signals, and quality state. | Treat aggregate/proxy data as engineering evidence, not billing proof. | +| Negative evidence | [apply_newsletter_output_shape_policy.py](../../tools/apply_newsletter_output_shape_policy.py), [build_output_shape_experiment_receipt.py](../../tools/build_output_shape_experiment_receipt.py) | Keeps tempting failed strategies from being repeated. The output-shape shape added up to `+2,295,172` tokens (`+$7.05`) instead of saving. | Promote only route-level wins that pass quality gates. | + +The integrated `-27%` and `-45%` movements are route-level results from the full stack working together. Isolated single-lever movement was often small, so the bundle attributes whole-route gains to the route, not to any one trick. The Phase 3 compact example is the clearest single-lever signal. + +### Generating A Newsletter With Lower Token Pressure + +1. Start from the prompt-rendered route, not the diagnostic harness. +2. Prepare the cycle cleanly so stale intermediates do not drive extra repair work. +3. Keep Phase 1 source identity and Phase 3 curation coverage explicit. +4. Reuse accepted artifacts instead of asking later phases to search again. +5. Use compact working sets only where source coverage and fallback are available. +6. Validate before claiming movement: newsletter validation, strict pipeline validation, and editorial scoring. +7. If a route gets cheaper but quality falls, repair the route or reject it while keeping the quality bar unchanged. + +### Applying The Pattern Elsewhere + +For another agentic workflow, copy the pattern rather than the numbers: + +- Separate research, synthesis, implementation, and validation phases. +- Make each phase write artifacts to disk so later phases can reuse accepted work. +- Add artifact identity, freshness, and source-scope checks before reuse. +- Define mandatory context floors before compacting prompts or working sets. +- Restrict tools when reuse is the goal, but keep fallback when readiness fails. +- Track route-level movement, including retries, repairs, validations, and failed attempts. +- Publish negative results as guardrails so teams do not repeat locally cheap but globally expensive strategies. ### Pipeline And Operator Surfaces @@ -52,6 +151,15 @@ The headline: the system got cheaper because it did less repeated work, not beca - Added live production and retained proof-run helpers for prompt-rendered newsletter runs. See [render_product_run_prompt.sh](../../tools/render_product_run_prompt.sh), [run_product_newsletter.sh](../../tools/run_product_newsletter.sh), and the Makefile targets in [Makefile](../../Makefile). - Added current source-intelligence guidance for the GitHub Copilot app and Copilot CLI. See [copilot-app.md](../../reference/source-intelligence/copilot-app.md) and [copilot-cli.md](../../reference/source-intelligence/copilot-cli.md). +### Broader System Changes Beyond Cost Optimization + +- The public/non-public boundary is now a first-class release surface. Snapshot allowlists, prune lists, and sensitive scans define what can leave the source repo. +- The newsletter validator now carries more product-specific drift checks, so public output quality does not depend only on human review. +- The source-intelligence layer was refreshed for Copilot CLI and Copilot app surfaces so future newsletters can reason over current product areas. +- The `upgrade-advisor` agent makes system-improvement recommendations bounded by workflow, harness, validation, and execution-surface evidence. +- Public tests now distinguish required public-safe suites from retained-fixture suites, which makes the public repo runnable without non-public run logs. +- The release bundle itself now acts as a handoff artifact: it pairs user guidance, technical change mapping, source inventory, validation evidence, and claim boundaries. + ### Published Outputs - Added the April 2026 generated newsletter alongside the existing February and May outputs. See [2026-04_april_newsletter.md](../../output/2026-04_april_newsletter.md). @@ -60,11 +168,21 @@ The headline: the system got cheaper because it did less repeated work, not beca ## Validation Summary - Public sensitive-pattern scan: pass. -- Public `make test-all`: required public suites pass; private retained-fixture suites skip when private fixtures are absent. +- Public `make test-all`: required public suites pass; retained-fixture suites skip when non-public fixtures are absent. - April newsletter validation: pass, 0 warnings. - May newsletter validation: pass, 0 warnings. -- Private full-suite validation before public sync: required suites pass, with only the expected private workspace fixture skip. +- Source full-suite validation before public sync: required suites pass, with only the expected retained workspace fixture skip. + +Recommended validation when adapting this release: + +```bash +git diff --check +make test-all +make validate-newsletter FILE=output/YYYY-MM_month_newsletter.md +bash tools/validate_pipeline_strict.sh START_DATE END_DATE --require-fresh --production-artifacts +bash tools/score-v2-rubric.sh output/YYYY-MM_month_newsletter.md +``` ## Publication Boundary -This release note intentionally excludes private run logs, private planning bursts, raw internal evidence, local machine paths, retained-run benchmark artifacts, and non-public source notes. Those materials remain in the private source repo and are represented here only as public-safe workflow descriptions. \ No newline at end of file +This release note intentionally excludes non-public run logs, planning bursts, raw evidence, local machine paths, retained-run benchmark artifacts, and non-public source notes. Those materials are represented here only as public-safe workflow descriptions. \ No newline at end of file diff --git a/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md b/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md index 358c29a..f930964 100644 --- a/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md +++ b/release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md @@ -1,13 +1,22 @@ # Product Feature Quick Hits For Cost Management +> **Bottom line:** A first-party-source-backed inventory of Copilot cost, visibility, routing, and governance features with conservative status labels. **For:** developers and Admin & FinOps owners auditing what is available and where. **Read time:** scan by `Group` column, about 5 minutes. + This is a first-party-source-backed inventory of features related to cost management, cost visibility, token efficiency, routing, and governance. Use the `Group` column for operational scanning before reading individual rows. Status labels are conservative: `GA` or `PREVIEW` appear only where the source explicitly supports that status. +GitHub Copilot billing and AI Credit claims should cite GitHub Docs, GitHub Blog, GitHub Changelog, or GitHub Well-Architected sources. Provider docs below are included for general token mechanics, context, caching, reasoning, batch, and instrumentation concepts; they are not Copilot billing sources. + | Group | Feature | Audience | Mechanism | Status | Source | Action | |---|---|---|---|---|---|---| | UBB readiness | Usage-based billing and AI Credits | Admin, finance | Budgeting and consumption model | Effective June 1, 2026 | https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/ | Prepare budgets, reports, and escalation paths. | | UBB readiness | GitHub Learn UBB module | Admin, enablement | UBB education | Published | https://learn.github.com/courses/gitHubusagebasedbillingmodule | Use as the first readiness resource. | | UBB readiness | April reports | Admin, finance | Baseline reporting | Available | https://github.blog/changelog/2026-05-12-april-reports-are-now-available-to-prepare-for-usage-based-billing | Pull baseline reports before policy changes. | | UBB readiness | Model pricing for GitHub Copilot | Admin, finance | Copilot-specific per-token model pricing converted to AI Credits | Docs | https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing | Use GitHub Copilot billing docs for model-cost discussions. | +| UBB readiness | Code completions and Next Edit Suggestions caveat | Developer, admin | Clarifies what is outside AI Credit billing for paid Copilot plans | Docs | https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing#code-completions | Reduce developer anxiety around normal completions. | +| UBB readiness | Copilot code review billing caveat | Admin, developer | Separates AI Credits and Actions-minute considerations | Docs | https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing#pricing-and-usage-cost-considerations-for-copilot-code-review | Track code review separately from chat and agent usage. | +| UBB readiness | Budget controls | Admin, finance | User-level budgets, hard stops, alerts, and overrides | Docs | https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls | Set defaults, power-user overrides, and alert ownership. | +| UBB readiness | Budget optimization | Admin, finance | Budget sizing and ongoing review | Docs | https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration | Review historical usage before tightening policy. | +| UBB readiness | GitHub Well-Architected AI credit management | Admin, FinOps | Layered budgets, attribution, and governance | Guidance | https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/ | Pair spend controls with model/context guidance. | | Model and policy | Auto model selection | Developer, platform | Task-aware routing and intent detection | Available | https://github.blog/changelog/2026-05-20-auto-model-selection-now-routes-based-on-your-task-in-vs-code | Prefer policy-aware routing over manual guessing. | | Model and policy | Auto model selection docs | Developer, platform | Routing semantics | Docs | https://docs.github.com/en/copilot/concepts/auto-model-selection | Teach what Auto does and does not prove. | | Model and policy | Model rules | Admin, platform | Organization model policy | PREVIEW | https://github.blog/changelog/2026-05-26-target-copilot-models-to-organizations-with-model-rules | Target model availability by policy while preview scope applies. | @@ -17,12 +26,14 @@ This is a first-party-source-backed inventory of features related to cost manage | Observability | GitHub-owned report URLs | Admin, finance | Reporting integration | Available | https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls | Stabilize report retrieval. | | Governance | Cloud-agent audit API | Admin, compliance | Auditability | PREVIEW | https://github.blog/changelog/2026-05-18-audit-repository-copilot-cloud-agent-configuration-via-the-rest-api | Audit repository cloud-agent settings before broad rollout. | | Observability | Copilot SDK OpenTelemetry | Platform, observability | Trace context and telemetry | Docs | https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry | Instrument custom agents; use telemetry as engineering signal, not billing proof. | +| Observability | OpenTelemetry GenAI semantic conventions | Platform, observability | Common vocabulary for GenAI traces | Spec | https://opentelemetry.io/docs/specs/semconv/gen-ai/ | Align custom instrumentation with standard GenAI attributes. | | Preview watchlist | Copilot SDK | Platform | Agent application surface | PREVIEW | https://github.blog/changelog/2026-04-02-copilot-sdk-in-public-preview | Pair SDK adoption with telemetry and preview controls. | | Developer efficiency | Prompt caching in VS Code | Developer | Stable context reuse | Available | https://code.visualstudio.com/updates/v1_118 | Treat as product behavior and engineering signal, not billing proof. | | Developer efficiency | Tool search in VS Code | Developer | Tool-selection efficiency | Available | https://code.visualstudio.com/updates/v1_119 | Scope tools and observe behavior. | | Developer efficiency | Terminal output compression | Developer, SRE | Context compression | PREVIEW | https://code.visualstudio.com/updates/v1_121 | Use for noisy terminal sessions where preview controls are acceptable. | | Model and policy | BYOK for VS Code Business and Enterprise | Developer, platform | Bring your own provider key into VS Code chat | Surface-specific | https://code.visualstudio.com/updates/v1_117 | Specify surface and billing/reporting context before making cost claims. | | Developer efficiency | Reasoning effort controls | Developer, platform | Model effort control | Surface-specific | https://code.visualstudio.com/updates/v1_121 | Test phase-by-phase and avoid generic savings claims. | +| Developer efficiency | GitHub optimize AI usage guide | Developer, team lead | Prompt clarity, phase separation, guardrails, and model/task fit | Docs | https://docs.github.com/en/copilot/tutorials/optimize-ai-usage | Use as the first developer enablement reference. | | Observability | Agent Debug and troubleshooting | Developer | Session visibility | Available | https://code.visualstudio.com/updates/v1_116 | Inspect behavior before optimizing. | | Observability | Chronicle session history | Developer, lead | Session recall and search | CLI release | https://github.com/github/copilot-cli/releases/tag/v1.0.49 | Use for local learning; keep cost-tips claims out unless public docs become explicit. | | Observability | Copilot CLI AI Credits display | Developer, admin | Cost visibility in CLI | CLI release | https://github.com/github/copilot-cli/releases/tag/v1.0.51 | Treat as visibility signal, not invoice proof. | @@ -37,6 +48,39 @@ This is a first-party-source-backed inventory of features related to cost manage | Security and governance | MCP secret scanning | Security | Tool ecosystem trust | GA | https://github.blog/changelog/2026-05-05-secret-scanning-with-github-mcp-server-is-now-generally-available | Include in MCP governance. | | Security and governance | MCP dependency scanning | Security | Tool ecosystem trust | PREVIEW | https://github.blog/changelog/2026-05-05-dependency-scanning-with-github-mcp-server-is-in-public-preview | Review MCP tool risk. | +## Provider Token Mechanics References + +These sources explain token economics and workflow design concepts that can inform custom agent systems. They are not authoritative for GitHub Copilot charges. + +| Provider | Topic | Source | Use | +|---|---|---|---| +| OpenAI | Pricing | https://developers.openai.com/api/docs/pricing | Understand direct API price structures for non-Copilot workflows. | +| OpenAI | Cost optimization | https://developers.openai.com/api/docs/guides/cost-optimization | Compare general levers such as model choice, batching, caching, and output control. | +| OpenAI | Prompt caching | https://developers.openai.com/api/docs/guides/prompt-caching | Understand why stable repeated prefixes can matter. | +| OpenAI | Conversation state | https://developers.openai.com/api/docs/guides/conversation-state | Manage long-running context and state. | +| OpenAI | Reasoning models | https://developers.openai.com/api/docs/guides/reasoning | Scope reasoning use to tasks that need it. | +| OpenAI | Evals | https://developers.openai.com/api/docs/guides/evals | Pair cost changes with quality measurement. | +| Anthropic | Pricing | https://platform.claude.com/docs/en/about-claude/pricing | Understand direct API price categories for non-Copilot workflows. | +| Anthropic | Prompt caching | https://platform.claude.com/docs/en/build-with-claude/prompt-caching | Understand cache-aware prompt design. | +| Anthropic | Token counting | https://platform.claude.com/docs/en/build-with-claude/token-counting | Estimate prompt and response size in custom workflows. | +| Anthropic | Context windows | https://platform.claude.com/docs/en/build-with-claude/context-windows | Design context floors and compaction rules. | +| Anthropic | Extended thinking | https://platform.claude.com/docs/en/build-with-claude/extended-thinking | Treat thinking/reasoning controls as task-specific. | +| Google Gemini | Pricing | https://ai.google.dev/gemini-api/docs/pricing | Understand direct Gemini API cost categories for non-Copilot workflows. | +| Google Gemini | Context caching | https://ai.google.dev/gemini-api/docs/caching | Understand cache lifecycle and repeated-context patterns. | +| Google Gemini | Token counting | https://ai.google.dev/gemini-api/docs/tokens | Estimate token use in custom Gemini workflows. | +| Google Gemini | Thinking | https://ai.google.dev/gemini-api/docs/thinking | Scope thinking budgets to task complexity. | +| Google Gemini | Batch API | https://ai.google.dev/gemini-api/docs/batch-api | Consider batch processing for offline custom workflows. | + +## Workflow Guidance Map + +| Question | First stop | Why | +|---|---|---| +| What is billed in Copilot? | GitHub Copilot models and pricing docs | GitHub is authoritative for AI Credits and Copilot feature behavior. | +| How should admins set controls? | Budget controls and Well-Architected AI credit management | Budget layers, overrides, and ownership decisions are operational. | +| How should developers reduce waste? | GitHub optimize AI usage guide plus [Customer Companion](CUSTOMER_COMPANION.md) | Workflow design, scoped context, and validation reduce rework. | +| How should custom agents be instrumented? | Copilot SDK OpenTelemetry and OpenTelemetry GenAI conventions | Custom harnesses need traceability before making cost claims. | +| How do provider concepts transfer? | Provider token mechanics references | Transfer concepts, not Copilot billing conclusions. | + ## Verification Flags | Item | Status | Bundle handling | @@ -45,3 +89,6 @@ This is a first-party-source-backed inventory of features related to cost manage | Grafana dashboards | No direct public Copilot/Grafana setup source verified | Treat as an observability gap, not a quick hit. | | Prompt caching savings magnitude | Product feature exists, billing impact not proved here | Describe mechanism, not savings. | | Tool search savings magnitude | Product source exists, scenario-specific magnitude unclear | Describe scoped tool efficiency, not universal token reduction. | +| Provider API pricing | Not Copilot billing authority | Use only for direct API or general token-mechanics context. | +| Model availability claims | High-change-rate surface | Re-check supported-models docs or changelog at publication time. | +| Legal, DPA, indemnity, or preview terms | Requires exact current source and owner review | Keep out of quick-hit recommendations unless verified. | diff --git a/release_bundle/2026-05_newsletter_cost_optimization/PUBLIC_SOURCES.md b/release_bundle/2026-05_newsletter_cost_optimization/PUBLIC_SOURCES.md new file mode 100644 index 0000000..7f3f661 --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/PUBLIC_SOURCES.md @@ -0,0 +1,82 @@ +# Public Sources + +> **Bottom line:** Every link in this bundle gathered in one place, grouped by topic. **For:** anyone who wants the canonical source list. **Read time:** reference only. + +All links in this bundle are customer-accessible public sources. + +## Billing, Budgets, And Usage + +- [GitHub Copilot is moving to usage-based billing](https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/) +- [Models and pricing for GitHub Copilot](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) +- [Getting started with budget controls](https://docs.github.com/en/copilot/tutorials/budgets/getting-started-with-budget-controls) +- [Optimizing your budget configuration](https://docs.github.com/en/copilot/tutorials/budgets/optimizing-your-budget-configuration) +- [Managing AI credits, GitHub Well-Architected](https://wellarchitected.github.com/library/governance/recommendations/managing-ai-credits/) +- [Improving agent quality to optimize AI usage](https://docs.github.com/en/copilot/tutorials/optimize-ai-usage) +- [April reports are now available to prepare for usage-based billing](https://github.blog/changelog/2026-05-12-april-reports-are-now-available-to-prepare-for-usage-based-billing) +- [Copilot usage metrics reports now use GitHub-owned download URLs](https://github.blog/changelog/2026-05-20-copilot-usage-metrics-reports-now-use-github-owned-download-urls) +- [Copilot usage metrics REST API](https://docs.github.com/rest/copilot/copilot-usage-metrics) +- [Using Copilot code review](https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review) + +## Copilot Features And Status + +- [Auto model selection now routes based on your task in VS Code](https://github.blog/changelog/2026-05-20-auto-model-selection-now-routes-based-on-your-task-in-vs-code) +- [Auto model selection docs](https://docs.github.com/en/copilot/concepts/auto-model-selection) +- [Target Copilot models to organizations with model rules](https://github.blog/changelog/2026-05-26-target-copilot-models-to-organizations-with-model-rules) +- [GitHub Copilot app is now available in technical preview](https://github.blog/changelog/2026-05-14-github-copilot-app-is-now-available-in-technical-preview/) +- [Copilot SDK in public preview](https://github.blog/changelog/2026-04-02-copilot-sdk-in-public-preview) +- [Copilot Spaces API now generally available](https://github.blog/changelog/2026-05-18-copilot-spaces-api-now-generally-available) +- [Semantic issue search in Copilot Chat](https://github.blog/changelog/2026-05-20-semantic-issue-search-in-copilot-chat) +- [Remote control for Copilot CLI sessions now generally available](https://github.blog/changelog/2026-05-18-remote-control-for-copilot-cli-sessions-now-generally-available-on-mobile-web-and-vs-code) + +## Observability And Workflow Controls + +- [OpenTelemetry instrumentation for Copilot SDK](https://docs.github.com/en/copilot/how-tos/copilot-sdk/observability/opentelemetry) +- [VS Code token efficiency notes](https://code.visualstudio.com/updates/v1_118) +- [VS Code OpenTelemetry and agent observability notes](https://code.visualstudio.com/updates/v1_119) +- [VS Code terminal controls notes](https://code.visualstudio.com/updates/v1_121) +- [VS Code Tool Search source](https://code.visualstudio.com/updates/v1_119) +- [Agent harnesses with GitHub Copilot in VS Code](https://code.visualstudio.com/blogs/2026/05/15/agent-harnesses-github-copilot-vscode) +- [VS Code release notes](https://code.visualstudio.com/updates) + +## Webinars And Enablement + +- [Understanding Budgets Webinar](https://www.youtube.com/watch?v=F2rJ55VVf44) +- [Token Optimization Webinar](https://www.youtube.com/watch?v=0IOcPqubpMc) + +## Provider Token Mechanics + +These links explain general token, context, caching, reasoning, and batch concepts for direct provider or custom-agent workflows. GitHub Copilot billing questions should use the GitHub Copilot billing sources above. + +- [OpenAI pricing](https://developers.openai.com/api/docs/pricing) +- [OpenAI cost optimization](https://developers.openai.com/api/docs/guides/cost-optimization) +- [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) +- [OpenAI conversation state](https://developers.openai.com/api/docs/guides/conversation-state) +- [OpenAI reasoning models](https://developers.openai.com/api/docs/guides/reasoning) +- [OpenAI evals](https://developers.openai.com/api/docs/guides/evals) +- [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing) +- [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) +- [Anthropic token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting) +- [Anthropic context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows) +- [Anthropic extended thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) +- [Google Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing) +- [Google Gemini context caching](https://ai.google.dev/gemini-api/docs/caching) +- [Google Gemini token counting](https://ai.google.dev/gemini-api/docs/tokens) +- [Google Gemini thinking](https://ai.google.dev/gemini-api/docs/thinking) +- [Google Gemini Batch API](https://ai.google.dev/gemini-api/docs/batch-api) +- [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) + +## Governance, Security, And Platform Controls + +- [Audit repository Copilot cloud agent configuration via the REST API](https://github.blog/changelog/2026-05-18-audit-repository-copilot-cloud-agent-configuration-via-the-rest-api) +- [Start Copilot cloud agent tasks via the REST API](https://github.blog/changelog/2026-05-13-start-copilot-cloud-agent-tasks-via-the-rest-api) +- [Secret scanning with GitHub MCP Server is now generally available](https://github.blog/changelog/2026-05-05-secret-scanning-with-github-mcp-server-is-now-generally-available) +- [Dependency scanning with GitHub MCP Server is in public preview](https://github.blog/changelog/2026-05-05-dependency-scanning-with-github-mcp-server-is-in-public-preview) +- [Code-to-cloud risk visibility with Microsoft Defender for Cloud is now generally available](https://github.blog/changelog/2026-05-05-code-to-cloud-risk-visibility-with-microsoft-defender-for-cloud-is-now-generally-available) +- [GitHub Code Quality repository enablement API](https://github.blog/changelog/2026-05-26-github-code-quality-repository-enablement-api) +- [Code coverage in pull requests is now in public preview](https://github.blog/changelog/2026-05-26-code-coverage-in-pull-requests-is-now-in-public-preview) +- [Enterprise Live Migrations is now in public preview](https://github.blog/changelog/2026-05-07-enterprise-live-migrations-is-now-in-public-preview) +- [New enterprise installation API now in public preview](https://github.blog/changelog/2026-05-13-new-enterprise-installation-api-now-in-public-preview) +- [Staged publishing and new install-time controls for npm](https://github.blog/changelog/2026-05-22-staged-publishing-and-new-install-time-controls-for-npm) +- [Expanded OIDC support for Dependabot and code scanning](https://github.blog/changelog/2026-05-19-expanded-oidc-support-for-dependabot-and-code-scanning) +- [Custom images for GitHub-hosted runners are now generally available](https://github.blog/changelog/2026-03-26-custom-images-for-github-hosted-runners-are-now-generally-available) +- [GitHub Enterprise Server signing key rotation](https://github.blog/security/investigating-unauthorized-access-to-githubs-internal-repositories/) diff --git a/release_bundle/2026-05_newsletter_cost_optimization/README.md b/release_bundle/2026-05_newsletter_cost_optimization/README.md new file mode 100644 index 0000000..2953c44 --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/README.md @@ -0,0 +1,24 @@ +# May 2026 Cost-Aware Copilot Usage Bundle + +> **Bottom line:** A customer-facing bundle on UBB readiness and cost-aware Copilot usage; start at the companion and branch by role. The headline result behind it: re-engineering this newsletter's own AI workflow cut roughly 27-45% of the direct provider tokens per run while quality gates held. **For:** developers and Admin & FinOps owners. **Read time:** 1 minute for this map. + +This customer-facing bundle accompanies the May 2026 GitHub Customer Newsletter. It focuses on UBB readiness, cost-aware Copilot usage, workflow design, product controls, and first-party source links. + +The shipped newsletter links to [CUSTOMER_COMPANION.md](CUSTOMER_COMPANION.md), so that file is the canonical landing page. [START_HERE.md](START_HERE.md) is a thin pointer back to the companion for readers browsing this directory. + +## Read Path + +| Need | File | Use when | +|---|---|---| +| Start here | [CUSTOMER_COMPANION.md](CUSTOMER_COMPANION.md) | You followed the shipped newsletter link or want the canonical companion and role routing. | +| Directory pointer | [START_HERE.md](START_HERE.md) | You are browsing the directory and want a quick pointer to the companion. | +| Developer enablement | [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md) | You are shaping team guidance for agentic workflows, with worked examples. | +| Admin and FinOps planning | [ADMIN_FINOPS_GUIDE.md](ADMIN_FINOPS_GUIDE.md) | You own budgets, reporting, policy, governance, attribution, or showback. | +| Technical deep dive | [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md) | You want the system changes, exact before/after token counts, code links, and an illustrative cost translation. | +| Product/source audit | [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md) | You need the feature inventory, source links, and provider token-mechanics references. | +| Newsletter excerpt | [NEWSLETTER_MAY_COST_SECTION.md](NEWSLETTER_MAY_COST_SECTION.md) | You want the exact newsletter section this bundle expands. | +| Source list | [PUBLIC_SOURCES.md](PUBLIC_SOURCES.md) | You want all public first-party links in one place. | + +## Boundary + +This bundle does not claim billing savings, durable percentage savings, model superiority, or fleet-wide transfer. Treat telemetry, usage reports, and workflow observations as engineering signals unless you are working from invoice-grade billing data. Any dollar figures in this bundle are illustrative API-equivalent estimates on direct provider tokens, not GitHub Copilot AI Credit bills. Where a feature is marked `GA` or `PREVIEW`, the label is tied to the linked first-party source. diff --git a/release_bundle/2026-05_newsletter_cost_optimization/START_HERE.md b/release_bundle/2026-05_newsletter_cost_optimization/START_HERE.md new file mode 100644 index 0000000..6f3de8f --- /dev/null +++ b/release_bundle/2026-05_newsletter_cost_optimization/START_HERE.md @@ -0,0 +1,16 @@ +# Start Here + +> **Bottom line:** The canonical landing page for this bundle is the customer companion. **For:** developers and Admin & FinOps owners browsing the release bundle directory. **Read time:** under 1 minute. + +The May 2026 cost-aware Copilot bundle is led by **[CUSTOMER_COMPANION.md](CUSTOMER_COMPANION.md)**. The shipped newsletter links there, and it carries the role routing, reading paths, and guidance for the whole bundle. + +Open [CUSTOMER_COMPANION.md](CUSTOMER_COMPANION.md) and use its **START HERE** section to route by role and time. + +Quick links: + +- [README.md](README.md) — bundle map and boundaries +- [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md) — cost-aware agentic workflows and worked examples +- [ADMIN_FINOPS_GUIDE.md](ADMIN_FINOPS_GUIDE.md) — budgets, reporting, governance, and FinOps +- [NEWSLETTER_SYSTEM_RELEASE_NOTES.md](NEWSLETTER_SYSTEM_RELEASE_NOTES.md) — technical deep dive with exact before/after numbers +- [PRODUCT_FEATURE_QUICK_HITS.md](PRODUCT_FEATURE_QUICK_HITS.md) — product feature inventory +- [PUBLIC_SOURCES.md](PUBLIC_SOURCES.md) — all public first-party links diff --git a/tools/public_snapshot_allowlist.txt b/tools/public_snapshot_allowlist.txt index c0d5086..bd35f34 100644 --- a/tools/public_snapshot_allowlist.txt +++ b/tools/public_snapshot_allowlist.txt @@ -25,11 +25,17 @@ output/.gitkeep workspace/.gitkeep workspace/archived/.gitkeep -# Public-safe release bundle layers only. Do not publish raw SOURCE_NOTES or -# internal evidence directories from release bundles. +# Public-safe release bundle layers only. Do not publish raw source-note or +# non-public evidence directories from release bundles. release_bundle/2026-02_newsletter_launch/public/ release_bundle/2026-04_newsletter_launch/START_HERE.md release_bundle/2026-04_newsletter_launch/run_april_production.sh +release_bundle/2026-05_newsletter_cost_optimization/README.md +release_bundle/2026-05_newsletter_cost_optimization/START_HERE.md release_bundle/2026-05_newsletter_cost_optimization/CUSTOMER_COMPANION.md +release_bundle/2026-05_newsletter_cost_optimization/ADMIN_FINOPS_GUIDE.md +release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_GUIDE.md release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURE_QUICK_HITS.md release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_SYSTEM_RELEASE_NOTES.md +release_bundle/2026-05_newsletter_cost_optimization/NEWSLETTER_MAY_COST_SECTION.md +release_bundle/2026-05_newsletter_cost_optimization/PUBLIC_SOURCES.md diff --git a/tools/public_snapshot_prune.txt b/tools/public_snapshot_prune.txt index 11df3f1..15fa4e3 100644 --- a/tools/public_snapshot_prune.txt +++ b/tools/public_snapshot_prune.txt @@ -48,6 +48,16 @@ release_bundle/2026-02_newsletter_launch/SOURCE_NOTES/ release_bundle/2026-04_newsletter_launch/SOURCE_NOTES/ release_bundle/2026-05_newsletter_cost_optimization/SOURCE_NOTES/ +# May cost bundle files retired during the Phase 2 consolidation (13 -> 9). +# Their content was merged into ADMIN_FINOPS_GUIDE.md, DEVELOPER_GUIDE.md, and +# the CUSTOMER_COMPANION executive-summary section. Prune the old target copies. +release_bundle/2026-05_newsletter_cost_optimization/EXECUTIVE_SUMMARY.md +release_bundle/2026-05_newsletter_cost_optimization/ADMIN_READINESS_GUIDE.md +release_bundle/2026-05_newsletter_cost_optimization/DEVELOPER_WORKFLOW_GUIDE.md +release_bundle/2026-05_newsletter_cost_optimization/FINOPS_PLAYBOOK.md +release_bundle/2026-05_newsletter_cost_optimization/WORKED_EXAMPLES.md +release_bundle/2026-05_newsletter_cost_optimization/PRODUCT_FEATURES_FOR_COST_MANAGEMENT.md + # Internal cost/proof helper surfaces are retained in the private source repo. tools/adjudicate_guard_replayability.py tools/build_phase3_compact_working_set_gate_packet.py