Skip to content

Route NuGet restore through CFS feed (SR21 CFSClean) - #1756

Open
bragi92 wants to merge 1 commit into
ci_prodfrom
kaveesh/sr21-cfs-nuget
Open

Route NuGet restore through CFS feed (SR21 CFSClean)#1756
bragi92 wants to merge 1 commit into
ci_prodfrom
kaveesh/sr21-cfs-nuget

Conversation

@bragi92

@bragi92 bragi92 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What

Routes NuGet package restore for the Windows agent build through the CFS-backed Azure Artifacts feed, instead of api.nuget.org.

Why

Pipeline 444 (ContainerInsights-MultiArch-MergedBranches) is flagged under MountainPass SR21 / SFI-ES4.2.4 — ICM 839785317, story 38937265.

1ES policy telemetry (GetPolicyViolationDetails(adoOrg='github-private', pipelineID='444')) shows the only CFSClean endpoint the pipeline reaches is api.nuget.org, from C:\Program Files\dotnet\dotnet.exe during the build base step on build_windows_2019 and build_windows_2022. Container Insights is in the Non-TCB cohort (HasTCB = false), which only has to satisfy the CFSClean policy — so this is the complete set of required changes.

Changes

  1. NuGet.config at repo root<clear /> plus a single source: the CFS-backed microsoft_PublicPackages project-scoped feed, which already has a NuGet Gallery upstream configured. There was no NuGet.config anywhere in the repo, so restore was defaulting to api.nuget.org.

  2. build/windows/Makefile.ps1 — removed:

    dotnet add package Newtonsoft.json
    dotnet add package BouncyCastle

    Both packages are already pinned as PackageReference in CertificateGenerator.csproj (Newtonsoft.Json 13.0.1, BouncyCastle 1.8.9). dotnet add package without a version contacts the package source to resolve latest and then rewrites the csproj — so these calls both caused the violation and silently un-pinned the dependency versions on every build. No behavior change from removing them.

  3. NuGetAuthenticate@1 added to both Windows jobs — the build shells out to dotnet from a script: step rather than DotNetCoreCLI@2, so nothing configures the credential provider for the feed today.

Validation

Build — PR build 122876: all 4 jobs green, both NuGetAuthenticate@1 tasks succeeded. Restore against the CFS feed resolved the pinned BouncyCastle 1.8.9 with 0 errors and dotnet publish produced CertificateGenerator.exe.

Scope of what can regressCertificateGenerator.csproj is the only .NET project in the repo, so the root NuGet.config reaches nothing else. At runtime main.ps1 only calls Generate-Certificates when USING_AAD_MSI_AUTH != true, so cert-auth mode is the sole path this change can affect (the AKS MSI default skips it entirely). All testing below therefore runs in cert-auth mode (isUsingAADAuth: "false") so the changed codepath actually executes.

Binary-level — reproduced the publish locally with this branch's NuGet.config + Makefile.ps1, then ran the binary: it emitted a valid 2048-bit sha256RSA self-signed cert + RSA key. Output binaries Newtonsoft.Json 13.0.1, BouncyCastle.Crypto 1.8.9, CertificateGenerator.exe.

Backdoor deployment

Built this branch via a manual (non-PR) run of pipeline 444 — build 123005 — and deployed cidev:win-3.6.0-1-g4b845429a-20260807201906 onto a Windows Server 2022 node pool, swapping only the ama-logs-windows daemonset image so the Windows agent is the single changed variable.

Pod Running 1/1, 0 restarts, same cert sequence as the production baseline:

Starting Windows in Cert Auth Mode
Successfully created self-signed certificate for agentGuid : {a1e0747c-...}
waiting response for registration request : OK
Certificate file found at C://oms.crt

Container logs — the data the Windows daemonset itself collects. A Windows workload emitting a marked line every second, compared over equal 10-min steady-state windows on the same node:

Metric (Computer == akswinp000000) Baseline ciprod:win-3.6.0 PR image
ContainerLog rows 587 587
...of which test-marker lines 587 587

Node-level Perf (also daemonset-collected, via cAdvisor on the local kubelet) likewise matched exactly at 305 rows over equal windows.

Earlier revisions of this section also listed ContainerInventory / KubePodInventory / KubeNodeInventory. Per review feedback those are collected by the replicaset, not the Windows daemonset — filtering them by Computer returns rows the (unchanged, Linux) replicaset emitted about the Windows node, so they were not valid evidence for this change and have been dropped.

Build 123005 reports failed, but only build_linux — all three Windows jobs succeeded and pushed. That failure is unrelated to this PR: a GCC ICE (internal compiler error: Segmentation fault ... cc1) compiling zstd-ruby on the QEMU-emulated aarch64 leg. The same commit passed build_linux in 122876, and the job also fails intermittently on ci_prod itself (builds 122434, 122097).

One correction to "Changes" #2 — removing dotnet add package is not fully behavior-neutral. dotnet add package Newtonsoft.json (no version) resolves 13.0.4, so the shipped agent moves 13.0.4 → the pinned 13.0.1. BouncyCastle is genuinely unchanged (1.8.9 is latest). Not a security regression: Newtonsoft's only advisory (CVE-2024-21907 / GHSA-5crp-9r3c-p9vr) has vulnerable range < 13.0.1, so 13.0.1 is the patched floor. Deterministic pinning is the intended outcome, but worth stating rather than "no behavior change".

  • After merge, confirm violations stop with:
    GetPolicyViolationDetails(adoOrg='github-private', pipelineID='444', ago(7d))
    | where PolicyViolation == 'CFSClean'

Out of scope

CFSClean2 (chocolatey, ghcr.io via trivy) and DefaultDeny (google-analytics, segment.io, Docker Desktop) are not required for the Non-TCB cohort and are intentionally left alone.

Resolution criteria

The ICM auto-resolves after zero CFSClean violations across at least 3 runs in a 7-day window, at which point the pipeline is locked into the policy. Deadline is Aug 21, 2026, so this needs to land by ~Aug 13 to leave room for the lock-in window.

Pipeline 444 (ContainerInsights-MultiArch-MergedBranches) is flagged under
MountainPass SR21 / SFI-ES4.2.4 for CFSClean violations. 1ES telemetry shows
the only CFSClean endpoint hit is api.nuget.org, from dotnet.exe during the
'build base' step on build_windows_2019 and build_windows_2022.

Changes:
- Add NuGet.config at repo root with <clear /> and only the CFS-backed
  Azure Artifacts feed (microsoft_PublicPackages), which already has a
  NuGet Gallery upstream. Without this, restore falls back to api.nuget.org.
- Drop 'dotnet add package Newtonsoft.json' and 'dotnet add package
  BouncyCastle' from build/windows/Makefile.ps1. Both are already pinned as
  PackageReference in CertificateGenerator.csproj (13.0.1 / 1.8.9); the
  unversioned 'dotnet add package' calls reach api.nuget.org to resolve the
  latest version and rewrite the csproj, un-pinning the versions at build time.
- Add NuGetAuthenticate@1 to both Windows jobs so the credential provider is
  configured for the feed. The build shells out to dotnet from a script rather
  than DotNetCoreCLI@2, so nothing wires up feed auth today.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ee239d9a-59a5-409c-bf8f-5a1e5d9a9d58
@bragi92
bragi92 requested a review from a team as a code owner August 6, 2026 20:02
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@bragi92

bragi92 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

@bragi92
bragi92 enabled auto-merge (squash) August 7, 2026 00:19
@rashmichandrashekar

Copy link
Copy Markdown
Contributor

Thanks for this pr bragi92 - is this fix working fine e2e? are windows pods coming up fine and sending data successfully?

@bragi92

bragi92 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

@rashmichandrashekar

Copy link
Copy Markdown
Contributor

bragi92 - thanks for adding the test cases. The data that should be tested is containerlog isn’t it? Since the other tables are collected by the replicaset?

@bragi92

bragi92 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

rashmichandrashekar (@rashmichandrashekar) Makes sense. Let me update the tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants