Skip to content

Commit bbf6303

Browse files
paulmedynskipaulmedynski
committed
Move symbol publishing into a dedicated stage and refactor build artifacts
- Extract symbol publishing from build jobs into a new publish_symbols stage (publish-symbols-stage.yml) with per-package jobs (publish-symbols-job.yml) - Reorganize JOB_OUTPUT subdirectories: assemblies/ (APIScan), packages/ (NuGet), symbols/ (PDBs shared by APIScan and symbol publishing) - Point APIScan at JOB_OUTPUT/assemblies and JOB_OUTPUT/symbols instead of separate apiScan/ tree; remove copy-apiscan-files-sqlclient-step.yml - Rename PACK_OUTPUT -> JOB_OUTPUT, PACK_INPUT -> JOB_INPUT for clarity - Extract inline PowerShell into Publish-Symbols.ps1 with structured error handling; add Pester tests - Rename variable group symbols-variables-v2 -> Symbols Publishing - Remove artifactPath variable from validate job in favor of packagesPath - Append $(System.JobAttempt) to symbol artifact names for retry safety - Guard publish_symbols stage against empty runs when no packages were built
1 parent 6db52a6 commit bbf6303

20 files changed

Lines changed: 916 additions & 321 deletions

‎.github/instructions/onebranch-pipeline-design.instructions.md‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,8 @@ Defined in `stages/build-stages.yml`. Four build stages plus validation, ordered
3636
- **`build_addons`** (Stage 4) — AKV Provider; `dependsOn: build_dependent`; downloads SqlClient + Abstractions + Logging artifacts
3737
- **`mds_package_validation`** — Validates signed SqlClient package; `dependsOn: build_dependent`; runs in parallel with Stage 4
3838

39+
Each build job copies PDB files into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published pipeline artifact alongside the NuGet packages in `$(JOB_OUTPUT)/packages/`.
40+
3941
Stage conditional rules:
4042
- Wrap stages/jobs in `${{ if }}` compile-time conditionals based on build parameters
4143
- `buildSqlClient` controls Stages 2, 3, validation, and Logging (when AKV is disabled)
@@ -45,17 +47,30 @@ Stage conditional rules:
4547

4648
## Job Templates
4749

48-
- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing
49-
- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing
50+
- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing → Copy PDBs to artifact
51+
- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing → Copy PDBs to artifact
5052
- **`validate-signed-package-job.yml`** — Validates signed MDS package (signature, strong names, folder structure, target frameworks)
5153
- **`publish-nuget-package-job.yml`** — Reusable release job using OneBranch `templateContext.type: releaseJob` with `inputs` for artifact download; pushes via `NuGetCommand@2`
54+
- **`publish-symbols-job.yml`** — Reusable symbols job: downloads a build artifact, locates PDBs under `symbols/`, and invokes `publish-symbols-step.yml`
5255

5356
When adding a new csproj-based package:
5457
- Use `build-signed-csproj-package-job.yml` with appropriate `packageName`, `packageFullName`, `versionProperties`, and `downloadArtifacts`
5558
- Add build and pack targets to `build.proj`
5659
- Add version variables to `variables/common-variables.yml`
5760
- Add artifact name variable to `variables/onebranch-variables.yml`
5861

62+
## Symbols Publishing Stage
63+
64+
- Defined in `stages/publish-symbols-stage.yml`; produces stage `publish_symbols`
65+
- Entire stage excluded at compile time when `publishSymbols` is false
66+
- `dependsOn` is conditional based on which `build*` parameters are set, mirroring the build stage dependency graph
67+
- One job per package (`publish-symbols-job.yml`), each downloading its build artifact and publishing PDBs from `symbols/`
68+
- Each package's PDBs are published separately with unique artifact names and version information
69+
- Build jobs copy PDBs into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published artifact
70+
- The `publish-symbols-step.yml` accepts a `symbolsFolder` parameter to point at the downloaded PDB location
71+
- The publish step calls an extracted `Publish-Symbols.ps1` script with structured error handling and diagnostic logging
72+
- Symbols publishing credentials come from the `Symbols Publishing` variable group
73+
5974
## Release Stage
6075

6176
- Defined in `stages/release-stages.yml`; produces stage `release_production` (official) or `release_test` (non-official) via `stageNameSuffix` parameter
@@ -98,8 +113,7 @@ When `isPreview` is true, pipeline resolves `effective*Version` variables to pre
98113
- When adding a new package, add GA version, preview version, and assembly file version entries
99114

100115
Variable groups:
101-
- `Release Variables` — release configuration (in `common-variables.yml`)
102-
- `Symbols publishing` — symbol publishing credentials (in `common-variables.yml`)
116+
- `Symbols Publishing` — symbol publishing credentials (in `onebranch-variables.yml`)
103117
- `ESRP Federated Creds (AME)` — ESRP signing credentials (in `common-variables.yml`)
104118

105119
## Code Signing (ESRP)
@@ -115,15 +129,19 @@ Variable groups:
115129

116130
- TSA: enabled only in official pipeline; disabled in non-official to avoid spurious alerts
117131
- ApiScan: enabled in both; currently `break: false` pending package registration
118-
- Each build job sets `ob_sdl_apiscan_*` variables pointing to `$(Build.SourcesDirectory)/apiScan/<PackageName>/`
132+
- Each build job sets `ob_sdl_apiscan_softwareFolder` to `$(JOB_OUTPUT)/assemblies` and `ob_sdl_apiscan_symbolsFolder` to `$(JOB_OUTPUT)/symbols`
119133
- CodeQL, SBOM, Policheck (`break: true`): enabled in both pipelines
120134
- asyncSdl `enabled: false` in both; individual sub-tools (CredScan, BinSkim, Armory, Roslyn) configured underneath
121135
- Policheck exclusions: `$(REPO_ROOT)\.config\PolicheckExclusions.xml`
122136
- CredScan suppressions: `$(REPO_ROOT)/.config/CredScanSuppressions.json`
123137

124138
## Artifact Conventions
125139

126-
- `ob_outputDirectory` set to `$(PACK_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
140+
- `ob_outputDirectory` set to `$(JOB_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
141+
- Each published artifact uses subdirectories to separate file types:
142+
- `assemblies/` — DLL assemblies for APIScan (preserving TFM folder structure)
143+
- `packages/` — NuGet packages (`.nupkg`, `.snupkg`)
144+
- `symbols/` — PDB symbol files (preserving TFM folder structure, shared by APIScan and symbol publishing)
127145
- Artifact names follow `drop_<stageName>_<jobName>` — defined in `variables/onebranch-variables.yml`
128146
- Downstream jobs download artifacts via `DownloadPipelineArtifact@2` into `$(Build.SourcesDirectory)/packages`
129147
- Downloaded packages serve as a local NuGet source for `dotnet restore`

‎eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml‎

Lines changed: 13 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -36,26 +36,6 @@ parameters:
3636
- name: signingEsrpConnectedServiceName
3737
type: string
3838

39-
# Symbols Publishing Parameters ------------------------------------------
40-
41-
- name: symbolsAzureSubscription
42-
type: string
43-
default: 'Symbols publishing Workload Identity federation service-ADO.Net'
44-
45-
- name: symbolsPublishProjectName
46-
type: string
47-
default: 'Microsoft.Data.SqlClient.SNI'
48-
49-
- name: symbolsPublishServer
50-
type: string
51-
52-
- name: symbolsPublishTokenUri
53-
type: string
54-
55-
- name: symbolsUploadAccount
56-
type: string
57-
default: 'SqlClientDrivers'
58-
5939
# OTHERS +=====================================
6040

6141
# Short package name used in the job name, display strings, filesystem paths, and as a suffix for
@@ -74,11 +54,6 @@ parameters:
7454
- name: packageFullName
7555
type: string
7656

77-
# The version of the package. This is used for symbol publishing. It is not used for the DLL or
78-
# NuGet package versions since those are supplied via the versionProperties parameter.
79-
- name: packageVersion
80-
type: string
81-
8257
# The MSBuild build target in build.proj (e.g. BuildLogging). If not specified, defaults to
8358
# Build<packageName>.
8459
- name: buildTarget
@@ -106,10 +81,6 @@ parameters:
10681
- name: assemblyFileVersion
10782
type: string
10883

109-
# True to publish symbols to private and public servers.
110-
- name: publishSymbols
111-
type: boolean
112-
11384
# Optional list of pipeline artifacts to download before building. Each entry is an object
11485
# with 'artifactName' (the pipeline artifact name) and 'displayName' (used in the task label).
11586
# This replaces hard-coded packageName conditionals so callers declare their own dependencies.
@@ -125,12 +96,12 @@ jobs:
12596

12697
variables:
12798
# Inform OneBranch that files put in this directory should be uploaded as artifacts.
128-
ob_outputDirectory: $(PACK_OUTPUT)
99+
ob_outputDirectory: $(JOB_OUTPUT)
129100

130101
# APIScan configuration for this Extension package
131102
ob_sdl_apiscan_enabled: true
132-
ob_sdl_apiscan_softwareFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/dlls
133-
ob_sdl_apiscan_symbolsFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/pdbs
103+
ob_sdl_apiscan_softwareFolder: $(JOB_OUTPUT)/assemblies
104+
ob_sdl_apiscan_symbolsFolder: $(JOB_OUTPUT)/symbols
134105
ob_sdl_apiscan_softwarename: ${{ parameters.packageFullName }}
135106
ob_sdl_apiscan_versionNumber: ${{ parameters.assemblyFileVersion }}
136107

@@ -149,7 +120,7 @@ jobs:
149120
displayName: Download ${{ artifact.displayName }}
150121
inputs:
151122
artifactName: ${{ artifact.artifactName }}
152-
targetPath: $(REPO_ROOT)/packages
123+
targetPath: $(JOB_INPUT)
153124

154125
# Install the .NET SDK.
155126
- template: /eng/pipelines/steps/install-dotnet.yml@self
@@ -181,24 +152,27 @@ jobs:
181152
esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
182153
pattern: '${{ parameters.packageFullName }}.dll'
183154

184-
# Copy signed/unsigned DLLs and PDBs to APIScan folders.
155+
# Copy DLLs to the assemblies/ subdirectory for APIScan.
185156
- task: CopyFiles@2
186157
displayName: Copy DLLs for APIScan
187158
inputs:
188159
SourceFolder: $(BUILD_OUTPUT)/Package/bin
189160
Contents: "**/${{ parameters.packageFullName }}.dll"
190-
TargetFolder: ${{ variables.ob_sdl_apiscan_softwareFolder }}
161+
TargetFolder: $(JOB_OUTPUT)/assemblies
191162
# We must preserve the folder structure since our C# projects may produce multiple
192163
# identically named DLLs for different target frameworks (e.g. netstandard2.0, net5.0,
193164
# etc.), and we need to keep those separate for APIScan to work correctly.
194165
flattenFolders: false
195166

167+
# Copy PDBs into the output directory so they are included in the published pipeline
168+
# artifact. The symbols publishing stage will download this artifact and publish PDBs
169+
# for this package using the files under symbols/.
196170
- task: CopyFiles@2
197-
displayName: Copy PDBs for APIScan
171+
displayName: Copy PDBs for APIScan and Symbols Publishing
198172
inputs:
199173
SourceFolder: $(BUILD_OUTPUT)/Package/bin
200-
Contents: "**/${{ parameters.packageFullName }}.pdb"
201-
TargetFolder: ${{ variables.ob_sdl_apiscan_symbolsFolder }}
174+
Contents: '**/${{ parameters.packageFullName }}.pdb'
175+
TargetFolder: $(JOB_OUTPUT)/symbols
202176
flattenFolders: false
203177

204178
# Pack the signed DLLs into NuGet package (NoBuild=true).
@@ -217,22 +191,5 @@ jobs:
217191
authSignCertName: '${{ parameters.signingAuthSignCertName }}'
218192
esrpClientId: '${{ parameters.signingEsrpClientId }}'
219193
esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
220-
searchPath: $(PACK_OUTPUT)
194+
searchPath: $(JOB_OUTPUT)/packages
221195
searchPattern: '${{ parameters.packageFullName}}.*nupkg'
222-
223-
# Publish symbols to servers
224-
# @TODO: Get these parameters from variables/libraries
225-
- ${{ if eq(parameters.publishSymbols, true) }}:
226-
- template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self
227-
parameters:
228-
artifactName: '${{ parameters.packageFullName }}_symbols_$(System.TeamProject)_$(Build.Repository.Name)_$(Build.SourceBranchName)_${{ parameters.packageVersion }}_$(System.TimelineId)'
229-
azureSubscription: '${{ parameters.symbolsAzureSubscription }}'
230-
packageName: '${{ parameters.packageFullName }}'
231-
publishProjectName: '${{ parameters.symbolsPublishProjectName }}'
232-
publishServer: '${{ parameters.symbolsPublishServer }}'
233-
publishToInternal: 'true'
234-
publishToPublic: 'true'
235-
publishTokenUri: '${{ parameters.symbolsPublishTokenUri }}'
236-
searchPattern: '**/${{ parameters.packageFullName }}*.pdb'
237-
uploadAccount: '${{ parameters.symbolsUploadAccount }}'
238-
version: '${{ parameters.packageVersion }}'

‎eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml‎

Lines changed: 26 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,11 @@
77
# Job that performs a build of Microsoft.Data.SqlClient using the build2.proj file.
88

99
parameters:
10-
- name: apiScanDllPath
11-
type: string
12-
13-
- name: apiScanPdbPath
14-
type: string
15-
1610
# Whether this build is for an "official" OneBranch pipeline. This is used to enable ESRP signing
1711
# on the artifacts of this job.
1812
- name: isOfficial
1913
type: boolean
2014

21-
- name: publishSymbols
22-
type: boolean
23-
2415
- name: signingAppRegistrationClientId
2516
type: string
2617

@@ -39,21 +30,6 @@ parameters:
3930
- name: signingEsrpConnectedServiceName
4031
type: string
4132

42-
- name: symbolsAzureSubscription
43-
type: string
44-
45-
- name: symbolsPublishProjectName
46-
type: string
47-
48-
- name: symbolsPublishServer
49-
type: string
50-
51-
- name: symbolsPublishTokenUri
52-
type: string
53-
54-
- name: symbolsUploadAccount
55-
type: string
56-
5733
# Package Parameters
5834
- name: abstractionsArtifactName
5935
type: string
@@ -80,9 +56,9 @@ jobs:
8056
type: windows
8157

8258
variables:
83-
ob_outputDirectory: '$(PACK_OUTPUT)'
84-
ob_sdl_apiscan_softwareFolder: ${{ parameters.apiScanDllPath }}
85-
ob_sdl_apiscan_symbolsFolder: ${{ parameters.apiScanPdbPath }}
59+
ob_outputDirectory: '$(JOB_OUTPUT)'
60+
ob_sdl_apiscan_softwareFolder: $(JOB_OUTPUT)/assemblies
61+
ob_sdl_apiscan_symbolsFolder: $(JOB_OUTPUT)/symbols
8662
ob_sdl_apiscan_softwarename: 'Microsoft.Data.SqlClient'
8763
ob_sdl_apiscan_versionNumber: ${{ parameters.sqlClientAssemblyFileVersion }}
8864

@@ -102,13 +78,13 @@ jobs:
10278
displayName: Download Microsoft.Data.SqlClient.Extensions.Abstractions Artifact
10379
inputs:
10480
artifactName: '${{ parameters.abstractionsArtifactName }}'
105-
targetPath: '$(REPO_ROOT)/packages'
81+
targetPath: '$(JOB_INPUT)'
10682

10783
- task: DownloadPipelineArtifact@2
10884
displayName: Download Microsoft.Data.SqlClient.Extensions.Logging Artifact
10985
inputs:
11086
artifactName: '${{ parameters.loggingArtifactName }}'
111-
targetPath: '$(REPO_ROOT)/packages'
87+
targetPath: '$(JOB_INPUT)'
11288

11389
# Install the .NET SDK
11490
- template: /eng/pipelines/steps/install-dotnet.yml@self
@@ -127,13 +103,6 @@ jobs:
127103
loggingPackageVersion: '${{ parameters.loggingPackageVersion }}'
128104
sqlClientPackageVersion: '${{ parameters.sqlClientPackageVersion }}'
129105

130-
# Copy the built DLLs and PDBs to the APIScan output folder for APIScanning post-build
131-
- template: /eng/pipelines/onebranch/steps/copy-apiscan-files-sqlclient-step.yml@self
132-
parameters:
133-
dllPath: '${{ parameters.apiScanDllPath }}'
134-
pdbPath: '${{ parameters.apiScanPdbPath }}'
135-
referenceType: 'Package'
136-
137106
# Sign the DLLs
138107
- ${{ if parameters.isOfficial }}:
139108
- template: /eng/pipelines/onebranch/steps/esrp-dll-signing-step.yml@self
@@ -146,6 +115,26 @@ jobs:
146115
esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
147116
pattern: 'Microsoft.Data.SqlClient*.dll'
148117

118+
# Copy DLLs to the assemblies/ subdirectory for APIScan.
119+
- task: CopyFiles@2
120+
displayName: Copy DLLs for APIScan
121+
inputs:
122+
SourceFolder: '$(BUILD_OUTPUT)/Microsoft.Data.SqlClient/Package-Release'
123+
Contents: '**/Microsoft.Data.SqlClient.dll'
124+
TargetFolder: '$(JOB_OUTPUT)/assemblies'
125+
flattenFolders: false
126+
127+
# Copy PDBs into the output directory so they are included in the published pipeline
128+
# artifact. The symbols publishing stage will download this artifact and publish PDBs
129+
# for this package using the files under symbols/.
130+
- task: CopyFiles@2
131+
displayName: Copy PDBs for APIScan and Symbols Publishing
132+
inputs:
133+
SourceFolder: '$(BUILD_OUTPUT)/Microsoft.Data.SqlClient/Package-Release'
134+
Contents: '**/Microsoft.Data.SqlClient*.pdb'
135+
TargetFolder: '$(JOB_OUTPUT)/symbols'
136+
flattenFolders: false
137+
149138
# Package the build output into a NuGet package
150139
- template: /eng/pipelines/onebranch/steps/pack-sqlclient-step.yml
151140
parameters:
@@ -163,20 +152,5 @@ jobs:
163152
authSignCertName: '${{ parameters.signingAuthSignCertName }}'
164153
esrpClientId: '${{ parameters.signingEsrpClientId }}'
165154
esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
166-
searchPath: '$(PACK_OUTPUT)'
155+
searchPath: '$(JOB_OUTPUT)/packages'
167156
searchPattern: 'Microsoft.Data.SqlClient.*nupkg'
168-
169-
- ${{ if parameters.publishSymbols }}:
170-
- template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self
171-
parameters:
172-
artifactName: 'Microsoft.Data.SqlClient_symbols_$(System.TeamProject)_$(Build.Repository.Name)_$(Build.SourceBranchName)_${{ parameters.sqlClientPackageVersion }}_$(System.TimelineId)'
173-
azureSubscription: '${{ parameters.symbolsAzureSubscription }}'
174-
packageName: 'Microsoft.Data.SqlClient'
175-
publishProjectName: '${{ parameters.symbolsPublishProjectName }}'
176-
publishServer: '${{ parameters.symbolsPublishServer }}'
177-
publishToInternal: true
178-
publishToPublic: true
179-
publishTokenUri: '${{ parameters.symbolsPublishTokenUri }}'
180-
searchPattern: '**/Microsoft.Data.SqlClient*.pdb' # @TODO: This seems very heavy
181-
uploadAccount: '${{ parameters.symbolsUploadAccount }}'
182-
version: '${{ parameters.sqlClientPackageVersion }}'

‎eng/pipelines/onebranch/jobs/publish-nuget-package-job.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,13 +65,13 @@ jobs:
6565

6666
variables:
6767
- name: ob_outputDirectory
68-
value: $(PACK_OUTPUT)
68+
value: $(JOB_OUTPUT)
6969

7070
- name: artifactPath
7171
value: $(Pipeline.Workspace)/${{ parameters.artifactName }}
7272

7373
- name: packageToPush
74-
value: $(artifactPath)/${{ coalesce(parameters.packagePath, format('{0}.*.nupkg', parameters.packageName)) }}
74+
value: $(artifactPath)/${{ coalesce(parameters.packagePath, format('packages/{0}.*.nupkg', parameters.packageName)) }}
7575

7676
# Template context inputs are used to pass parameters to the deployment job since it doesn't
7777
# automatically download pipeline artifacts.

0 commit comments

Comments
 (0)