Skip to content

fix(ci-serialisation): report every outcome, and quieten the baseline builds #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sakanni merged 2 commits into from
Aug 25, 2026
Merged

fix(ci-serialisation): report every outcome, and quieten the baseline builds #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffba0b1
test(ci-serialisation): capture three structural defects in the action
sakanni Aug 24, 2026
d7ae5c1
fix(ci-serialisation): report every outcome, and quieten the baseline…
sakanni Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 50 additions & 76 deletions .github/actions/ci-serialisation/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,10 @@ runs:
run: |
git fetch origin ${{ inputs.base_ref }}
git checkout ${{ inputs.base_ref }}
Remove-Item "C:\ProgramData\BHoM\Assemblies\*" -Recurse -Force -ErrorAction SilentlyContinue
# No assemblies reset here. resolve-dependencies removes and recreates both
# ProgramData\BHoM\Assemblies and ProgramData\BHoM\Upgrades in its own Prepare
# folders step, which is the next step and runs unconditionally. A reset here
# would cover one of those two directories and be redundant with the step after it.

# prefer_branch is what makes this a baseline. Without it the head branch is preferred on
# both legs, because resolve-dependencies takes PR_BRANCH from the event payload and that
Expand All @@ -223,7 +226,10 @@ runs:
shell: pwsh
run: |
dotnet restore "${{ steps.solution.outputs.path }}"
dotnet build "${{ steps.solution.outputs.path }}" --no-restore -c Release --nologo -m
# -clp:ErrorsOnly for the same reason as the branch leg above, and with more force:
# this compiles the base branch, so every warning belongs to code the author did
# not write and cannot act on.
dotnet build "${{ steps.solution.outputs.path }}" --no-restore -c Release --nologo -m -clp:ErrorsOnly

# Re-infer against the baseline checkout: Verification.sln may declare
# different Release variants on branch versus base (e.g. the branch
Expand All @@ -240,7 +246,7 @@ runs:
shell: pwsh
run: |
dotnet restore "${{ steps.ver_sln.outputs.path }}"
dotnet build "${{ steps.ver_sln.outputs.path }}" --no-restore -c ${{ steps.ver_config_baseline.outputs.config }} --nologo
dotnet build "${{ steps.ver_sln.outputs.path }}" --no-restore -c ${{ steps.ver_config_baseline.outputs.config }} --nologo -clp:ErrorsOnly

# --- Comparison ---

Expand All @@ -266,83 +272,51 @@ runs:
"verdict=$verdict" | Out-File $env:GITHUB_OUTPUT -Append
if ($code -ne 0) { exit $code }

# Distinguish an infrastructure failure from a serialisation regression. A
# dependency build failure otherwise shows as a red serialisation check with no
# indication that serialisation never ran.
- name: Write to Job Summary (did not run)
if: always() && steps.changed.outputs.count != '0' && (steps.deps_branch.conclusion == 'failure' || steps.build_branch.conclusion == 'failure')
shell: pwsh
run: |
$step = if ("${{ steps.deps_branch.conclusion }}" -eq 'failure') { 'Resolve dependencies (branch)' } else { 'Build primary repo (branch)' }
@(
"### Serialisation did not run"
""
"Failed in: $step"
""
"The dependency closure did not build, so no serialisation comparison was made."
"This is not a serialisation finding. See the ci-build check on this pull request"
"for the same failure."
) | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8

- name: Write to Job Summary (branch passed)
if: always() && steps.changed.outputs.count != '0' && steps.branch_run.conclusion == 'success' && steps.branch_run.outputs.status != 'Error'
shell: pwsh
run: |
@(
"### Serialisation"
""
"| Status |"
"|---|"
"| No serialisation failures (${{ steps.branch_run.outputs.status }}) |"
""
"Objects exercised: ${{ steps.branch_run.outputs.population }} across ${{ steps.branch_run.outputs.assemblies }} loaded assemblies, ${{ steps.branch_run.outputs.legs }} legs. Failures reported by the runner: ${{ steps.branch_run.outputs.failures }}."
""
) | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8

- name: Write to Job Summary (pre-existing failures)
if: always() && steps.changed.outputs.count != '0' && steps.compare.outputs.verdict == 'ok'
# One summary step, reached on every path.
#
# This was five steps, each gated on its own combination of step conclusions and outputs.
# The conditions had to enumerate the states worth reporting, and between them they named
# five of this action's eighteen functional steps, so a failure anywhere else matched no
# condition and the check went red with an empty summary. Seven of the unnamed steps run
# before serialisation is attempted, which is the case a reader is least able to diagnose.
#
# The decision moved to a script so it can be tested without running a serialisation check,
# and so the branch that handles "none of the above" is a branch in code rather than a
# condition nobody wrote.
#
# Values go through env: rather than into the script arguments, so a step output cannot be
# interpolated into a command line.
- name: Write to Job Summary
if: always() && steps.changed.outputs.count != '0'
shell: pwsh
env:
DEPS_BRANCH_CONCLUSION: ${{ steps.deps_branch.conclusion }}
BUILD_BRANCH_CONCLUSION: ${{ steps.build_branch.conclusion }}
BRANCH_RUN_CONCLUSION: ${{ steps.branch_run.conclusion }}
BRANCH_STATUS: ${{ steps.branch_run.outputs.status }}
COMPARE_VERDICT: ${{ steps.compare.outputs.verdict }}
POPULATION: ${{ steps.branch_run.outputs.population }}
ASSEMBLIES: ${{ steps.branch_run.outputs.assemblies }}
LEGS: ${{ steps.branch_run.outputs.legs }}
FAILURES: ${{ steps.branch_run.outputs.failures }}
run: |
@(
"### Serialisation"
""
"| Status |"
"|---|"
"| Serialisation failures detected. All are pre-existing on the base branch, so this job passes. |"
""
"Objects exercised: ${{ steps.branch_run.outputs.population }} across ${{ steps.branch_run.outputs.assemblies }} loaded assemblies, ${{ steps.branch_run.outputs.legs }} legs. Failures reported by the runner: ${{ steps.branch_run.outputs.failures }}."
""
) | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8
# GITHUB_ACTION_PATH is .github/actions/ci-serialisation, so ../../scripts is
# .github/scripts, the same path ci-build and ci-versioning use for shared scripts.
$script = Join-Path $env:GITHUB_ACTION_PATH "../../scripts/Write-SerialisationSummary.ps1"
. $script

- name: Write to Job Summary (regression detected)
if: always() && steps.changed.outputs.count != '0' && steps.compare.outputs.verdict == 'regression'
shell: pwsh
run: |
@(
"### Serialisation"
""
"| Status |"
"|---|"
"| Serialisation regression detected. Affected types are in the step log. |"
""
"Objects exercised: ${{ steps.branch_run.outputs.population }} across ${{ steps.branch_run.outputs.assemblies }} loaded assemblies, ${{ steps.branch_run.outputs.legs }} legs. Failures reported by the runner: ${{ steps.branch_run.outputs.failures }}."
""
) | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8
$lines = Get-SerialisationSummary `
-DepsBranchConclusion $env:DEPS_BRANCH_CONCLUSION `
-BuildBranchConclusion $env:BUILD_BRANCH_CONCLUSION `
-BranchRunConclusion $env:BRANCH_RUN_CONCLUSION `
-BranchStatus $env:BRANCH_STATUS `
-CompareVerdict $env:COMPARE_VERDICT `
-Population $env:POPULATION `
-Assemblies $env:ASSEMBLIES `
-Legs $env:LEGS `
-Failures $env:FAILURES

- name: Write to Job Summary (serialisation unverifiable)
if: always() && steps.changed.outputs.count != '0' && (steps.branch_run.conclusion == 'failure' || steps.compare.outputs.verdict == 'baseline-unusable')
shell: pwsh
run: |
@(
"### Serialisation"
""
"| Status |"
"|---|"
"| Serialisation could not be verified. This is a CI runner failure, not a defect in this pull request. |"
""
"Objects exercised: ${{ steps.branch_run.outputs.population }} across ${{ steps.branch_run.outputs.assemblies }} loaded assemblies, ${{ steps.branch_run.outputs.legs }} legs. Failures reported by the runner: ${{ steps.branch_run.outputs.failures }}."
""
) | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8
$lines | Out-File $env:GITHUB_STEP_SUMMARY -Append -Encoding utf8

- name: Upload results
if: always() && steps.changed.outputs.count != '0'
Expand Down
133 changes: 133 additions & 0 deletions .github/scripts/Write-SerialisationSummary.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Write-SerialisationSummary.ps1 — the serialisation check's job-summary decision, on its
# own so it can be tested without running a serialisation check.
#
# Dot-sourced by .github/actions/ci-serialisation/action.yml and by
# .github/scripts/tests/Write-SerialisationSummary.Tests.ps1. Defines a function and does
# nothing else, so dot-sourcing has no side effects.

function Get-SerialisationSummary {
<#
.SYNOPSIS
Turns the serialisation check's step outcomes into the lines of its job summary.

.DESCRIPTION
This replaced five separate summary steps, each gated on its own combination of step
conclusions and outputs. The problem with that shape was not any one condition: it was
that the conditions had to enumerate the states worth reporting, and a state nobody
enumerated produced no summary at all. Between them they named five of the action's
eighteen functional steps, so a failure anywhere else left the check red and the summary
empty, with nothing to tell the reader whether serialisation had even been attempted.

A single function reached on every path cannot have that hole. The last branch is a
catch-all, so an outcome nobody anticipated still produces a readable summary naming the
phase it stopped in, rather than silence.

Every parameter is a string because that is what a workflow expression yields. An absent
value arrives as the empty string, and the empty string is meaningful here: it is how a
step that never ran is distinguished from one that ran and failed.

.PARAMETER DepsBranchConclusion
Conclusion of the branch-leg dependency resolution.

.PARAMETER BuildBranchConclusion
Conclusion of the branch-leg build of the calling repository.

.PARAMETER BranchRunConclusion
Conclusion of the branch-leg serialisation run. Empty when it never ran.

.PARAMETER BranchStatus
Status the branch-leg runner reported: Pass, Warning or Error. Empty when it never ran.

.PARAMETER CompareVerdict
Verdict of the comparison step: ok, regression, baseline-unusable. Empty when the
comparison never ran, which is every path that stopped before it.

.PARAMETER Population, Assemblies, Legs, Failures
Coverage figures the branch-leg runner reported. Emitted only when the branch leg
produced them, so a summary never claims coverage the run did not measure.

.OUTPUTS
[string[]] markdown lines, ready to append to the step summary.
#>
[CmdletBinding()]
param(
[string]$DepsBranchConclusion = '',
[string]$BuildBranchConclusion = '',
[string]$BranchRunConclusion = '',
[string]$BranchStatus = '',
[string]$CompareVerdict = '',
[string]$Population = '',
[string]$Assemblies = '',
[string]$Legs = '',
[string]$Failures = ''
)

# The dependency closure or the caller's own build failed, so serialisation never started.
# Reported separately from every other failure because it is not a serialisation finding
# at all and the same failure is already on the build check, where it belongs.
if ($DepsBranchConclusion -eq 'failure' -or $BuildBranchConclusion -eq 'failure') {
$step = if ($DepsBranchConclusion -eq 'failure') { 'Resolve dependencies (branch)' }
else { 'Build primary repo (branch)' }
return @(
"### Serialisation did not run"
""
"Failed in: $step"
""
"The dependency closure did not build, so no serialisation comparison was made."
"This is not a serialisation finding. See the ci-build check on this pull request"
"for the same failure."
)
}

$status =
if ($BranchRunConclusion -eq 'success' -and $BranchStatus -ne 'Error') {
"No serialisation failures ($BranchStatus)"
}
elseif ($CompareVerdict -eq 'ok') {
"Serialisation failures detected. All are pre-existing on the base branch, so this job passes."
}
elseif ($CompareVerdict -eq 'regression') {
"Serialisation regression detected. Affected types are in the step log."
}
elseif ($BranchRunConclusion -eq 'failure' -or $CompareVerdict -eq 'baseline-unusable') {
"Serialisation could not be verified. This is a CI runner failure, not a defect in this pull request."
}
else {
# The catch-all, and the reason this function exists. Everything above describes a
# state someone thought of; this describes the rest. It names the phase rather than
# the step, because deriving the phase needs only the signals already passed in,
# whereas naming the step would mean giving every step an id and threading each
# conclusion through — which is the enumeration that failed in the first place.
$phase =
if ($BranchRunConclusion -eq '' -or $BranchRunConclusion -eq 'skipped') {
"while preparing the branch leg, before serialisation ran"
}
elseif ($BranchStatus -eq 'Error') {
"while preparing the baseline leg, after the branch leg reported failures"
}
else {
"at a point this summary cannot identify"
}
"Serialisation did not reach a verdict. The check stopped $phase. Look at the failed step in the log; this is a CI failure, not a finding about this pull request."
}

$lines = @(
"### Serialisation"
""
"| Status |"
"|---|"
"| $status |"
""
)

# Coverage is evidence that the run examined something, so a green summary can be told
# apart from a vacuous one. Only emitted when the branch leg actually reported figures:
# printing "Objects exercised: across loaded assemblies" for a run that never happened
# would be worse than printing nothing.
if ($Population -ne '') {
$lines += "Objects exercised: $Population across $Assemblies loaded assemblies, $Legs legs. Failures reported by the runner: $Failures."
$lines += ""
}

return $lines
}
Loading
Loading