Skip to content

Add DetectShaderErrors attribute for detecting shader errors during tests - #204

Draft
nowsprinting wants to merge 14 commits into
masterfrom
feature/detect-shader-errors
Draft

Add DetectShaderErrors attribute for detecting shader errors during tests#204
nowsprinting wants to merge 14 commits into
masterfrom
feature/detect-shader-errors

Conversation

@nowsprinting

@nowsprinting nowsprinting commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

Adds DetectShaderErrorsAttribute, an NUnit test attribute that detects shader problems while a test is running and fails the test when one is found — cases that don't produce a compile error or an error log on their own:

  • Shader fallback (warning log only)
  • Material's shader reference missing (Hidden/InternalErrorShader)
  • Shader exists but is not supported on the running environment (!Shader.isSupported)
  • Renderer's material reference missing (null slot)

Detection combines two methods: monitoring Application.logMessageReceived for fallback warnings, and a periodic scan of active Renderers, uGUI Graphics, RenderSettings.skybox, and enabled Skybox components (the per-camera override of RenderSettings.skybox) for the other cases. The scan also always runs once more when the test ends, so a shader error isn't missed even in a test that finishes within a few frames. Checked materials are cached per (material, shader) pair, so a material whose shader is swapped mid-test is re-validated instead of being skipped.

Can be placed on the test method, the test class, or the test assembly; works with sync Test, async Test, and UnityTest. See the README for usage and known limitations (sync-Test fallback-warning timing, draw paths the material scan does not reach, and runtime GPU errors with no log output at all, which are out of scope for this attribute).

Test plan

  • TestHelper.RuntimeInternals.Tests (ShaderErrorDetection namespace) and TestHelper.Tests (DetectShaderErrorsAttributeTest) — 83/83 passing
  • CI

nowsprinting and others added 6 commits August 2, 2026 19:20
Types and method signatures only (no logic yet); test-first
implementation continues in the next commit.
Covers the log-monitoring and hierarchy-scanning detection methods,
the session lifecycle, and the DetectShaderErrors attribute. Confirmed
red against the no-op skeleton (STATUS: OK).
Wire up DetectShaderErrors attribute end-to-end: log-monitor and
hierarchy-scan detection methods, session orchestration, and the
InvalidShaderException reporting path.

Unity's log dispatch never delivers a log raised from inside an
already-executing Application.logMessageReceived listener, so an
exception thrown by FallbackWarningLogMonitor's reporter is otherwise
silently lost; DeferredExceptionLogger relays it to Debug.LogException
on the next frame instead, outside the nested dispatch. A reentrancy
guard on the monitor prevents that relayed log from being observed a
second time by the same listener.
Attach_ShaderFallbackWarningLogged_LogsInvalidShaderException and
AttachToUnityTest_ShaderFallbackWarningLogged_LogsInvalidShaderException
became identical (same UnityTest setup, same exception-log assertion)
after the former was changed from [Test] to [UnityTest] during Step 3
implementation debugging. Keep the more accurately named
AttachToUnityTest_ test, carrying over its Category("Acceptance")
marker and explanatory comment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/simplify pass: extract CheckedMaterialCache.TryMarkCheckedError and
HiddenGameObjectFactory.CreateHidden<T> to remove duplication across
the three material scanners and the two hidden-ticker classes; make
scanner finding lists lazily allocated; remove FallbackWarningLogMonitor's
_isHandling reentrancy guard after confirming empirically it never
changes any test outcome (the LogType.Warning filter alone already
excludes the deferred exception's redispatch.

/resolve-diagnostics pass: suppress the two iterator-never-returns
warnings on the intentionally infinite scan/drain coroutines, implement
InvalidShaderException's standard constructor set, and clear a few
unrelated redundant-code warnings in touched test files.
EOF
)
Add the README entry for DetectShaderErrorsAttribute, alphabetically
between CreateScene and FocusGameView. Widen AttributeUsage to allow
assembly-level placement, matching the other outer-action attributes
in this package; general consumer projects don't intentionally exercise
broken shaders the way this package's own scanner/validator tests do,
so there's no self-conflict concern for them.

Rename hierarchy scan to material scan throughout the doc comments and
README to match the existing class names (MaterialScanRunner,
IMaterialScanner). Document two limitations: a fallback warning during
a fully synchronous Test method can attribute the failure to a later
test (Unity defers the log by one frame), and shader runtime errors
with no log output (NaN, invalid pixel output, GPU-side computation
anomalies) can't be detected by this attribute at all.
Copilot AI review requested due to automatic review settings August 2, 2026 10:29
@github-actions github-actions Bot added the enhancement New feature or request label Aug 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new NUnit/Unity Test Framework attribute to proactively detect and fail tests on shader issues that often only surface as warnings or scene state problems, supported by a small runtime-internals subsystem (log monitoring + periodic scene scanning) and comprehensive tests/fixtures.

Changes:

  • Introduces DetectShaderErrorsAttribute to monitor shader fallback warnings and scan scene materials during test execution.
  • Adds runtime internals for shader error detection (log source/monitor, scanners, cache, runner, exception logging).
  • Adds test coverage plus a shader fixture resource and README documentation for usage/limitations.

Reviewed changes

Copilot reviewed 61 out of 76 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Tests/Shaders/Resources/UnsupportedShader.shader.meta Unity asset metadata for the unsupported-shader test fixture.
Tests/Shaders/Resources/UnsupportedShader.shader Shader fixture intended to compile but be unsupported at runtime in most environments.
Tests/Shaders/Resources.meta Unity folder metadata for shader test resources.
Tests/Shaders.meta Unity folder metadata for shader test assets.
Tests/RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporterTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporterTest.cs Tests that reporter throws the expected exception with the expected message.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyShaderErrorReporter.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyShaderErrorReporter.cs Spy reporter test double for verifying reported messages.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyMaterialScanner.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyMaterialScanner.cs Spy scanner test double to control findings and observe scan calls.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/FakeLogMessageSource.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/FakeLogMessageSource.cs Fake log source enabling synchronous test verification of log handler behavior.
Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles.meta Unity folder metadata for test doubles.
Tests/RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScannerTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScannerTest.cs Integration tests for skybox material scanning behavior.
Tests/RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSessionTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSessionTest.cs Tests for session lifecycle, log monitoring, final scan behavior, and idempotency.
Tests/RuntimeInternals/ShaderErrorDetection/RendererMaterialScannerTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/RendererMaterialScannerTest.cs Integration tests for renderer material scanning (error shader, unsupported shader, null slots).
Tests/RuntimeInternals/ShaderErrorDetection/MaterialValidatorTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/MaterialValidatorTest.cs Tests for shader/material validation decision logic (supported/unsupported/null slot rules).
Tests/RuntimeInternals/ShaderErrorDetection/MaterialScanRunnerTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/MaterialScanRunnerTest.cs Play mode tests validating scan tick cadence and persistence across scene loads.
Tests/RuntimeInternals/ShaderErrorDetection/GraphicMaterialScannerTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/GraphicMaterialScannerTest.cs Integration tests for uGUI Graphic material scanning.
Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPatternTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPatternTest.cs Tests for fallback-warning pattern matching across wording variations.
Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitorTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitorTest.cs Tests that only warning-type fallback logs are reported and stop() unsubscribes.
Tests/RuntimeInternals/ShaderErrorDetection/CheckedMaterialCacheTest.cs.meta Unity test script metadata.
Tests/RuntimeInternals/ShaderErrorDetection/CheckedMaterialCacheTest.cs Tests caching behavior to avoid repeated validations within a session.
Tests/RuntimeInternals/ShaderErrorDetection.meta Unity folder metadata for shader error detection tests.
Tests/Runtime/Attributes/DetectShaderErrorsAttributeTest.cs.meta Unity test script metadata.
Tests/Runtime/Attributes/DetectShaderErrorsAttributeTest.cs End-to-end tests verifying the attribute’s behavior across async tests and UnityTests.
RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporter.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporter.cs Reporter implementation that fails tests by throwing on first reported issue.
RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScanner.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScanner.cs Scanner for RenderSettings.skybox material validation.
RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSession.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSession.cs Orchestrates per-test session lifecycle: monitor + runner + scanners + reporter.
RuntimeInternals/ShaderErrorDetection/RendererMaterialScanner.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/RendererMaterialScanner.cs Scanner for active Renderer materials, including null-slot detection.
RuntimeInternals/ShaderErrorDetection/MaterialValidator.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/MaterialValidator.cs Central decision logic for identifying error shaders/materials and ignorable null slots.
RuntimeInternals/ShaderErrorDetection/MaterialScanRunner.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/MaterialScanRunner.cs Hidden coroutine driver that invokes scans on a fixed frame interval.
RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs Interface for reporting detected shader errors (throwing vs spying).
RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs Exception type used to surface shader problems to the test framework.
RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs Interface for scanners that inspect subsets of scene state for shader/material issues.
RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs Abstraction for log message sources to enable deterministic tests.
RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs Utility for hidden DontDestroyOnLoad driver objects.
RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs Scanner for active uGUI Graphic materials.
RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs Formats hierarchy paths for error messages.
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs Regex-based matcher for shader fallback warning messages.
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs Monitors warnings from the log stream and reports shader fallback warnings.
RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs Defers exception logging to the next frame to avoid nested log-dispatch suppression.
RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs Per-session cache preventing repeated material validation across scanners/ticks.
RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs Bridges Application.logMessageReceived to subscribers and handles exception relaying.
RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs.meta Unity script metadata.
RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs Version-conditional wrapper to find active scene objects using non-obsolete APIs.
RuntimeInternals/ShaderErrorDetection.meta Unity folder metadata for shader error detection internals.
Runtime/Attributes/DetectShaderErrorsAttribute.cs.meta Unity script metadata.
Runtime/Attributes/DetectShaderErrorsAttribute.cs Public NUnit attribute that wires up a per-test shader error detection session.
README.md Adds user-facing documentation for the new attribute, usage, and limitations.
Files not reviewed (15)
  • Runtime/Attributes/DetectShaderErrorsAttribute.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs.meta: Generated file
  • RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs.meta: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs Outdated
@github-actions

This comment has been minimized.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

The scan only reaches active Renderers, active uGUI Graphics, and
RenderSettings.skybox; materials on other draw paths (Graphics.DrawMesh,
CommandBuffer, VFX Graph, Blit/post-processing, Terrain, Projector/Decal,
CanvasRenderer.SetMaterial without a Graphic, GL/IMGUI/UI Toolkit) are
covered by log monitoring only, and only when they log a fallback warning.
@github-actions

This comment has been minimized.

@nowsprinting
nowsprinting marked this pull request as draft August 2, 2026 11:54
nowsprinting and others added 4 commits August 2, 2026 21:19
Test First for two detection gaps in DetectShaderErrors:
- SkyboxComponentMaterialScannerTest covers scanning enabled Skybox
  components on active GameObjects (error/supported/no material,
  disabled component, inactive GameObject)
- CheckedMaterialCacheTest covers re-validation when a checked
  material's shader is swapped, plus null-shader key handling
- RendererMaterialScannerTest covers a second scan reporting a
  material whose shader was swapped to an error shader
- DetectShaderErrorsAttributeTest covers end-to-end detection of a
  broken Skybox component material via the default session wiring

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e key

Scan enabled Skybox components on active GameObjects — a Skybox
component overrides RenderSettings.skybox per camera, so a broken
material there was previously invisible to the material scan. A Camera
on the same GameObject is not required: a broken skybox material on a
camera-less GameObject is almost certainly a mistake worth reporting.

Key CheckedMaterialCache by (material, shader) pair instead of the
material alone, so a material whose shader is swapped to a broken one
after it was already validated in the same session is re-validated
instead of being skipped by the cache.
Contains the EntityId/InstanceID branch and the null-object sentinel in
one place instead of duplicating the ternary across both #if branches
of the composite-key computation.
@github-actions

This comment has been minimized.

CI runs the Standalone Linux player with GraphicsDeviceType.Null (no real
GPU), where Shader.isSupported is false for every shader. That makes
MaterialValidator.IsErrorShader flag even correctly supported shaders
(UI/Default, Sprites/Default, Skybox/Procedural, ...) as errors, failing
8 tests that assert a supported material produces no finding. Headless
execution is not a supported target for this feature, so these tests are
excluded on LinuxPlayer instead. The shared exclusion reason is factored
into LinuxHeadlessGpuUnsupportedAttribute to avoid repeating the same
rationale at every call site.
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Code Metrics Report

master (43ab8f1) #204 (afff0c2) +/-
Coverage 84.5% 85.6% +1.0%
Code to Test Ratio 1:1.4 1:1.4 +0.0
Test Execution Time 5m49s 5m51s +2s
Details
  |                     | master (43ab8f1) | #204 (afff0c2) |  +/-  |
  |---------------------|------------------|----------------|-------|
+ | Coverage            |            84.5% |          85.6% | +1.0% |
  |   Files             |               65 |             83 |   +18 |
  |   Lines             |             2411 |           2748 |  +337 |
+ |   Covered           |             2039 |           2353 |  +314 |
+ | Code to Test Ratio  |            1:1.4 |          1:1.4 |  +0.0 |
  |   Code              |             4073 |           4695 |  +622 |
+ |   Test              |             5887 |           6950 | +1063 |
- | Test Execution Time |            5m49s |          5m51s |   +2s |

Code coverage of files in pull request scope (0.0% → 93.1%)

Files Coverage +/- Status
Runtime/Attributes/DetectShaderErrorsAttribute.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs 92.1% +92.1% added
RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs 82.1% +82.1% added
RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs 89.4% +89.4% added
RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs 33.3% +33.3% added
RuntimeInternals/ShaderErrorDetection/MaterialScanRunner.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/MaterialValidator.cs 87.0% +87.0% added
RuntimeInternals/ShaderErrorDetection/RendererMaterialScanner.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSession.cs 93.7% +93.7% added
RuntimeInternals/ShaderErrorDetection/SkyboxComponentMaterialScanner.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScanner.cs 100.0% +100.0% added
RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporter.cs 100.0% +100.0% added

Reported by octocov

A shader whose only SubShader is unsupported but declares a Fallback reports
isSupported == true, so the material scan cannot flag it by design; the shader
fallback warning log is the only remaining detection path. This test documents
that expectation and currently fails on at least Unity 6000.4.12f1 / Metal /
Editor, where the fallback happens silently and no warning log is emitted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants