Add DetectShaderErrors attribute for detecting shader errors during tests - #204
Draft
nowsprinting wants to merge 14 commits into
Draft
Add DetectShaderErrors attribute for detecting shader errors during tests#204nowsprinting wants to merge 14 commits into
nowsprinting wants to merge 14 commits into
Conversation
Types and method signatures only (no logic yet); test-first implementation continues in the next commit.
Covers the log-monitoring and hierarchy-scanning detection methods, the session lifecycle, and the DetectShaderErrors attribute. Confirmed red against the no-op skeleton (STATUS: OK).
Wire up DetectShaderErrors attribute end-to-end: log-monitor and hierarchy-scan detection methods, session orchestration, and the InvalidShaderException reporting path. Unity's log dispatch never delivers a log raised from inside an already-executing Application.logMessageReceived listener, so an exception thrown by FallbackWarningLogMonitor's reporter is otherwise silently lost; DeferredExceptionLogger relays it to Debug.LogException on the next frame instead, outside the nested dispatch. A reentrancy guard on the monitor prevents that relayed log from being observed a second time by the same listener.
Attach_ShaderFallbackWarningLogged_LogsInvalidShaderException and
AttachToUnityTest_ShaderFallbackWarningLogged_LogsInvalidShaderException
became identical (same UnityTest setup, same exception-log assertion)
after the former was changed from [Test] to [UnityTest] during Step 3
implementation debugging. Keep the more accurately named
AttachToUnityTest_ test, carrying over its Category("Acceptance")
marker and explanatory comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/simplify pass: extract CheckedMaterialCache.TryMarkCheckedError and HiddenGameObjectFactory.CreateHidden<T> to remove duplication across the three material scanners and the two hidden-ticker classes; make scanner finding lists lazily allocated; remove FallbackWarningLogMonitor's _isHandling reentrancy guard after confirming empirically it never changes any test outcome (the LogType.Warning filter alone already excludes the deferred exception's redispatch. /resolve-diagnostics pass: suppress the two iterator-never-returns warnings on the intentionally infinite scan/drain coroutines, implement InvalidShaderException's standard constructor set, and clear a few unrelated redundant-code warnings in touched test files. EOF )
Add the README entry for DetectShaderErrorsAttribute, alphabetically between CreateScene and FocusGameView. Widen AttributeUsage to allow assembly-level placement, matching the other outer-action attributes in this package; general consumer projects don't intentionally exercise broken shaders the way this package's own scanner/validator tests do, so there's no self-conflict concern for them. Rename hierarchy scan to material scan throughout the doc comments and README to match the existing class names (MaterialScanRunner, IMaterialScanner). Document two limitations: a fallback warning during a fully synchronous Test method can attribute the failure to a later test (Unity defers the log by one frame), and shader runtime errors with no log output (NaN, invalid pixel output, GPU-side computation anomalies) can't be detected by this attribute at all.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new NUnit/Unity Test Framework attribute to proactively detect and fail tests on shader issues that often only surface as warnings or scene state problems, supported by a small runtime-internals subsystem (log monitoring + periodic scene scanning) and comprehensive tests/fixtures.
Changes:
- Introduces
DetectShaderErrorsAttributeto monitor shader fallback warnings and scan scene materials during test execution. - Adds runtime internals for shader error detection (log source/monitor, scanners, cache, runner, exception logging).
- Adds test coverage plus a shader fixture resource and README documentation for usage/limitations.
Reviewed changes
Copilot reviewed 61 out of 76 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Tests/Shaders/Resources/UnsupportedShader.shader.meta | Unity asset metadata for the unsupported-shader test fixture. |
| Tests/Shaders/Resources/UnsupportedShader.shader | Shader fixture intended to compile but be unsupported at runtime in most environments. |
| Tests/Shaders/Resources.meta | Unity folder metadata for shader test resources. |
| Tests/Shaders.meta | Unity folder metadata for shader test assets. |
| Tests/RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporterTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporterTest.cs | Tests that reporter throws the expected exception with the expected message. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyShaderErrorReporter.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyShaderErrorReporter.cs | Spy reporter test double for verifying reported messages. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyMaterialScanner.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/SpyMaterialScanner.cs | Spy scanner test double to control findings and observe scan calls. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/FakeLogMessageSource.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles/FakeLogMessageSource.cs | Fake log source enabling synchronous test verification of log handler behavior. |
| Tests/RuntimeInternals/ShaderErrorDetection/TestDoubles.meta | Unity folder metadata for test doubles. |
| Tests/RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScannerTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScannerTest.cs | Integration tests for skybox material scanning behavior. |
| Tests/RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSessionTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSessionTest.cs | Tests for session lifecycle, log monitoring, final scan behavior, and idempotency. |
| Tests/RuntimeInternals/ShaderErrorDetection/RendererMaterialScannerTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/RendererMaterialScannerTest.cs | Integration tests for renderer material scanning (error shader, unsupported shader, null slots). |
| Tests/RuntimeInternals/ShaderErrorDetection/MaterialValidatorTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/MaterialValidatorTest.cs | Tests for shader/material validation decision logic (supported/unsupported/null slot rules). |
| Tests/RuntimeInternals/ShaderErrorDetection/MaterialScanRunnerTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/MaterialScanRunnerTest.cs | Play mode tests validating scan tick cadence and persistence across scene loads. |
| Tests/RuntimeInternals/ShaderErrorDetection/GraphicMaterialScannerTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/GraphicMaterialScannerTest.cs | Integration tests for uGUI Graphic material scanning. |
| Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPatternTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPatternTest.cs | Tests for fallback-warning pattern matching across wording variations. |
| Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitorTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitorTest.cs | Tests that only warning-type fallback logs are reported and stop() unsubscribes. |
| Tests/RuntimeInternals/ShaderErrorDetection/CheckedMaterialCacheTest.cs.meta | Unity test script metadata. |
| Tests/RuntimeInternals/ShaderErrorDetection/CheckedMaterialCacheTest.cs | Tests caching behavior to avoid repeated validations within a session. |
| Tests/RuntimeInternals/ShaderErrorDetection.meta | Unity folder metadata for shader error detection tests. |
| Tests/Runtime/Attributes/DetectShaderErrorsAttributeTest.cs.meta | Unity test script metadata. |
| Tests/Runtime/Attributes/DetectShaderErrorsAttributeTest.cs | End-to-end tests verifying the attribute’s behavior across async tests and UnityTests. |
| RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporter.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/ThrowingShaderErrorReporter.cs | Reporter implementation that fails tests by throwing on first reported issue. |
| RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScanner.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/SkyboxMaterialScanner.cs | Scanner for RenderSettings.skybox material validation. |
| RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSession.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/ShaderErrorDetectionSession.cs | Orchestrates per-test session lifecycle: monitor + runner + scanners + reporter. |
| RuntimeInternals/ShaderErrorDetection/RendererMaterialScanner.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/RendererMaterialScanner.cs | Scanner for active Renderer materials, including null-slot detection. |
| RuntimeInternals/ShaderErrorDetection/MaterialValidator.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/MaterialValidator.cs | Central decision logic for identifying error shaders/materials and ignorable null slots. |
| RuntimeInternals/ShaderErrorDetection/MaterialScanRunner.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/MaterialScanRunner.cs | Hidden coroutine driver that invokes scans on a fixed frame interval. |
| RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs | Interface for reporting detected shader errors (throwing vs spying). |
| RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs | Exception type used to surface shader problems to the test framework. |
| RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs | Interface for scanners that inspect subsets of scene state for shader/material issues. |
| RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs | Abstraction for log message sources to enable deterministic tests. |
| RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs | Utility for hidden DontDestroyOnLoad driver objects. |
| RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs | Scanner for active uGUI Graphic materials. |
| RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs | Formats hierarchy paths for error messages. |
| RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs | Regex-based matcher for shader fallback warning messages. |
| RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs | Monitors warnings from the log stream and reports shader fallback warnings. |
| RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs | Defers exception logging to the next frame to avoid nested log-dispatch suppression. |
| RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs | Per-session cache preventing repeated material validation across scanners/ticks. |
| RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs | Bridges Application.logMessageReceived to subscribers and handles exception relaying. |
| RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs.meta | Unity script metadata. |
| RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs | Version-conditional wrapper to find active scene objects using non-obsolete APIs. |
| RuntimeInternals/ShaderErrorDetection.meta | Unity folder metadata for shader error detection internals. |
| Runtime/Attributes/DetectShaderErrorsAttribute.cs.meta | Unity script metadata. |
| Runtime/Attributes/DetectShaderErrorsAttribute.cs | Public NUnit attribute that wires up a per-test shader error detection session. |
| README.md | Adds user-facing documentation for the new attribute, usage, and limitations. |
Files not reviewed (15)
- Runtime/Attributes/DetectShaderErrorsAttribute.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/ActiveObjectFinder.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/ApplicationLogMessageSource.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/CheckedMaterialCache.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/DeferredExceptionLogger.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/FallbackWarningLogMonitor.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/FallbackWarningLogPattern.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/GameObjectPathFormatter.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/GraphicMaterialScanner.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/HiddenGameObjectFactory.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/ILogMessageSource.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/IMaterialScanner.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/IShaderErrorReporter.cs.meta: Generated file
- RuntimeInternals/ShaderErrorDetection/InvalidShaderException.cs.meta: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This comment has been minimized.
This comment has been minimized.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
The scan only reaches active Renderers, active uGUI Graphics, and RenderSettings.skybox; materials on other draw paths (Graphics.DrawMesh, CommandBuffer, VFX Graph, Blit/post-processing, Terrain, Projector/Decal, CanvasRenderer.SetMaterial without a Graphic, GL/IMGUI/UI Toolkit) are covered by log monitoring only, and only when they log a fallback warning.
This comment has been minimized.
This comment has been minimized.
nowsprinting
marked this pull request as draft
August 2, 2026 11:54
Test First for two detection gaps in DetectShaderErrors: - SkyboxComponentMaterialScannerTest covers scanning enabled Skybox components on active GameObjects (error/supported/no material, disabled component, inactive GameObject) - CheckedMaterialCacheTest covers re-validation when a checked material's shader is swapped, plus null-shader key handling - RendererMaterialScannerTest covers a second scan reporting a material whose shader was swapped to an error shader - DetectShaderErrorsAttributeTest covers end-to-end detection of a broken Skybox component material via the default session wiring Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e key Scan enabled Skybox components on active GameObjects — a Skybox component overrides RenderSettings.skybox per camera, so a broken material there was previously invisible to the material scan. A Camera on the same GameObject is not required: a broken skybox material on a camera-less GameObject is almost certainly a mistake worth reporting. Key CheckedMaterialCache by (material, shader) pair instead of the material alone, so a material whose shader is swapped to a broken one after it was already validated in the same session is re-validated instead of being skipped by the cache.
Contains the EntityId/InstanceID branch and the null-object sentinel in one place instead of duplicating the ternary across both #if branches of the composite-key computation.
This comment has been minimized.
This comment has been minimized.
CI runs the Standalone Linux player with GraphicsDeviceType.Null (no real GPU), where Shader.isSupported is false for every shader. That makes MaterialValidator.IsErrorShader flag even correctly supported shaders (UI/Default, Sprites/Default, Skybox/Procedural, ...) as errors, failing 8 tests that assert a supported material produces no finding. Headless execution is not a supported target for this feature, so these tests are excluded on LinuxPlayer instead. The shared exclusion reason is factored into LinuxHeadlessGpuUnsupportedAttribute to avoid repeating the same rationale at every call site.
A shader whose only SubShader is unsupported but declares a Fallback reports isSupported == true, so the material scan cannot flag it by design; the shader fallback warning log is the only remaining detection path. This test documents that expectation and currently fails on at least Unity 6000.4.12f1 / Metal / Editor, where the fallback happens silently and no warning log is emitted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
DetectShaderErrorsAttribute, an NUnit test attribute that detects shader problems while a test is running and fails the test when one is found — cases that don't produce a compile error or an error log on their own:Hidden/InternalErrorShader)!Shader.isSupported)Detection combines two methods: monitoring
Application.logMessageReceivedfor fallback warnings, and a periodic scan of activeRenderers, uGUIGraphics,RenderSettings.skybox, and enabledSkyboxcomponents (the per-camera override ofRenderSettings.skybox) for the other cases. The scan also always runs once more when the test ends, so a shader error isn't missed even in a test that finishes within a few frames. Checked materials are cached per (material, shader) pair, so a material whose shader is swapped mid-test is re-validated instead of being skipped.Can be placed on the test method, the test class, or the test assembly; works with sync
Test, asyncTest, andUnityTest. See the README for usage and known limitations (sync-Testfallback-warning timing, draw paths the material scan does not reach, and runtime GPU errors with no log output at all, which are out of scope for this attribute).Test plan
TestHelper.RuntimeInternals.Tests(ShaderErrorDetectionnamespace) andTestHelper.Tests(DetectShaderErrorsAttributeTest) — 83/83 passing