fix: eliminate flaky unit test failures caused by sys.modules import … - #57
Open
ashnanze wants to merge 5 commits into
Open
fix: eliminate flaky unit test failures caused by sys.modules import …#57ashnanze wants to merge 5 commits into
ashnanze wants to merge 5 commits into
Conversation
…order races Root cause: each test file independently stubbed sys.modules at import time. When pytest discovered files in varying order, whichever ran first corrupted stubs for others. Changes: - Add conftest.py with centralized session-scoped stubs (try real import first, stub on failure) - Remove inline sys.modules manipulation from test_correlation_id, test_custom, test_utils_ - Use __path__=[] on mock packages so Python's import system can traverse them - All 63 tests now pass consistently regardless of discovery order
ashnanze
requested review from
anagg929,
bgriddaluru,
jianyunt and
jmccormick7
as code owners
July 14, 2026 22:48
|
The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR. Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions). pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>
|
|
Replace [System.Text.Json.JsonDocument]::Parse() with ConvertFrom-Json and add null checks before parsing. When 'az connectedk8s show' returns null (transient failure or cluster not ready), the retry loop now logs and continues instead of throwing ArgumentNullException.
The resource group hit the 800 connectedClusters quota because failed CI runs leave orphaned clusters. This adds: - Cleanup.ps1: deletes clusters older than 2 hours - Pipeline step: runs cleanup before each test suite
Prevents orphaned connectedClusters when tests fail mid-run. The step uses condition: always() so it executes regardless of test outcome.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: Eliminate flaky unit test failures caused by sys.modules import order races
Problem
Unit tests in
azext_connectedk8s/tests/unittests/were intermittently failing depending on pytest's file discovery order. Each test file independently stubbedsys.modulesat import time — whichever file was imported first "won" the race, corrupting stubs for others.Root Cause
MagicMock()objects used as module stubs don't have__path__, so Python's import system can't traverse them as packages (e.g.,azure.cli.corefails whenazureis a plain MagicMock)sys.modules.setdefault()could install stubs before real modules were importable, preventing legitimate imports from succeedingFix
conftest.py(new)__import__()first, only stubs on failure. Uses__path__=[]on mocks so they behave as packages.test_correlation_id.pysys.path.insert/import os, sys— conftest handles ittest_custom.pysys.path.insert/import os, sys— conftest handles ittest_utils_.pysys.path.insert; guards against leftover MagicMock stubs for_utilsVerification