Import restrictions on test modules by nccatoni · Pull Request #7408 · DataDog/system-tests

nccatoni · 2026-07-29T08:04:20Z

Motivation

Restricting imports in test modules makes it possible to enforce certain system-test usage patterns that would otherwise require manual review. There are currently two targeted patterns:

Preventing imports across test modules.
Restricting pytest imports to a limited feature set, notably to prevent the use of native pytest decorators instead of the framework-provided decorators.

Changes

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

datadog-datadog-prod-us1 · 2026-07-29T08:11:56Z

✨ Unblock PR with BitsAI

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 6 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / End-to-end #1 / poc 1 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions
❌ tests.integrations.test_dsm.Test_DsmRabbitmq_TopicExchange.test_dsm_rabbitmq[poc] from system_tests_suite
ValueError: Checkpoint has not been found, please have a look in logs

self = &lt;tests.integrations.test_dsm.Test_DsmRabbitmq_TopicExchange object at 0x7f2d54bafd10&gt;

    def test_dsm_rabbitmq(self):
        assert self.r.text == &#34;ok&#34;
    
        parent_hash = 14115675228093516275
        DsmHelper.assert_checkpoint_presence(
            hash_=parent_hash,
...
Testing the test | all-jobs-are-green — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

DataDog/system-tests | Debian_12_amd64.SID0: [test-app-php] — 🔄 Retry may pass, looks flaky

View more details · View in GitLab
View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5d02d6f | Docs | View more details | Give us feedback!}

…irectly

github-actions · 2026-08-12T12:28:09Z

CODEOWNERS have been resolved as:

.importlinter                                                           @DataDog/system-tests-core
utils/format/__init__.py                                                @DataDog/system-tests-core
utils/format/import_linter_contracts.py                                 @DataDog/system-tests-core
utils/pytest.py                                                         @DataDog/system-tests-core
.cursor/rules/code-format-standards.mdc                                 @DataDog/system-tests-core
.cursor/rules/pr-review.mdc                                             @DataDog/system-tests-core
.cursor/rules/repository-structure.mdc                                  @DataDog/system-tests-core
.cursor/rules/test-activation.mdc                                       @DataDog/system-tests-core
format.sh                                                               @DataDog/system-tests-core
requirements.txt                                                        @DataDog/system-tests-core
tests/ai_guard/conftest.py                                              @DataDog/k9-ai-guard
tests/appsec/waf/test_addresses.py                                      @DataDog/asm-libraries
tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk
tests/integration_frameworks/conftest.py                                @DataDog/system-tests-reviewers
tests/integration_frameworks/llm/anthropic/test_anthropic_ai_guard.py   @DataDog/ml-observability
tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py        @DataDog/ml-observability
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py     @DataDog/ml-observability
tests/integration_frameworks/llm/google_genai/test_google_genai_apm.py  @DataDog/ml-observability
tests/integration_frameworks/llm/google_genai/test_google_genai_llmobs.py  @DataDog/ml-observability
tests/integration_frameworks/llm/openai/test_openai_ai_guard.py         @DataDog/ml-observability
tests/integration_frameworks/llm/openai/test_openai_apm.py              @DataDog/ml-observability
tests/integration_frameworks/llm/openai/test_openai_llmobs.py           @DataDog/ml-observability
tests/parametric/conftest.py                                            @DataDog/system-tests-reviewers
tests/parametric/otel_env_vars/test_otel_sdk_disabled.py                @DataDog/system-tests-reviewers
tests/parametric/otel_env_vars/test_otel_service_name.py                @DataDog/system-tests-reviewers
tests/parametric/test_128_bit_traceids.py                               @DataDog/system-tests-reviewers
tests/parametric/test_config_consistency.py                             @DataDog/apm-sdk-capabilities
tests/parametric/test_crashtracking.py                                  @DataDog/system-tests-reviewers
tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-reviewers
tests/parametric/test_extract_behavior.py                               @DataDog/system-tests-reviewers
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/test_span_enrichment.py                       @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_headers_b3.py                                     @DataDog/system-tests-reviewers
tests/parametric/test_headers_b3multi.py                                @DataDog/system-tests-reviewers
tests/parametric/test_headers_baggage.py                                @DataDog/system-tests-reviewers
tests/parametric/test_headers_none.py                                   @DataDog/system-tests-reviewers
tests/parametric/test_headers_opm.py                                    @DataDog/system-tests-reviewers
tests/parametric/test_headers_precedence.py                             @DataDog/system-tests-reviewers
tests/parametric/test_headers_tracecontext.py                           @DataDog/system-tests-reviewers
tests/parametric/test_headers_tracestate_dd.py                          @DataDog/system-tests-reviewers
tests/parametric/test_library_tracestats.py                             @DataDog/system-tests-reviewers
tests/parametric/test_llm_observability/conftest.py                     @DataDog/ml-observability
tests/parametric/test_llm_observability/test_llm_observability.py       @DataDog/ml-observability
tests/parametric/test_llm_observability/test_llm_observability_dne.py   @DataDog/ml-observability
tests/parametric/test_llm_observability/utils.py                        @DataDog/ml-observability
tests/parametric/test_otel_api_interoperability.py                      @DataDog/system-tests-reviewers
tests/parametric/test_otel_env_vars.py                                  @DataDog/system-tests-reviewers
tests/parametric/test_otel_logs.py                                      @DataDog/system-tests-reviewers
tests/parametric/test_otel_metrics.py                                   @DataDog/system-tests-reviewers
tests/parametric/test_otel_span_methods.py                              @DataDog/system-tests-reviewers
tests/parametric/test_otel_span_with_baggage.py                         @DataDog/system-tests-reviewers
tests/parametric/test_otel_tracer.py                                    @DataDog/system-tests-reviewers
tests/parametric/test_otel_tracestate_sampling.py                       @DataDog/system-tests-reviewers
tests/parametric/test_otlp_trace_metrics.py                             @DataDog/system-tests-reviewers
tests/parametric/test_parametric_endpoints.py                           @DataDog/system-tests-reviewers
tests/parametric/test_partial_flushing.py                               @DataDog/system-tests-reviewers
tests/parametric/test_process_discovery.py                              @DataDog/system-tests-reviewers
tests/parametric/test_sampling_delegation.py                            @DataDog/system-tests-reviewers
tests/parametric/test_sampling_manual.py                                @DataDog/system-tests-reviewers
tests/parametric/test_sampling_span_tags.py                             @DataDog/system-tests-reviewers
tests/parametric/test_span_events.py                                    @DataDog/system-tests-reviewers
tests/parametric/test_span_links.py                                     @DataDog/system-tests-reviewers
tests/parametric/test_span_sampling.py                                  @DataDog/system-tests-reviewers
tests/parametric/test_startup_logs.py                                   @DataDog/system-tests-reviewers
tests/parametric/test_telemetry.py                                      @DataDog/system-tests-reviewers
tests/parametric/test_trace_filters.py                                  @DataDog/system-tests-reviewers
tests/parametric/test_trace_sampling.py                                 @DataDog/system-tests-reviewers
tests/parametric/test_tracer.py                                         @DataDog/system-tests-reviewers
tests/parametric/test_tracer_flare.py                                   @DataDog/system-tests-reviewers
tests/parametric/utils.py                                               @DataDog/system-tests-reviewers
tests/stats/test_stats.py                                               @DataDog/system-tests-reviewers
tests/test_library_conf.py                                              @DataDog/system-tests-reviewers
tests/test_the_test/scenarios.json                                      @DataDog/system-tests-reviewers
tests/test_the_test/test_ai_guard_span_helpers.py                       @DataDog/system-tests-reviewers
tests/test_the_test/test_build_base_images.py                           @DataDog/system-tests-reviewers
tests/test_the_test/test_build_pipeline.py                              @DataDog/system-tests-reviewers
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-reviewers
tests/test_the_test/test_decorators.py                                  @DataDog/system-tests-reviewers
tests/test_the_test/test_deserializer.py                                @DataDog/system-tests-reviewers
tests/test_the_test/test_docker_run_cleanup.py                          @DataDog/system-tests-reviewers
tests/test_the_test/test_docker_scenario.py                             @DataDog/system-tests-reviewers
tests/test_the_test/test_easy_win.py                                    @DataDog/system-tests-reviewers
tests/test_the_test/test_external_gitlab_pipeline.py                    @DataDog/system-tests-reviewers
tests/test_the_test/test_features.py                                    @DataDog/system-tests-reviewers
tests/test_the_test/test_force_option.py                                @DataDog/system-tests-reviewers
tests/test_the_test/test_get_image_list.py                              @DataDog/system-tests-reviewers
tests/test_the_test/test_github_nightly.py                              @DataDog/system-tests-reviewers
tests/test_the_test/test_json_report.py                                 @DataDog/system-tests-reviewers
tests/test_the_test/test_junit.py                                       @DataDog/system-tests-reviewers
tests/test_the_test/test_manifest.py                                    @DataDog/system-tests-reviewers
tests/test_the_test/test_minimal_number_of_scenarios.py                 @DataDog/system-tests-reviewers
tests/test_the_test/test_mock_backend_v2.py                             @DataDog/system-tests-reviewers
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/feature-flagging-and-experimentation-sdk
tests/test_the_test/test_scrubber.py                                    @DataDog/system-tests-reviewers
tests/test_the_test/test_slow_and_crash_decorators.py                   @DataDog/system-tests-reviewers
tests/test_the_test/test_telemetry_heartbeat.py                         @DataDog/system-tests-reviewers
tests/test_the_test/test_update_mirror_images.py                        @DataDog/system-tests-reviewers
tests/test_the_test/test_version.py                                     @DataDog/system-tests-reviewers
utils/ci/gitlab/main.yml                                                @DataDog/system-tests-core

zacharycmontoya

LGTM for the parametric tests

nccatoni mentioned this pull request Jul 29, 2026

Add rule and checks to validate no test cross-imports are performed #7335

Closed

5 tasks

nccatoni added 5 commits August 12, 2026 14:22

Creating pytest facade

54c74d3

Fixes

a8bdb28

Replacing skip_if_xfail by the correct decorator

ef72341

Adding import check rules for test in test imports and using pytest d…

c4d4b7b

…irectly

Update runner image tag

7dfc221

nccatoni force-pushed the nccatoni/pytest-facade branch from b66c820 to 7dfc221 Compare August 12, 2026 12:26

pytest import cleanup

8e2d950

nccatoni marked this pull request as ready for review August 12, 2026 13:29

nccatoni requested review from a team as code owners August 12, 2026 13:29

nccatoni requested review from sameerank, vjfridge and zacharycmontoya and removed request for a team August 12, 2026 13:29

nccatoni changed the title ~~Import linting~~ Impose import restriction on test modules Aug 12, 2026

nccatoni added the changelog label Aug 12, 2026

nccatoni changed the title ~~Impose import restriction on test modules~~ Import restrictions on test modules Aug 12, 2026

cbeauchesne requested changes Aug 18, 2026

View reviewed changes

Comment thread tests/test_the_test/test_slow_and_crash_decorators.py

Comment thread utils/scripts/shellcheck.sh Outdated

Comment thread utils/pytest.py Outdated

Comment thread utils/format/import_linter_contracts.py

cbeauchesne mentioned this pull request Aug 19, 2026

Prevent the usage of xfail in tests files #7538

Merged

5 tasks

nccatoni force-pushed the nccatoni/pytest-facade branch from 173b528 to 8e2d950 Compare August 20, 2026 14:58

nccatoni added 2 commits August 26, 2026 15:07

Merge branch 'main' into nccatoni/pytest-facade

7ac44be

Restore shellcheck

0d9e711

nccatoni added 2 commits August 26, 2026 15:18

Ban xfail

75cb418

Move importlinter policy to dedicated utils dir

82b2043

nccatoni requested a review from cbeauchesne August 26, 2026 13:24

cbeauchesne approved these changes Aug 26, 2026

View reviewed changes

nccatoni added 2 commits August 26, 2026 15:44

Adding the facade to new imports

5132784

format

f9c238a

nccatoni enabled auto-merge (squash) August 26, 2026 13:50

Merge branch 'main' into nccatoni/pytest-facade

db1129c

zacharycmontoya approved these changes Sep 1, 2026

View reviewed changes

nccatoni added 5 commits September 7, 2026 14:34

Merge branch 'main' into nccatoni/pytest-facade

857549e

Merge branch 'main' into nccatoni/pytest-facade

1a75463

Fix

7a286e0

Merge branch 'main' into nccatoni/pytest-facade

4841b89

Merge branch 'main' into nccatoni/pytest-facade

29a6a80

nccatoni requested a review from a team as a code owner September 11, 2026 12:21

nccatoni requested review from TonyCTHsu and removed request for a team September 11, 2026 12:21

nccatoni added 2 commits September 11, 2026 14:38

Fixing new imports

6f4240c

Merge branch 'main' into nccatoni/pytest-facade

8c31364

nccatoni disabled auto-merge September 14, 2026 13:28

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed and removed mergequeue-status: waiting labels Sep 14, 2026

nccatoni added 2 commits September 14, 2026 15:34

Merge branch 'main' into nccatoni/pytest-facade

325ebc9

fix

5d02d6f

nccatoni force-pushed the nccatoni/pytest-facade branch from cb1db5c to 5d02d6f Compare September 14, 2026 13:47

Navigation Menu

Uh oh!

Import restrictions on test modules - #7408

Import restrictions on test modules#7408
nccatoni wants to merge 22 commits into
mainfrom
nccatoni/pytest-facade

nccatoni commented Jul 29, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 29, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zacharycmontoya left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nccatoni commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 29, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

github-actions Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nccatoni commented Jul 29, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 29, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

github-actions Bot commented Aug 12, 2026 •

edited

Loading