Introduce target artifact staging for Python

docs/execute/binaries.md

@@ -2,6 +2,30 @@ By default, system tests will build a [weblog](../edit/weblog.md) image that shi
 
 But we often want to run system tests against unmerged changes. The general approach is to identify the git commit hash that contains your changes and use this commit hash to download a targeted build of the tracer. Note: ensure that the commit is pushed to a remote branch first, and when taking the commit hash, ensure you use the full hash. You can identify the commit hash using `git log` or from the github UI.
 
+## Target artifact staging
+
+Python is the first target using the target artifact staging framework. The existing
+compatibility command continues to work:
+
+```bash
+./utils/scripts/load-binary.sh python <dev|prod|custom>
+```
+
+The equivalent direct command is:
+
+```bash
+python3 utils/scripts/stage-target-artifacts.py python <dev|prod|custom>
+```
+
+Staging writes bounded text selectors and records generated-file ownership in
+`binaries/.target-artifacts-manifest.json`. It refuses to overwrite manual files,
+changed generated entries, symlinks, or conflicting selectors in `binaries/`.
+Development and production selectors for the same target are treated as mutually
+exclusive automatically.
+Switching to `custom` removes unchanged generated Python selectors while preserving
+manual payloads. Other targets continue to use their existing loading behavior until
+they are migrated separately.
+
 
 ## Agent
 

docs/glossary.md

@@ -1,5 +1,17 @@
 # Glossary
 
+## Target artifact staging
+
+- target artifact: The library, layer, image, module, release, or workflow artifact selected for a system-tests target such as `python`, `java`, `c`, or `cpp_nginx`.
+- dependency artifact: A supporting artifact that is not the selected test target, such as the Datadog Agent image.
+- overlay artifact: A supplemental artifact layered onto tests without being the selected test target, such as the WAF rule set.
+- artifact staging: The step that resolves target artifact inputs and writes generated artifact entries into `binaries/` before a Docker build or test run consumes them.
+- artifact entry: A generated text file in `binaries/` that tells an installer which target artifact to use.
+- bounded artifact selector: A selector with stable meaning, such as a commit SHA, release tag, package version, or OCI digest.
+- selection marker: A generated artifact entry that records the bounded selector when another entry must use a provider-specific fetch selector.
+- payload override: A manual payload placed in `binaries/`, such as a jar, wheel, archive, native module, or local checkout, that takes precedence over generated artifact entries.
+- artifact manifest: The generated `binaries/.target-artifacts-manifest.json` file that tracks ownership and hashes for generated artifact entries.
+
 ## Test activation/deactivation
 
 - successful: A test is successful if none of its assertions are failing

docs/internals/README.md

@@ -11,6 +11,7 @@ All about system-tests deep internals. For those of you who are not afraid of ge
 
 - [MITM certificate](recreating_MITM_certificate.md) -- how to recreate the proxy certificate
 - [Core dump generation](generate-core-dump.md) -- generating core dumps for debugging
+- [Target artifact staging](target-artifact-staging-spec.md) -- target-owned artifact selection model, manifest behavior, and maintainer contract
 
 ### Recreating protobuf schemas
 

docs/internals/target-artifact-staging-spec.md

@@ -0,0 +1,70 @@
+# Target artifact staging
+
+Target artifact staging resolves a test target to bounded, inspectable entries in
+`binaries/` before a build consumes them. The first migration covers Python; other
+targets continue to use `utils/scripts/load-binary.sh` until migrated separately.
+
+## Contract
+
+Each migrated target provides `utils/build/docker/<target>/artifact.py` with `Dev`
+and `Prod` implementations. They declare every filename they may emit, declare
+resolver inputs, and map the resolved values to text entries without performing
+network or filesystem side effects themselves.
+
+The shared orchestrator owns external lookups and writes the generated entries. It
+also maintains `binaries/.target-artifacts-manifest.json`, which records the owner
+and content hash of every generated file. Staging:
+
+- verifies that previously generated entries still match their recorded hashes;
+- refreshes entries previously owned by the same target;
+- removes stale entries owned by that target;
+- preserves entries owned by other targets; and
+- refuses to overwrite unowned files, changed generated entries, symlinks, conflicting
+  selectors, or entries owned by another target.
+
+The filenames declared by a target's `Dev` and `Prod` implementations define its
+selector family. Staging rejects any manual selector in that family that the selected
+environment did not emit, while allowing multiple entries emitted together to coexist.
+Individual entries do not need to name their conflicts, and declaring filenames does
+not resolve the inactive environment's external inputs.
+
+Selectors should be bounded, such as a commit SHA, release tag, package version, or
+OCI digest. If an installer must consume a mutable provider selector, the target must
+also emit a bounded selection marker with `provider_fetch_entries`.
+
+The `custom` environment does not resolve or create selectors because an upstream or
+local artifact bundle is already the source of truth. It removes unchanged generated
+selectors previously owned by the target so they cannot override that custom payload.
+
+## Commands
+
+The canonical entry point is:
+
+```bash
+python3 utils/scripts/stage-target-artifacts.py <target> <dev|prod|custom>
+```
+
+During migration, the existing compatibility command delegates migrated targets to
+the same implementation:
+
+```bash
+./utils/scripts/load-binary.sh <target> <dev|prod|custom>
+```
+
+## Python demonstration
+
+For `python dev`, the configured `LIBRARY_TARGET_BRANCH` (default: `main`) resolves
+to a commit SHA and produces `python-load-from-s3`. For `python prod`, the latest
+published `ddtrace` package version produces `python-load-from-pip`. Existing Python
+installer behavior consumes both files, so no installer change is needed.
+
+## Adding a target
+
+1. Add the target's `artifact.py` with `Dev` and `Prod` implementations.
+2. Reuse shared resolvers, or add a resolver with isolated unit coverage.
+3. Emit text entries only; keep payload downloads in existing build/install steps.
+4. Preserve local payload overrides and add public-contract tests for the target.
+5. Route only that target through the compatibility loader.
+
+GitLab job integration and Buildx remote caching are intentionally handled in later
+changes after target migrations are reviewed.

tests/test_the_test/test_load_binary.py

@@ -5,13 +5,15 @@
 import subprocess
 
 from utils import scenarios
+from utils.target_artifacts.orchestrator import MANIFEST_FILENAME
 
 
 SCRIPT = Path("utils/scripts/load-binary.sh")
 C_LIBRARY_PROD_IMAGE = "install.datadoghq.com/apm-library-c-package:latest"
 C_INJECTOR_PROD_IMAGE = "install.datadoghq.com/apm-inject-package:latest"
 C_LIBRARY_SHA = "1" * 40
 C_INJECTOR_SHA = "2" * 40
+PYTHON_SHA = "3" * 40
 
 
 def _write_executable(path: Path, contents: str) -> None:
@@ -158,3 +160,44 @@ def test_missing_package_fails_with_clear_error(self, tmp_path: Path) -> None:
 
         assert result.returncode != 0
         assert "OCI package does not exist or is not accessible" in result.stderr
+
+
+@scenarios.test_the_test
+class Test_LoadBinaryPython:
+    def test_development_branch_uses_target_artifact_staging(self, tmp_path: Path) -> None:
+        binaries_dir = tmp_path / "binaries"
+        env = {
+            **os.environ,
+            "BINARIES_DIR": str(binaries_dir),
+            "LIBRARY_TARGET_BRANCH": PYTHON_SHA,
+        }
+
+        result = subprocess.run(
+            ["bash", str(SCRIPT), "python", "dev"],
+            check=False,
+            capture_output=True,
+            text=True,
+            env=env,
+        )
+
+        assert result.returncode == 0, result.stderr
+        assert (binaries_dir / "python-load-from-s3").read_text(encoding="utf-8") == f"{PYTHON_SHA}\n"
+        assert (binaries_dir / MANIFEST_FILENAME).exists()
+
+    def test_custom_environment_preserves_manual_python_artifacts(self, tmp_path: Path) -> None:
+        binaries_dir = tmp_path / "binaries"
+        binaries_dir.mkdir()
+        manual_artifact = binaries_dir / "python-load-from-s3"
+        manual_artifact.write_text("manual\n", encoding="utf-8")
+
+        result = subprocess.run(
+            ["bash", str(SCRIPT), "python", "custom"],
+            check=False,
+            capture_output=True,
+            text=True,
+            env={**os.environ, "BINARIES_DIR": str(binaries_dir)},
+        )
+
+        assert result.returncode == 0, result.stderr
+        assert manual_artifact.read_text(encoding="utf-8") == "manual\n"
+        assert not (binaries_dir / MANIFEST_FILENAME).exists()