diff --git a/docs/execute/binaries.md b/docs/execute/binaries.md index 1e0c535029c..8c46a4ff3b7 100644 --- a/docs/execute/binaries.md +++ b/docs/execute/binaries.md @@ -2,6 +2,30 @@ By default, system tests will build a [weblog](../edit/weblog.md) image that shi But we often want to run system tests against unmerged changes. The general approach is to identify the git commit hash that contains your changes and use this commit hash to download a targeted build of the tracer. Note: ensure that the commit is pushed to a remote branch first, and when taking the commit hash, ensure you use the full hash. You can identify the commit hash using `git log` or from the github UI. +## Target artifact staging + +Python is the first target using the target artifact staging framework. The existing +compatibility command continues to work: + +```bash +./utils/scripts/load-binary.sh python +``` + +The equivalent direct command is: + +```bash +python3 utils/scripts/stage-target-artifacts.py python +``` + +Staging writes bounded text selectors and records generated-file ownership in +`binaries/.target-artifacts-manifest.json`. It refuses to overwrite manual files, +changed generated entries, symlinks, or conflicting selectors in `binaries/`. +Development and production selectors for the same target are treated as mutually +exclusive automatically. +Switching to `custom` removes unchanged generated Python selectors while preserving +manual payloads. Other targets continue to use their existing loading behavior until +they are migrated separately. + ## Agent diff --git a/docs/glossary.md b/docs/glossary.md index 32b5afeca05..5543748414c 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,5 +1,17 @@ # Glossary +## Target artifact staging + +- target artifact: The library, layer, image, module, release, or workflow artifact selected for a system-tests target such as `python`, `java`, `c`, or `cpp_nginx`. +- dependency artifact: A supporting artifact that is not the selected test target, such as the Datadog Agent image. +- overlay artifact: A supplemental artifact layered onto tests without being the selected test target, such as the WAF rule set. +- artifact staging: The step that resolves target artifact inputs and writes generated artifact entries into `binaries/` before a Docker build or test run consumes them. +- artifact entry: A generated text file in `binaries/` that tells an installer which target artifact to use. +- bounded artifact selector: A selector with stable meaning, such as a commit SHA, release tag, package version, or OCI digest. +- selection marker: A generated artifact entry that records the bounded selector when another entry must use a provider-specific fetch selector. +- payload override: A manual payload placed in `binaries/`, such as a jar, wheel, archive, native module, or local checkout, that takes precedence over generated artifact entries. +- artifact manifest: The generated `binaries/.target-artifacts-manifest.json` file that tracks ownership and hashes for generated artifact entries. + ## Test activation/deactivation - successful: A test is successful if none of its assertions are failing diff --git a/docs/internals/README.md b/docs/internals/README.md index 96c3989f17b..ca5e34e09dd 100644 --- a/docs/internals/README.md +++ b/docs/internals/README.md @@ -11,6 +11,7 @@ All about system-tests deep internals. For those of you who are not afraid of ge - [MITM certificate](recreating_MITM_certificate.md) -- how to recreate the proxy certificate - [Core dump generation](generate-core-dump.md) -- generating core dumps for debugging +- [Target artifact staging](target-artifact-staging-spec.md) -- target-owned artifact selection model, manifest behavior, and maintainer contract ### Recreating protobuf schemas diff --git a/docs/internals/target-artifact-staging-spec.md b/docs/internals/target-artifact-staging-spec.md new file mode 100644 index 00000000000..fcc73db54ea --- /dev/null +++ b/docs/internals/target-artifact-staging-spec.md @@ -0,0 +1,70 @@ +# Target artifact staging + +Target artifact staging resolves a test target to bounded, inspectable entries in +`binaries/` before a build consumes them. The first migration covers Python; other +targets continue to use `utils/scripts/load-binary.sh` until migrated separately. + +## Contract + +Each migrated target provides `utils/build/docker//artifact.py` with `Dev` +and `Prod` implementations. They declare every filename they may emit, declare +resolver inputs, and map the resolved values to text entries without performing +network or filesystem side effects themselves. + +The shared orchestrator owns external lookups and writes the generated entries. It +also maintains `binaries/.target-artifacts-manifest.json`, which records the owner +and content hash of every generated file. Staging: + +- verifies that previously generated entries still match their recorded hashes; +- refreshes entries previously owned by the same target; +- removes stale entries owned by that target; +- preserves entries owned by other targets; and +- refuses to overwrite unowned files, changed generated entries, symlinks, conflicting + selectors, or entries owned by another target. + +The filenames declared by a target's `Dev` and `Prod` implementations define its +selector family. Staging rejects any manual selector in that family that the selected +environment did not emit, while allowing multiple entries emitted together to coexist. +Individual entries do not need to name their conflicts, and declaring filenames does +not resolve the inactive environment's external inputs. + +Selectors should be bounded, such as a commit SHA, release tag, package version, or +OCI digest. If an installer must consume a mutable provider selector, the target must +also emit a bounded selection marker with `provider_fetch_entries`. + +The `custom` environment does not resolve or create selectors because an upstream or +local artifact bundle is already the source of truth. It removes unchanged generated +selectors previously owned by the target so they cannot override that custom payload. + +## Commands + +The canonical entry point is: + +```bash +python3 utils/scripts/stage-target-artifacts.py +``` + +During migration, the existing compatibility command delegates migrated targets to +the same implementation: + +```bash +./utils/scripts/load-binary.sh +``` + +## Python demonstration + +For `python dev`, the configured `LIBRARY_TARGET_BRANCH` (default: `main`) resolves +to a commit SHA and produces `python-load-from-s3`. For `python prod`, the latest +published `ddtrace` package version produces `python-load-from-pip`. Existing Python +installer behavior consumes both files, so no installer change is needed. + +## Adding a target + +1. Add the target's `artifact.py` with `Dev` and `Prod` implementations. +2. Reuse shared resolvers, or add a resolver with isolated unit coverage. +3. Emit text entries only; keep payload downloads in existing build/install steps. +4. Preserve local payload overrides and add public-contract tests for the target. +5. Route only that target through the compatibility loader. + +GitLab job integration and Buildx remote caching are intentionally handled in later +changes after target migrations are reviewed. diff --git a/tests/test_the_test/test_load_binary.py b/tests/test_the_test/test_load_binary.py index 149f2f47bb3..f5ff335d1cc 100644 --- a/tests/test_the_test/test_load_binary.py +++ b/tests/test_the_test/test_load_binary.py @@ -5,6 +5,7 @@ import subprocess from utils import scenarios +from utils.target_artifacts.orchestrator import MANIFEST_FILENAME SCRIPT = Path("utils/scripts/load-binary.sh") @@ -12,6 +13,7 @@ C_INJECTOR_PROD_IMAGE = "install.datadoghq.com/apm-inject-package:latest" C_LIBRARY_SHA = "1" * 40 C_INJECTOR_SHA = "2" * 40 +PYTHON_SHA = "3" * 40 def _write_executable(path: Path, contents: str) -> None: @@ -158,3 +160,44 @@ def test_missing_package_fails_with_clear_error(self, tmp_path: Path) -> None: assert result.returncode != 0 assert "OCI package does not exist or is not accessible" in result.stderr + + +@scenarios.test_the_test +class Test_LoadBinaryPython: + def test_development_branch_uses_target_artifact_staging(self, tmp_path: Path) -> None: + binaries_dir = tmp_path / "binaries" + env = { + **os.environ, + "BINARIES_DIR": str(binaries_dir), + "LIBRARY_TARGET_BRANCH": PYTHON_SHA, + } + + result = subprocess.run( + ["bash", str(SCRIPT), "python", "dev"], + check=False, + capture_output=True, + text=True, + env=env, + ) + + assert result.returncode == 0, result.stderr + assert (binaries_dir / "python-load-from-s3").read_text(encoding="utf-8") == f"{PYTHON_SHA}\n" + assert (binaries_dir / MANIFEST_FILENAME).exists() + + def test_custom_environment_preserves_manual_python_artifacts(self, tmp_path: Path) -> None: + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + manual_artifact = binaries_dir / "python-load-from-s3" + manual_artifact.write_text("manual\n", encoding="utf-8") + + result = subprocess.run( + ["bash", str(SCRIPT), "python", "custom"], + check=False, + capture_output=True, + text=True, + env={**os.environ, "BINARIES_DIR": str(binaries_dir)}, + ) + + assert result.returncode == 0, result.stderr + assert manual_artifact.read_text(encoding="utf-8") == "manual\n" + assert not (binaries_dir / MANIFEST_FILENAME).exists() diff --git a/tests/test_the_test/test_target_artifacts.py b/tests/test_the_test/test_target_artifacts.py new file mode 100644 index 00000000000..e94aff6717b --- /dev/null +++ b/tests/test_the_test/test_target_artifacts.py @@ -0,0 +1,1287 @@ +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from typing import TYPE_CHECKING, Any + +import pytest +import requests + +from utils import scenarios +from utils.target_artifacts.models import ( + ArtifactResolver, + BranchReference, + GitHubActionsArtifactReference, + GitHubReleaseReference, + LiteralValue, + ModuleVersion, + OciImageReference, + ReleaseAsset, + ResolvedArtifactInput, + TargetArtifactError, +) +from utils.target_artifacts.orchestrator import MANIFEST_FILENAME, load_target_environment, stage_target +from utils.target_artifacts.resolvers import ( + CratesLatestResolver, + EnvResolver, + GitHubActionsArtifactResolver, + GitHubBranchResolver, + GitHubLatestReleaseResolver, + GoModuleLatestResolver, + NpmLatestResolver, + OciDigestResolver, + PypiLatestResolver, + RubygemsLatestResolver, +) + +if TYPE_CHECKING: + from collections.abc import Callable + +SHA = "1" * 40 +DIGEST = "sha256:" + ("2" * 64) +OTHER_SHA = "3" * 40 + + +class StubResponse: + def __init__( + self, + payload: object, + *, + status_error: requests.RequestException | None = None, + json_error: ValueError | None = None, + ) -> None: + self.payload = payload + self.status_error = status_error + self.json_error = json_error + + def raise_for_status(self) -> None: + if self.status_error is not None: + raise self.status_error + + def json(self) -> object: + if self.json_error is not None: + raise self.json_error + return self.payload + + +def _stub_get_json( + monkeypatch: pytest.MonkeyPatch, + payloads: dict[str, dict[str, Any]] | Callable[[str, dict[str, str]], dict[str, Any]], +) -> list[tuple[str, dict[str, str]]]: + calls: list[tuple[str, dict[str, str]]] = [] + + def fake_get_json(url: str, headers: dict[str, str]) -> dict[str, Any]: + calls.append((url, dict(headers))) + if callable(payloads): + return payloads(url, headers) + return payloads[url] + + monkeypatch.setattr("utils.target_artifacts.resolvers._get_json", fake_get_json) + return calls + + +def _completed_process( + args: list[str], + *, + returncode: int = 0, + stdout: str = "", + stderr: str = "", +) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(args=args, returncode=returncode, stdout=stdout, stderr=stderr) + + +class FakeResolver: + def resolve(self, artifact_resolver: ArtifactResolver, env: dict[str, str]) -> ResolvedArtifactInput: + if isinstance(artifact_resolver, EnvResolver): + return LiteralValue( + name=artifact_resolver.name, + value=env.get(artifact_resolver.variable_name, artifact_resolver.default_value), + ) + if isinstance(artifact_resolver, GitHubBranchResolver): + return BranchReference( + name=artifact_resolver.name, + repository=artifact_resolver.repository, + branch=env.get(artifact_resolver.variable_name, artifact_resolver.default_value), + sha=SHA, + ) + if isinstance(artifact_resolver, GitHubLatestReleaseResolver): + return GitHubReleaseReference( + name=artifact_resolver.name, + repository=artifact_resolver.repository, + tag_name="v1.2.3", + ) + if isinstance(artifact_resolver, GitHubActionsArtifactResolver): + return GitHubActionsArtifactReference( + name=artifact_resolver.name, + repository=artifact_resolver.repository, + workflow=artifact_resolver.workflow, + branch=env.get(artifact_resolver.variable_name, artifact_resolver.default_value), + commit_sha=SHA, + run_id=123, + run_url="https://github.example/run", + artifact_id=456, + artifact_name=artifact_resolver.artifact_name, + archive_download_url="https://github.example/artifact.zip", + ) + if isinstance(artifact_resolver, OciDigestResolver): + image = env.get( + artifact_resolver.variable_name, + artifact_resolver.image or artifact_resolver.default_value, + ) + last_slash = image.rfind("/") + last_colon = image.rfind(":") + repository = image[:last_colon] if last_colon > last_slash else image + return OciImageReference( + name=artifact_resolver.name, + image=image, + digest=DIGEST, + reference=f"{repository}@{DIGEST}", + ) + if isinstance( + artifact_resolver, + (NpmLatestResolver, PypiLatestResolver, RubygemsLatestResolver, CratesLatestResolver), + ): + return ModuleVersion(name=artifact_resolver.name, module=artifact_resolver.package, version="1.2.3") + if isinstance(artifact_resolver, GoModuleLatestResolver): + return ModuleVersion(name=artifact_resolver.name, module=artifact_resolver.module, version="v1.2.3") + raise AssertionError(f"Unhandled input resolver: {type(artifact_resolver).__name__}") + + +def _write_target_module(repo_root: Path, body: str) -> None: + target_dir = repo_root / "utils" / "build" / "docker" / "fake" + target_dir.mkdir(parents=True) + (target_dir / "artifact.py").write_text(body, encoding="utf-8") + + +def _manifest_entries(binaries_dir: Path) -> dict[str, object]: + manifest = json.loads((binaries_dir / MANIFEST_FILENAME).read_text(encoding="utf-8")) + return manifest["entries"] + + +@scenarios.test_the_test +class Test_TargetArtifactStaging: + def test_custom_environment_is_noop(self, tmp_path: Path) -> None: + binaries_dir = tmp_path / "binaries" + + stage_target( + "does-not-exist", + "custom", + repo_root=tmp_path, + binaries_dir=binaries_dir, + process_env={}, + ) + + assert not binaries_dir.exists() + + def test_custom_environment_clears_owned_selectors(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("generated",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("generated", "selector"),) + +class Prod(Dev): + pass +""", + ) + binaries_dir = tmp_path / "binaries" + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + (binaries_dir / "manual.whl").write_text("payload", encoding="utf-8") + + stage_target("fake", "custom", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert not (binaries_dir / "generated").exists() + assert (binaries_dir / "manual.whl").read_text(encoding="utf-8") == "payload" + assert _manifest_entries(binaries_dir) == {} + + def test_manifest_refreshes_owned_files_and_preserves_other_targets(self, tmp_path: Path) -> None: + module_path = tmp_path / "utils" / "build" / "docker" / "fake" + module_path.mkdir(parents=True) + artifact_module = module_path / "artifact.py" + artifact_module.write_text( + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("kept", "stale") + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("kept", "one"), text_entry("stale", "old")) + +class Prod: + def artifact_entry_filenames(self): + return ("kept",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("kept", "two"),) +""", + encoding="utf-8", + ) + other_module = tmp_path / "utils" / "build" / "docker" / "other" / "artifact.py" + other_module.parent.mkdir(parents=True) + other_module.write_text( + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("other",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("other", "target"),) + +class Prod(Dev): + pass +""", + encoding="utf-8", + ) + + binaries_dir = tmp_path / "binaries" + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + stage_target("other", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + stage_target("fake", "prod", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert (binaries_dir / "kept").read_text(encoding="utf-8") == "two\n" + assert not (binaries_dir / "stale").exists() + assert (binaries_dir / "other").read_text(encoding="utf-8") == "target\n" + assert set(_manifest_entries(binaries_dir)) == {"kept", "other"} + + def test_unowned_file_is_not_overwritten(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("manual",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("manual", "generated"),) + +class Prod(Dev): + pass +""", + ) + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + (binaries_dir / "manual").write_text("user\n", encoding="utf-8") + + with pytest.raises(Exception, match="Refusing to overwrite unowned artifact entry 'manual'"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert (binaries_dir / "manual").read_text(encoding="utf-8") == "user\n" + + @pytest.mark.parametrize("next_environment", ["dev", "custom"]) + def test_changed_owned_file_is_not_replaced_or_removed(self, tmp_path: Path, next_environment: str) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("generated",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("generated", "original"),) + +class Prod(Dev): + pass +""", + ) + binaries_dir = tmp_path / "binaries" + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + generated = binaries_dir / "generated" + generated.write_text("changed\n", encoding="utf-8") + + with pytest.raises(TargetArtifactError, match="Refusing to modify changed artifact entry 'generated'"): + stage_target( + "fake", + next_environment, + repo_root=tmp_path, + binaries_dir=binaries_dir, + ) + + assert generated.read_text(encoding="utf-8") == "changed\n" + + @pytest.mark.parametrize("link_target", ["existing", "broken"]) + def test_artifact_entry_symlink_is_rejected(self, tmp_path: Path, link_target: str) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("selector",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("selector", "generated"),) + +class Prod(Dev): + pass +""", + ) + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + external = tmp_path / "external" + if link_target == "existing": + external.write_text("outside\n", encoding="utf-8") + (binaries_dir / "selector").symlink_to(external) + + with pytest.raises(TargetArtifactError, match="symlink"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert link_target != "broken" or not external.exists() + if link_target == "existing": + assert external.read_text(encoding="utf-8") == "outside\n" + + def test_duplicate_resolver_names_fail_before_resolution( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.resolvers import EnvResolver + +class Dev: + def artifact_entry_filenames(self): + return () + + def artifact_inputs(self, env): + return (EnvResolver(name="duplicate"), EnvResolver(name="duplicate")) + + def artifact_entries(self, resolved_inputs): + return () + +class Prod(Dev): + pass +""", + ) + resolve_calls = 0 + + def fake_resolve(_resolver: EnvResolver, _env: dict[str, str]) -> LiteralValue: + nonlocal resolve_calls + resolve_calls += 1 + return LiteralValue(name="duplicate", value="value") + + monkeypatch.setattr(EnvResolver, "resolve", fake_resolve) + + with pytest.raises(TargetArtifactError, match=r"Duplicate artifact input name.*duplicate"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=tmp_path / "binaries") + + assert resolve_calls == 0 + + def test_dynamic_target_module_supports_dataclasses(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from __future__ import annotations +from dataclasses import dataclass + +@dataclass +class Dev: + value: str = "selector" + + def artifact_entry_filenames(self): + return () + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return () + +class Prod(Dev): + pass +""", + ) + + target_environment = load_target_environment(tmp_path, "fake", "dev") + + assert target_environment.value == "selector" # type: ignore[attr-defined] + + def test_manual_dev_selector_blocks_prod_without_resolving_dev(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("dev-selector",) + + def artifact_inputs(self, env): + raise AssertionError("dev inputs must not be inspected while staging prod") + + def artifact_entries(self, resolved_inputs): + raise AssertionError("dev entries must not be produced while staging prod") + +class Prod: + def artifact_entry_filenames(self): + return ("prod-selector",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("prod-selector", "prod"),) +""", + ) + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + (binaries_dir / "dev-selector").write_text("manual\n", encoding="utf-8") + + with pytest.raises(TargetArtifactError, match=r"conflicting selector 'dev-selector'.*not owned"): + stage_target("fake", "prod", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert not (binaries_dir / "prod-selector").exists() + + def test_manual_prod_selector_blocks_dev_without_resolving_prod(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("dev-selector",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("dev-selector", "dev"),) + +class Prod: + def artifact_entry_filenames(self): + return ("prod-selector",) + + def artifact_inputs(self, env): + raise AssertionError("prod inputs must not be inspected while staging dev") + + def artifact_entries(self, resolved_inputs): + raise AssertionError("prod entries must not be produced while staging dev") +""", + ) + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + (binaries_dir / "prod-selector").write_text("manual\n", encoding="utf-8") + + with pytest.raises(TargetArtifactError, match=r"conflicting selector 'prod-selector'.*not owned"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert not (binaries_dir / "dev-selector").exists() + + def test_selected_environment_can_emit_multiple_entries(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("dev-selector", "dev-marker") + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("dev-selector", "dev"), text_entry("dev-marker", "bounded")) + +class Prod: + def artifact_entry_filenames(self): + return ("prod-selector",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("prod-selector", "prod"),) +""", + ) + binaries_dir = tmp_path / "binaries" + + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert (binaries_dir / "dev-selector").read_text(encoding="utf-8") == "dev\n" + assert (binaries_dir / "dev-marker").read_text(encoding="utf-8") == "bounded\n" + + @pytest.mark.parametrize("filename", ["", "../outside", "nested/entry", MANIFEST_FILENAME]) + def test_invalid_declared_filename_is_rejected_before_resolution(self, tmp_path: Path, filename: str) -> None: + _write_target_module( + tmp_path, + f""" +class Dev: + def artifact_entry_filenames(self): + return ({filename!r},) + + def artifact_inputs(self, env): + raise AssertionError("invalid declarations must fail before resolution") + + def artifact_entries(self, resolved_inputs): + return () + +class Prod(Dev): + pass +""", + ) + + with pytest.raises(TargetArtifactError, match="Invalid artifact entry filename"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=tmp_path / "binaries") + + def test_undeclared_emitted_filename_is_rejected(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ("declared",) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry("undeclared", "value"),) + +class Prod(Dev): + pass +""", + ) + + with pytest.raises(TargetArtifactError, match=r"emitted undeclared artifact entry filename.*undeclared"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=tmp_path / "binaries") + + @pytest.mark.parametrize("filename", ["", "../outside", "nested/entry", MANIFEST_FILENAME]) + def test_invalid_entry_filename_is_rejected(self, tmp_path: Path, filename: str) -> None: + _write_target_module( + tmp_path, + f""" +from utils.target_artifacts.entry_helpers import text_entry + +class Dev: + def artifact_entry_filenames(self): + return ({filename!r},) + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return (text_entry({filename!r}, "generated"),) + +class Prod(Dev): + pass +""", + ) + + with pytest.raises(TargetArtifactError, match="Invalid artifact entry filename"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=tmp_path / "binaries") + + def test_manifest_cannot_delete_files_outside_binaries(self, tmp_path: Path) -> None: + _write_target_module( + tmp_path, + """ +class Dev: + def artifact_entry_filenames(self): + return () + + def artifact_inputs(self, env): + return () + + def artifact_entries(self, resolved_inputs): + return () + +class Prod(Dev): + pass +""", + ) + outside = tmp_path / "outside" + outside.write_text("keep\n", encoding="utf-8") + binaries_dir = tmp_path / "binaries" + binaries_dir.mkdir() + manifest = { + "version": 1, + "entries": {"../outside": {"owner": {"target": "fake", "environment": "dev"}}}, + } + (binaries_dir / MANIFEST_FILENAME).write_text(json.dumps(manifest), encoding="utf-8") + + with pytest.raises(TargetArtifactError, match="Invalid artifact entry filename"): + stage_target("fake", "dev", repo_root=tmp_path, binaries_dir=binaries_dir) + + assert outside.read_text(encoding="utf-8") == "keep\n" + + def test_github_release_resolver_wraps_request_failures(self, monkeypatch: pytest.MonkeyPatch) -> None: + def fail_get(*_args: object, **_kwargs: object) -> object: + raise requests.ConnectionError("network unavailable") + + monkeypatch.setattr("utils.target_artifacts.resolvers.requests.get", fail_get) + resolver = GitHubLatestReleaseResolver(name="release", repository="DataDog/dd-trace-py") + + with pytest.raises(TargetArtifactError, match="Unable to resolve artifact metadata"): + resolver.resolve({}) + + def test_env_resolver_resolves_env_input(self) -> None: + resolver = EnvResolver(name="value", variable_name="STAGED_VALUE", default_value="default") + + resolved = resolver.resolve({"STAGED_VALUE": "from-env"}) + + assert resolved == LiteralValue(name="value", value="from-env") + + @pytest.mark.parametrize( + ("resolver_type", "resolved_type"), + [ + (EnvResolver, LiteralValue.__name__), + (GitHubBranchResolver, BranchReference.__name__), + (GitHubLatestReleaseResolver, GitHubReleaseReference.__name__), + (GitHubActionsArtifactResolver, GitHubActionsArtifactReference.__name__), + (OciDigestResolver, OciImageReference.__name__), + (NpmLatestResolver, ModuleVersion.__name__), + (PypiLatestResolver, ModuleVersion.__name__), + (RubygemsLatestResolver, ModuleVersion.__name__), + (CratesLatestResolver, ModuleVersion.__name__), + (GoModuleLatestResolver, ModuleVersion.__name__), + ], + ) + def test_artifact_resolver_docstring_names_resolved_input_type( + self, + resolver_type: type[ArtifactResolver], + resolved_type: str, + ) -> None: + assert resolver_type.__doc__ is not None + assert resolved_type in resolver_type.__doc__ + + +@scenarios.test_the_test +class Test_TargetArtifactResolvers: + def test_github_requests_include_auth_header_when_token_is_provided( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + calls: list[tuple[str, dict[str, str], int]] = [] + + def fake_get(url: str, *, headers: dict[str, str], timeout: int) -> StubResponse: + calls.append((url, dict(headers), timeout)) + return StubResponse({"tag_name": "v1.2.3"}) + + monkeypatch.setattr("utils.target_artifacts.resolvers.requests.get", fake_get) + + resolved = GitHubLatestReleaseResolver(name="release", repository="DataDog/example").resolve( + {"GITHUB_TOKEN": "secret-token"}, + ) + + assert resolved.tag_name == "v1.2.3" + assert calls == [ + ( + "https://api.github.com/repos/DataDog/example/releases/latest", + { + "Accept": "application/vnd.github.v3+json", + "Authorization": "Bearer secret-token", + }, + 30, + ), + ] + + def test_get_json_wraps_non_json_payloads(self, monkeypatch: pytest.MonkeyPatch) -> None: + def fake_get(_url: str, *, headers: dict[str, str], timeout: int) -> StubResponse: + assert headers == {"Accept": "application/vnd.github.v3+json"} + assert timeout == 30 + return StubResponse({}, json_error=ValueError("invalid json")) + + monkeypatch.setattr("utils.target_artifacts.resolvers.requests.get", fake_get) + resolver = GitHubLatestReleaseResolver(name="release", repository="DataDog/example") + + with pytest.raises(TargetArtifactError, match="Unable to parse artifact metadata"): + resolver.resolve({}) + + def test_github_branch_resolver_accepts_full_sha_without_network(self, monkeypatch: pytest.MonkeyPatch) -> None: + def fail_get_json(url: str, headers: dict[str, str]) -> dict[str, Any]: + raise AssertionError(f"Unexpected GitHub request to {url} with {headers}") + + monkeypatch.setattr("utils.target_artifacts.resolvers._get_json", fail_get_json) + resolver = GitHubBranchResolver( + name="library_branch", + repository="DataDog/dd-trace-py", + variable_name="LIBRARY_TARGET_BRANCH", + ) + + resolved = resolver.resolve({"LIBRARY_TARGET_BRANCH": SHA}) + + assert resolved == BranchReference( + name="library_branch", + repository="DataDog/dd-trace-py", + branch=SHA, + sha=SHA, + ) + + def test_github_branch_resolver_resolves_quoted_branch_name(self, monkeypatch: pytest.MonkeyPatch) -> None: + branch = "feature/space branch" + expected_url = "https://api.github.com/repos/DataDog/dd-trace-py/branches/feature%2Fspace%20branch" + calls = _stub_get_json( + monkeypatch, + { + expected_url: { + "commit": { + "sha": OTHER_SHA, + }, + }, + }, + ) + resolver = GitHubBranchResolver( + name="library_branch", + repository="DataDog/dd-trace-py", + variable_name="LIBRARY_TARGET_BRANCH", + ) + + resolved = resolver.resolve({"GITHUB_TOKEN": "secret-token", "LIBRARY_TARGET_BRANCH": branch}) + + assert resolved == BranchReference( + name="library_branch", + repository="DataDog/dd-trace-py", + branch=branch, + sha=OTHER_SHA, + ) + assert calls == [ + ( + expected_url, + { + "Accept": "application/vnd.github.v3+json", + "Authorization": "Bearer secret-token", + }, + ), + ] + + def test_github_branch_resolver_rejects_missing_branch(self) -> None: + resolver = GitHubBranchResolver(name="library_branch", repository="DataDog/dd-trace-py") + + with pytest.raises(TargetArtifactError, match="Missing branch for input 'library_branch'"): + resolver.resolve({}) + + def test_github_branch_resolver_rejects_invalid_sha(self, monkeypatch: pytest.MonkeyPatch) -> None: + _stub_get_json( + monkeypatch, + { + "https://api.github.com/repos/DataDog/dd-trace-py/branches/main": { + "commit": { + "sha": "not-a-sha", + }, + }, + }, + ) + resolver = GitHubBranchResolver( + name="library_branch", + repository="DataDog/dd-trace-py", + default_value="main", + ) + + with pytest.raises(TargetArtifactError, match="did not resolve to a commit SHA"): + resolver.resolve({}) + + def test_github_latest_release_resolver_includes_assets(self, monkeypatch: pytest.MonkeyPatch) -> None: + _stub_get_json( + monkeypatch, + { + "https://api.github.com/repos/DataDog/dd-trace-java/releases/latest": { + "tag_name": "v1.2.3", + "assets": [ + { + "name": "dd-java-agent.jar", + "browser_download_url": "https://github.example/dd-java-agent.jar", + }, + ], + }, + }, + ) + resolver = GitHubLatestReleaseResolver( + name="release", + repository="DataDog/dd-trace-java", + include_assets=True, + ) + + resolved = resolver.resolve({}) + + assert resolved == GitHubReleaseReference( + name="release", + repository="DataDog/dd-trace-java", + tag_name="v1.2.3", + assets=( + ReleaseAsset( + name="dd-java-agent.jar", + browser_download_url="https://github.example/dd-java-agent.jar", + ), + ), + ) + + def test_github_latest_release_resolver_rejects_missing_assets(self, monkeypatch: pytest.MonkeyPatch) -> None: + _stub_get_json( + monkeypatch, + { + "https://api.github.com/repos/DataDog/dd-trace-java/releases/latest": { + "tag_name": "v1.2.3", + }, + }, + ) + resolver = GitHubLatestReleaseResolver( + name="release", + repository="DataDog/dd-trace-java", + include_assets=True, + ) + + with pytest.raises(TargetArtifactError, match="did not include assets"): + resolver.resolve({}) + + def test_github_actions_artifact_resolver_selects_matching_artifact( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + expected_runs_url = ( + "https://api.github.com/repos/DataDog/httpd-datadog/actions/workflows/dev.yml/runs" + "?branch=feature%2Fbranch&status=completed&per_page=100" + ) + expected_artifacts_url = "https://api.github.example/runs/123/artifacts?per_page=100" + calls = _stub_get_json( + monkeypatch, + { + expected_runs_url: { + "workflow_runs": [ + { + "conclusion": "failure", + }, + { + "conclusion": "success", + "artifacts_url": "https://api.github.example/runs/123/artifacts", + "head_sha": SHA, + "id": 123, + "html_url": "https://github.example/DataDog/httpd-datadog/actions/runs/123", + }, + ], + }, + expected_artifacts_url: { + "artifacts": [ + { + "id": 456, + "name": "logs", + "archive_download_url": "https://github.example/logs.zip", + }, + { + "id": 789, + "name": "mod_datadog_artifact.zip", + "archive_download_url": "https://github.example/mod_datadog_artifact.zip", + }, + ], + }, + }, + ) + resolver = GitHubActionsArtifactResolver( + name="workflow_artifact", + repository="DataDog/httpd-datadog", + workflow="dev.yml", + artifact_name="mod_datadog_artifact", + variable_name="LIBRARY_TARGET_BRANCH", + ) + + resolved = resolver.resolve({"LIBRARY_TARGET_BRANCH": "feature/branch"}) + + assert resolved == GitHubActionsArtifactReference( + name="workflow_artifact", + repository="DataDog/httpd-datadog", + workflow="dev.yml", + branch="feature/branch", + commit_sha=SHA, + run_id=123, + run_url="https://github.example/DataDog/httpd-datadog/actions/runs/123", + artifact_id=789, + artifact_name="mod_datadog_artifact.zip", + archive_download_url="https://github.example/mod_datadog_artifact.zip", + ) + assert [url for url, _headers in calls] == [expected_runs_url, expected_artifacts_url] + + def test_github_actions_artifact_resolver_errors_when_only_failed_runs_exist( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + _stub_get_json( + monkeypatch, + { + "https://api.github.com/repos/DataDog/httpd-datadog/actions/workflows/dev.yml/runs" + "?branch=main&status=completed&per_page=100": { + "workflow_runs": [ + { + "conclusion": "failure", + }, + ], + }, + }, + ) + resolver = GitHubActionsArtifactResolver( + name="workflow_artifact", + repository="DataDog/httpd-datadog", + workflow="dev.yml", + artifact_name="mod_datadog_artifact", + default_value="main", + ) + + with pytest.raises(TargetArtifactError, match="No completed workflow run found"): + resolver.resolve({}) + + def test_github_actions_artifact_resolver_errors_when_artifact_is_missing( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + _stub_get_json( + monkeypatch, + { + "https://api.github.com/repos/DataDog/httpd-datadog/actions/workflows/dev.yml/runs" + "?branch=main&status=completed&per_page=100": { + "workflow_runs": [ + { + "conclusion": "success", + "artifacts_url": "https://api.github.example/runs/123/artifacts", + "head_sha": SHA, + "id": 123, + "html_url": "https://github.example/DataDog/httpd-datadog/actions/runs/123", + }, + ], + }, + "https://api.github.example/runs/123/artifacts?per_page=100": { + "artifacts": [ + { + "id": 456, + "name": "logs", + "archive_download_url": "https://github.example/logs.zip", + }, + ], + }, + }, + ) + resolver = GitHubActionsArtifactResolver( + name="workflow_artifact", + repository="DataDog/httpd-datadog", + workflow="dev.yml", + artifact_name="mod_datadog_artifact", + default_value="main", + ) + + with pytest.raises(TargetArtifactError, match="No artifact containing 'mod_datadog_artifact' found"): + resolver.resolve({}) + + def test_oci_digest_resolver_accepts_pinned_digest_without_docker(self, monkeypatch: pytest.MonkeyPatch) -> None: + def fail_run( + args: list[str], *, capture_output: bool, check: bool, text: bool + ) -> subprocess.CompletedProcess[str]: + raise AssertionError(f"Unexpected docker invocation: {args}, {capture_output}, {check}, {text}") + + monkeypatch.setattr("utils.target_artifacts.resolvers.subprocess.run", fail_run) + image = f"registry.example.com/team/app@{DIGEST}" + resolver = OciDigestResolver(name="image", image=image) + + resolved = resolver.resolve({}) + + assert resolved == OciImageReference( + name="image", + image=image, + digest=DIGEST, + reference=image, + ) + + def test_oci_digest_resolver_builds_digest_reference_for_registry_with_port( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + image = "registry.example.com:5000/team/app:latest" + + def fake_run( + args: list[str], + *, + capture_output: bool, + check: bool, + text: bool, + ) -> subprocess.CompletedProcess[str]: + assert args == ["docker", "buildx", "imagetools", "inspect", image] + assert capture_output is True + assert check is False + assert text is True + return _completed_process(args, stdout=f"Name: {image}\nDigest: {DIGEST}\n") + + monkeypatch.setattr("utils.target_artifacts.resolvers.subprocess.run", fake_run) + resolver = OciDigestResolver(name="image", image=image) + + resolved = resolver.resolve({}) + + assert resolved == OciImageReference( + name="image", + image=image, + digest=DIGEST, + reference=f"registry.example.com:5000/team/app@{DIGEST}", + ) + + @pytest.mark.parametrize( + ("run_result", "match"), + [ + (FileNotFoundError(), "docker was not found"), + (_completed_process([], returncode=1, stderr="denied"), "denied"), + (_completed_process([], stdout="Name: registry.example.com/app\n"), "Unable to find OCI digest"), + ], + ) + def test_oci_digest_resolver_wraps_docker_failures( + self, + monkeypatch: pytest.MonkeyPatch, + run_result: subprocess.CompletedProcess[str] | FileNotFoundError, + match: str, + ) -> None: + def fake_run( + args: list[str], + *, + capture_output: bool, + check: bool, + text: bool, + ) -> subprocess.CompletedProcess[str]: + assert capture_output is True + assert check is False + assert text is True + if isinstance(run_result, FileNotFoundError): + raise run_result + return _completed_process( + args, returncode=run_result.returncode, stdout=run_result.stdout, stderr=run_result.stderr + ) + + monkeypatch.setattr("utils.target_artifacts.resolvers.subprocess.run", fake_run) + resolver = OciDigestResolver(name="image", image="registry.example.com/app:latest") + + with pytest.raises(TargetArtifactError, match=match): + resolver.resolve({}) + + @pytest.mark.parametrize( + ("resolver", "payload", "expected"), + [ + ( + NpmLatestResolver(name="package", package="@datadog/browser-core"), + {"version": "1.2.3"}, + ModuleVersion(name="package", module="@datadog/browser-core", version="1.2.3"), + ), + ( + PypiLatestResolver(name="package", package="ddtrace"), + {"info": {"version": "2.3.4"}}, + ModuleVersion(name="package", module="ddtrace", version="2.3.4"), + ), + ( + RubygemsLatestResolver(name="package", package="datadog"), + {"version": "3.4.5"}, + ModuleVersion(name="package", module="datadog", version="3.4.5"), + ), + ( + CratesLatestResolver(name="package", package="datadog-opentelemetry"), + {"crate": {"max_stable_version": "0.1.0", "max_version": "0.2.0"}}, + ModuleVersion(name="package", module="datadog-opentelemetry", version="0.1.0"), + ), + ], + ) + def test_package_registry_resolvers_return_versions( + self, + monkeypatch: pytest.MonkeyPatch, + resolver: NpmLatestResolver | PypiLatestResolver | RubygemsLatestResolver | CratesLatestResolver, + payload: dict[str, Any], + expected: ModuleVersion, + ) -> None: + calls = _stub_get_json(monkeypatch, lambda _url, _headers: payload) + + resolved = resolver.resolve({}) + + assert resolved == expected + assert len(calls) == 1 + + def test_crates_latest_resolver_falls_back_to_max_version(self, monkeypatch: pytest.MonkeyPatch) -> None: + _stub_get_json( + monkeypatch, + lambda _url, _headers: { + "crate": { + "max_stable_version": None, + "max_version": "0.2.0", + }, + }, + ) + resolver = CratesLatestResolver(name="package", package="datadog-opentelemetry") + + resolved = resolver.resolve({}) + + assert resolved == ModuleVersion(name="package", module="datadog-opentelemetry", version="0.2.0") + + def test_crates_latest_resolver_sends_descriptive_user_agent(self, monkeypatch: pytest.MonkeyPatch) -> None: + calls = _stub_get_json( + monkeypatch, + lambda _url, _headers: { + "crate": { + "max_stable_version": "0.1.0", + }, + }, + ) + resolver = CratesLatestResolver(name="package", package="datadog-opentelemetry") + + resolver.resolve({}) + + assert calls == [ + ( + "https://crates.io/api/v1/crates/datadog-opentelemetry", + { + "Accept": "application/json", + "User-Agent": "system-tests-target-artifacts (https://github.com/DataDog/system-tests)", + }, + ), + ] + + @pytest.mark.parametrize( + ("resolver", "payload", "match"), + [ + ( + NpmLatestResolver(name="package", package="dd-trace"), + {}, + "NPM package dd-trace did not include a version", + ), + ( + PypiLatestResolver(name="package", package="ddtrace"), + {}, + "Expected PyPI package ddtrace info to be an object", + ), + ( + RubygemsLatestResolver(name="package", package="datadog"), + {"version": ""}, + "RubyGems package datadog did not include a version", + ), + ( + CratesLatestResolver(name="package", package="datadog-opentelemetry"), + {"crate": {}}, + "crate datadog-opentelemetry did not include a version", + ), + ], + ) + def test_package_registry_resolvers_reject_missing_versions( + self, + monkeypatch: pytest.MonkeyPatch, + resolver: NpmLatestResolver | PypiLatestResolver | RubygemsLatestResolver | CratesLatestResolver, + payload: dict[str, Any], + match: str, + ) -> None: + _stub_get_json(monkeypatch, lambda _url, _headers: payload) + + with pytest.raises(TargetArtifactError, match=match): + resolver.resolve({}) + + def test_go_module_latest_resolver_returns_version(self, monkeypatch: pytest.MonkeyPatch) -> None: + module = "github.com/DataDog/dd-trace-go/v2" + + def fake_run( + args: list[str], + *, + capture_output: bool, + check: bool, + text: bool, + ) -> subprocess.CompletedProcess[str]: + assert args == ["go", "list", "-m", "-json", f"{module}@latest"] + assert capture_output is True + assert check is False + assert text is True + return _completed_process(args, stdout='{"Version": "v1.2.3"}') + + monkeypatch.setattr("utils.target_artifacts.resolvers.subprocess.run", fake_run) + resolver = GoModuleLatestResolver(name="module", module=module) + + resolved = resolver.resolve({}) + + assert resolved == ModuleVersion(name="module", module=module, version="v1.2.3") + + @pytest.mark.parametrize( + ("run_result", "match"), + [ + (FileNotFoundError(), "go was not found"), + (_completed_process([], returncode=1, stderr="module not found"), "module not found"), + (_completed_process([], stdout="{not-json"), "Unable to parse Go module metadata"), + ( + _completed_process([], stdout='{"Path": "github.com/DataDog/dd-trace-go/v2"}'), + "did not include a version", + ), + ], + ) + def test_go_module_latest_resolver_wraps_go_failures( + self, + monkeypatch: pytest.MonkeyPatch, + run_result: subprocess.CompletedProcess[str] | FileNotFoundError, + match: str, + ) -> None: + def fake_run( + args: list[str], + *, + capture_output: bool, + check: bool, + text: bool, + ) -> subprocess.CompletedProcess[str]: + assert capture_output is True + assert check is False + assert text is True + if isinstance(run_result, FileNotFoundError): + raise run_result + return _completed_process( + args, returncode=run_result.returncode, stdout=run_result.stdout, stderr=run_result.stderr + ) + + monkeypatch.setattr("utils.target_artifacts.resolvers.subprocess.run", fake_run) + resolver = GoModuleLatestResolver(name="module", module="github.com/DataDog/dd-trace-go/v2") + + with pytest.raises(TargetArtifactError, match=match): + resolver.resolve({}) + + +@scenarios.test_the_test +class Test_TargetArtifactModules: + @pytest.mark.parametrize("environment", ["dev", "prod"]) + def test_python_staging_emits_a_bounded_selector(self, environment: str) -> None: + target_environment = load_target_environment(Path.cwd(), "python", environment) + env = {"LIBRARY_TARGET_BRANCH": "feature-branch"} if environment == "dev" else {} + resolver = FakeResolver() + resolved = { + artifact_resolver.name: resolver.resolve(artifact_resolver, env) + for artifact_resolver in target_environment.artifact_inputs(env) + } + + entries = target_environment.artifact_entries(resolved) + + assert len(entries) == 1 + assert target_environment.artifact_entry_filenames() == (entries[0].filename,) + if environment == "dev": + assert entries[0].filename == "python-load-from-s3" + assert entries[0].content == f"{SHA}\n" + else: + assert entries[0].filename == "python-load-from-pip" + assert entries[0].content == "ddtrace==1.2.3\n" diff --git a/utils/__init__.py b/utils/__init__.py index 56fe17b8fae..9bad69eba76 100644 --- a/utils/__init__.py +++ b/utils/__init__.py @@ -2,25 +2,31 @@ # This product includes software developed at Datadog (https://www.datadoghq.com/). # Copyright 2021 Datadog, Inc. -# singletons -from utils._weblog import weblog, HttpResponse -from utils._context.core import context -from utils._context._scenarios import scenarios, scenario_groups -from utils._decorators import ( - bug, - irrelevant, - missing_feature, - rfc, - flaky, - incomplete_test_app, - slow, - scenario_crash, - auxiliary_test, -) -from utils._logger import logger -from utils import interfaces, _remote_config as remote_config -from utils.interfaces._core import ValidationError -from utils._features import features +from __future__ import annotations + +from importlib import import_module +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + import utils._remote_config as remote_config + from utils import interfaces + from utils._context._scenarios import scenario_groups as scenario_groups + from utils._context._scenarios import scenarios as scenarios + from utils._context.core import context as context + from utils._decorators import auxiliary_test as auxiliary_test + from utils._decorators import bug as bug + from utils._decorators import flaky as flaky + from utils._decorators import incomplete_test_app as incomplete_test_app + from utils._decorators import irrelevant as irrelevant + from utils._decorators import missing_feature as missing_feature + from utils._decorators import rfc as rfc + from utils._decorators import scenario_crash as scenario_crash + from utils._decorators import slow as slow + from utils._features import features as features + from utils._logger import logger as logger + from utils._weblog import HttpResponse as HttpResponse + from utils._weblog import weblog as weblog + from utils.interfaces._core import ValidationError as ValidationError __all__ = [ "HttpResponse", @@ -43,3 +49,36 @@ "slow", "weblog", ] + +_LAZY_EXPORTS = { + "HttpResponse": ("utils._weblog", "HttpResponse"), + "ValidationError": ("utils.interfaces._core", "ValidationError"), + "auxiliary_test": ("utils._decorators", "auxiliary_test"), + "bug": ("utils._decorators", "bug"), + "context": ("utils._context.core", "context"), + "features": ("utils._features", "features"), + "flaky": ("utils._decorators", "flaky"), + "incomplete_test_app": ("utils._decorators", "incomplete_test_app"), + "interfaces": ("utils.interfaces", None), + "irrelevant": ("utils._decorators", "irrelevant"), + "logger": ("utils._logger", "logger"), + "missing_feature": ("utils._decorators", "missing_feature"), + "remote_config": ("utils._remote_config", None), + "rfc": ("utils._decorators", "rfc"), + "scenario_crash": ("utils._decorators", "scenario_crash"), + "scenario_groups": ("utils._context._scenarios", "scenario_groups"), + "scenarios": ("utils._context._scenarios", "scenarios"), + "slow": ("utils._decorators", "slow"), + "weblog": ("utils._weblog", "weblog"), +} + + +def __getattr__(name: str) -> object: + if name not in _LAZY_EXPORTS: + raise AttributeError(f"module 'utils' has no attribute '{name}'") + + module_name, attribute_name = _LAZY_EXPORTS[name] + module = import_module(module_name) + value = module if attribute_name is None else getattr(module, attribute_name) + globals()[name] = value + return value diff --git a/utils/build/docker/python/artifact.py b/utils/build/docker/python/artifact.py new file mode 100644 index 00000000000..9af29c5006f --- /dev/null +++ b/utils/build/docker/python/artifact.py @@ -0,0 +1,23 @@ +from __future__ import annotations + + +from utils.target_artifacts.entry_helpers import text_entry +from utils.target_artifacts.models import SimpleTarget +from utils.target_artifacts.resolvers import GitHubBranchResolver, PypiLatestResolver + + +class Dev(SimpleTarget): + inputs = ( + GitHubBranchResolver( + name="library_branch", + repository="DataDog/dd-trace-py", + variable_name="LIBRARY_TARGET_BRANCH", + default_value="main", + ), + ) + entries = (text_entry("python-load-from-s3", "{library_branch.sha}"),) + + +class Prod(SimpleTarget): + inputs = (PypiLatestResolver(name="ddtrace", package="ddtrace"),) + entries = (text_entry("python-load-from-pip", "ddtrace=={ddtrace.version}"),) diff --git a/utils/scripts/load-binary.sh b/utils/scripts/load-binary.sh index 378da8b92ea..6e0f6b630a7 100755 --- a/utils/scripts/load-binary.sh +++ b/utils/scripts/load-binary.sh @@ -152,6 +152,7 @@ fi TARGET=$1 VERSION=${2:-'dev'} +BINARIES_DIR=${BINARIES_DIR:-binaries} GITHUB_TOKEN="${GITHUB_TOKEN:-}" GITHUB_AUTH_HEADER=() @@ -161,7 +162,15 @@ fi echo "Load $VERSION binary for $TARGET" -cd "${BINARIES_DIR:-binaries}/" +if [ "$TARGET" = "python" ]; then + python3 utils/scripts/stage-target-artifacts.py \ + "$TARGET" "$VERSION" \ + --binaries-dir "$BINARIES_DIR" \ + --repo-root . + exit 0 +fi + +cd "$BINARIES_DIR/" if [ "$TARGET" = "c" ]; then if [ "$VERSION" = "prod" ]; then @@ -218,13 +227,6 @@ elif [ "$TARGET" = "dotnet" ]; then ../utils/scripts/docker_base_image.sh "ghcr.io/datadog/dd-trace-dotnet/dd-trace-dotnet:${NORMALIZED_BRANCH}" . -elif [ "$TARGET" = "python" ]; then - assert_version_is_dev - - LIBRARY_TARGET_BRANCH="${LIBRARY_TARGET_BRANCH:-main}" - echo "Using $LIBRARY_TARGET_BRANCH in S3 for DataDog/dd-trace-py" - echo "$LIBRARY_TARGET_BRANCH" > python-load-from-s3 - elif [ "$TARGET" = "ruby" ]; then assert_version_is_dev diff --git a/utils/scripts/stage-target-artifacts.py b/utils/scripts/stage-target-artifacts.py new file mode 100755 index 00000000000..8f07aec28db --- /dev/null +++ b/utils/scripts/stage-target-artifacts.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 + +from __future__ import annotations + +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO_ROOT)) + +from utils.target_artifacts.cli import main # noqa: E402 + +raise SystemExit(main()) diff --git a/utils/target_artifacts/__init__.py b/utils/target_artifacts/__init__.py new file mode 100644 index 00000000000..38f0b1a076d --- /dev/null +++ b/utils/target_artifacts/__init__.py @@ -0,0 +1,34 @@ +from .models import ( + ArtifactEntry, + ArtifactResolver, + BranchReference, + GitHubActionsArtifactReference, + GitHubReleaseReference, + LiteralValue, + ModuleVersion, + OciImageReference, + ReleaseAsset, + ResolvedArtifactInput, + SimpleTarget, + TargetArtifactEnvironment, + TargetArtifactError, +) +from .orchestrator import MANIFEST_FILENAME, stage_target + +__all__ = [ + "MANIFEST_FILENAME", + "ArtifactEntry", + "ArtifactResolver", + "BranchReference", + "GitHubActionsArtifactReference", + "GitHubReleaseReference", + "LiteralValue", + "ModuleVersion", + "OciImageReference", + "ReleaseAsset", + "ResolvedArtifactInput", + "SimpleTarget", + "TargetArtifactEnvironment", + "TargetArtifactError", + "stage_target", +] diff --git a/utils/target_artifacts/__main__.py b/utils/target_artifacts/__main__.py new file mode 100644 index 00000000000..eb53e2f31b2 --- /dev/null +++ b/utils/target_artifacts/__main__.py @@ -0,0 +1,3 @@ +from .cli import main + +raise SystemExit(main()) diff --git a/utils/target_artifacts/cli.py b/utils/target_artifacts/cli.py new file mode 100644 index 00000000000..eeb160d09a4 --- /dev/null +++ b/utils/target_artifacts/cli.py @@ -0,0 +1,28 @@ +from __future__ import annotations + +import argparse +import os +import sys +from pathlib import Path + +from .models import TargetArtifactError +from .orchestrator import stage_target + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="stage-target-artifacts") + parser.add_argument("target", help="Target artifact name, such as python, java, or custom") + parser.add_argument("environment", nargs="?", default="dev", help="dev, prod, or custom") + parser.add_argument("--binaries-dir", default=os.environ.get("BINARIES_DIR", "binaries")) + parser.add_argument("--repo-root", default=".") + + args = parser.parse_args(argv) + repo_root = Path(args.repo_root) + binaries_dir = Path(args.binaries_dir) + + try: + stage_target(args.target, args.environment, repo_root=repo_root, binaries_dir=binaries_dir) + except TargetArtifactError as exc: + sys.stderr.write(f"{exc}\n") + return 1 + return 0 diff --git a/utils/target_artifacts/entry_helpers.py b/utils/target_artifacts/entry_helpers.py new file mode 100644 index 00000000000..52fe781ecfd --- /dev/null +++ b/utils/target_artifacts/entry_helpers.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +import json + +from .models import ( + ArtifactEntry, + GitHubActionsArtifactReference, + TargetArtifactError, +) + + +def text_entry( + filename: str, + content: str, +) -> ArtifactEntry: + return ArtifactEntry( + filename=filename, + content=f"{content.rstrip()}\n", + ) + + +def json_entry(filename: str, payload: dict[str, object]) -> ArtifactEntry: + if not filename.endswith(".json"): + raise TargetArtifactError(f"JSON artifact entry '{filename}' must use a .json extension") + return ArtifactEntry(filename=filename, content=f"{json.dumps(payload, sort_keys=True)}\n") + + +def gha_artifact_entry(filename: str, artifact: GitHubActionsArtifactReference) -> ArtifactEntry: + """Create a JSON artifact entry from a GitHub Actions artifact reference.""" + return json_entry( + filename, + { + "archive_download_url": artifact.archive_download_url, + "artifact_id": artifact.artifact_id, + "artifact_name": artifact.artifact_name, + "commit_sha": artifact.commit_sha, + "repository": artifact.repository, + "run_id": artifact.run_id, + "run_url": artifact.run_url, + "workflow": artifact.workflow, + }, + ) + + +def provider_fetch_entries( + *, + fetch_filename: str, + fetch_selector: str, + marker_filename: str, + bounded_selector: str, +) -> tuple[ArtifactEntry, ArtifactEntry]: + """Create a provider fetch entry plus its bounded selection marker. + + Some providers require installer-facing fetch selectors that are not + themselves bounded, such as branch-derived package image tags. The fetch + entry is consumed by the build to retrieve the provider artifact, while the + marker entry records the bounded selector used for cache identity. + """ + return ( + text_entry(fetch_filename, fetch_selector), + text_entry(marker_filename, bounded_selector), + ) diff --git a/utils/target_artifacts/models.py b/utils/target_artifacts/models.py new file mode 100644 index 00000000000..eb658d0155b --- /dev/null +++ b/utils/target_artifacts/models.py @@ -0,0 +1,141 @@ +from dataclasses import dataclass +from typing import Protocol, runtime_checkable + + +class TargetArtifactError(Exception): + """Expected target artifact configuration or resolution failure.""" + + +@dataclass(frozen=True) +class ArtifactEntry: + filename: str + content: str + + +@dataclass(frozen=True) +class LiteralValue: + name: str + value: str + + +@dataclass(frozen=True) +class BranchReference: + name: str + repository: str + branch: str + sha: str + + +@dataclass(frozen=True) +class ReleaseAsset: + name: str + browser_download_url: str + + +@dataclass(frozen=True) +class GitHubReleaseReference: + name: str + repository: str + tag_name: str + assets: tuple[ReleaseAsset, ...] = () + + +@dataclass(frozen=True) +class GitHubActionsArtifactReference: + name: str + repository: str + workflow: str + branch: str + commit_sha: str + run_id: int + run_url: str + artifact_id: int + artifact_name: str + archive_download_url: str + + +@dataclass(frozen=True) +class OciImageReference: + name: str + image: str + digest: str + reference: str + + +@dataclass(frozen=True) +class ModuleVersion: + name: str + module: str + version: str + + +type ResolvedArtifactInput = ( + LiteralValue + | BranchReference + | GitHubReleaseReference + | GitHubActionsArtifactReference + | OciImageReference + | ModuleVersion +) + + +class ArtifactResolver(Protocol): + """Resolver implementations document their resolved model type in their class docstring.""" + + @property + def name(self) -> str: + """Resolved input name.""" + ... + + def resolve(self, env: dict[str, str], /) -> ResolvedArtifactInput: + """Resolve one declared artifact input.""" + ... + + +@runtime_checkable +class TargetArtifactEnvironment(Protocol): + def artifact_entry_filenames(self) -> tuple[str, ...]: + """Declare every artifact entry filename this environment can emit.""" + ... + + def artifact_inputs( + self, + env: dict[str, str], + ) -> tuple[ArtifactResolver, ...]: + """Declare the inputs needed to produce artifact entries.""" + ... + + def artifact_entries( + self, + resolved_inputs: dict[str, ResolvedArtifactInput], + ) -> tuple[ArtifactEntry, ...]: + """Return text artifact entries from resolved inputs.""" + ... + + +class SimpleTarget: + """Declarative base for targets with static inputs and template entries. + + Subclasses set ``inputs`` (a tuple of resolvers) and ``entries`` (a tuple + of ``ArtifactEntry`` whose ``content`` uses ``{resolver_name.field}`` + format placeholders). The orchestrator resolves the inputs and formats + each entry's content with the resolved values. + """ + + inputs: tuple[ArtifactResolver, ...] = () + entries: tuple[ArtifactEntry, ...] = () + + def artifact_inputs(self, _env: dict[str, str]) -> tuple[ArtifactResolver, ...]: + return self.inputs + + def artifact_entry_filenames(self) -> tuple[str, ...]: + return tuple(entry.filename for entry in self.entries) + + def artifact_entries(self, resolved_inputs: dict[str, ResolvedArtifactInput]) -> tuple[ArtifactEntry, ...]: + return tuple( + ArtifactEntry( + filename=entry.filename, + content=entry.content.format(**resolved_inputs), + ) + for entry in self.entries + ) diff --git a/utils/target_artifacts/orchestrator.py b/utils/target_artifacts/orchestrator.py new file mode 100644 index 00000000000..1be950a4c0f --- /dev/null +++ b/utils/target_artifacts/orchestrator.py @@ -0,0 +1,254 @@ +from __future__ import annotations + +import hashlib +import importlib.util +import json +import os +import sys +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from .models import ( + ArtifactEntry, + TargetArtifactEnvironment, + TargetArtifactError, +) + +if TYPE_CHECKING: + from types import ModuleType + +MANIFEST_FILENAME = ".target-artifacts-manifest.json" +MANIFEST_VERSION = 1 + + +def stage_target( + target: str, + environment: str, + *, + repo_root: Path | None = None, + binaries_dir: Path | None = None, + process_env: dict[str, str] | None = None, +) -> None: + root = Path.cwd() if repo_root is None else repo_root + output_dir = Path(os.environ.get("BINARIES_DIR", "binaries")) if binaries_dir is None else binaries_dir + output_dir = output_dir if output_dir.is_absolute() else root / output_dir + + env = dict(os.environ if process_env is None else process_env) + + if environment not in {"dev", "prod"}: + if environment == "custom": + manifest_path = output_dir / MANIFEST_FILENAME + if manifest_path.exists() or manifest_path.is_symlink(): + write_artifact_entries(output_dir, target, environment, ()) + return + raise TargetArtifactError(f"Unknown target artifact environment: {environment}") + + target_environments = load_target_environments(root, target) + target_environment = target_environments[environment] + environment_filenames = { + name: _validate_declared_filenames(target, name, target_env.artifact_entry_filenames()) + for name, target_env in target_environments.items() + } + artifact_inputs = target_environment.artifact_inputs(env) + input_names = [artifact_resolver.name for artifact_resolver in artifact_inputs] + duplicate_names = sorted({name for name in input_names if input_names.count(name) > 1}) + if duplicate_names: + raise TargetArtifactError(f"Duplicate artifact input name(s): {', '.join(duplicate_names)}") + resolved_inputs = {artifact_resolver.name: artifact_resolver.resolve(env) for artifact_resolver in artifact_inputs} + entries = target_environment.artifact_entries(resolved_inputs) + undeclared_filenames = sorted({entry.filename for entry in entries} - set(environment_filenames[environment])) + if undeclared_filenames: + raise TargetArtifactError( + f"{target}.{environment} emitted undeclared artifact entry filename(s): {', '.join(undeclared_filenames)}" + ) + selector_filenames = tuple(filename for filenames in environment_filenames.values() for filename in filenames) + write_artifact_entries(output_dir, target, environment, entries, selector_filenames=selector_filenames) + + +def load_target_environment(repo_root: Path, target: str, environment: str) -> TargetArtifactEnvironment: + return load_target_environments(repo_root, target)[environment] + + +def load_target_environments(repo_root: Path, target: str) -> dict[str, TargetArtifactEnvironment]: + module_path = repo_root / "utils" / "build" / "docker" / target / "artifact.py" + if not module_path.exists(): + raise TargetArtifactError(f"No target artifact module found for '{target}' at {module_path}") + + module = _load_module(module_path, f"system_tests_target_artifacts_{target}") + result: dict[str, TargetArtifactEnvironment] = {} + for environment, class_name in (("dev", "Dev"), ("prod", "Prod")): + environment_class = getattr(module, class_name, None) + if environment_class is None: + raise TargetArtifactError(f"Target artifact module for '{target}' does not define {class_name}") + + instance = environment_class() + if not isinstance(instance, TargetArtifactEnvironment): + raise TargetArtifactError(f"{target}.{class_name} does not implement TargetArtifactEnvironment") + result[environment] = instance + return result + + +def write_artifact_entries( + binaries_dir: Path, + target: str, + environment: str, + entries: tuple[ArtifactEntry, ...], + *, + selector_filenames: tuple[str, ...] = (), +) -> None: + manifest = _read_manifest(binaries_dir) + manifest_entries = _manifest_entries(manifest) + new_entries = _dedupe_entries(entries) + owner = {"target": target, "environment": environment} + + for filename in manifest_entries: + _validate_filename(filename) + + for filename in selector_filenames: + _validate_filename(filename) + + for filename, metadata in manifest_entries.items(): + if _same_target(metadata.get("owner"), target): + _validate_owned_file(binaries_dir / filename, filename, metadata) + + for filename in new_entries: + _validate_filename(filename) + existing_owner = manifest_entries.get(filename, {}).get("owner") + path = binaries_dir / filename + if path.is_symlink(): + raise TargetArtifactError(f"Refusing to write artifact entry through symlink '{filename}'") + if existing_owner is not None and not _same_target(existing_owner, target): + owner_target = ( + existing_owner.get("target", "") if isinstance(existing_owner, dict) else "" + ) + raise TargetArtifactError(f"Artifact entry '{filename}' is already owned by target '{owner_target}'") + if path.exists() and existing_owner is None: + raise TargetArtifactError(f"Refusing to overwrite unowned artifact entry '{filename}'") + + for selector_filename in set(selector_filenames) - set(new_entries): + selector_path = binaries_dir / selector_filename + if selector_path.is_symlink(): + raise TargetArtifactError(f"Refusing conflicting selector symlink '{selector_filename}'") + if selector_path.exists(): + selector_owner = manifest_entries.get(selector_filename, {}).get("owner") + if not _same_target(selector_owner, target): + raise TargetArtifactError( + f"Refusing conflicting selector '{selector_filename}' because it is not owned by target '{target}'" + ) + + for filename, metadata in list(manifest_entries.items()): + owner_data = metadata.get("owner") + if _same_target(owner_data, target) and filename not in new_entries: + path = binaries_dir / filename + if path.exists(): + path.unlink() + del manifest_entries[filename] + + binaries_dir.mkdir(parents=True, exist_ok=True) + for filename, entry in new_entries.items(): + path = binaries_dir / filename + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(entry.content, encoding="utf-8") + manifest_entries[filename] = { + "owner": owner, + "sha256": hashlib.sha256(entry.content.encode("utf-8")).hexdigest(), + } + + manifest["version"] = MANIFEST_VERSION + manifest["entries"] = dict(sorted(manifest_entries.items())) + manifest_content = f"{json.dumps(manifest, indent=2, sort_keys=True)}\n" + (binaries_dir / MANIFEST_FILENAME).write_text(manifest_content, encoding="utf-8") + + +def _load_module(module_path: Path, module_name: str) -> ModuleType: + spec = importlib.util.spec_from_file_location(module_name, module_path) + if spec is None or spec.loader is None: + raise TargetArtifactError(f"Unable to import target artifact module at {module_path}") + module = importlib.util.module_from_spec(spec) + previous_module = sys.modules.get(module_name) + sys.modules[module_name] = module + try: + spec.loader.exec_module(module) + except BaseException: + if previous_module is None: + del sys.modules[module_name] + else: + sys.modules[module_name] = previous_module + raise + return module + + +def _read_manifest(binaries_dir: Path) -> dict[str, Any]: + path = binaries_dir / MANIFEST_FILENAME + if path.is_symlink(): + raise TargetArtifactError(f"Artifact manifest {path} must not be a symlink") + if not path.exists(): + return {"version": MANIFEST_VERSION, "entries": {}} + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise TargetArtifactError(f"Unable to read artifact manifest {path}") from exc + if not isinstance(payload, dict): + raise TargetArtifactError(f"Artifact manifest {path} is not an object") + if payload.get("version") != MANIFEST_VERSION: + raise TargetArtifactError(f"Unsupported artifact manifest version in {path}") + return payload + + +def _manifest_entries(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]: + entries = manifest.get("entries") + if not isinstance(entries, dict): + raise TargetArtifactError("Artifact manifest entries must be an object") + return {str(name): _metadata(metadata) for name, metadata in entries.items()} + + +def _metadata(value: object) -> dict[str, Any]: + if not isinstance(value, dict): + raise TargetArtifactError("Artifact manifest entry metadata must be an object") + return value + + +def _dedupe_entries(entries: tuple[ArtifactEntry, ...]) -> dict[str, ArtifactEntry]: + result: dict[str, ArtifactEntry] = {} + for entry in entries: + if entry.filename in result: + raise TargetArtifactError(f"Duplicate artifact entry '{entry.filename}'") + result[entry.filename] = entry + return result + + +def _validate_declared_filenames(target: str, environment: str, filenames: tuple[str, ...]) -> tuple[str, ...]: + duplicates = sorted({filename for filename in filenames if filenames.count(filename) > 1}) + if duplicates: + raise TargetArtifactError( + f"{target}.{environment} declares duplicate artifact entry filename(s): {', '.join(duplicates)}" + ) + for filename in filenames: + _validate_filename(filename) + return filenames + + +def _validate_filename(filename: str) -> None: + path = Path(filename) + if not filename or path.name != filename or filename == MANIFEST_FILENAME: + raise TargetArtifactError(f"Invalid artifact entry filename '{filename}'") + + +def _validate_owned_file(path: Path, filename: str, metadata: dict[str, Any]) -> None: + if path.is_symlink(): + raise TargetArtifactError(f"Refusing to modify owned artifact entry symlink '{filename}'") + if not path.exists(): + return + expected_hash = metadata.get("sha256") + if not isinstance(expected_hash, str): + raise TargetArtifactError(f"Owned artifact entry '{filename}' has no valid recorded hash") + try: + actual_hash = hashlib.sha256(path.read_bytes()).hexdigest() + except OSError as exc: + raise TargetArtifactError(f"Unable to verify owned artifact entry '{filename}'") from exc + if actual_hash != expected_hash: + raise TargetArtifactError(f"Refusing to modify changed artifact entry '{filename}'") + + +def _same_target(owner: object, target: str) -> bool: + return isinstance(owner, dict) and owner.get("target") == target diff --git a/utils/target_artifacts/resolvers.py b/utils/target_artifacts/resolvers.py new file mode 100644 index 00000000000..06d321dab43 --- /dev/null +++ b/utils/target_artifacts/resolvers.py @@ -0,0 +1,378 @@ +from __future__ import annotations + +import json +import re +import subprocess +from dataclasses import dataclass +from typing import Any +from urllib.parse import quote + +import requests + +from .models import ( + BranchReference, + GitHubActionsArtifactReference, + GitHubReleaseReference, + LiteralValue, + ModuleVersion, + OciImageReference, + ReleaseAsset, + TargetArtifactError, +) + +REQUEST_TIMEOUT_SECONDS = 30 +FULL_SHA_PATTERN = re.compile(r"^[0-9a-f]{40}$") +CRATES_IO_HEADERS = { + "Accept": "application/json", + "User-Agent": "system-tests-target-artifacts (https://github.com/DataDog/system-tests)", +} + + +@dataclass(frozen=True) +class EnvResolver: + """Resolve an environment variable to LiteralValue.""" + + name: str + variable_name: str = "" + default_value: str = "" + + def resolve(self, env: dict[str, str]) -> LiteralValue: + value = env.get(self.variable_name, self.default_value) + return LiteralValue(name=self.name, value=value) + + +class _GitHubResolver: + @staticmethod + def _github_headers(env: dict[str, str]) -> dict[str, str]: + headers = {"Accept": "application/vnd.github.v3+json"} + token = env.get("GITHUB_TOKEN", "") + if token: + headers["Authorization"] = f"Bearer {token}" + return headers + + def _github_get(self, url: str, env: dict[str, str]) -> dict[str, Any]: + return _get_json(url, self._github_headers(env)) + + +@dataclass(frozen=True) +class GitHubBranchResolver(_GitHubResolver): + """Resolve a GitHub branch or commit SHA to BranchReference.""" + + name: str + repository: str + variable_name: str = "" + default_value: str = "" + + def resolve(self, env: dict[str, str]) -> BranchReference: + branch = env.get(self.variable_name, self.default_value) + if not branch: + raise TargetArtifactError(f"Missing branch for input '{self.name}'") + if FULL_SHA_PATTERN.match(branch): + return BranchReference( + name=self.name, + repository=self.repository, + branch=branch, + sha=branch, + ) + + payload = self._github_get( + f"https://api.github.com/repos/{self.repository}/branches/{quote(branch, safe='')}", + env, + ) + commit = _mapping(payload.get("commit"), f"branch '{branch}' commit") + sha = commit.get("sha") + if not isinstance(sha, str) or FULL_SHA_PATTERN.match(sha) is None: + raise TargetArtifactError(f"Branch '{branch}' in {self.repository} did not resolve to a commit SHA") + return BranchReference( + name=self.name, + repository=self.repository, + branch=branch, + sha=sha, + ) + + +@dataclass(frozen=True) +class GitHubLatestReleaseResolver(_GitHubResolver): + """Resolve the latest GitHub release to GitHubReleaseReference.""" + + name: str + repository: str + include_assets: bool = False + + def resolve(self, env: dict[str, str]) -> GitHubReleaseReference: + payload = self._github_get(f"https://api.github.com/repos/{self.repository}/releases/latest", env) + tag_name = payload.get("tag_name") + if not isinstance(tag_name, str) or not tag_name: + raise TargetArtifactError(f"Latest release for {self.repository} did not include a tag") + + assets: tuple[ReleaseAsset, ...] = () + if self.include_assets: + raw_assets = payload.get("assets") + if not isinstance(raw_assets, list): + raise TargetArtifactError(f"Latest release for {self.repository} did not include assets") + assets = tuple(_release_asset(asset) for asset in raw_assets) + + return GitHubReleaseReference( + name=self.name, + repository=self.repository, + tag_name=tag_name, + assets=assets, + ) + + +@dataclass(frozen=True) +class GitHubActionsArtifactResolver(_GitHubResolver): + """Resolve a GitHub Actions workflow artifact to GitHubActionsArtifactReference.""" + + name: str + repository: str + workflow: str + artifact_name: str + variable_name: str = "" + default_value: str = "" + ignore_failed_workflow: bool = True + + def resolve(self, env: dict[str, str]) -> GitHubActionsArtifactReference: + branch = env.get(self.variable_name, self.default_value) + if not branch: + raise TargetArtifactError(f"Missing workflow branch for input '{self.name}'") + + runs_payload = self._github_get( + "https://api.github.com/repos/" + f"{self.repository}/actions/workflows/{self.workflow}/runs" + f"?branch={quote(branch, safe='')}&status=completed&per_page=100", + env, + ) + runs = runs_payload.get("workflow_runs") + if not isinstance(runs, list): + raise TargetArtifactError(f"Workflow runs were not returned for {self.repository}") + + selected_run: dict[str, Any] | None = None + for run in runs: + run_mapping = _mapping(run, "workflow run") + if self.ignore_failed_workflow and run_mapping.get("conclusion") == "failure": + continue + selected_run = run_mapping + break + + if selected_run is None: + raise TargetArtifactError(f"No completed workflow run found for {self.repository}@{branch}") + + artifacts_url = selected_run.get("artifacts_url") + if not isinstance(artifacts_url, str): + raise TargetArtifactError("Selected workflow run did not include artifacts_url") + artifacts_payload = self._github_get(f"{artifacts_url}?per_page=100", env) + artifacts = artifacts_payload.get("artifacts") + if not isinstance(artifacts, list): + raise TargetArtifactError("Workflow artifacts were not returned") + + selected_artifact: dict[str, Any] | None = None + for artifact in artifacts: + artifact_mapping = _mapping(artifact, "workflow artifact") + artifact_name = artifact_mapping.get("name") + if isinstance(artifact_name, str) and self.artifact_name in artifact_name: + selected_artifact = artifact_mapping + break + + if selected_artifact is None: + raise TargetArtifactError(f"No artifact containing '{self.artifact_name}' found for {self.repository}") + + return GitHubActionsArtifactReference( + name=self.name, + repository=self.repository, + workflow=self.workflow, + branch=branch, + commit_sha=_required_str(selected_run, "head_sha"), + run_id=_required_int(selected_run, "id"), + run_url=_required_str(selected_run, "html_url"), + artifact_id=_required_int(selected_artifact, "id"), + artifact_name=_required_str(selected_artifact, "name"), + archive_download_url=_required_str(selected_artifact, "archive_download_url"), + ) + + +@dataclass(frozen=True) +class OciDigestResolver: + """Resolve an OCI image tag to OciImageReference.""" + + name: str + image: str = "" + variable_name: str = "" + default_value: str = "" + + def resolve(self, env: dict[str, str]) -> OciImageReference: + image = env.get(self.variable_name, self.image or self.default_value) + if not image: + raise TargetArtifactError(f"Missing OCI image for input '{self.name}'") + if "@sha256:" in image: + digest = image.rsplit("@", 1)[1] + return OciImageReference( + name=self.name, + image=image, + digest=digest, + reference=image, + ) + + try: + result = subprocess.run( + ["docker", "buildx", "imagetools", "inspect", image], + capture_output=True, + check=False, + text=True, + ) + except FileNotFoundError as exc: + raise TargetArtifactError("Unable to resolve OCI digest: docker was not found") from exc + if result.returncode != 0: + raise TargetArtifactError(f"Unable to resolve OCI digest for {image}: {result.stderr.strip()}") + + digest = "" + for line in result.stdout.splitlines(): + stripped = line.strip() + if stripped.startswith("Digest:"): + digest = stripped.removeprefix("Digest:").strip() + break + + if not digest.startswith("sha256:"): + raise TargetArtifactError(f"Unable to find OCI digest for {image}") + + last_slash = image.rfind("/") + last_colon = image.rfind(":") + repository = image[:last_colon] if last_colon > last_slash else image + return OciImageReference( + name=self.name, + image=image, + digest=digest, + reference=f"{repository}@{digest}", + ) + + +@dataclass(frozen=True) +class NpmLatestResolver: + """Resolve the latest npm package version to ModuleVersion.""" + + name: str + package: str + + def resolve(self, _env: dict[str, str]) -> ModuleVersion: + payload = _get_json(f"https://registry.npmjs.org/{quote(self.package, safe='@/')}/latest", {}) + version = payload.get("version") + if not isinstance(version, str) or not version: + raise TargetArtifactError(f"NPM package {self.package} did not include a version") + return ModuleVersion(name=self.name, module=self.package, version=version) + + +@dataclass(frozen=True) +class PypiLatestResolver: + """Resolve the latest PyPI package version to ModuleVersion.""" + + name: str + package: str + + def resolve(self, _env: dict[str, str]) -> ModuleVersion: + payload = _get_json(f"https://pypi.org/pypi/{quote(self.package, safe='')}/json", {}) + info = _mapping(payload.get("info"), f"PyPI package {self.package} info") + version = info.get("version") + if not isinstance(version, str) or not version: + raise TargetArtifactError(f"PyPI package {self.package} did not include a version") + return ModuleVersion(name=self.name, module=self.package, version=version) + + +@dataclass(frozen=True) +class RubygemsLatestResolver: + """Resolve the latest RubyGems package version to ModuleVersion.""" + + name: str + package: str + + def resolve(self, _env: dict[str, str]) -> ModuleVersion: + payload = _get_json(f"https://rubygems.org/api/v1/gems/{quote(self.package, safe='')}.json", {}) + version = payload.get("version") + if not isinstance(version, str) or not version: + raise TargetArtifactError(f"RubyGems package {self.package} did not include a version") + return ModuleVersion(name=self.name, module=self.package, version=version) + + +@dataclass(frozen=True) +class CratesLatestResolver: + """Resolve the latest crates.io package version to ModuleVersion.""" + + name: str + package: str + + def resolve(self, _env: dict[str, str]) -> ModuleVersion: + payload = _get_json(f"https://crates.io/api/v1/crates/{quote(self.package, safe='')}", CRATES_IO_HEADERS) + crate = _mapping(payload.get("crate"), f"crate {self.package}") + version = crate.get("max_stable_version") or crate.get("max_version") + if not isinstance(version, str) or not version: + raise TargetArtifactError(f"crate {self.package} did not include a version") + return ModuleVersion(name=self.name, module=self.package, version=version) + + +@dataclass(frozen=True) +class GoModuleLatestResolver: + """Resolve the latest Go module version to ModuleVersion.""" + + name: str + module: str + + def resolve(self, _env: dict[str, str]) -> ModuleVersion: + try: + result = subprocess.run( + ["go", "list", "-m", "-json", f"{self.module}@latest"], + capture_output=True, + check=False, + text=True, + ) + except FileNotFoundError as exc: + raise TargetArtifactError("Unable to resolve Go module: go was not found") from exc + if result.returncode != 0: + raise TargetArtifactError(f"Unable to resolve Go module {self.module}: {result.stderr.strip()}") + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise TargetArtifactError(f"Unable to parse Go module metadata for {self.module}") from exc + version = payload.get("Version") + if not isinstance(version, str) or not version: + raise TargetArtifactError(f"Go module {self.module} did not include a version") + return ModuleVersion(name=self.name, module=self.module, version=version) + + +def _get_json(url: str, headers: dict[str, str]) -> dict[str, Any]: + try: + response = requests.get(url, headers=dict(headers), timeout=REQUEST_TIMEOUT_SECONDS) + response.raise_for_status() + except requests.RequestException as exc: + raise TargetArtifactError(f"Unable to resolve artifact metadata from {url}: {exc}") from exc + try: + payload = response.json() + except ValueError as exc: + raise TargetArtifactError(f"Unable to parse artifact metadata from {url}") from exc + return _mapping(payload, f"response from {url}") + + +def _mapping(value: object, description: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise TargetArtifactError(f"Expected {description} to be an object") + return value + + +def _release_asset(value: object) -> ReleaseAsset: + item = _mapping(value, "release asset") + return ReleaseAsset( + name=_required_str(item, "name"), + browser_download_url=_required_str(item, "browser_download_url"), + ) + + +def _required_str(value: dict[str, Any], key: str) -> str: + result = value.get(key) + if not isinstance(result, str) or not result: + raise TargetArtifactError(f"Expected '{key}' to be a non-empty string") + return result + + +def _required_int(value: dict[str, Any], key: str) -> int: + result = value.get(key) + if not isinstance(result, int): + raise TargetArtifactError(f"Expected '{key}' to be an integer") + return result