Maven dependency resolution on real Maven APIs#8240
Draft
jkschneider wants to merge 34 commits into
Draft
Conversation
…chmarks Studies of rewrite-maven's custom resolution and the embeddable Maven stack (resolver 2.x + Maven 3.9 model-builder), the canonical design (DESIGN.md), Phase 0 spec, runnable feasibility/provenance/cycle spikes, corpus tooling, and comparative benchmark baselines.
…resolve A download failure in any single scope discarded the entire MavenResolutionResult marker, losing the resolvable scopes. The aggregated MavenDownloadingExceptions now carries the partial result and MavenParser attaches it alongside the ParseExceptionResult, so the complete model survives with the failure surfaced (parity ledger L-P0-004).
Shape-aware ResolutionSnapshot/ResolutionDiff, determinism and identity gates, serialized-LST compatibility fixtures, a MockWebServer-backed synthetic behavior corpus (mirrors, auth, snapshots, negative caching, file repos, HTML-index derivation), the inert resolution-engine selector, and the parity ledger seeded with the first seven divergences.
…spine (Phase 1) Resolver 2.0.20 + Maven 3.9.16 model-builder relocated under org.openrewrite.maven.engine.shaded with an empty-allowlist jar guard, MavenEngine bootstrap whose session template mirrors Maven 3.9's own session factory, and HttpSenderTransporter as the sole network transport (per-session sender, structural no-bypass).
The Maven model builder reads real XML; parsed Pom objects are lossy. Raw pom bytes become a cacheable region keyed like the parsed-Pom region, added as default methods so host implementations keep compiling. RocksDB stores positives only under a prefixed keyspace so raw XML can never be misread as Smile-serialized Pom entries; the Pom.modelVersion bump stays deferred to the descriptor-region change.
Classified KEEP_REWRITE as a transport-layer augmentation: it only produces an answer where Maven produces an error, never a different answer where Maven has one.
Pure-translation settings seam: request-repository assembly with today's ordering, mirror and credential semantics, session selectors for descriptor-discovered repositories agreeing with the request path, per-server transport config, and EffectiveSettings for model building. ReactorWorkspace serves printed-XML raw models with the three-tier reactor matching and an epoch counter for re-resolution invalidation. Consumes the engine's shadowRuntimeElements variant explicitly and corrects its advertised TargetJvmVersion to the true Java 8. (cherry picked from commit 3c56b49568017ee3cf33f506a5bfa18f2215348d)
…ase 2) EngineEffectivePom builds effective models from real XML through the seams: CacheBridge supplies pom bytes from the pluggable cache with fall-through to the resolver path (HttpSenderTransporter I/O, scratch LRM, per-repo attribution), EngineModelResolver handles parents/BOMs with range support and recessive repository aggregation, and reactor poms flow through ReactorWorkspace (bypassing the session model cache, whose relativePath cross-check distrusts in-memory sources; staleness is guarded by the workspace's own epoch-cleared cache). Cycles and self-parents fail Maven-identically via ModelParityErrorMapper.
…(Phase 2) EffectivePomMapper projects ModelBuildingResult into ResolvedPom per the canonical design: raw-lineage properties with injected-property overlay, InputLocation-joined dependency management threading the original ManagedDependency instances, GA-keyed requested-dependency merge preserving instance identity, and super-POM-contributed entries filtered to keep the projected model declared/inherited-only (ledger L-P2-B2-001). BomGavAttributor stamps imported-BOM provenance by effective-DM membership, correct for BOMs that inherit their management. Differential gate: 10/11 parity fixtures project identically to the legacy engine; the eleventh differs only by the Maven-correct plugin ordering fixes (L-P0-002/003).
…r (Phase 2) MavenEngineResolution routes effective-pom construction: legacy (default, untouched), maven (EngineEffectivePom + mappers), or shadow — both engines run, the projected poms diff under the ledgered mask registry, and any unexplained difference or asymmetric failure throws with the rendered diff while legacy's result is returned. PomXmlRegistry carries printed document bytes from MavenParser and UpdateMavenModel (with reactor-epoch invalidation) so the engine always builds from real XML. Parity snapshot classes move to main sources to serve the comparator. Shadow census on MavenParserTest: 41 clean, 37+1 masked-expected flips, 31 open findings recorded for the Phase-2 burn-down.
Twenty-two engine/mapper fixes: projected gav interpolation with raw-vs-interpolated modelId reconciliation for property-versioned poms, reactor and inherited BOM imports served through the workspace with provenance attribution, RELEASE/LATEST metaversion resolution, injected parser properties overriding declared ones, and the DM projection deduplicated by g:a:classifier:type (proven against maven-model-builder source to be output shape, not Maven semantics). Nine parity-correct strictness cases where the engine fails Maven-identically are ledgered as outcome masks (L-P2-D-001..008) that flip at cutover. MavenParserTest in shadow: zero unexplained differences.
…fixes (Phase 2) Root cause of the 90-test shadow failure wave: recipes that mutate the requested Pom and call resolve() directly (ChangeParentPom's scanner) bypassed both PomXmlRegistry refresh points, so the engine built from stale initial-parse XML. The registry now binds each payload to its source Pom and serves bytes only while the stored source equals the requested instance, falling back to PomToModelConverter otherwise — covering every direct-resolve path uniformly. Residuals: declaring-pom empty-vs-null property representation, injected properties no longer leak onto downloaded poms, the dependency-resolution pass is guarded legacy-only for Phase 2, bomGav matching interpolates import coordinates, and UpdateMavenModel re-reads profile properties. Ledger L-P2-E-001/002 (outcome + value-conditioned metaversion masks); L-P2-E-003 (nested import-BOM version selection) is a real engine defect deliberately left red for the Phase-3 collection work.
…machinery (Phase 3) EngineDependencyCollector runs the single widest-scope verbose collect (per the large-reactor benchmark decision) with winner/loser and premanaged state intact. EngineDescriptorReader implements the descriptor SPI over EngineEffectivePom with Maven 3.9's exact tolerance profile (missing dep pom tolerated, unresolvable parent/BOM fails, relocation loop guarded); the direct-fails/transitive-warns split is applied post-collect as rewrite's reporting layer. PinnedVersionResolver honors pinned snapshots transitively without metadata reads. (cherry picked from commit 7be030f30b6337e4e1b5e61fe95eafb874f50e83)
…ndencies (Phase 3) DependencyGraphMapper projects the single widest-scope verbose graph into the four pure per-scope lists with the identity contracts intact: requested instances threaded at depth 0, winners shared within a scope and never across scopes, legacy ordering and membership semantics, and direct failures aggregated with the partial result per the complete-model contract. ExclusionAttributor recomputes effectiveExclusions as reporting. Scopes differential: 12/12 fixtures zero unexplained. L-P2-E-003 inverted on investigation: the engine's DM superset comes from Maven correctly activating a property-negation profile (infinispan-bom community) that legacy's env-var-based activation drops — reclassified ALIGN_TO_MAVEN with a hermetic pinning test and a directional dm-superset mask that can never hide a value divergence or a drop.
…t shadow (Phase 3) MavenResolutionResult.resolveDependencies dispatches legacy (default), maven (single collect projected per scope, preserving the partial-result and per-GA dedup contracts), or shadow — now diffing the complete snapshot including dependency graphs, with re-entrancy guarded. The first full census with scopes compared found five gap mechanisms (ledgered L-P3-C-001..005, left red and unmasked): exclusion attribution, reactor-transitive attribution, closure divergence, and requested-shape threading. Also fixes a real pre-existing concurrency race: parallel tests publishing different bodies for the same GAV into the process-global local repository (reproduced in legacy mode) now serialize through the publish-run-cleanup window.
Exclusion attribution rebuilt over descriptor-time declared-dependency capture with path-based shallowest-ancestor attribution; reactor and warm-pool attribution made cumulative across collects; requested threading interpolates property coordinates before matching; and the scope of test-reachable compile transitives now replicates legacy's classpath derivation after ground truth showed aether's scope promotion disagreeing with both Maven and legacy. Census 381 to 172, dominated by one pre-existing effective-pom attribution class (L-P3-D-004) plus the machine-state-dependent local-repository attribution class (L-P3-D-002) and depth-tolerance residue (L-P3-D-003), all ledgered red.
The dmRequestedRef volume leader was a re-resolution identity subtlety, not attribution: legacy's no-change gate returns the caller's instance still threading pre-edit BOM declarations, so the engine now threads against the caller's pom when threading inputs are value-equal. Local repository attribution replicates legacy's file-repo gate exactly (machine-independence proven against a populated ~/.m2). Jar-type pom-404 failures align to the arbitrated behavior: any depth, but only resolved-graph members, root-attributed for per-scope reporting. Three more pre-existing classes found and fixed along the way (duplicate declaration retention, parent-inherited requested threading, shaded super-POM plugin-groupId leak). Remaining shadow failures are all ledger-classified: Phase-5 ALIGN flips, strictness outcomes, mediation residue, and two newly ledgered OPEN gaps (L-P3-E-004/005). Legacy full suite: 1583/0. Maven-mode worst-three classes: 77 percent.
…orpus The corpus runner consumes the local build (Maven Local snapshot wiring; composite build blocked by a Gradle classloader-scope bug), passes the engine selector through, records per-entry shadow verdicts instead of aborting, and tops up the HTTP store with the engine's checksum and HEAD request profile so hermetic replay covers both engines with zero misses. First census: 10 of 20 entries clean or masked-expected; the other 10 classified to ledgered residues plus four new evidenced gap classes (project.parent.version interpolation in threaded requested, os-profile activation machine-dependence, mirror attribution convention, system-scoped profile dependency retention). Determinism verified in-JVM and across JVMs. (cherry picked from commit c69218c271b19ceb8dbd622727098e476025acef)
…on dedup (Phase 3) RELEASE and LATEST seeds resolve through merged metadata before collection so descriptor reads carry concrete versions, pinned hermetically. Ground truth from real Maven arbitrated the duplicate scope/classifier/type dedup: the engine is Maven-identical in all three shapes and legacy is the outlier — those pins flip at cutover. One new bug ledgered open (L-P3-E-006, intra-pom duplicate declaration raw threading) after a blunt fix regressed 81 tests and was reverted; the remaining mediation value divergences stay red under drop-safety.
…se 3) The project.parent namespace now interpolates in threaded transitive requested coordinates (dubbo's class eliminated). Mirror attribution resolves through the parent-merged effective-model repositories so the declared mirror is attributed as legacy does. The file-exists-activated system-scoped dependency arbitrated Maven-correct. OS/JDK profile activation becomes an embedder input: MavenExecutionContextView.setActivationSystemProperties, defaulting to host values (Maven parity), pinnable by hosts wanting fleet-determinism — corpus runner pins a canonical snapshot; the directional ALIGN mask awaits sign-off. Synthetic request-log assertions are legacy-scoped where shadow's dual-engine traffic invalidates exact counts. Census 94, every entry ledger-classified.
… (Phase 4) Ground truth dissolved the engine-drops-members premise: dual-engine flat-set probes show identical per-scope sets on every recurring victim; the divergences are nested tree shape and attribution where the engine matches real Maven and legacy is the outlier — L-P3-C-004 reclassified from bulk residue to individually arbitrated ALIGN. The one genuine engine bug: a dependency reached at depth 1 as runtime and depth 2 as compile belongs on the compile classpath (Maven never narrows scope across paths); the mapper now trusts the wider aether edge scope while preserving the no-promote guard. Pinned by CompileScopeWideningTest. Clean full-census confirmation deferred: Central rate-limiting degraded mid-slice; hermetic gates green and unchanged.
…ss hygiene (Phase 4) Transitive requested classifiers thread the declaring pom's raw value — hermetic repro inverted the assumed direction (legacy preserves empty classifier elements). The recipe re-resolution residue reclassified: commons-io 1.3.2 is a relocation the engine follows Maven-correctly and legacy ignores — registered as an ALIGN flip with fixture. The intra-pom duplicate hypothesis was disproven by hermetic repro and ledgered as such. The synthetic transport suite pins legacy in shadow mode since its dual-run reds are already-ledgered value divergences, not new findings.
…Phase 4) The end-to-end gate FAILS: maven mode is 2.3-4.6x slower than legacy across all tiers with no crossover, ~6x allocation at maven-15, and an FD-exhaustion crash on BOM-heavy reactors from unreleased materialized pom streams — recorded as flip blockers with measured cost centers (benchmarks/integrated-2026-07-09). The raw pipeline resolves the same warm reactor ~25x faster than the integrated path, so the gap is per-iteration reconstruction, not adapter arithmetic. Debt closed: EngineCollectSystemSupplier now extends the public engine supplier so one class owns the transport monopoly and RRF-off (pinned), and metadata reads route through the pluggable region with tri-state semantics (pinned, zero-network warm metaversion).
… (Phase 4) The file-descriptor exhaustion was aether's file-based named-lock factory holding a FileChannel per artifact across context lifetimes — in-JVM locks are the correct choice for a private per-run scratch repository and were also the top measured cost. Super-POM re-reads collapse to a shared singleton builder, descriptor builds share the session cache (72 percent hit rate), imported-BOM attribution caches per epoch instead of rebuilding per module, and model sources are byte-backed instead of materialized. Warm ratios improve from 2.3-4.6x to 1.5-1.9x with allocation within 1.4x and stable descriptors at 2g. Gate still short of targets: the residual is structural (root-build fresh cache for servedBy attribution; ModelBuilder per-invocation weight) — named with evidence for the next slice.
A ctx-lifetime memo above DefaultModelBuilder, keyed by reactor epoch, active profiles, and content hash, serves every repeated build — the re-resolution loop that dominates recipe time drops from up to 2.64x to 1.04-1.10x of legacy with zero model rebuilds, invalidating wholesale on any mutation (pinned by a memo-hit-plus-invalidation test). Root-cache sharing was measured unnecessary and documented rather than built. The remaining warm-parse gap (1.26-1.49x) is proven structural: ~3,215 full model builds per warm camel-400 parse that legacy's RawPom path never pays; closing it requires a descriptor-only projection, a design trade against the parity guarantee, recorded as the Phase-5 decision input.
Transitive descriptor model builds now run without location tracking, matching DefaultArtifactDescriptorReader.loadPom exactly; the root and BOM-attribution builds keep full provenance, and the grades are cache-partitioned by construction. Correctness proven identical across the differential suites and corpus replay. The A/B answered the open question negatively: tracking was not the warm cost — per-build attribution puts it in ModelBuilder's inheritance assembly, interpolation, and import processing, confirming the structural floor.
The internal selector now defaults to MAVEN; the legacy resolver stays in-tree for one release as revert-by-release insurance, reachable only through the internal dev/CI key. SHADOW returns the engine result in every routing (it must never change behavior relative to the default), and the single-scope projection routes through the engine in shadow mode as well.
ChangeParentPom's speculative trial resolve and UpdateMavenModel's
re-resolution both meet Maven's model validation now: a transiently
invalid document (managed version renamed before its dependency, a new
parent that no longer manages a version the visitor later repairs)
keeps the previous resolution instead of failing the run.
UpdateMavenProjectPropertyJavaVersion reads the parent's requested pom
plugins, since the effective model interpolates ${java.version}-style
placeholders from session system properties exactly as Maven does
(ledger row L-P5-A-001).
…e 5) The 216 tests that pinned legacy resolution behavior now assert what real Maven does: fixtures gain pom packaging where Maven requires it (aggregators, parents), tolerance tests assert Maven's model-validation failures (self-parent, expression cycles, missing versions, systemPath), duplicate declarations collapse on Maven's g:a:classifier:type conflict key, relocations are followed, dependencyManagement metaversions stay literal, and projected repositories carry only declared/inherited entries. The parity comparison harness and the synthetic transport suite pin the legacy engine explicitly where they document its semantics until its deletion. Sixteen tests stay deliberately red on engine-side gaps tracked for arbitration before release.
…s (Phase 5) Code comments no longer reference parity-ledger row ids or its classification vocabulary; the masks registry keeps ids as its build-enforced data format. Adds ADR 0010 documenting the resolution engine decision and backfills the ADR index.
…them (Phase 5) Maven's interpolation value sources are user properties, then model properties, then system properties; only injected (user) properties override a pom-declared value. Inverts the legacy pin that had a bare system property clobbering the pom.
Everything the pin inversion surfaced as a genuine engine-side gap: - datedSnapshotVersion carries the timestamped form for genuine snapshots, derived from the already-cached metadata region with the legacy downloader's exact precedence; releases still duplicate the plain version deliberately. - Transitive classifiers interpolate through the declaring pom's property lineage, like versions already did; an explicit empty classifier still threads as "". - HTML-index metadata derivation now exists on the engine path (RegionMetadataResolver), with the same self-disable semantics and region caching; the synthetic suite asserts it engine-side. - downloadError fires once per definitively-failed pom download at the CacheBridge seam. - The transporter retries a connection-level http failure once over https and remembers the answer, replacing legacy's eager per-repo probe. - Invalid descriptors of resolvable artifacts are tolerated at every depth, matching Maven's uniform ArtifactDescriptorPolicy; only missing POMs keep failing. - Maven-1 poms skip dependency resolution again (deliberate leniency, like the modelVersion injection). - Non-pom-type scope=import entries stay unexpanded exactly as Maven warns-and-retains them; legacy had over-expanded on scope alone. - RemoveRedundantDependencyVersions compares interpolated plugin versions and respects inherited=false parent management; PrintMavenAsDot dedups edges by value rather than instance identity.
…es (Phase 5) A synthetic single-dependency pom can be model-invalid under Maven's validation exactly when a recipe edits away the constraint or platform that supplied a version; record the failure like a download error and fall back to the originally resolved dependency instead of failing the run. The recorded exception type keeps the real class name — consumers treat it as an opaque display string. Also forwards the dev/CI engine selector to rewrite-gradle's forked test JVMs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces rewrite-maven's custom dependency-resolution algorithm (~4.5k lines:
ResolvedPom.Resolver,doResolveDependencies,VersionRequirement) with real Maven APIs: Maven Resolver 2.0.20 and Maven 3.9.16's model builder, shaded into a newrewrite-maven-enginemodule. Any divergence from Maven's resolution behavior is a bug, not a feature — recipes now see the same effective POMs, dependency graphs, and failuresmvnitself produces.The frozen
org.openrewrite.maven.tree.*API, wire format, and LST identity are unchanged. No user-facing configuration flags: this is a single flip-over. The legacy resolver stays in-tree, unreachable through any supported surface, for one release (revert-by-release), then gets deleted.Design record:
doc/adr/0010-maven-resolution-engine.md. Full plan, benchmarks, and corpus tooling:maven-resolution-plan/.Behavior changes
Every deliberate change is catalogued in
doc/maven-resolution-ledger.md(one row per divergence, each with a pinning test asserting the Maven-identical behavior) and summarized user-facing inmaven-resolution-plan/RELEASE-NOTES.md. Highlights:<os>/<jdk>/<file>conditions;MavenExecutionContextView.setActivationSystemPropertiespins activation inputs for reproducible fleets.groupId:artifactId:classifier:type; duplicate declarations collapse per Maven's model normalizer.<distributionManagement><relocation>) are followed.LATEST/RELEASE) in<dependencyManagement>stay literal, as Maven defers them.systemPathfail per file viaParseExceptionResultwithout aborting the parse run.Architecture
rewrite-maven-engine: shaded stack behindorg.openrewrite.maven.engine.shaded.*; a jar guard test forbids un-relocated classes.HttpSenderTransporteris the sole network transport (HttpSenderinjection remains a hard host requirement);MavenPomCache/MavenArtifactCacheremain the pluggable persistence seams.org.openrewrite.maven.internal.engine: the adapters —MavenEngineResolution(routing facade),EngineEffectivePom/EffectivePomMapper(effective model → frozenResolvedPom),EngineDependencyCollector/DependencyGraphMapper(one verbose collect → per-scopeResolvedDependencytrees),CacheBridge/RegionMetadataResolver(cache regions, HTML-index metadata derivation).internal.parity) diffs both engines under a mask registry in which every mask must cite a ledger row; it proved parity during construction and guards optimization after it.Validation
:rewrite-maven:test(1615 tests) and:rewrite-maven-engine:testgreen on the new default.${revision}reactor): hermetic double-replay, byte-deterministic, zero store misses.:rewrite-gradle:testgreen (the Gradle marker's synthetic-pom path degrades gracefully on Maven-strict validation).mvn3.9.x source and live runs where the two disagreed.Performance
Steady-state recipe re-resolution is at parity (1.04–1.10×). Warm full parse costs 1.26–1.49× — the structural price of real
DefaultModelBuilderbuilds for transitive descriptors. Cold-network parses benefit from parallel prefetch. Benchmarks:maven-resolution-plan/benchmarks/.Known residue (deliberate, ledgered, pre-deletion work)
${...}settings credentials still leak on the pom-bytes path (legacy downloader seam).Pom.modelVersion3→4 bump is deferred until the descriptor-region format lands (LST-consumer coordination).