[Epic] Dynamic dependency management — restartless module and third-party JARs

[ThuF]

Summary

Introduce a first-class dependency management mechanism in the platform: a project declares Maven dependencies (another platform module packaged as an AOT JAR, or a third-party library such as the AWS SDK for Java) in project.json, and the platform resolves and activates them — on the module tier without a restart.

This generalizes the mechanism proven by the AOT compiled-modules work (#6400) and the /modules drop-in directory (#6592): those made module JARs a supported boot-time input; this epic makes dependencies a supported runtime input.

Motivation

Downstream products (BusinessIntents) ship every module as an AOT-compiled JAR with a META-INF/dirigible/<project>/.compiled marker and a registry payload. Today the JAR set is fixed at image build time (-Dloader.path=/modules is read once at JVM launch), so adding a module or any third-party Java library means: rebuild image → redeploy. There is also no authoring surface at all for a third-party Java dependency — a project cannot declare "I need software.amazon.awssdk:s3".

Architecture (agreed design)

project.json (type:"maven" deps)
        │
        ▼
DependencySynchronizer ──► embedded Maven Resolver ──► local repository (~/.m2 layout)
        │                                                      │ (immutable versioned paths)
        ▼                                                      ▼
             ModulesClassLoader — generation N+1 (URLClassLoader, parent = app CL)
        │                    │                     │                        │
        ▼                    ▼                     ▼                        ▼
ClasspathExpander    CompiledModuleClass-   JavaLoader rebuild      JS / Flowable / Camel
expand(jar)→registry Provider.rediscover()  (registry .java vs      via the holder chain
                     →installCompiledModules new classpath)

Key insight: the engine already funnels every consumer of client code through one seam
(ClientClassLoaderHolder, already consumed by Flowable's ClientAwareClassLoader, Camel's
DirigibleJavaInvokerImpl, and the Beans API), and JavaLoader's generation model is documented
as source-agnostic with an idempotent installCompiledModules(...). Restartless
dependencies = giving the classpath itself the same generation lifecycle client classes already
have.

Two dependency scopes with honest contracts:

Scope	Mechanism	Add	Upgrade/remove	For
module (default)	swappable ModulesClassLoader	restartless	restartless	platform modules, pure-Java libs
platform (opt-in)	Instrumentation.appendToSystemClassLoaderSearch via launcher agent	restartless	restart	JDBC drivers, JNI-bearing libs

Child issues

1. Dependency resolver component and 'maven' dependency type in project.json #6777 — Dependency resolver component + maven dependency type in project.json — embedded Maven
   Artifact Resolver, global local repository, boot-time resolution. Value: "declare + restart"
   replaces "rebuild image + redeploy". Zero classloader risk.
2. Swappable modules classloader — restartless add/upgrade/remove of dependency JARs #6778 — Swappable modules classloader — generation lifecycle for the resolved JAR set; five
   integration-point changes; restartless add/upgrade/remove on the module tier.
3. 'platform' dependency scope — restartless add via launcher agent #6779 — platform scope via launcher agent — Instrumentation append for dependencies that must be
   visible to platform code (JDBC drivers, JNI).
4. Dependency lockfile, provided-BOM, and shadowing report #6780 — Lockfile, provided-BOM, and dependency report — reproducibility, shadowing detection, and
   the trust story.
5. Swappable modules classloader (#6787): the swap is inert behind loader.path, and the watcher blocks the shared scheduler #6789 — Review findings on PR Swappable modules classloader — restartless add/upgrade/remove of dependency JARs #6787 — remediation for phase 2 (Swappable modules classloader — restartless add/upgrade/remove of dependency JARs #6778): the swap is inert behind
   loader.path, and the watcher blocks the shared scheduler, plus nine further correctness,
   lifecycle and observability defects.
6. The dependencies watcher re-warns about a legacy typeless project.json entry every five seconds #6827 — Watcher re-warns about a legacy typeless project.json entry — hygiene finding from the
   phase 4 verification: tolerate the legacy guid-only project reference silently and warn about a
   genuinely unknown type once per declaration change, not once per watcher tick.

Phases 1–4 are each independently shippable, in order. Phase 1 has standalone value; phases 2–4
build on it. Items 5 and 6 are remediation and hygiene findings, not phases of their own.

Documentation

Published on dirigible.io:

• Maven dependencies — declaring maven entries in project.json, the module / platform scopes, repositories and credentials.
• How dependency versions are decided — declared → mediated → provided/shadowed → locked; the lockfile, frozen mode and the provided-BOM.
• Dependencies view — the Operations-perspective surface.
• Environment variables — the DIRIGIBLE_DEPENDENCIES_* / DIRIGIBLE_MAVEN_* reference.

Non-goals

• No change to the production posture of downstream products: immutable images remain the
  recommended prod deployment; the dynamic mechanism is guarded by a configuration flag
  (DIRIGIBLE_DEPENDENCIES_DYNAMIC, default per-distribution).
• No Spring bean scanning / auto-configuration from dropped JARs. Module classes flow through
  ComponentContainer and the JavaClassConsumers, as today.
• No child-first classloading. Parent-first delegation stays; version shadowing is reported
  (issue 4), not worked around invisibly.
• pom.xml / Gradle are not adopted as authoring formats. project.json is the single
  authored descriptor; Maven supplies coordinates, transport, and the local-repository layout via
  the embedded resolver (no mvn binary, no generated hidden POMs).

Cross-cutting requirements for every child PR

• Examples in the PR: the PR description must contain a worked end-to-end example
  (the project.json before/after, the command or UI action, and the observable result, e.g. log
  lines / REST responses). New public classes carry javadoc with a usage example.
• Examples in the code: every PR ships integration tests under
  tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/ (suffix IT,
  extending org.eclipse.dirigible.tests.base.IntegrationTest from tests-framework), using
  hermetic fixture repositories (file:// Maven repo layout built by the test — no network).
• License header (EPL-2.0, see licensing-header.txt) on every new file; format with
  dirigible-formatter.xml.

Prior art / references

• feat(engine-java): classpath loading of AOT compiled modules (no runtime javac) #6400 — AOT compiled modules: JavaLoader.installCompiledModules, CompiledModuleClassProvider,
  the .compiled marker.
• AOT compiled modules: drop-in /modules classpath directory in the runtime image (PropertiesLauncher loader.path) — follow-up to #6400 #6592 — /modules drop-in via PropertiesLauncher (build/application/Dockerfile).
• Embedded resolver precedent: JBang, Groovy Grape, ShrinkWrap Resolver (Apache Maven Artifact
  Resolver used programmatically).