♻️ refactor: rename toFHIR → Ignifyr and split the reactor into Community/Enterprise editions - #301
Open
sinaci wants to merge 58 commits into
Open
♻️ refactor: rename toFHIR → Ignifyr and split the reactor into Community/Enterprise editions#301sinaci wants to merge 58 commits into
sinaci wants to merge 58 commits into
Conversation
…afmt config + plugin
…g, and docs. Wholesale rename of the legacy tofhir name: Maven coordinates (io.onfhir:tofhir-* -> io.ignifyr:ignifyr-*), packages (io.tofhir.* -> io.ignifyr.*), ToFhir* classes -> Ignifyr*, module directories, HOCON namespaces (ignifyr / ignifyr-redcap), REST base path /ignifyr, ignifyr-db folder, Docker images srdc/ignifyr-*, IGNIFYR_* env vars, log/logger names, docs and agent guides. Migration notes added to README. Kept until sibling repos rename: srdc/tofhir-web image, tofhir-redcap endpoint value and repo links, SwaggerHub toFHIR docs link. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ries Add io.ignifyr.engine.spi: an aggregate IgnifyrExtension SPI discovered via java.util.ServiceLoader, an ExtensionRegistry indexing contributions by model class, and provider contracts for source connectors, sinks, terminology/identity services and CLI commands. The engine's own implementations register through the same mechanism via CoreExtension (META-INF/services), so moving a feature between editions becomes a module move with no engine code change. Convert the four hard-coded dispatch sites to registry lookups (implementations unchanged, all still in-engine): DataSourceReaderFactory (removed; SourceHandler queries the registry and raises an actionable MissingConnectorException), FhirWriterFactory and IntegratedServiceFactory (registry-backed facades), CommandFactory (registry-backed). Job JSON still parses everywhere because all settings/model classes remain in the engine. Fold in warts: delete the unimplemented AVRO content-type stub, make the dead --db CLI flag work (also accept --db-path), default spark.app.name to "Ignifyr", drop the unused org.reflections direct dependency, and remove the stale ignifyr-server-common_2.13 dependencyManagement entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne jar The plugin split makes the engine a library that connector modules depend on, so the engine can no longer bundle those connectors into its own shaded jar (that would be a dependency cycle). Introduce ignifyr-cli: a downstream assembly module that depends on the engine (and, as they are extracted, the community connector/format plugins) and produces the shaded ignifyr-engine-standalone.jar (Main-Class io.ignifyr.engine.Boot, ServicesResourceTransformer merges every bundled module's META-INF/services). Remove the shade execution from ignifyr-engine (it stays a plain library + test-jar). Point the docker/engine build files and CLAUDE.md at ignifyr-cli/target for the jar. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…in module Move SqlSourceReader out of the engine into a new ignifyr-connector-sql module (package io.ignifyr.connector.sql), registered via the IgnifyrExtension SPI and ServiceLoader (SqlConnectorExtension + META-INF/services). The engine drops the reader, its CoreExtension registration, and the PostgreSQL/DB2 driver dependencies; JDBC drivers now live with the connector (PostgreSQL bundled; other drivers added by the distribution). ignifyr-cli depends on the new module so the community standalone jar still bundles it. A lightweight ServiceLoader registration spec verifies discovery. The full-pipeline SqlSourceTest (H2 + onFHIR container) is parked pending the Phase 2 testkit, which will share the engine's folder-based test fixtures across modules; the reader logic is an unchanged relocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts own plugin module Move FhirServerDataSourceReader out of the engine into a new (enterprise-bound) ignifyr-connector-fhir-server module (package io.ignifyr.connector.fhirserver), registered via the IgnifyrExtension SPI + ServiceLoader. The engine drops the reader, its CoreExtension registration, and the io.onfhir:fhir-api-spark-source dependency — which also lifts that spark-on-fhir SNAPSHOT out of the community engine's dependency path. The server (the enterprise runtime that executes jobs and infers schemas) now depends explicitly on ignifyr-connector-sql and ignifyr-connector-fhir-server, since it no longer gets those readers transitively through the engine. A lightweight registration spec verifies discovery; the full-pipeline FhirServerSourceTest (onFHIR container) is parked for the Phase 2 testkit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…module SchedulingTest loads test-schedule-mappingjob.json, whose job uses a SQL source. With the SQL reader now in ignifyr-connector-sql (which the engine's own test classpath cannot depend on — reactor cycle), the test fails at runtime with the actionable MissingConnectorException. Park it (preserved in git history) and re-home it in the Phase 1.5 scheduling module (ignifyr-runtime-scheduling), which can depend on ignifyr-connector-sql without a cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…om the engine Move KafkaSourceReader out of the engine into a new (enterprise-bound) ignifyr-connector-kafka module (package io.ignifyr.connector.kafka), registered via the IgnifyrExtension SPI. The Spark Kafka connector (spark-sql-kafka-0-10) and the Kafka client libraries leave the community engine with it. Introduce a StreamingFailureDescriptor SPI hook: RunningJobRegistry no longer imports org.apache.kafka UnknownTopicOrPartitionException or casts KafkaSource — on a streaming query failure it asks the registered descriptors to translate the error, and the Kafka connector provides the descriptor that names the missing topic(s). Streaming-query tracking itself stays in the engine (plain Spark SQL). The server (enterprise runtime) depends on the new connector. A registration spec verifies both the source connector and the failure descriptor are discovered; KafkaSourceIntegrationTest is parked for the Phase 2 testkit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nstallable capability Introduce a StreamingExecutionProvider SPI and a MappingTaskPipeline seam in the engine, and move Spark structured-streaming execution into a new (enterprise-bound) ignifyr-runtime-streaming module: StreamingSinkHandler (the foreachBatch wrapper over the engine's batch SinkHandler.writeMappingResult) and StreamingJobExecutor (the former FhirMappingJobManager.startMappingJobStream body). FhirMappingJobManager now implements MappingTaskPipeline and its startMappingJobStream delegates to the installed streaming provider, throwing a clear MissingCapabilityException when none is present — so the community CLI cleanly refuses streaming jobs while the enterprise server (which depends on the new module) runs them. SinkHandler.writeStream is removed from the engine; IgnifyrEngine only starts the file-stream archiver timer when a streaming runtime is installed. The streaming SinkHandler test moves with the code (rate source + mock, no fixtures). FileStreamingTest is parked for the file+formats+testkit batch (it needs both the streaming runtime and the file connector). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…allable capability Move cron4j-based periodic scheduling out of the engine into a new enterprise ignifyr-runtime-scheduling module behind a SchedulerProvider SPI, mirroring the streaming seam. The community engine now carries no scheduling dependency; a job with schedulingSettings and no provider fails with a clear MissingCapabilityException. Engine (community): - New SPI SchedulerProvider + IgnifyrExtension.schedulerProvider + ExtensionRegistry.scheduler (at-most-one, like the streaming capability). - FhirMappingJobManager loses its mappingJobScheduler ctor param and scheduleMappingJob/ runnableMappingJob/getScheduledTimeRange; IFhirMappingJobManager loses scheduleMappingJob. - RunningJobRegistry sheds the scheduling half (scheduledTasks + registerSchedulingJob/ descheduleJobExecution/isScheduled/getScheduledExecutions + the DESCHEDULED shutdown hook); handleCompletedBatchJob is now public so the module's scheduler listener can call it. - MappingJobScheduler deleted; cron4j dependency removed from the engine pom. (mvn dependency:tree -pl ignifyr-engine | grep cron4j -> empty) Enterprise module (ignifyr-runtime-scheduling): - Cron4jSchedulerProvider: cron validation, SQL/plain start-time, last-sync bookkeeping, and the scheduling runnable (the old FhirMappingJobManager scheduling body + MappingJobScheduler). - ScheduledJobRegistry: the scheduled-tasks map + register/deschedule/isScheduled/ getScheduledExecutions + the whenTerminated DESCHEDULED logging. - SchedulingRuntimeExtension registers the provider via ServiceLoader. - Lightweight no-Docker SchedulingRuntimeExtensionSpec (discovery + empty-registry state). Callers rewired onto ExtensionRegistry.scheduler: server ExecutionService (runJob scheduling branch + getExecutions + descheduleJobExecution) and CLI CommandLineInterface.runJob. Server pom gains the module (server = enterprise runtime); ignifyr-cli does not (scheduling is enterprise). Behavior notes (both intended, verified benign): the server no longer requires ignifyr.db for non-scheduled jobs (an unconditional, mislabeled guard was removed); CLI-scheduled jobs now register with the running-job registry and archive per fire, unifying them with the server path. The Docker/SQL/onFHIR SchedulingTest stays parked and re-homes with the testkit batch (IgnifyrTestSpec builds repos from folder URIs that do not survive test-jar consumption). Full mvn -B verify green (all 12 modules, Docker). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…to its own module Move the offline OMOP terminology-mapping generator (Concept, GeneratorDBAdapter, TerminologyMappingGenerator) out of the engine into a new enterprise ignifyr-terminology-tools module (package io.ignifyr.tools.terminology). It is standalone tooling with its own main — not on any runtime path — and is kept out of the community distribution because it targets a local OMOP Postgres database with hard-coded dev connection details (GeneratorDBAdapter). - New module ignifyr-terminology-tools depends on ignifyr-engine (for LocalTerminologyService.ConceptMapFileColumns), jackson-dataformat-csv, and the PostgreSQL driver. Not a dependency of the CLI or the server (offline tool). - TerminologyMappingGeneratorTest re-homes cleanly: it uses only mockito (no IgnifyrTestSpec, no Docker), so it runs as a normal module unit test. - jackson-dataformat-csv stays in the engine (still used by CsvUtil + LocalTerminologyService). Engine unit 84/84 (was 86; the 2 generator tests moved to the new module); new module 2/2; whole reactor (13 modules) builds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… enterprise module Move the structured-log encoder (MapMarkerToLogstashMarkerEncoder) and the Logstash/Fluentd dependencies out of the community engine into a new enterprise ignifyr-observability module (package io.ignifyr.observability.logback). The community engine now logs plain console + rolling files; structured (Logstash JSON) audit encoding and Fluentd shipping to the EFK stack are an enterprise concern, referenced from the server's logback.xml. - New standalone module ignifyr-observability (no engine dependency) carrying logback-classic, logstash-logback-encoder, logback-more-appenders, and fluent-logger. - Engine pom drops logstash-logback-encoder (encoder moved) and fluent-logger (Fluentd transport, used only by the server's DataFluentAppender). It KEEPS logback-more-appenders: ExecutionLogger and the mapping-result models use its MapMarker type to attach structured data to log events (community logging); only the JSON encoding + shipping that consume those markers are enterprise. - Engine logback.xml: the mapping/execution audit appender (FILE-AUDIT) now uses a plain PatternLayoutEncoder instead of LogstashEncoder + the MapMarker encoder, and the dangling FLUENT / ASYNC-AUDIT-FLUENT appender-refs (never defined in the engine config) are removed. - Server depends on ignifyr-observability (dropping its own fluent-logger + logback-more-appenders) and its logback.xml (main + test) references the encoder at its new io.ignifyr.observability.logback package. Whole reactor (14 modules) builds; full mvn -B verify green (Docker), including the server boot that initializes the Fluentd/Logstash logback pipeline from the observability module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ispatch seam - New io.ignifyr.engine.execution.MappingJobLauncher: the one home of the batch / streaming / scheduled 3-way dispatch. CommandLineInterface.runJob is rewired onto it (the server's ExecutionService follows next). Every launch now registers with RunningJobRegistry, so 'run --job' batch executions gain execution tracking + post-batch archiving (same deliberate improvement the scheduled CLI path got in the scheduling extraction). Streaming now wins over schedulingSettings on the (nonsensical) combination of both, matching the server's precedence. - Model virtual dispatch instead of type matches: MappingSourceBinding gains withPreprocessSql and FhirMappingTask.substituteBatchParameters (dedupes the copies in FhirMappingJobManager and the server); MappingJobSourceSettings gains withAsStream so callers can force batch reads without matching on KafkaSourceSettings. - SPI: StreamingFailureDescriptor renamed to SourceFailureDescriptor (describeStreamingFailure) + new defaulted describeBatchTaskFailure hook; the Kafka connector now describes unknown-topic failures for ad-hoc task runs too. - SPI: new SourceSchemaInferrer trait + ExtensionRegistry.schemaInferrers (keyed by settings class, forced at init) for connector-native schema inference; the SQL connector contributes one in the next commit.
…to the extension SPI - ExecutionService.runJob delegates its batch/streaming/scheduled dispatch to the engine's MappingJobLauncher (checkpoint clearing included); only the HTTP-level guards (already-running, already-scheduled) stay in the server. - testMappingWithJob no longer imports Kafka types: streaming sources are forced to batch via the model-level withAsStream(false) (now also covering streaming file sources, which previously slipped through as unbounded streams), and unknown-topic failures are described through the connectors' SourceFailureDescriptor.describeBatchTaskFailure hook. - SchemaDefinitionService.inferSchema dispatches through ExtensionRegistry.schemaInferrers instead of matching on SqlSourceSettings; the JDBC-metadata inference (getTableSchema/mapSqlTypeToSpark) moves from the engine's SchemaConverter into the SQL connector's new SqlSchemaInferrer. - Drops the server's duplicated substituteBatchParameters (model method now). Server suite green: 14 suites, 96 tests, 0 failures (Docker/onFHIR included).
…er-extension SPI
- New IgnifyrServerExtension SPI in ignifyr-server-common (ServiceLoader-
discovered, like the engine's IgnifyrExtension): modules contribute root
routes, schema-import routes (persisting through a narrow SchemaImportSink),
and the version of the external component they integrate (for /metadata).
IgnifyrServer/IgnifyrServerEndpoint/ProjectEndpoint/SchemaDefinitionEndpoint/
MetadataService are rewired onto the discovered extensions; the hard-coded
Option[RedCapServiceConfig] threading is gone.
- New enterprise ignifyr-redcap module carrying the whole REDCap surface:
the /redcap proxy endpoint+service+config (gated on the unchanged
'ignifyr-redcap' HOCON block), the /projects/{id}/schemas/redcap
data-dictionary import, RedCapUtil, and the extract-redcap-schemas CLI
command — contributed through BOTH SPIs (engine CliCommandProvider + server
extension). The server depends on it (enterprise runtime); the community CLI
does not and now reports the providing module for the unknown command.
- Engine Boot loses its REDCap string branch: any non-built-in command token
resolves through ExtensionRegistry.cliCommands, with flags translated to
positional args by the new CliCommandProvider.argsFromOptions hook;
nextArg parses generic --flag value pairs; Help appends extension-contributed
helpText lines instead of hard-coding the REDCap usage.
- RedCapUtil finally gets a direct unit test (module spec, no Docker); the
endpoint-level REDCap import stays covered by SchemaEndpointTest.
Gates green: engine 84, redcap 4, server 96 tests (Docker/onFHIR included).
The flag's only behaviour was gating environment-variable resolution of the Kafka sourceUri; the resolver now resolves it unconditionally (topic names already were). The custom Kafka settings serializer ignores an obsolete "asRedCap" key in existing job JSONs, so they keep parsing everywhere. README's RedCAP section and api.yaml drop the field accordingly. Full `mvn -B verify` green across the 16-module reactor (Docker included); community CLI smoke: extract-redcap-schemas reports the ignifyr-redcap module to install; server fat jar merges both extension SPI service files.
SinkContentTypes lived as a nested object inside the FileSystemWriter companion (the write layer), yet the model class FileSystemSinkSettings imported it back from there — a model->write back-edge that would become a reactor cycle once FileSystemWriter is extracted into ignifyr-connector-file. Promote SinkContentTypes to a top-level object in io.ignifyr.engine.model (mirroring SourceContentTypes, already there), delete the now-empty FileSystemWriter companion, and repoint every consumer (the writer, the engine writer test, and five server test suites) at the model FQN. The DELTA_LAKE="delta" constant stays in the community model so any job JSON naming contentType="delta" keeps parsing everywhere; only the Delta write handler + delta-spark dep migrate to enterprise in a later step. No behavior change.
…dule Create ignifyr-testkit and relocate the cross-module test harness out of the engine so downstream plugin modules can reuse it without the engine forming a reactor cycle (engine tests must never depend on a module that depends on the engine). testkit (src/main) now owns IgnifyrTestSpec, OnFhirTestContainer and the shared classpath fixtures (test-mappings, test-schemas, terminology-service, test-data/loop.json). IgnifyrTestSpec's folder-repository wiring is made jar-safe: a fixture folder served as a jar: URL is materialized to a temp directory first, so consumption works both inside the reactor (file: on target/classes) and against a published testkit jar (where new File(jar:...) would otherwise throw). Its test-support libraries (scalatest, mockito, h2, testcontainers) are compile-scope so a single test-scope dependency on the testkit gives a consumer everything transitively. The three pure-engine behavioural suites that used the harness (FhirMappingFolderRepositoryTest, FhirPathMappingFunctionsTest, LocalTerminologyServiceTest) move to the testkit's own tests. ExtensionRegistrySpec is decoupled from IgnifyrTestSpec (it only needed the shared Spark session) and stays in the engine. FhirMappingJobManagerTest is parked (it needs both the harness and the file connector); it is re-homed into ignifyr-connector-file when that module is extracted. Retire the engine test-jar: drop the maven-jar-plugin test-jar goal and the root test-jar dependency entry, and point the server's test scope at the testkit (its sole use of the old test-jar was OnFhirTestContainer).
…ngTest With the shared testkit in place, the two heavy fixture-driven suites parked during the connector-sql / scheduling extractions return to their owning modules, each in the io.ignifyr.integrationtest package and run in the integration-test phase (a test/integration-test scalatest split keeps `mvn test` Docker-free; the lightweight registration specs stay in the unit phase). SqlSourceTest -> ignifyr-connector-sql, verbatim (its named arguments already match the concrete FhirMappingJobManager signature); its SQL fixtures + test-sql-mappingjob.json move with it. SchedulingTest -> ignifyr-runtime-scheduling, rewritten onto the extracted Cron4jSchedulerProvider SPI: the provider owns and starts cron4j internally, so the test schedules through scheduleMappingJob(...) and tears down via descheduleJobExecution(...) instead of managing a Scheduler and the removed MappingJobScheduler / 6-arg manager. It depends on connector-sql (test) for the SqlSource reader and writes scheduler state under a temp directory. Both modules gain a test application.conf; the shared mappings/schemas come from the testkit. The engine's orphaned SQL/scheduling fixtures move out with them.
…ble format sub-SPI
Move FileDataSourceReader and FileSystemWriter out of the engine into the new
community ignifyr-connector-file module (package io.ignifyr.connector.file),
contributed through the IgnifyrExtension SPI. CoreExtension no longer registers
the file source/sink; the engine now carries no file-connector code.
Introduce a connector-local file-format sub-SPI, owned by connector-file and
discovered through its own ServiceLoader so the engine stays ignorant of file
formats:
- FileSourceFormat / FileSinkFormat traits + FileFormatRegistry (keyed by
content type, fail-fast on duplicates) + FileSinkSupport (the shared
partition-by-resource-type / HDFS write machinery) + FileFormatHints /
MissingFileFormatException (actionable "install ..." errors).
- The reader/writer become thin: the reader resolves the path, validates the
streaming directory, adds the streaming filename-logging column and applies
`distinct`, then delegates the per-content-type read; the writer just
dispatches to the sink format for its content type.
- Community formats ship here: csv/tsv/parquet source and ndjson/csv/parquet
sink (FHIR bulk output). JSON/NDJSON *source* and the Delta *sink* handlers
live here for now too; they are split into the enterprise format-json /
format-delta modules in the following steps.
connector-file is bundled into ignifyr-cli (community) and ignifyr-server
(enterprise runtime). The file reader/writer suites move here
(io.ignifyr.connector.file, unit) with the unsupported-content-type expectation
updated to MissingFileFormatException and the writer test's accidental
delta-spark encoder import replaced by the standard Spark encoders; the
onFHIR-backed FhirMappingJobManagerTest is un-parked into
io.ignifyr.integrationtest with the file fixtures. ExtensionRegistrySpec drops
its file source/sink assertions (now covered by FileConnectorExtensionSpec).
…se module Move the JSON/NDJSON file *source* format handler out of the community file connector into the new enterprise ignifyr-format-json module (package io.ignifyr.format.json). It contributes a FileSourceFormat through the connector's format sub-SPI via its own META-INF/services entry, so JSON/NDJSON files can be used as mapping input only when this module is installed; the community file connector now discovers just csv/tsv/parquet sources. The community file *sink* still writes NDJSON — only the source side is enterprise. The module depends on ignifyr-connector-file (the FileSourceFormat sub-SPI it implements) and is bundled into ignifyr-server (enterprise runtime), not the community CLI. The JSON/NDJSON read cases + their fixtures move from the connector's FileDataSourceReaderTest into ignifyr-format-json's JsonSourceFormatTest; the connector's reader test keeps the csv/tsv/parquet/zip cases.
…Spark wiring
Move the Delta Lake file *sink* format out of the community file connector into
the new enterprise ignifyr-format-delta module (package io.ignifyr.format.delta),
carrying the delta-spark dependency with it, and make the Delta Spark-session
wiring contributed rather than hardcoded — completing the removal of Delta from
the community edition.
New sparkConfContributions SPI hook: IgnifyrExtension gains a
sparkConfContributions accessor; ExtensionRegistry aggregates it across installed
modules (comma-concatenating the additive spark.sql.extensions, failing fast on
any other conflicting key); IgnifyrConfig.createSparkConf folds it in below the
user's `spark { }` config. The two hardcoded Delta entries
(spark.sql.extensions=DeltaSparkSessionExtension, spark.sql.catalog.spark_catalog
=DeltaCatalog) leave sparkConfDefaults and are now contributed by
ignifyr-format-delta's DeltaFormatExtension, so the community engine builds its
SparkSession with no Delta wiring and no delta-spark on the classpath.
delta-spark is removed from the engine pom and declared in ignifyr-format-delta.
The Delta writer contributes a FileSinkFormat via the connector's format sub-SPI
(its own META-INF/services entry); the community connector now discovers only
ndjson/csv/parquet sinks. ignifyr-format-delta is bundled into ignifyr-server
(enterprise), not the community CLI. The Delta write cases move from the
connector's FileSystemWriterTest into ignifyr-format-delta's DeltaSinkFormatTest;
the sizeable shared FhirMappingResult test dataset is extracted to the testkit
(FhirMappingResultFixtures) so both modules use one copy instead of duplicating it.
…fail-fast Follow-ups from an adversarial review of the batch (all low severity; behaviour and edition-boundary review came back clean): - Surface the "install the '…' module" hint for a missing file format as the top-level error instead of burying it. MissingExtensionException is un-sealed and MissingFileFormatException now extends it, so SourceHandler's read path rethrows any MissingExtensionException as-is (matching the connector-missing and write-path behaviour) rather than wrapping it in a generic "Source cannot be read" FhirMappingException. - Fail fast on a duplicate file-format registration: FileConnectorExtension. initialize() forces FileFormatRegistry to materialize at engine startup (it only reads the classpath), mirroring ExtensionRegistry.init(), so a duplicate content-type provider errors at boot instead of mid-job. - Cover the install-hint branch: FileConnectorExtensionSpec now asserts that an uninstalled JSON source / Delta sink raises MissingFileFormatException naming the exact enterprise module coordinates (previously only the generic "unsupported" path was exercised, with no message assertion). - Document the real re-entrancy trap on IgnifyrExtension.initialize(): it can run while the shared SparkSession is mid-build (via sparkConfContributions), so it must not touch IgnifyrConfig.sparkSession — the caveat previously lived only on sparkConfContributions.
The suite asserted a hardcoded folder/file count against the shared context root (base_path="."), so its children count depended on whatever transient folders other suites (repository dirs, logs, checkpoints) leave in the context root during a full reactor run — producing intermittent off-by-one failures. Point the test at a uniquely-named fixture directory it creates inside the context root and tears down in afterAll, with a known layout (3 folders + a top-level file + a nested file). The assertions are now deterministic and order-independent regardless of what else lives in the module working dir.
`list-plugins` (one-shot `java -jar ignifyr-engine-standalone.jar list-plugins` or interactive) enumerates the installed IgnifyrExtension modules discovered through ServiceLoader and everything each contributes — source connectors, sinks, terminology/identity services, CLI commands, schema inferrers, streaming/scheduling capabilities and Spark-conf keys. It is the machine-checkable form of "which plugins ship in this distribution?", a CI gate on the community vs enterprise edition boundary. Boot dispatches it without constructing the engine (it reads only the registry, so no workspace or mapping-job config is needed just to inspect the plugins). Adds a defaulted `IgnifyrExtension.extraCapabilities` display hook for contributions the engine's typed registries cannot introspect on their own; the file connector overrides it to surface its own file-format sub-registry (the discovered source/sink formats), so those appear in the listing too.
Add a maven-enforcer `bannedDependencies` rule (defined once in the root pluginManagement as the `ban-enterprise-deps` execution) and opt every community module into it (engine, common, connector-sql, connector-file, cli, testkit). The rule fails the build if a community module pulls in a library that belongs to an enterprise-only module: Kafka (connector-kafka), cron4j (runtime-scheduling), Delta Lake (format-delta), the DB2 JCC driver, or Logstash/Fluentd log shipping (observability). searchTransitive catches indirect pulls too. logback-more-appenders is deliberately NOT banned — the community engine uses its MapMarker type for structured logging; only the Logstash JSON encoder and Fluentd shipping are enterprise. Enterprise modules simply do not opt in, so they use these libraries freely. Makes an edition-boundary regression a build failure rather than a silent leak into the community distribution.
- ExtensionHints.describe: a missing sink no longer reads "No FHIR writer registered for source type ..."; the noun now matches the lookup kind. - SinkHandler: the write-problems accumulator name mapped over the characters of mappingTaskName (String.map), producing a garbled name; plain interpolation instead. - FileSinkSupport (connector-file): partition-by-resource-type now skips mapped results that carry no resourceType discriminator, with a warning — instead of writing a literal "null" directory (local path) or throwing on resourceType.get (HDFS path). Covered by a new FileSystemWriterTest case. - Document FhirMappingResult.resourceType as the generic sink-routing discriminator (a FHIR resource type or a non-FHIR target/table name), never validated against FHIR.
…sink SPI naming Sinks now mirror the source connectors — one module per sink, discovered through the IgnifyrExtension SPI. The engine ships no concrete I/O at all: - New ignifyr-sink-fhir (community): FhirRepositoryWriter moves out of the engine together with the FHIR-server-backed terminology/identity service providers keyed by FhirRepositorySinkSettings (the same server connection doubles as both). CoreExtension shrinks to the built-in CLI commands plus the local (CSV-backed) terminology service. No dependency change: the FhirRepositorySinkSettings model references onfhir-client types, so onfhir-client stays an engine dependency — this is a structural move. - New ignifyr-sink-file (community): FileSystemWriter, FileSinkSupport and the FileSinkFormat sub-SPI (ndjson/csv/parquet handlers + its own ServiceLoader registry and install hints) move out of ignifyr-connector-file, which is now source-only. ignifyr-format-delta re-points its FileSinkFormat services entry and dependency to the new module. - Sink SPI naming neutralized before the API freezes at the first Central release (the template output has always been opaque JSON; CSV/relational/ OMOP targets are on the roadmap): FhirSinkSettings -> SinkSettings, BaseFhirWriter -> BaseSinkWriter, FhirWriterFactory -> SinkWriterFactory. Concrete model class names (FhirRepositorySinkSettings, FileSystemSinkSettings) are job-JSON discriminators and stay unchanged. - ExtensionHints now points at io.ignifyr:ignifyr-sink-fhir / -sink-file; both modules ship in ignifyr-cli and ignifyr-server; the integration suites that write to onFHIR (connector-sql, connector-file, runtime-scheduling) gain a test-scoped ignifyr-sink-fhir dependency.
Empty ignifyr-sink-omop skeleton reserving the module and its ServiceLoader registration for the upcoming "map to OMOP" feature: an OmopSinkSettings-keyed writer typing rows against versioned OMOP CDM schemas (5.3/5.4/6.0) with FK-ordered table writes, plus an OMOP-vocabulary-backed terminology service (design recorded in the edition-split plan). Registers nothing yet; ships in ignifyr-server only.
writePartitionedByResourceType special-cased hdfs:// and wrote newline-joined raw JSON regardless of the content type, so a parquet or Delta sink on an hdfs:// path silently produced .txt files (and ndjson lost numOfPartitions, options and partitioningColumns). Spark's DataFrameWriter resolves hdfs:// natively, so the branch is deleted and one path serves every scheme. Each resource type is now written from its own filtered Dataset, keeping the payloads off the driver where the old grouped path collect_list-ed them all.
…to fit the new test classifications.
…tain environments
…+ test gaps - engine: registerBatchJob attached handleCompletedBatchJob with onComplete, which only schedules it, so a caller awaiting the raw job future raced the archiver. The one-shot batch CLI calls System.exit(0) right after that await, so with archiveMode archive/delete the processed inputs could be left in place. registerBatchJob now returns a Future chained with andThen, and MappingJobLauncher hands that back as MappingJobLaunch.Batch, so awaiting it means archiving is done. - test-flow: the ~6 MB jar listings made `echo "$VAR" | grep -q` invert under pipefail — grep exits at the first match and closes the pipe, echo dies of SIGPIPE, and the successful match evaluates as false. The enumeration guard therefore fired against a jar that `jar tf` lists fine and the whole jar-content section was silently skipped (passed: 19 instead of 41). Had it run, a marker actually present in the community jar would also have been reported as "excludes", so a real enterprise leak would have passed. Matched with bash substring tests. - build: Spark 3.5 reaches into JDK internals (StorageUtils -> sun.nio.ch.DirectBuffer), which JDK 17+ denies by default. spark-submit adds the module opens via JavaModuleOptions, but these suites run Spark embedded in the JVM scalatest-maven-plugin forks, so nothing adds them and every Spark-backed suite aborts. Added as argLine; harmless on the JDK 11 target thanks to -XX:+IgnoreUnrecognizedVMOptions.
The `pull_request` trigger was filtered to base branch main, so no CI ran on PRs opened
against the integration branches we actually target — including the current one. Dropped
the filter.
The long tier is now pull-request-only (plus workflow_dispatch): a push to main would just
repeat the tier its own PR already ran. Local runs stay opt-in via ${skipITs}, so a plain
`mvn install`/`verify` is short and Docker-free and `mvn verify -DskipITs=false` opts in.
Previously merged test-extension PR invalidated several claims: verify no longer runs the integration tier (skipITs gates it), ignifyr-server's onFHIR suites moved to io.ignifyr.integrationtest so mvn test is now Docker-free reactor-wide, ignifyr-cli gained test sources for the edition-separation spec, and testkit pins its suites. Point the root guide at test-flow/README.md rather than duplicating it, and record that MappingJobLaunch.Batch now completes only after post-batch archiving.
- IgnifyrConfig: the user's `spark { }` block was merged with `++`, which replaces on collision,
so it overwrote the contributions ExtensionRegistry had deliberately concatenated. Setting
spark.sql.extensions for any unrelated reason silently dropped Delta's session extension —
measured with a competing user value, the effective config went from `com.example.NotReal` to
`io.delta.sql.DeltaSparkSessionExtension,com.example.NotReal`. List-valued keys are now joined
across all three layers, generalised to a set as requested in review. Keys that only look
list-valued stay out: extraClassPath is path-separator delimited, and
spark.sql.catalog.spark_catalog is a genuine scalar a user may legitimately override.
- ExtensionRegistry.init(): also materialize streaming and scheduler, so a duplicate capability
module fails at startup rather than at first job launch. streaming had been covered only
incidentally, by IgnifyrEngine reading it to decide whether to start the archive timer.
- StreamingSinkHandler: pass the throwable, not e.getMessage. A String selects scala-logging's
`error(String, Any*)` overload, whose args fill `{}` placeholders, so the fully-interpolated
message discarded both the argument and the stack trace — on the deliberate swallow path, where
that log line is the only record a failed micro-batch leaves.
Okanmercan99
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidated rename + edition-split branch — both touch the same files, so they land together.
Rename (clean break): packages
io.ignifyr.*, coordinatesio.ignifyr:ignifyr-*, HOCONignifyr.*, Dockersrdc/ignifyr-*, docs; version →2.0-SNAPSHOT. Legacytofhirintentionally survives only in thesrdc/tofhir-webimage, thetofhir-redcapsibling service, the SwaggerHub link, and git history.Edition split: the reactor is now module-separable into a public Community Edition (
ignifyr-cli+ engine/common/sql/file connectors, Apache-2.0) and a private Enterprise Edition (ignifyr-server+ fhir-server/kafka/json/delta/streaming/scheduling/redcap/observability/terminology). Everything plugs in through theIgnifyrExtensionServiceLoader SPI — the engine never names a connector or format directly. A maven-enforcerbannedDependenciesgate keeps enterprise-only libs out of community modules. Moving a feature between editions is a one-folder move with zero engine changes.This is the module-level split only. Physical extraction to
ignifyr-enterpriseis Phase 2, not this PR.Testing:⚠️ Requires JDK 11 — under JDK 21 the Spark reflective path surfaces as a spurious
mvn -B verifygreen.IllegalAccessErrorinExtensionRegistrySpec; unsupported-JDK symptom, not a defect.33 commits · 496 files · +18,952 / −11,381