From 4116f0d89970d6b9cb304f1dad1c8224a16d5897 Mon Sep 17 00:00:00 2001 From: delchev Date: Wed, 19 Aug 2026 12:30:07 +0300 Subject: [PATCH] fix(intent): an expansion's generated rows do not outlive their master (#6821) `expansions` states that the generated child set is OWNED by the expansion - a span change replaces every row pointing at the master - and then binds only two of the master's three events. There was no handler for the master's delete, so the rows the expansion generated were the one part of a deleted record that survived it: orphans pointing at an id that no longer resolves, and still counted by every roll-up, report and balance that counted them while the record existed. A leave request's day rows kept spending the balance of a request that was gone from the list. Nothing else closed the gap. The issue read it as deployment-dependent (a refused delete where the constraint is enforced, orphans where it is not), but on this platform there is only one outcome: `SchemasSynchronizer.parseImpl` drops every foreign key a `.schema` declares BY DESIGN - referential integrity is a business-layer check - so the rows were always silently orphaned, never protected. A composition does not imply the cleanup either: it governs how a child is reached and edited, not what becomes of generated rows when the thing that generated them ceases to exist. Each expansion now also contributes an `OnDelete` handler on the master's `-deleted` topic, removing the rows selected by the expansion's own back-reference and nothing else. It deletes them one at a time THROUGH the child repository, so each row's delete event fires and whatever reacts to a hand-deleted row - a roll-up on the child, a capacity guard, a downstream aggregate - reacts identically; a bulk statement would leave exactly the stale totals this exists to prevent. It is idempotent by construction: the delete event is published after the master row is gone, so a redelivery finds an empty child set. No write-back to the master, whose row (and `count` field) is gone. The database cascade the issue offers as an alternative is deliberately not taken: it would fire for rows the expansion does not own, apply to hand-entered children too, bypass the child's layer so the events the totals depend on never fire, and encode a rule the model neither states nor can see. The cleanup is its own glue collection (`expansionCleanups`) and its own template, because a template source renders once per collection entry and the cleanup's body shares nothing with the regeneration's. Registering it in `GlueGenerator` is not optional bookkeeping - an unregistered collection falls through to the whole-model branch and emits one file of unresolved placeholders. `IntentEngineIT` asserts the emitted handler (topic, criteria, per-row delete through the child repository, no master write-back, every placeholder resolved). `IntentEmissionCoverageIT` asserts the promise at the only layer that can show it: a real published app whose retainer expands into three monthly rows on create and has none left once the retainer is deleted - the emitted source can be perfect and still be subscribed to a topic nothing publishes to. Also corrects the module guide's note on the DELETE-to-409 mapping, which still described FK constraint emission as pending work rather than as the decision it is, and mis-stated the mechanism (the schema template does emit `foreignKey` structures; the synchronizer drops them). --- components/engine/engine-intent/CLAUDE.md | 2 +- .../intent/generator/GlueIntentGenerator.java | 34 +++++++-- .../template/service/model/GlueGenerator.java | 21 +++++- .../events/ExpansionCleanup.java.template | 50 +++++++++++++ .../template/template.js | 7 ++ .../tests/api/IntentEmissionCoverageIT.java | 73 +++++++++++++++++++ .../integration/tests/api/IntentEngineIT.java | 18 ++++- 7 files changed, 193 insertions(+), 12 deletions(-) create mode 100644 components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/events/ExpansionCleanup.java.template diff --git a/components/engine/engine-intent/CLAUDE.md b/components/engine/engine-intent/CLAUDE.md index 16987dfcca4..f72fe61759d 100644 --- a/components/engine/engine-intent/CLAUDE.md +++ b/components/engine/engine-intent/CLAUDE.md @@ -391,7 +391,7 @@ Semantics worth knowing: - **`postings:` (top-level) = declarative posting (source-document status → generated local document + computed items).** The accounting "documents → ledger" capability, generalized (spike-derived; see the driving suite's spike findings). `PostingIntent` + parser `validatePostings` (creates = local document owning a composition items child; backReference = its to-one to the source, the at-most-once guard; event trigger `onTransition` with a mandatory `when: " == "` status guard, or `onCreate` for a source with NO status lifecycle - a booked payment - binding the `-created` topic with the `when` guard optional (#6421); item cells = `rule()` refs into a single-selector rule entity or Calc arithmetic over the source; row `when: ==|!= `). `GlueIntentGenerator.buildPostings` pre-renders EVERYTHING as Java expressions (the expansions convention — the template stays shape-only): topic + re-load coordinates via `CrossModelSupport`, guard, header assignments (copy / literal / `{placeholder}` concat), `ruleRow.` refs, `Calc.eval("", source, )` amounts with the scale from the LOCAL item field, null-safe Calc row guards. `postings` glue collection → the pipeline's collection case (source gen folder = sanitized model alias, topic keeps the RAW perspective) → `Posting.java.template`: a `MessageHandler` on `---transitioned` (#6220's channel) that re-loads the source by id (the payload lacks later-step data — the stamped number), guards, resolves the rule row (missing row / null referenced column → SKIP, the unposted worklist), and writes target + items through the repositories — so numbering / status `init:` / `checks:` fire on the created document. **Idempotent + resumable, not transactional** (the cloud-native consistency model — there is NO cross-step DB rollback; each write commits on its own): the back-reference identifies an existing post, so a redelivery of a COMPLETE post (item count ≥ the derived `expectedItems`) is a no-op, and a redelivery of a HALF-post (an item write failed after the target was saved) clears the partial items and rebuilds the full set — it never throws or half-posts. Concurrent-redelivery de-duplication is best-effort (a check-then-act on the back-reference) until a real UNIQUE key on the back-reference lands with schema constraint emission. Storno/negation mode LANDED as **`reverses:`** (paired with the `transitions:` void primitive - the "void-document event" is a transition into the void status): a reversal posting inherits creates/backReference/rule/map/items from the reversed sibling, negates every item amount expression on the SAME side (`Calc.eval("-()", ...)` - red storno), locates the original through the empty `storno:` self-link (none -> fail-soft skip), stamps the link on its creation, and both handlers' idempotency guards discriminate by that link (reversal counts linked rows, the sibling counts unlinked ones - `stornoProperty`/`stornoFilterProperty` in the glue). The explicit manual Reverse action (no source void) remains a follow-up. Compensation, not a transaction, is how a bad post is unwound. - **Lifecycle-aware aggregates: seed-row `stage:` + report `scope:` + symbolic status names (#6645).** An aggregate over an entity carrying a `function: EntityStatus` was **wrong by default** - drafts nobody had issued, cancelled and voided (анулиране) rows all landed in the sum unless the author remembered a magic-number status predicate in `filter:`, and nothing said so (the motivating case: a voided invoice kept its 2000 in "Revenue this month" because the report declared dimensions + measures and no `filter`, so the emitted query had no `WHERE` at all). Four coordinated pieces, all in `LifecycleStages` + `ReportIntentGenerator.scopePredicate` + `StatusSymbolResolver`: (1) a status **seed row** classifies what the status MEANS with a closed-vocabulary `stage: draft|live|cancelled|void` - metadata, never a column (the CSV generator only emits declared fields + referenced FKs, and `CsvimIntentGeneratorTest` pins that); (2) a report declares `scope: all` or a stage name, emitted as `."" IN ()` ANDed onto the filter; (3) with the nomenclature classified, an **aggregating** report **defaults to `live`** - but only when its dimensions/`filter` do not already reference the status (a breakdown BY status must keep its draft rows, and an authored predicate is authoritative), so an existing model is byte-identical until it adopts `stage:`; (4) every site that names a status accepts the **seeded name** (`from: [ISSUED]`, `setStatus: VOIDED`, `init: DRAFT`, `setRelationField` `value:`, `abortOn.status`, a check's `status`/`setStatus`, `immutableWhen`, a posting's `event.when`, a report's `filter`) - resolved on the **raw YAML tree before the typed Gson mapping** (the `rejectRemovedNumberKeys` precedent), so every validator, generator and template keeps seeing plain integers. **Why names matter more than they look:** an id is positional, so inserting a status mid-nomenclature shifts every later id and silently retargets every guard authored against the old numbering - that is how a `reverses:` posting guarded `when: "Status == 8"` stopped matching a Void that now writes 9, leaving the ledger with a receivable for a document that no longer existed, with well-formed Java emitted throughout. **Boundaries, deliberate:** the nomenclature must be seeded IN THIS MODEL - the parser holds one file and no repository, so a **cross-model** status can neither be stage-scoped nor named (both fail loudly naming the numeric-id fallback; cross-model symbols need the name→id map on the generated `.model` and are follow-up work). A symbolic **ordering** comparison (`Status >= ISSUED`) is rejected - names have no order, that is what `scope:` is for. A nomenclature that declares its own `stage` property collides with the marker and is rejected rather than guessed. Nothing is emitted into the `.model` for `stage` - no consumer needs it yet (the Harmonia badge's `statusVariant` keyword guess is the obvious future one). **Part 3, the cheap half that catches everything the other three cannot:** when a report aggregates over a lifecycle entity and neither declares `scope:` nor filters on the status AND the nomenclature is unclassified, generation records a `context.addIssue` warning - surfaced in the generate response's `warnings` and now in the **Intent Editor**'s own amber strip (it used to discard them on success; the Builder shell already showed them). That warning, not the default, is what turns an invisible modelling omission into a visible one. - **`lifecycle:` on an entity = the declarative state machine (#6714).** The whole set of legal status edges, declared once over the entity's `function: EntityStatus` nomenclature (`edges: [{ from: DRAFT, to: [ISSUED, CANCELLED] }, ...]`, either side a seeded name or an id) and **enforced on every status write**. The gap it closes: the status machinery was a set of point constructs - `init:` names the start, a `transitions:` button guards the flips that go through THAT button, a workflow `setRelationField` writes one unguarded, a `checks:` rejection files another - and nothing declared which edges were legal at all, so any other writer (a workflow branch, a glue action, a plain REST call) could jump a document from any status to any other and nothing noticed. **Enforcement lives in the generated REPOSITORY, deliberately** (`Repository.java.template`: `LIFECYCLE_EDGES` + `enforceLifecycle` / `enforceLifecycleMove` / `enforceLifecycleStart`, `ValidationException` -> 400) - it is the ONE choke point every writer passes through: `update` (the REST payload), `updateWithoutEvent` (system writes), and `updateProperties` (which `updateProperty`, and therefore the transition controller, the workflow setters and `updateDerived`, all route through - so the targeted-write overrides are now emitted for a lifecycle entity too, not only for `documentChecks`/`hasLabel`). Guarding the transition endpoints instead would have left every other writer free, which is the whole defect. `enforceLifecycleStart` (emitted only when the status relation declares `init:`) additionally refuses a CREATE filed anywhere but at the start - entering the lifecycle mid-graph skips it rather than travelling it - and is placed BEFORE the aggregate-guard macros in `save()` so an `outcome: reject` can still file the record where the model says. Emission is three scalars on the entity map (`lifecycleStatusProperty`, `lifecycleEdges` as `1>2,1>9` pairs, `lifecycleStatusNames` as `1=DRAFT,...` so a rejection reads "cannot move from ISSUED to DRAFT" instead of quoting positional ids, plus `lifecycleInitialStatus`) - scalars, so they reach the `.edm` twin like `immutableStatusValues`. **Parse-time is where the other status sites are made to agree** (`validateLifecycles`): every `from` of a `transitions:` entry must reach its `setStatus` along an edge (a button is presentation over the graph), and a status written by a `setRelationField` step or forced by a check's rejection must be one some edge reaches - which is what catches a reject path transiting through an approved status when the file is read. **Deliberate boundaries:** no `on:` key - the graph is always over the EntityStatus relation, so naming it would be redundant, and YAML 1.1 reads a bare `on` as the boolean `true` (it would arrive as the key `true` and bind to nothing), so `rejectLifecycleOn` refuses it in the raw-tree preprocessing rather than dropping it silently; a cross-model nomenclature is seeded in its owner model and so is its lifecycle (refused, naming that); the nomenclature must be seeded here (the ids are validated against the seeds); no reachability check - one nomenclature may serve two entities with different graphs, so "unreachable here" is not an error. -- **`immutableWhen:` / `immutable:` on an entity = user-write immutability.** `immutableWhen: "Status == 2"` (a boolean expression over EntityStatus seed ids, terms joined with `||`) makes update/delete through the generated REST controller answer 409 CONFLICT while the record's `function: EntityStatus` FK satisfies it; `immutable: true` is the unconditional append-only variant (mutually exclusive with `immutableWhen`; a non-existent id still yields 404, not 409). Emitted as the entity-level `immutableStatusProperty` + `immutableStatusValues` (or `immutableAlways`) model attrs; `requireMutable` fetches the existing row before writing. Repository writes are deliberately unaffected — the workflow (storno generation, roll-ups, ProcessId write-back) keeps working; this guards the USER surface, per the accounting audit-trail requirement (corrections are reversals, never edits). **The UI is gated up front, not just on the 409:** each of the three generated controllers (power / partner / my) also exposes a **`GET /{id}/mutable`** pre-check (`{"mutable": true|false}` via the shared `isMutable`, scoped like its reads), and every Harmonia surface consumes it — the manage form and document pages ask it on edit load and force the read-only preview mode with a "Read-only" title badge (so a directly typed `/edit` URL opens read-only), the partner/my form + document pages disable their controls (`fieldset :disabled`) and hide Save/Delete/item actions, while the browse tables (manage list, master) gate row Edit/Delete through a **baked `isRowImmutable(row)`** computed from the row's status FK against the generation-time immutable ids — no per-row API call, same generated-from-the-same-attrs no-drift argument as the client `validationSchema`. The pre-check fails OPEN (an outage must not lock the UI); the PUT/DELETE 409 stays the authoritative guard. Covered by `IntentEmissionCoverageIT` (endpoint tokens + page tokens + mutable=false/true over REST). Parser requires an EntityStatus relation. Alongside it (no DSL): every generated controller now maps a **database constraint violation on DELETE to 409** ("referenced by other records") instead of a 500. Caveat discovered while verifying: the generated schema currently emits **no FK constraints at all** (`constraints: []` on every table — same-model included), so this mapping only engages once constraint emission lands; whether to emit them (a suite-wide data-integrity semantics change: deploy order, CSVIM import order, existing tables unaffected by ALTER) is a separate decision, raised with the accounting findings. Date-based period locking (records whose date falls in a Locked period) is deliberately NOT part of this — its shape needs the real fiscal-period module and follows as its own PR. **The lock reaches the master's composition CHILDREN (#6695).** It was per-entity, and a child declares no immutability of its own — while its generated repository writes THROUGH to the master, recomputing `net`/`vat`/`total` on every `save`/`update`/`delete`. So `POST`/`PUT`/`DELETE` on a line of an ISSUED invoice succeeded over REST and silently rewrote the document's totals after the number was stamped, the immutable snapshot taken and the ledger posted — the UI forbade it, REST permitted it, and the permitted operation was the one `immutableWhen` exists to prevent. `ModelParameterProcessor.inheritMasterLock` now propagates the master's `immutableAlways` / `immutableStatusProperty` + values onto each direct composition child as a `masterLock` map (master entity + FK property + its `…Entity`/`…Repository` classes, resolved through the composition FK's perspective exactly as the personal/partner inheritance does), and all three generated controllers (power / partner / my) emit a `requireMasterMutable` that loads the master and answers the same 409 — on create (the payload's FK), on update (the STORED master *and* the incoming one, so a line cannot be moved into a locked document either), on delete, and on an attachment upload. Engine writers stay exempt by construction: they go through the repository, not the controller — which is why the issue-time snapshot generator (`Attachments.store` + `repository.save`) is untouched. The opt-out is the flag #6700 already introduced: `locksWithMaster: false` on the child (settlement is a different lifecycle from content), so the affordance and the REST guard are governed by one declaration and cannot drift apart. Only the DIRECT child is covered — that is the shape that writes through to the master. It composes with the prompted `generates` action (#6685): that create runs through the TARGET's repository, not a controller, so a guided create against a post-issue child keeps working on a locked document exactly as its per-record button (deliberately not gated on mutability) implies — the panel and the action remain the two separate answers to "this collection must go on being recorded". `IntentEmissionCoverageIT` carries both controls: `EntryLine` (silent → inherits) is refused create/update/delete on a POSTED entry and the master's total is asserted UNMOVED, while `CampaignNote` (`locksWithMaster: false`) still posts to a locked campaign. +- **`immutableWhen:` / `immutable:` on an entity = user-write immutability.** `immutableWhen: "Status == 2"` (a boolean expression over EntityStatus seed ids, terms joined with `||`) makes update/delete through the generated REST controller answer 409 CONFLICT while the record's `function: EntityStatus` FK satisfies it; `immutable: true` is the unconditional append-only variant (mutually exclusive with `immutableWhen`; a non-existent id still yields 404, not 409). Emitted as the entity-level `immutableStatusProperty` + `immutableStatusValues` (or `immutableAlways`) model attrs; `requireMutable` fetches the existing row before writing. Repository writes are deliberately unaffected — the workflow (storno generation, roll-ups, ProcessId write-back) keeps working; this guards the USER surface, per the accounting audit-trail requirement (corrections are reversals, never edits). **The UI is gated up front, not just on the 409:** each of the three generated controllers (power / partner / my) also exposes a **`GET /{id}/mutable`** pre-check (`{"mutable": true|false}` via the shared `isMutable`, scoped like its reads), and every Harmonia surface consumes it — the manage form and document pages ask it on edit load and force the read-only preview mode with a "Read-only" title badge (so a directly typed `/edit` URL opens read-only), the partner/my form + document pages disable their controls (`fieldset :disabled`) and hide Save/Delete/item actions, while the browse tables (manage list, master) gate row Edit/Delete through a **baked `isRowImmutable(row)`** computed from the row's status FK against the generation-time immutable ids — no per-row API call, same generated-from-the-same-attrs no-drift argument as the client `validationSchema`. The pre-check fails OPEN (an outage must not lock the UI); the PUT/DELETE 409 stays the authoritative guard. Covered by `IntentEmissionCoverageIT` (endpoint tokens + page tokens + mutable=false/true over REST). Parser requires an EntityStatus relation. Alongside it (no DSL): every generated controller now maps a **database constraint violation on DELETE to 409** ("referenced by other records") instead of a 500. Scope of that mapping: the schema template does emit `type: "foreignKey"` structures, but `SchemasSynchronizer.parseImpl` drops them **by design** — a foreign key never becomes a database constraint on this platform, because a constraint binds insert/delete ORDER into the schema where seeds, imports, regeneration and deletes would all have to obey an ordering nothing in the model asked for; referential integrity is a business-layer check. Only the **unique** keys are carried over (`carryUniqueConstraints`, #6793), so the 409 engages for a business-key collision and never for a reference. Anything that must not outlive the record it points at therefore needs an explicit handler — which is what an expansion's `OnDelete` cleanup is (#6821). Date-based period locking (records whose date falls in a Locked period) is deliberately NOT part of this — its shape needs the real fiscal-period module and follows as its own PR. **The lock reaches the master's composition CHILDREN (#6695).** It was per-entity, and a child declares no immutability of its own — while its generated repository writes THROUGH to the master, recomputing `net`/`vat`/`total` on every `save`/`update`/`delete`. So `POST`/`PUT`/`DELETE` on a line of an ISSUED invoice succeeded over REST and silently rewrote the document's totals after the number was stamped, the immutable snapshot taken and the ledger posted — the UI forbade it, REST permitted it, and the permitted operation was the one `immutableWhen` exists to prevent. `ModelParameterProcessor.inheritMasterLock` now propagates the master's `immutableAlways` / `immutableStatusProperty` + values onto each direct composition child as a `masterLock` map (master entity + FK property + its `…Entity`/`…Repository` classes, resolved through the composition FK's perspective exactly as the personal/partner inheritance does), and all three generated controllers (power / partner / my) emit a `requireMasterMutable` that loads the master and answers the same 409 — on create (the payload's FK), on update (the STORED master *and* the incoming one, so a line cannot be moved into a locked document either), on delete, and on an attachment upload. Engine writers stay exempt by construction: they go through the repository, not the controller — which is why the issue-time snapshot generator (`Attachments.store` + `repository.save`) is untouched. The opt-out is the flag #6700 already introduced: `locksWithMaster: false` on the child (settlement is a different lifecycle from content), so the affordance and the REST guard are governed by one declaration and cannot drift apart. Only the DIRECT child is covered — that is the shape that writes through to the master. It composes with the prompted `generates` action (#6685): that create runs through the TARGET's repository, not a controller, so a guided create against a post-issue child keeps working on a locked document exactly as its per-record button (deliberately not gated on mutability) implies — the panel and the action remain the two separate answers to "this collection must go on being recorded". `IntentEmissionCoverageIT` carries both controls: `EntryLine` (silent → inherits) is refused create/update/delete on a POSTED entry and the master's total is asserted UNMOVED, while `CampaignNote` (`locksWithMaster: false`) still posts to a locked campaign. - **`checks:` on an entity = declarative cross-field / cross-line validations (the double-entry shape).** Three kinds (`CheckIntent`): row-level `exactlyOne` (`fields:` — exactly one non-null; emitted PascalCased into the `.model` `checks` list and enforced in the generated REST `validate()` with 400) and document-level `itemsSumEqual` (`over:` two item fields whose sums must match) / `itemsMin` (`count:`), both REQUIRING a `status:` gate (an EntityStatus seed id) — parser-enforced, because an ungated sum check would forbid drafting a document item by item. The EDM generator precomputes everything template-side (`buildChecks`: items entity + back-FK via the composition child, `statusProperty`, PascalCased fields); `ModelParameterProcessor` splits `rowChecks`/`documentChecks`; the **DAO repository** enforces document checks in `save`/`update`/**`updateWithoutEvent`** whenever the persisted entity carries the gate status — so the workflow setter flipping DRAFT→POSTED hits `enforceChecks` and an unbalanced document FAILS the write instead of silently posting: it throws the SDK `org.eclipse.dirigible.sdk.db.ValidationException`, which the client-controller dispatcher (`ControllerInvoker`) maps to **HTTP 400** with the authored message on a REST create/update, and which rolls back the task completion on the BPMN path (the capacity guard on roll-ups throws the same). `recalculate()` deliberately bypasses it (it persists the recomputed totals through the BASE targeted write, `super.updateProperties(id, totals)`, so a document still being assembled line by line never fails its own gate). No Harmonia-side mirror in v1 — the task-completion error surfaces the authored message. - **`resolves:` = the effective-dated register lookup (#6712).** The enterprise shape with no declarative form before it: a register says "X applied to Y from A to B" (a vehicle assignment, a price list, a contract in force, an org assignment), a record carries the match key(s) and a date, and a to-one must be filled from the row whose period covers that date. Nothing else in the DSL reaches it - `dependsOn` is a UI-time copy with equality matching only, a `decision` condition is a single comparison, and `setField` writes constants - so every application hand-wrote the same delegate. Authored as `{ event: { onCreate|onUpdate: , when? }, set: , from: , match: { : , ... }, between: { start?, end?, value }, outcome?: , found?/notFound?/ambiguous?: { setStatus } }`; `ResolveIntent` -> `GlueIntentGenerator.buildResolves` -> the `resolves` glue collection -> `Resolve.java.template`, a `@Component MessageHandler` on the record's event topic. **All three outcomes are first-class, and that is the point of the construct:** exactly one covering row fills the relation, NO covering row and MORE THAN ONE covering row both leave it unset (an automation that silently picks one of two candidates is worse than none - the ambiguous register goes back to a human). Each outcome may route the record by `setStatus` (seed id or seeded name, resolved by `StatusSymbolResolver` like every other status site), and the attempt is **observable**: `outcome:` stamps `found`/`notFound`/`ambiguous` into a string field of the record - queryable, filterable in a list view, and readable by a process `decision` - and the handler logs the keys and the date it checked. **Decisions worth keeping:** the value copied is derived, not authored - the register must carry exactly ONE to-one to the same target as `set:`, and zero or two is a validation error rather than a guess (the same refusal, one altitude up); a record that already carries the relation is skipped, so a manual correction is never overwritten and a re-delivered event is a no-op; the write is a single targeted `updateProperties` of the relation + the outcome + the status, so no `-updated` re-fires and no concurrent write to another column is reverted; period bounds are optional on either side (open-ended = still valid), the end is INCLUSIVE, and a date-only bound covers its whole day (the generated `millis`/`endExclusive` helpers put a `LocalDate` and an `Instant` on one epoch-milli axis, UTC). v1 is same-model (`from:` must be declared here) and binds to `onCreate`/`onUpdate` only - `onDelete` is refused, there is nothing left to fill. The parser refuses a `when` guard it cannot render rather than degrading it to an always-open guard. - **Every DERIVED write is targeted (document totals, `rollups:`, `aggregates:`) — the last member of the lost-update family.** A recompute reads a row, changes the one or two columns it computes, and persists. Persisting the WHOLE row silently reverts any concurrent write to another column of that row: the trigger `ProcessId` variant was fixed in #6226 and the workflow setter/writer variant in #6306, and the recompute variant was live-reproduced against a roll-up (REST-create a parent, PUT another column immediately after → 200, but a re-read shows the OLD value; the recompute had read the row before the PUT and wrote its stale snapshot after it). All three recompute sites now write only what they computed: `Repository.recalculate(Object)` collects the document totals into a map and calls the base `super.updateProperties` (no gate checks, no `-updated` — exactly the previous `super.update` semantics minus the merge); `Rollup.java.template` and `Aggregate.java.template` collect each recomputed column into a `derived` map and persist through the generated **`updateDerived(id, values)`**, which routes through `updateProperties` (so a `checks:` entity still runs its gate and a labelled entity still refreshes its `Name`) and then re-publishes `---updated` — the event contract the old full-row `update()` provided, which TRANSITIVE roll-ups above the row depend on. Two invariants when touching these: a column assigned in the recompute must also be put into `derived` (a capacity roll-up writes count + balance + status), and an EMPTY `derived` map means nothing is persisted, so the map is what the emission oracle asserts. Covered by the `IntentEmissionCoverageIT` derived-write assertions (Bill document totals, `ClaimLineClaimRollupOnCreate`, `LedgerTotalAggregateOnCreate`). diff --git a/components/engine/engine-intent/src/main/java/org/eclipse/dirigible/components/intent/generator/GlueIntentGenerator.java b/components/engine/engine-intent/src/main/java/org/eclipse/dirigible/components/intent/generator/GlueIntentGenerator.java index 4ad6244f5e3..a246fa268de 100644 --- a/components/engine/engine-intent/src/main/java/org/eclipse/dirigible/components/intent/generator/GlueIntentGenerator.java +++ b/components/engine/engine-intent/src/main/java/org/eclipse/dirigible/components/intent/generator/GlueIntentGenerator.java @@ -107,7 +107,9 @@ public void generate(IntentGenerationContext context) { List> outbound = buildOutbound(model, byName, compositionParents, settings, context); List> stepEvents = buildStepEvents(model, compositionParents, settings); List> rollups = buildRollups(model, byName, compositionParents, settings, context); - List> expansions = buildExpansions(model, byName, compositionParents, settings); + ExpansionHandlers expansionHandlers = buildExpansions(model, byName, compositionParents, settings); + List> expansions = expansionHandlers.regenerations(); + List> expansionCleanups = expansionHandlers.cleanups(); List> settlements = buildSettlements(model, byName, compositionParents, settings, context); List> generates = buildGenerates(model, byName, compositionParents, settings, context); List> transitions = buildTransitions(model, byName, compositionParents, settings, context); @@ -154,6 +156,7 @@ public void generate(IntentGenerationContext context) { glue.put("stepEvents", stepEvents); glue.put("rollups", rollups); glue.put("expansions", expansions); + glue.put("expansionCleanups", expansionCleanups); glue.put("settlements", settlements); glue.put("generates", generates); // The event-driven subset (issue #6711) - the SAME descriptors, filtered, so the listener and @@ -2475,15 +2478,17 @@ private static Map rollupEntry(Map base, String } /** - * Period expansions: per expansion, two handlers - on the master's create and update events - that - * (re)generate the child rows for the span. Everything type-dependent (the defaults literals, the - * count write-back) is pre-rendered here as Java lines so the template stays shape-only; the child - * rows go through the child repository, so their create/delete events fire and downstream - * roll-ups/guards run exactly as for hand-entered rows. + * Period expansions: per expansion, three handlers - on the master's create and update events, that + * (re)generate the child rows for the span, and on its delete event, that removes them again. + * Everything type-dependent (the defaults literals, the count write-back) is pre-rendered here as + * Java lines so the template stays shape-only; the child rows go through the child repository, so + * their create/delete events fire and downstream roll-ups/guards run exactly as for hand-entered + * rows. */ - private static List> buildExpansions(IntentModel model, Map byName, + private static ExpansionHandlers buildExpansions(IntentModel model, Map byName, Map compositionParents, IntentSettings settings) { List> expansions = new ArrayList<>(); + List> cleanups = new ArrayList<>(); for (ExpansionIntent expansion : model.getExpansions()) { if (expansion.getName() == null || expansion.getName() .isBlank()) { @@ -2558,8 +2563,21 @@ private static List> buildExpansions(IntentModel model, Map< String className = IntentNaming.pascalIdentifier(expansion.getName()) + "Expansion"; expansions.add(rollupEntry(base, className + "OnCreate", "")); expansions.add(rollupEntry(base, className + "OnUpdate", "-updated")); + cleanups.add(rollupEntry(base, className + "OnDelete", "-deleted")); } - return expansions; + return new ExpansionHandlers(expansions, cleanups); + } + + /** + * The handlers an intent's expansions contribute, split by the template that renders them: the + * (re)generation pair per expansion, and the cleanup that removes the generated rows when their + * master is deleted. They are two collections rather than one because a template source renders + * once per collection entry, and the cleanup's body shares nothing with the regeneration's. + * + * @param regenerations the create/update handlers + * @param cleanups the master-delete handlers + */ + private record ExpansionHandlers(List> regenerations, List> cleanups) { } /** Pre-rendered Java assignment lines for the expansion's literal child defaults. */ diff --git a/components/ide/ide-template/src/main/java/org/eclipse/dirigible/components/ide/template/service/model/GlueGenerator.java b/components/ide/ide-template/src/main/java/org/eclipse/dirigible/components/ide/template/service/model/GlueGenerator.java index d9392925489..001ae4a6378 100644 --- a/components/ide/ide-template/src/main/java/org/eclipse/dirigible/components/ide/template/service/model/GlueGenerator.java +++ b/components/ide/ide-template/src/main/java/org/eclipse/dirigible/components/ide/template/service/model/GlueGenerator.java @@ -45,8 +45,8 @@ class GlueGenerator { /** The names of the collections this generator handles. */ private static final List COLLECTIONS = List.of("triggers", "resolvers", "fieldLoaders", "assignees", "timerLoaders", "waits", "aborts", "setters", "writers", "notifications", "schedules", "integrations", "inbound", "inboundMessages", "inboundFiles", - "outbound", "stepEvents", "rollups", "expansions", "settlements", "generates", "generateEvents", "transitions", "sends", - "posts", "aggregates", "postings", "printFeeders", "snapshots", "numbering", "resolves"); + "outbound", "stepEvents", "rollups", "expansions", "expansionCleanups", "settlements", "generates", "generateEvents", + "transitions", "sends", "posts", "aggregates", "postings", "printFeeders", "snapshots", "numbering", "resolves"); /** The renderer. */ private final ModelTemplateRenderer renderer; @@ -102,6 +102,7 @@ List generate(String collection, GenerationTemplateMetadataSource case "outbound" -> each(collection, source, content, model, parameters, GlueGenerator::bindOutbound); case "stepEvents" -> each(collection, source, content, model, parameters, GlueGenerator::bindStepEvent); case "expansions" -> each(collection, source, content, model, parameters, GlueGenerator::bindExpansion); + case "expansionCleanups" -> each(collection, source, content, model, parameters, GlueGenerator::bindExpansionCleanup); case "settlements" -> each(collection, source, content, model, parameters, GlueGenerator::bindSettlement); // Both collections carry the SAME create-from descriptors (generateEvents is the // event-driven subset), so they share one binding - the listener and the create-from it @@ -525,6 +526,22 @@ private static void bindExpansion(Map item, Map context.put("topicSuffix", strOr(item, "topicSuffix", "")); } + /** + * Binds an expansion cleanup - the handler that removes an expansion's generated rows when their + * master is deleted. It needs only the master's identity and the child set's criteria, so the span, + * unit, defaults and spread the regeneration binds are deliberately absent. + * + * @param item the descriptor + * @param context the template context + * @param parameters the generation parameters + */ + private static void bindExpansionCleanup(Map item, Map context, Map parameters) { + copy(context, item, "className", "masterEntity", "masterPerspective", "masterPk", "childEntity", "criteriaExpression"); + context.put("javaMasterPerspective", sanitize(item, "masterPerspective")); + context.put("javaChildPerspective", sanitize(item, "childPerspective")); + context.put("topicSuffix", strOr(item, "topicSuffix", "-deleted")); + } + /** * Binds an auto-settlement - the listener and delegate pair that applies a payment to an invoice * through their junction. diff --git a/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/events/ExpansionCleanup.java.template b/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/events/ExpansionCleanup.java.template new file mode 100644 index 00000000000..bb08b959a80 --- /dev/null +++ b/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/events/ExpansionCleanup.java.template @@ -0,0 +1,50 @@ +package gen.events.${javaGenFolderName}; + +import org.eclipse.dirigible.components.data.store.java.repository.Criteria; +import org.eclipse.dirigible.sdk.component.Component; +import org.eclipse.dirigible.sdk.messaging.ListenerKind; +import org.eclipse.dirigible.sdk.messaging.MessageHandler; +import org.eclipse.dirigible.sdk.utils.Json; + +import gen.${javaGenFolderName}.data.${javaMasterPerspective}.${masterEntity}Entity; +import gen.${javaGenFolderName}.data.${javaChildPerspective}.${childEntity}Entity; +import gen.${javaGenFolderName}.data.${javaChildPerspective}.${childEntity}Repository; + +/** + * Removes the generated ${childEntity} rows of a deleted ${masterEntity}. + * + * Generated from the intent expansions block - do not edit; it is re-generated with the application. + * The expansion OWNS the child set, so the master's delete has to take that set with it. A foreign + * key never becomes a database constraint on this platform - referential integrity is a + * business-layer check - so nothing else would stop the rows from outliving their master: they would + * survive as orphans pointing at an id that no longer exists and keep feeding the roll-ups, reports + * and balances the live rows fed. Rows are removed through the child repository, so each one's + * delete event fires and downstream roll-ups and guards run exactly as for a hand-deleted row. + */ +@Component("${javaGenFolderName}_${className}") +public class ${className} implements MessageHandler { + + @Override + public String destination() { + return "${projectName}-${masterPerspective}-${masterEntity}${topicSuffix}"; + } + + @Override + public ListenerKind kind() { + return ListenerKind.TOPIC; + } + + @Override + public void onMessage(String message) { + ${masterEntity}Entity master = Json.parse(message, ${masterEntity}Entity.class); + if (master == null || master.${masterPk} == null) { + return; + } + // The delete event is published after the master row is gone, so re-delivery finds an empty + // child set and is a no-op - the handler is idempotent without a guard of its own. + ${childEntity}Repository children = new ${childEntity}Repository(); + for (${childEntity}Entity row : children.findAll(${criteriaExpression})) { + children.delete(row); + } + } +} diff --git a/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/template/template.js b/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/template/template.js index d4c8c3f64af..11f805c8cd1 100644 --- a/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/template/template.js +++ b/components/template/template-application-events-java/src/main/resources/META-INF/dirigible/template-application-events-java/template/template.js @@ -143,6 +143,13 @@ export function getTemplate(parameters) { engine: "velocity", collection: "expansions" }, + { + location: "/template-application-events-java/events/ExpansionCleanup.java.template", + action: "generate", + rename: "gen/events/{{javaGenFolderName}}/{{className}}.java", + engine: "velocity", + collection: "expansionCleanups" + }, { location: "/template-application-events-java/events/PrintFeeder.java.template", action: "generate", diff --git a/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEmissionCoverageIT.java b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEmissionCoverageIT.java index fc4107c23c9..004be8565b2 100644 --- a/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEmissionCoverageIT.java +++ b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEmissionCoverageIT.java @@ -568,6 +568,28 @@ class IntentEmissionCoverageIT extends IntegrationTest { - { name: id, type: integer, primaryKey: true, generated: true } - { name: note, type: string, length: 200 } + # expansions: the generated child set is OWNED by the expansion, so the master's DELETE + # has to take it with it (#6821). Nothing else would - a foreign key never becomes a + # database constraint on this platform - so the rows would otherwise outlive the record + # and keep counting. Asserted at runtime, both halves: the rows appear on create and are + # gone once the master is deleted. + - name: Retainer + fields: + - { name: id, type: integer, primaryKey: true, generated: true } + - { name: note, type: string, length: 200 } + - { name: startDate, type: date, required: true } + - { name: endDate, type: date, required: true } + - { name: fee, type: decimal, required: true } + - { name: periods, type: integer, readOnly: true } + + - name: RetainerPeriod + fields: + - { name: id, type: integer, primaryKey: true, generated: true } + - { name: dueDate, type: date } + - { name: amount, type: decimal } + relations: + - { name: Retainer, kind: manyToOne, to: Retainer, composition: true, required: true } + # BPM events wave 2 (abortOn): an approval whose confirm task is cancelled the moment # the record is voided via the CancelApproval transition (reusing the EntryStatus seeds: # DRAFT 1 / CANCELLED 3). Closes the orphaned-Inbox-task hole. @@ -718,6 +740,16 @@ class IntentEmissionCoverageIT extends IntegrationTest { rollups: - { name: claimCost, entity: ClaimLine, via: Claim, field: totalCost, op: sum, of: cost } + expansions: + - name: retainer-periods + from: Retainer + into: RetainerPeriod + unit: month + between: { start: startDate, end: endDate } + map: { dueDate: period } + spread: { total: fee, into: amount, round: 2 } + count: periods + # collection-driven generation: the monthly job creates one Claim per Person and, # under each, one ClaimLine per working day of the month (amount defaulted). schedules: @@ -3322,9 +3354,50 @@ private void assertRuntimeEnforcement() { assertManyToManyRuntime(); assertInboundSourcesRuntime(); assertOutboundDepartureRuntime(); + assertExpansionLifecycleRuntime(); assertBpmEventsRuntime(); } + /** + * An expansion's generated rows, over their whole life (#6821): they appear when the master is + * created and they are gone once it is deleted. The delete half is the one that was missing - the + * construct bound create and update only, and because a foreign key never becomes a database + * constraint here, the rows simply survived as orphans still counted by every roll-up and report. + * Only the runtime shows it: the emitted handler can be present and still be subscribed to a topic + * nothing publishes to. + */ + private void assertExpansionLifecycleRuntime() { + String retainerApi = API + "/retainer/RetainerController"; + String periodApi = API + "/retainer/RetainerPeriodController"; + AtomicInteger retainerId = new AtomicInteger(); + restAssuredExecutor.execute(() -> retainerId.set(given().contentType("application/json") + .body("{\"Note\":\"expanded\",\"StartDate\":\"2026-01-15\",\"EndDate\":\"2026-03-15\",\"Fee\":300}") + .when() + .post(retainerApi) + .then() + .statusCode(200) + .extract() + .path("Id"))); + // A month span over three months yields a row per month, each carrying its share of the fee. + restAssuredExecutor.execute(() -> given().when() + .get(periodApi + "?Retainer=" + retainerId.get()) + .then() + .statusCode(200) + .body("$", hasSize(3)), + 30); + + restAssuredExecutor.execute(() -> given().when() + .delete(retainerApi + "/" + retainerId.get()) + .then() + .statusCode(200)); + restAssuredExecutor.execute(() -> given().when() + .get(periodApi + "?Retainer=" + retainerId.get()) + .then() + .statusCode(200) + .body("$", hasSize(0)), + 30); + } + /** * The non-HTTP inbound arrivals end to end (#6537): a JSON record sent to the declared queue, and * one dropped as a file into the polled folder, both turn into rows through the entity's own diff --git a/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEngineIT.java b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEngineIT.java index 1d37a33f222..b010429b7c6 100644 --- a/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEngineIT.java +++ b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/IntentEngineIT.java @@ -2139,11 +2139,14 @@ void expansion_generates_the_span_handlers_and_the_status_badge_stack() { .then() .statusCode(200)); - // The glue carries the two per-event expansion handlers with the pre-rendered Java pieces. + // The glue carries the per-event expansion handlers with the pre-rendered Java pieces - the + // (re)generation pair plus the cleanup that takes the generated rows down with their master. String glue = contentOf("loans.glue"); assertTrue(glue.contains("\"expansions\""), "the .glue should carry the expansions collection"); assertTrue(glue.contains("InstallmentsExpansionOnCreate"), "an OnCreate handler entry is expected"); assertTrue(glue.contains("InstallmentsExpansionOnUpdate"), "an OnUpdate handler entry is expected"); + assertTrue(glue.contains("\"expansionCleanups\""), "the .glue should carry the expansionCleanups collection"); + assertTrue(glue.contains("InstallmentsExpansionOnDelete"), "an OnDelete cleanup entry is expected"); // The EntityStatus relation lands as the DOCUMENT_STATUS widget on a NON-document entity. String model = contentOf("loans.model"); @@ -2165,6 +2168,19 @@ void expansion_generates_the_span_handlers_and_the_status_badge_stack() { String onUpdate = contentOf("gen/events/loans/InstallmentsExpansionOnUpdate.java"); assertTrue(onUpdate.contains("intent-test-Loan-Loan-updated\""), "the OnUpdate handler binds the -updated topic"); + // The master's delete removes the rows the expansion generated. Nothing else would: a foreign + // key never becomes a database constraint, so the rows would otherwise outlive their master as + // orphans and keep feeding the roll-ups and reports. They go through the child repository, so + // each row's delete event still fires. + String onDelete = contentOf("gen/events/loans/InstallmentsExpansionOnDelete.java"); + assertTrue(onDelete.contains("intent-test-Loan-Loan-deleted\""), "the OnDelete handler binds the -deleted topic"); + assertTrue(onDelete.contains("LoanInstallmentRepository children = new LoanInstallmentRepository()"), + "the cleanup must delete through the child repository so the per-row delete events fire"); + assertTrue(onDelete.contains("Criteria.create().eq(\"Loan\", master.Id)"), "the cleanup must scope to the master's own rows"); + assertTrue(onDelete.contains("children.delete(row)"), "the cleanup must delete every generated row"); + assertFalse(onDelete.contains("updateProperty"), "the cleanup must not write back to the master - the master row is gone"); + assertFalse(onDelete.contains("${"), "the cleanup template must render every placeholder"); + // Harmonia UI: the status renders as the title-bar badge (not an editable input) and the // calculated field previews live via the calc evaluator with the date functions. generateFromModel("template-application-ui-harmonia-java/template/template.js", "loans.model");