[Version: 3.13.1] Attachments stuck in "Scanning" status after server crash — no retry on restart

[ansgarlichter]

Describe the bug
Files uploaded to composition attachment entities can get permanently stuck in Scanning status if the server crashes while a malware scan is in progress. On the next server start, no mechanism exists to detect and retry these orphaned scans.

To Reproduce

1. Upload a file to an entity that uses the Attachments aspect
2. Kill the server while ScanAttachmentsFile is being processed (before updateStatus writes Clean/Infected/Failed)
3. Restart the server
4. The attachment remains in Scanning status indefinitely — no retry is triggered

Expected behavior
On startup, any attachment rows with status = 'Scanning' should be detected and re-queued for scanning, since the scan that set that status clearly never completed.

Additional context
We implemented a workaround on the application side by hooking into cds.once('served') and querying all @_is_media_data entities for stuck rows, then re-emitting ScanAttachmentsFile for each. However this belongs in the plugin itself.

If you agree with this, I am happy to support further by submitting a PR.

[github-actions]

👋 Hello @ansgarlichter, thank you for submitting this issue. Our team is reviewing your report and will follow up with you as soon as possible.

[ansgarlichter]

We investigated the issue further and identified the root cause:

Attachments get stuck in state Scanning

#432 (Clean→Scanning under load) is the race: the un-awaited cds.tx(..., "Scanning") in rescan() commits after the spawned scan's Clean. #442 fixes it by removing the redundant pre-emit write (the scanner already sets Scanning as its first awaited step). We hit the same race and confirm the diagnosis — we patched it downstream by awaiting the cds.tx calls to confirm the issue.

rescan() in lib/generic-handlers.js (v3.13.1) sets status to Scanning in a non-awaited cds.tx(...), then spawns the scan:

cds.tx(async () => await malwareScanner.updateStatus(target, keys, "Scanning"))  // (A) NOT awaited
cds.spawn(async () => await malwareScanner.emit("ScanAttachmentsFile", { target, keys }))  // (B)

The comment claims "Scanning" is committed before the emit, but since A isn't awaited there is no happens-before guarantee. Per @sap/cds 10.0.3:

• cds.tx(fn) runs fn synchronously now, but its COMMIT is an un-awaited, IO-gated promise — nothing drives it to completion (see as well https://pages.github.tools.sap/cap/docs/node.js/cds-tx#srv-tx).
• cds.spawn(fn) starts one turn later (setImmediate) in its own tx, and its chain is actively awaited to commit.

B then does heavy, connection-hungry work (S3 fetch, HTTP scan, its own Scanning write, then Clean). Under load this starves A's orphaned commit, so A's Scanning commits after B's Clean → last-writer-wins → row stuck in Scanning. It's load-dependent/intermittent (matches #432's "0–4 of 10–15 files, different each run").

Evidence (single request id):
XX:XX:25.233 Updated scan status to Clean for , [ ... ID='7ef45298-...' ]
XX:XX:25.249 Updated scan status to Scanning for undefined, {"ID":"7ef45298-..."}

The for undefined fingerprints A: rescan() passes req.target.name (a string), so updateStatus's ${target.name} logs undefined.

Original issue

This issue (#481) is a distinct failure mode #442/#432 do not address: a crash after Scanning is committed but before Clean/Failed leaves the row stuck in Scanning. On the outbox — I checked with @sap/cds 10.0.3:

• With the default in-memory outbox, a crash loses the in-flight ScanAttachmentsFile entirely → stuck in Scanning forever. This is the case that bites us.
• With a persistent outbox, the emit is a cds.outbox.Messages row. A worker crash leaves it as status='processing'; it becomes processable again once lastAttemptTimestamp < now - timeout (default timeout: '1h', maxAttempts: 10) and is redelivered on the next poll/startup sweep. So a persistent outbox does recover this — but only after the timeout window, and only up to maxAttempts. So this is largely resolvable by recommending/documenting persistent outbox for the malware-scanner service. That said, two gaps remain worth a plugin-level safeguard: (1) apps on the default in-memory outbox get no recovery, and (2) even with persistent outbox, recovery is delayed by the timeout. A lightweight startup reconciliation (query @_is_media_data rows in Scanning, re-emit ScanAttachmentsFile, guarded against multi-instance duplicate emits) covers both, and we already run exactly this downstream.