Summary
File: src/flowx/motifs/collapser.py (collapse_motifs)
When one pipeline contains two or more matches of the same motif (e.g. two separate "File Existence Validation" groups), the collapser tracks inserted motifs in a set[str] keyed by motif_id alone (if motif_id in inserted_motifs: continue). The second match hits the guard and is skipped entirely, so:
- its member activities are never added to
motif_task_keys, and
_rewire_dependencies cannot redirect edges pointing at them.
Downstream tasks that depended on the dropped activities keep a depends_on edge to a task that no longer exists.
Observed failure (deploy-time)
Error: cannot create resources.jobs.<job>: Invalid dependency graph for task
'DataExtractionTableList', job specification does not contain task with key
'OlDBVersiontoCurrentDBVersion' (400 INVALID_PARAMETER_VALUE)
The motif_<id> task_key also collides between the two matches.
Impact
Silent data loss + invalid DAG: a whole group of activities is dropped from the output, and the resulting job fails to create. No error or warning at conversion time.
Proposed fix
Track each detected match by identity (id(detected)) and mint a unique task_key per occurrence (motif_X, motif_X_2, …) via a task_key_suffix, recording every member's task_key -> motif mapping so dependency rewiring still works.
Test gap
No test asserts that a motif matching twice in one pipeline produces two distinct nodes with rewired dependencies. Fix PR adds coverage.
Summary
File:
src/flowx/motifs/collapser.py(collapse_motifs)When one pipeline contains two or more matches of the same motif (e.g. two separate "File Existence Validation" groups), the collapser tracks inserted motifs in a
set[str]keyed bymotif_idalone (if motif_id in inserted_motifs: continue). The second match hits the guard and is skipped entirely, so:motif_task_keys, and_rewire_dependenciescannot redirect edges pointing at them.Downstream tasks that depended on the dropped activities keep a
depends_onedge to a task that no longer exists.Observed failure (deploy-time)
The
motif_<id>task_key also collides between the two matches.Impact
Silent data loss + invalid DAG: a whole group of activities is dropped from the output, and the resulting job fails to create. No error or warning at conversion time.
Proposed fix
Track each detected match by identity (
id(detected)) and mint a uniquetask_keyper occurrence (motif_X,motif_X_2, …) via atask_key_suffix, recording every member'stask_key -> motifmapping so dependency rewiring still works.Test gap
No test asserts that a motif matching twice in one pipeline produces two distinct nodes with rewired dependencies. Fix PR adds coverage.