Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The Mermaid hierarchy includes a couple of labels that are inconsistent with the stable failure identifiers/meanings, which can mislead readers cross-referencing the diagram and catalog.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new documentation artifact defining the DVA failure taxonomy across evaluation, AoV issuance, and AoV verification, including stable identifiers, detectability, feedback recipients, recommended actions, and relationships. This supports aligning future backend/frontend behavior and operator workflows without changing runtime behavior.
Changes:
- Introduces a failure hierarchy (text + Mermaid) to organize outcomes, faults, and trust limitations.
- Defines a failure catalog with per-failure detection points, feedback locations/recipients, and recovery guidance.
- Documents a proposed structured “feedback record” contract shape and cross-failure relationship mapping.
File summaries
| File | Description |
|---|---|
| docs/failure-model.md | Adds the DVA failure/feedback model, including hierarchy diagrams, catalog entries, and a proposed structured contract. |
Review details
Suppressed comments (1)
docs/failure-model.md:78
- The Mermaid diagram labels
VLAHASHas "VLA version mismatch", but the corresponding stable code in the hierarchy/catalog isVLA_COMMITMENT_MISMATCH. Aligning the label avoids confusion when cross-referencing the diagram with the catalog.
VERIFY --> COMMIT[Commitment faults]
COMMIT --> DATAHASH[Data commitment mismatch]
COMMIT --> VLAHASH[VLA version mismatch]
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| VERIFY --> CRYPTO[Cryptographic faults] | ||
| CRYPTO --> INVALIDSIG[Invalid signature] | ||
| INVALIDSIG -. observed during verification .-> SIGCHECK[Signature verification failed] |
bzp99
changed the title
bcdad94 to
1953753
Compare
| ```text | ||
| DVA outcome | ||
| ├── Expected negative outcome | ||
| │ └── DATA_REQUIREMENT_NOT_MET | ||
| ├── Evaluation faults | ||
| │ ├── Execution faults | ||
| │ │ ├── EVALUATION_OPERATIONAL_FAULT | ||
| │ │ └── EVALUATION_TRANSIENT_FAULT | ||
| │ └── Requirement-definition faults | ||
| │ ├── EVALUATION_LOGIC_SYNTAX_INVALID | ||
| │ ├── EVALUATION_LOGIC_INCORRECT | ||
| │ └── REQUIREMENT_MISUNDERSTOOD | ||
| ├── AoV issuance faults | ||
| │ └── AOV_ISSUANCE_OPERATIONAL_FAULT | ||
| └── AoV verification faults | ||
| ├── Structural | ||
| │ └── AOV_MALFORMED | ||
| ├── Cryptographic | ||
| │ ├── AOV_SIGNATURE_INVALID | ||
| │ └── SIGNATURE_VERIFICATION_FAILED | ||
| ├── Commitment | ||
| │ ├── DATA_COMMITMENT_MISMATCH | ||
| │ └── VLA_COMMITMENT_MISMATCH | ||
| └── Trust limitation | ||
| └── AOV_CLAIMS_FORGED | ||
| ``` | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| DVA[DVA outcome] | ||
| DVA --> NEG[Expected negative outcome] | ||
| NEG --> DATA[Data does not fulfil requirement] | ||
|
|
||
| DVA --> EVAL[Evaluation faults] | ||
| EVAL --> EXEC[Execution faults] | ||
| EXEC --> OP[Operational or environmental fault] | ||
| EXEC --> TRANSIENT[Transient or probabilistic fault] | ||
| EVAL --> DEF[Requirement definition faults] | ||
| DEF --> SYNTAX[Invalid evaluation syntax] | ||
| DEF --> LOGIC[Incorrect implementation logic] | ||
| DEF --> MEANING[Misunderstood requirement] | ||
|
|
||
| DVA --> ISSUE[AoV issuance faults] | ||
| ISSUE --> ISSUEOP[Operational issuance fault] | ||
|
|
||
| DVA --> VERIFY[AoV verification faults] | ||
| VERIFY --> STRUCT[Structural faults] | ||
| STRUCT --> MALFORMED[Malformed AoV] | ||
| VERIFY --> CRYPTO[Cryptographic faults] | ||
| CRYPTO --> INVALIDSIG[Invalid signature] | ||
| INVALIDSIG -. observed during verification .-> SIGCHECK[Signature verification failed] | ||
| VERIFY --> COMMIT[Commitment faults] | ||
| COMMIT --> DATAHASH[Data commitment mismatch] | ||
| COMMIT --> VLAHASH[VLA version mismatch] | ||
| VERIFY --> TRUST[Trust limitation] | ||
| TRUST --> FORGED[Forged evaluation claims] | ||
|
|
||
| SYNTAX -. may become an execution fault if not caught earlier .-> OP | ||
| ``` |
There was a problem hiding this comment.
Instead of this, I strongly suggest sketching up one or more fault tree(s) as I expect that to help more with communication than a custom representation.
At the top, we have the undesired outcome event like Attestation cannot be generated and we can drill down to the potential causes; eg, there was an error during the AoV VC issuance OR there was some error during the evaluation of some requirement OR there was an error while retrieveing the associated VLA. Most of these intermediate events can be further broken down: for instance, what could cause an error while retrieving the VLA? Perhaps it was that the VLA manager was not reachable. Or that it was reachable but returned a 404 for the sought VLA. And so on.
| SYNTAX -. may become an execution fault if not caught earlier .-> OP | ||
| ``` | ||
|
|
||
| ## Failure Catalog |
There was a problem hiding this comment.
This catalogue is very useful and I think all of the identifiers here should be present in the fault tree(s). Then this can be a more detailed explanation of those events along with annotations on how they can be detected and what feedback can be given where, etc.
| ## Feedback Matrix | ||
|
|
||
| | Failure | Primary interface | Recipient | Recommended response | | ||
| |---|---|---|---| | ||
| | `DATA_REQUIREMENT_NOT_MET` | Provider dashboard | Provider | Correct the data or review the requirement. | | ||
| | `EVALUATION_OPERATIONAL_FAULT` | Dashboard and logs | Operator | Restore the evaluator and retry. | | ||
| | `EVALUATION_TRANSIENT_FAULT` | Evaluation details | Provider, operator | Repeat under controlled conditions and compare attempts. | | ||
| | `EVALUATION_LOGIC_SYNTAX_INVALID` | VLA Manager | VLA author | Block the requirement and correct the implementation. | | ||
| | `EVALUATION_LOGIC_INCORRECT` | VLA Manager tests | VLA author | Review rendered logic and known examples. | | ||
| | `REQUIREMENT_MISUNDERSTOOD` | VLA review | VLA author, participants | Review business meaning and representative examples. | | ||
| | `AOV_ISSUANCE_OPERATIONAL_FAULT` | Provider dashboard and logs | Operator | Restore signing or persistence and issue again safely. | | ||
| | `AOV_MALFORMED` | Consumer dashboard | Consumer | Reject and request a replacement. | | ||
| | `AOV_SIGNATURE_INVALID` | Consumer dashboard | Consumer, operator | Reject and investigate issuer, key, or corruption. | | ||
| | `SIGNATURE_VERIFICATION_FAILED` | Verification history | Consumer, operator | Reject and inspect the specific cryptographic reason. | | ||
| | `DATA_COMMITMENT_MISMATCH` | Consumer dashboard | Consumer, operator | Reject and investigate exchanged data or routing. | | ||
| | `VLA_COMMITMENT_MISMATCH` | Consumer dashboard | Consumer, operator | Reject and retrieve the referenced VLA version. | | ||
| | `AOV_CLAIMS_FORGED` | Optional re-evaluation | Consumer, security operator | Investigate the signer and disputed claims. | |
There was a problem hiding this comment.
I don't see the purpose of this if it can be derived from the data catalogue above.
| ## Proposed Feedback Record | ||
|
|
||
| Components need the same structured information even when they present it differently. The following shape is a proposed contract for implementation work. It is not a claim that every field is already returned by the APIs. | ||
|
|
||
| ```json | ||
| { | ||
| "code": "DATA_COMMITMENT_MISMATCH", | ||
| "outcome": "ERROR", | ||
| "stage": "AOV_VERIFICATION", | ||
| "category": "COMMITMENT_FAULT", | ||
| "parentCode": "AOV_VERIFICATION_FAULT", | ||
| "message": "The attestation refers to different data.", | ||
| "evidence": { | ||
| "algorithm": "sha-256", | ||
| "expected": "...", | ||
| "actual": "..." | ||
| }, | ||
| "detectability": "AUTOMATIC", | ||
| "recipients": ["DATA_CONSUMER", "OPERATOR"], | ||
| "retryable": false, | ||
| "suggestedAction": "Reject the attestation and investigate the exchanged data." | ||
| } | ||
| ``` | ||
|
|
||
| `code`, `outcome`, `stage`, and `category` are stable machine-readable values. `message` and `suggestedAction` are user-facing text. `evidence` contains safe structured details returned by the component that detected the problem. It must not expose private keys, credentials, raw secrets, or unnecessary personal data. | ||
|
|
||
| The backend should provide the most specific code it can justify. If it cannot identify the cause, it should return an explicit unknown code for the stage rather than asking the frontend to infer a cause from exception text. |
There was a problem hiding this comment.
As we discussed in our meeting, I don’t think there should be any specialized feedback record. The individual modules all have their own mechanisms of reporting errors: APIs return error responses, frontends show error messages, etc. We do not have to have a unified error feedback format across the whole of DVA.
| ## Relationships Between Failures | ||
|
|
||
| | Source | Relationship | Target | Meaning | | ||
| |---|---|---|---| | ||
| | `EVALUATION_TRANSIENT_FAULT` | specialization of | `EVALUATION_OPERATIONAL_FAULT` | It is an evaluation execution fault whose result may change across controlled attempts. | | ||
| | `EVALUATION_LOGIC_SYNTAX_INVALID` | may cause | `EVALUATION_OPERATIONAL_FAULT` | Invalid logic becomes a runtime execution fault if VLA creation fails to block it. | | ||
| | `SIGNATURE_VERIFICATION_FAILED` | observes | `AOV_SIGNATURE_INVALID` | The verification event exposes the underlying cryptographic problem. | | ||
| | `DATA_COMMITMENT_MISMATCH` | invalidates use of | AoV for received data | A valid signature cannot make the credential applicable to different data. | | ||
| | `VLA_COMMITMENT_MISMATCH` | invalidates use of | AoV for referenced agreement | The credential cannot prove requirements from another VLA version. | | ||
| | Optional consumer re-evaluation | may reveal | `AOV_CLAIMS_FORGED` | Re-evaluation can expose a disagreement but is not part of normal signature verification. | |
There was a problem hiding this comment.
Frankly, I think fault trees should obsolete these relationships.
Summary
This documents the failure modes identified across evaluation, AoV issuance, and verification.
It defines stable identifiers, detection points, feedback locations, responsible parties, recovery actions, relationships, and known trust limitations. It also records proposed answers to the open design questions and includes a visual hierarchy.
No runtime behavior is changed.
Review focus
CC @bzp99