[case] domux-safety-gate - output-aware gate exposes input-only safety blind spots

[yangmengze608-afk]

Ref #20

Summary

Adds a reproducible Domux safety-boundary case based on a real frozen Tesla T4 / NF4 run at
revision 6c71a32f4d624cadfd9fce9d10240d8068e53456.

The case:

• corrects the historical 81.25% format compliance claim to legacy action-vocabulary
  acceptance; all 48 samples and 53 non-empty lines are structurally seven-field;
• freezes the original input-aware v1 and adds a parser-only ablation;
• implements v2 output-side risk and input-output consistency inspection across all seven
  fields, including multi-line maximum severity and malformed fail-closed behavior;
• compares v1, parser-fixed v1, and v2 on the same frozen raw Domux outputs;
• adds a 256-pair real-output mismatch attack, a 20-case development fault suite, and an
  independently generated 84-case one-shot held-out evaluation;
• reports held-out failures without retuning the frozen v2 gate.

Headline held-out result: 51/84 exact, Macro F1 0.6152, 49/56 risky labels intercepted,
0/23 block labels passed as allow, and 18/28 false interventions. These are bounded test-set
results, not a claim of production safety or Domux model accuracy.

Reproduction

cd cases/domux-safety-gate
python -m unittest -v \
  test_safety_gate.py test_safety_gate_v1.py test_dataset.py test_evaluate_safety.py
python verify_evidence.py --dataset example_safety_commands.jsonl \
  --responses evidence/domux_raw.jsonl --report evidence/safety_report.json
python verify_v2_evidence.py

No model weights, HF tokens, private cache paths, or private household data are included.

Published Discussion

https://huggingface.co/iFlytekOpenSource/Domux/discussions/4

[CLAassistant]

All committers have signed the CLA.

[FenjuFu]

技术内容是三份里对「指标诚实 + 包含边界」最较真的一份，几点很好：

• 明确指出 v1 的 93.75% 是假象（未校验实际执行 payload），并用 output-aware gate 复核；
• Wilson 95% CI、样本量标注清楚；
• 诚实说明 NF4 只是运行配置、没有 matched BF16 control，故不做量化因果归因；
• fault-injection、cross-pair attack 证据齐全。

其余标准：Discussion #4 公开可达、标题合规；未见权重文件；扫描 evidence/ 未见 token / 内网 IP / 家目录。

需要整改（阻塞点，1 条）：cases/domux-safety-gate/ 里混入了大量过程 / 草稿文件，应从提交中移除，只保留符合 cases/TEMPLATE 的正式产物。至少包括：

DCO_HISTORY_REWRITE.md
PR_BODY_DRAFT.md
PR_DESCRIPTION_DRAFT.md
README.SUBMISSION_DRAFT.md
README.draft.md
EXTERNAL_REVIEW_PACKET_2026-08-25.md
FINAL_AUDIT_2026-08-25.md
CURRENT_STATUS.md
DISCUSSION_DRAFT.md
V1_BASELINE.md

这些草稿不属于案例本体，清理后案例目录会干净很多，也更符合 cases/ 的框架约定。清理后即为有效案例。最终评奖由 @FenjuFu 统一裁定。

[yangmengze608-afk]

已按建议完成清理，感谢审阅。

• 已移除评论中点名的 10 个过程/草稿文件；
• 另移除重复的日期化运行记录 RUN_RESULTS_2026-08-25.md；
• 当前 case 仅保留正式 README.md、数据卡、notebook、源码、测试、依赖与可复核 evidence；
• 重新运行 23 项测试、原始证据校验、v2/held-out 校验及官方 validate_cases.py，全部通过；
• DCO 与 CLA 门禁保持通过。

整改提交：e39df7f。

[FenjuFu]

??????? e39df7f?????? 10 ???/????????? RUN_RESULTS_2026-08-25.md ????? diff ?????????????? README?????notebook????????????? evidence?

?????????????Discussion #4 ??????? revision ? T4/NF4 ??????????? Wilson ??????????? matched BF16 control?held-out ??? false intervention ??????DCO ? CLA ????

??????????????????????????????????????????????????????

[FenjuFu]

复审确认（评奖前最终核对）——上轮阻塞点已解除：此前 cases/domux-safety-gate/ 混入的约 10 个过程 / 草稿 md（DCO_HISTORY_REWRITE / PR_*_DRAFT / README.draft / EXTERNAL_REVIEW_PACKET / FINAL_AUDIT / CURRENT_STATUS / V1_BASELINE 等）已全部移除，现仅保留正式产物（README / DATASET_CARD / evidence/v2 / 测试 / verify 脚本），目录整洁。

技术侧此前已核：自曝 v1 93.75% 假象、Wilson CI、NF4 无 matched BF16 对照故不做因果归因——指标与失败分析的诚实度高。HF Discussion #4 公开。有效案例。

最终评奖将在活动评奖期（8-31 – 9-6）统一公布。