Skip to content

Commit c4303e1

Browse files
Seungpyo1007Seungpyo1007
committed
feat(verify): add tiered data verification layer (Tier 0 offline scoring)
Adds app/verify/, an existence/trust verification layer that sits above the structural validator (app/validate.py, untouched). It answers "does this record describe a real, existing device/part — confidently enough to set verified:true?" to lift the ~1.2% verified ratio. - Tier 0 (offline, deterministic, all ~102k records): completeness + cross-field consistency (signals.py) + source-host trust (hosts.py) + provenance -> a green/yellow/red band. Full scores cached to gitignored data/_verify/state/; the tracked data/_verify/ledger.jsonl is reserved for promotion decisions. - Tier 1 (http_check.py): source_urls HTTP liveness, urllib + ThreadPool, per-host rate limit, resumable TTL cache. - Tier 2 (crossref.py): external cross-reference under a strict exact-heading rule (no fuzzy matching; ambiguous candidates never auto-promote). - Tier 3 (promote.py): hybrid escalation + surgical verified:false->true write-back (only that token, atomic, LF-preserved, never clobbers curated data). CLI: python -m app.verify score|report|check-urls|crossref|promote. CI: non-blocking verify-offline job in validate-data.yml; scheduled/manual verify-network.yml for network tiers with a diff-scope guard. Validates that the offline scorer reproduces the human-curated verified CPU set (40 tests pass). Refs #1
1 parent 75f7b76 commit c4303e1

21 files changed

Lines changed: 2173 additions & 0 deletions

‎.github/workflows/validate-data.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,15 @@ on:
77
paths:
88
- "data/**"
99
- "app/validate.py"
10+
- "app/verify/**"
11+
- "tests/verify/**"
1012
push:
1113
branches: [main]
1214
paths:
1315
- "data/**"
1416
- "app/validate.py"
17+
- "app/verify/**"
18+
- "tests/verify/**"
1519

1620
jobs:
1721
self-validate:
@@ -24,6 +28,28 @@ jobs:
2428
- name: Self-check (bundled validator)
2529
run: python -m app.validate
2630

31+
# Non-blocking existence/trust signal: scores the records changed in this PR
32+
# with the Tier 0 offline verifier and prints a band histogram. Informational
33+
# only — never gates the merge (continue-on-error).
34+
verify-offline:
35+
runs-on: ubuntu-latest
36+
continue-on-error: true
37+
env:
38+
PYTHONIOENCODING: utf-8
39+
steps:
40+
- uses: actions/checkout@v4
41+
with:
42+
fetch-depth: 0
43+
- uses: actions/setup-python@v5
44+
with:
45+
python-version: "3.12"
46+
- name: Tier 0 verification report (changed records)
47+
run: |
48+
git fetch origin main --depth=1 || true
49+
python -m app.verify score --changed --no-cache
50+
- name: Verifier unit tests
51+
run: python -m pytest tests/verify -q -m "not slow"
52+
2753
engine-validate:
2854
needs: self-validate
2955
uses: GetTechAPI/TechEngine/.github/workflows/validate-data.yml@main

‎.github/workflows/verify-network.yml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
name: verify-network
2+
3+
# Network verification tiers (source-URL liveness + external cross-reference) and
4+
# verified promotion. NEVER runs on pull_request — these tiers hit external sites,
5+
# are rate-limited, and must not gate a merge. Scheduled + manual only. Promotions
6+
# are written on a branch and opened as a PR for human review; the job hard-guards
7+
# that nothing but `verified` flags and the ledger changed.
8+
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
apply:
13+
description: "Flip verified->true and open a PR (otherwise dry-run only)"
14+
type: boolean
15+
default: false
16+
max_urls:
17+
description: "Frontier records to URL-check"
18+
default: "2000"
19+
max_crossref:
20+
description: "Yellow/red records to cross-reference"
21+
default: "500"
22+
schedule:
23+
- cron: "0 4 * * 1" # Mondays 04:00 UTC
24+
25+
permissions:
26+
contents: write
27+
pull-requests: write
28+
29+
jobs:
30+
verify-network:
31+
runs-on: ubuntu-latest
32+
env:
33+
PYTHONIOENCODING: utf-8
34+
steps:
35+
- uses: actions/checkout@v4
36+
with:
37+
fetch-depth: 0
38+
39+
- uses: actions/setup-python@v5
40+
with:
41+
python-version: "3.12"
42+
43+
# Resumable caches (URL + crossref). Recomputable, so a miss is harmless.
44+
- name: Restore verify caches
45+
uses: actions/cache@v4
46+
with:
47+
path: data/_verify/state
48+
key: verify-state-${{ github.run_id }}
49+
restore-keys: verify-state-
50+
51+
- name: Tier 0 score (writes scores cache)
52+
run: python -m app.verify score
53+
54+
- name: Tier 1 source-URL liveness
55+
run: python -m app.verify check-urls --max ${{ github.event.inputs.max_urls || '2000' }}
56+
57+
- name: Tier 2 external cross-reference
58+
run: python -m app.verify crossref --max ${{ github.event.inputs.max_crossref || '500' }}
59+
60+
- name: Tier 3 promote (dry-run)
61+
run: python -m app.verify promote
62+
63+
- name: Tier 3 promote (apply)
64+
if: ${{ github.event.inputs.apply == 'true' }}
65+
run: python -m app.verify promote --apply
66+
67+
- name: Structural validator self-check
68+
if: ${{ github.event.inputs.apply == 'true' }}
69+
run: python -m app.validate
70+
71+
# Guard: the only tracked changes may be `verified` toggles in data/**.json
72+
# plus the promotion ledger. Anything else fails the run loudly.
73+
- name: Guard diff scope
74+
if: ${{ github.event.inputs.apply == 'true' }}
75+
run: |
76+
python - <<'PY'
77+
import subprocess, sys
78+
out = subprocess.run(["git", "diff", "--unified=0", "--", "data/"],
79+
capture_output=True, text=True).stdout
80+
bad = []
81+
for line in out.splitlines():
82+
if line.startswith(("+++", "---", "@@", "diff ", "index ")):
83+
continue
84+
if line.startswith(("+", "-")) and line[1:].strip():
85+
body = line[1:].strip().rstrip(",")
86+
if body not in ('"verified": true', '"verified": false'):
87+
bad.append(line)
88+
if bad:
89+
print("Unexpected non-verified changes:")
90+
print("\n".join(bad[:50]))
91+
sys.exit(1)
92+
print("diff scope OK: only verified toggles")
93+
PY
94+
95+
- name: Open promotion PR
96+
if: ${{ github.event.inputs.apply == 'true' }}
97+
env:
98+
GH_TOKEN: ${{ secrets.TECHAPI_TOKEN || secrets.GITHUB_TOKEN }}
99+
run: |
100+
set -e
101+
if git diff --quiet -- data/; then
102+
echo "no promotions to commit"; exit 0
103+
fi
104+
branch="verify/promote-${{ github.run_id }}"
105+
git config user.name "github-actions[bot]"
106+
git config user.email "github-actions[bot]@users.noreply.github.com"
107+
git checkout -b "$branch"
108+
git add data/
109+
git commit -m "data(verify): promote records to verified via cross-reference
110+
111+
Auto-promotions from the verification layer (green+live-T1 or crossref-confirm).
112+
Each flip is verified:false->true only; see data/_verify/ledger.jsonl. Refs #1"
113+
git push origin "$branch"
114+
gh pr create --base main --head "$branch" \
115+
--title "data(verify): verified promotions ($(date -u +%Y-%m-%d))" \
116+
--body "Automated verified promotions from \`app.verify promote\`. Each change flips only the \`verified\` flag; structural validator passed and diff scope guarded. Review before merge. Refs #1"

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,10 @@ env/
3131
# Note: data/_staging/ (raw collected candidate pool) is intentionally tracked —
3232
# comprehensive data collection is a purpose of this repo.
3333

34+
# Verification layer caches: full Tier 0 scores + network caches are cheap to
35+
# recompute. Only data/_verify/ledger.jsonl (the promotion audit trail) is tracked.
36+
data/_verify/state/
37+
3438
# Testing / coverage
3539
.pytest_cache/
3640
.coverage

‎app/verify/__init__.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
"""TechAPI data *verification* layer (§ existence/trust, sits above structural validation).
2+
3+
``app.validate`` answers "is this record well-formed?". ``app.verify`` answers
4+
"does this record describe a real, actually-existing device/part — confidently
5+
enough to mark it ``verified``?".
6+
7+
It is a separate, additive layer: the structural validator (``app/validate.py``)
8+
stays the fast CI gate and is never rewritten. Verification is tiered:
9+
10+
* Tier 0 — offline deterministic plausibility score over the whole dataset
11+
(``offline``/``signals``/``hosts``); bands records green/yellow/red.
12+
* Tier 1 — ``source_urls`` HTTP liveness (``http_check``).
13+
* Tier 2 — external cross-reference under an exact-heading rule (``crossref``).
14+
* Tier 3 — hybrid escalation + safe ``verified:true`` write-back (``promote``).
15+
16+
Decisions are recorded append-only in ``data/_verify/ledger.jsonl`` so runs are
17+
incremental and resumable.
18+
"""

‎app/verify/__main__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
"""``python -m app.verify`` entry point."""
2+
3+
import sys
4+
5+
from .cli import main
6+
7+
if __name__ == "__main__":
8+
sys.exit(main())

0 commit comments

Comments
 (0)