Turn a company's documents into a knowledge graph where every claim is traced to weighted evidence — and what's missing is visible. The gap is often the output.
Deep-tech documentation mixes measured facts, design intent and marketing narrative. This pipeline extracts the claims, entities, risks and relationships from your documents (PDF, Word, PowerPoint, Markdown, HTML or plain text), puts a human review gate in front of everything (nothing becomes a fact until you accept it), links claims to evidential statements in other documents with weighted SUPPORTS/CONTRADICTS edges, and loads the result into Neo4j — where you can ask: which claims have no support? where do documents disagree? what should we validate next?
- Local-first — runs entirely on open-source models via Ollama; nothing leaves your machine. Any OpenAI-compatible endpoint (vLLM, LM Studio, Groq, OpenRouter, OpenAI) or Gemini plugs in through config.
- Shareable when you need it — add accounts (
pipeline/accounts/users.py add you@example.com) and the UI grows an email/password gate, a sign-out button and a 60-minute idle timeout. With no accounts it stays exactly as it was: local, no login. - Human review gate — a local web UI to accept/edit/reject every extracted item, with the source passage (page/line), a verbatim evidence quote and the model's reasoning next to each decision. Duplicate detection and alias-cluster review included.
- Evidence linking — claims are linked to supporting/contradicting statements in
other documents; every edge carries
weight = source trust × relevance. - Ask the graph in plain English — the local LLM translates questions to Cypher, shows its reasoning and the query, and runs it read-only. Canned questions and raw Cypher too.
- See the graph — results render as an interactive graph (click to inspect, double-click to expand), with Table and Raw views alongside, and a database panel showing what the topic contains.
- Ranking & evaluation — score every claim by evidence, rank documents by evidential value vs narrative, measure extractor precision from your own review verdicts.
- Survives a misbehaving model — every call has a generation ceiling derived from its own input, truncated responses are salvaged rather than discarded, and a stuck chunk is retried once then skipped. A looping model costs one chunk, not a run.
- Says when it is broken —
python3 pipeline/ops/health.pychecks the data root, disk, that the model is present (not merely reachable), the graph, stale locks and the login gate, each with the command that fixes it. - API-first — everything the UI does is a documented HTTP endpoint
(
/api/v1, OpenAPI 3.1 at/api/v1/openapi.json), with per-user tokens and read/write/admin scopes. The UI is a client, not a special case. - Company-agnostic — claim layers, document types, trust priors and filename
hints live in
pipeline/config.json, not in code. Nothing is tuned to one corpus: ceilings scale with your inputs, timeouts are one env knob.
Extraction is the only expensive part, and its cost is generation speed — the model writes far more than it reads. Measured, not estimated:
| reviewing, asking the graph | extracting a 20,000-char paper | |
|---|---|---|
| laptop, 8 GB RAM, no GPU | fine | ~2 hours |
| laptop, 16 GB + GPU/Metal | fine | ~30 min |
| 4 vCPU cloud VM, no GPU | fine | 5 h 26 m — timed, all six stages |
| a hosted API instead of Ollama | fine | minutes, but ~80 calls per document |
Floor: 2 cores, 8 GB RAM, 15 GB disk (5 GB model + 1.5 GB GLiNER weights + Neo4j + your corpus). Below 8 GB the model pages to disk and it stops being slow and becomes hopeless. Reviewing needs none of this — it is a web page.
Ask before you start, rather than watching a progress line for three hours:
python3 pipeline/ops/capacity.py --topic mytopic # every document, this machineIt reports chunks, model calls and expected time per document, plus anything in the
way: no model installed, a context window too small for a chunk, a model bigger than
RAM, an exhausted API quota, another run already holding the model. The same scan
runs before every pipeline start (in the UI, as a dialog you can dismiss; over the
API as GET …/estimate and check_only). Nothing it says blocks a run — it
tells you, you decide. It learns: once a stage completes, later estimates come from
what this machine actually did rather than from a token rate.
Extraction is resume-safe and survives sign-out, so starting a long run and coming back is a normal way to use it.
Both run the same code. The cloud one exists so somebody else can review — not because the local one is missing anything.
| Local | Cloud | |
|---|---|---|
| start | ./run.sh |
infra/scripts/up.sh |
| stop | Ctrl-C | infra/scripts/down.sh |
| model | local qwen2.5:7b, offline | the same local model, on the VM |
| accounts | optional | required (public URL) |
| cost | nothing | ~£85/month running, ~£5 paused |
| guide | docs/15-local.md | docs/11-cloud-sop.md |
./run.sh # check everything, then open http://localhost:8765
./run.sh --check # just the checks; starts nothingrun.sh installs nothing and starts no database — it verifies what is running,
names the command for anything that is not, and then launches the UI with the
right interpreter. That last part matters more than it sounds: see
the one thing that catches everybody.
Prerequisites: Python 3.10+, Homebrew (macOS) or equivalent. Full walkthrough in docs/15-local.md.
1. System tools + the local model (~5 GB disk, one-time):
brew install poppler # pdftotext — PDF to text
brew install ollama # local LLM runtime
brew install uv # runs GLiNER / neo4j driver in isolated envs (no global installs)
ollama pull qwen2.5:7b # the default local model
ollama run qwen2.5:7b "Say ready." # health check2. Neo4j — install Neo4j Desktop, create a local instance, start it, then:
cp pipeline/.env.example pipeline/.env # and paste your Neo4j password into it3. Your documents — put PDFs under data/ (any folders) and point
pipeline/config.json → corpus_dirs at them, e.g. ["data/docs", "data/papers"].
While you're there, adapt doc_types (trust priors) and layers to your domain —
or keep the defaults. data/ is gitignored: your documents never enter version control.
No pip installs: the pipeline is stdlib-only; heavy dependencies (GLiNER, neo4j
driver) run via uv run --with <pkg>.
# extract + analyze (resume-safe; the llm stage takes hours on a large corpus)
python3 pipeline/stages/extract.py
uv run --with gliner python pipeline/stages/analyze.py --stage gliner --topic $TOPIC
python3 pipeline/stages/analyze.py --stage llm --topic $TOPIC
python3 pipeline/stages/analyze.py --stage explain --topic $TOPIC # evidence quotes + reasoning
python3 pipeline/stages/analyze.py --stage canonical --topic $TOPIC # cross-document alias clusters
python3 pipeline/stages/pagemap.py # page/line grounding
# review — the human gate (web UI at http://localhost:8765)
uv run --with neo4j python pipeline/web/ui.py
# link claims to evidence across documents, then review the links (UI → Links tab)
python3 pipeline/stages/linker.py --stage pair --topic $TOPIC
python3 pipeline/stages/linker.py --stage judge --topic $TOPIC
# load ONLY what you accepted into Neo4j, then rank & evaluate
uv run --with neo4j python pipeline/stages/load_neo4j.py --topic $TOPIC
python3 pipeline/stages/score.py --topic $TOPICFull step-by-step with timings and the detached-run pattern: docs/03-sop.md.
| doc | what |
|---|---|
| docs/12-user-guide.md | start here — using the platform, reviewing, asking the graph, deploying it |
| docs/13-api.md | the HTTP API: tokens, scopes, all 82 endpoints, worked recipes |
| docs/14-system-design.md | how a document becomes evidence — the pipeline diagram |
| docs/00-overview.md | what this is and the three principles |
| docs/01-architecture.md | stages, file formats, invariants |
| docs/02-setup.md | installation + full config reference |
| docs/03-sop.md | run it, in order, copy-paste |
| docs/04-review-guide.md | the human gate: UI, keys, judgment |
| docs/05-graph-queries.md | ask the graph (incl. plain English → Cypher) |
| docs/06-scoring.md | claim & document ranking, KG health |
| docs/07-adapting.md | point it at your own domain |
| docs/08-troubleshooting.md | when something misbehaves |
| docs/09-roadmap.md | what's built, what's next, known limits |
| docs/10-deep-dive.md | how each stage works, in detail |
| docs/15-local.md | running it locally: one command, first-time install, troubleshooting |
| docs/11-cloud-sop.md | running it on GCP: up, down, day-to-day |
| schema.md | the graph schema: nodes, edges, weights, decisions |
| CONTRIBUTING.md | contributing: setup, the four extension points, what gets a PR sent back |
Seven top-level folders, one job each.
| folder | what lives here |
|---|---|
pipeline/ |
the whole application, in eight named packages (below) |
docs/ |
numbered documentation,00–15, plus arch/ (editable draw.io diagrams). Start at 12-user-guide, API at 13-api, design at 14-system-design |
benchmark/ |
thefrozen benchmark that chose each extraction approach. Never edited to suit new needs; it is the evidence behind every model decision |
qa/ |
qa_suite.py — 874 checks against a real server on a throwaway data root. Run before any deploy |
infra/ |
deploying it on GCP:terraform/ (the infrastructure), scripts/ (up.sh, down.sh), proxy/ (the Cloud Run front door) |
data/ |
your input documents. Gitignored — they never enter version control |
notes/ |
local-only working notes. Gitignored |
Eight packages, named for what they do. A newcomer's first question is usually "where does a document go in?" or "where is the agent?" — the folder names answer both without opening a file.
core/ plib.py paths, topics, atomic writes — every module's floor
schema.py the ontology: what a document is read through
events.py append-only activity log
config.json what you adapt: corpus dirs, layers, doc types, LLM
toolbox/ llm.py pluggable backend + generation ceilings + salvage
providers.py which backends exist, and their adapters
orchestrate.py single / fallback / parallel / ensemble
prompts.py every prompt and few-shot, versioned
keystore.py finds an API key without storing one
agents/ cypher.py English -> Cypher, iterate on the result
answer.py rows -> prose grounded in those rows
settings.py the settings helper, grounded in the catalogue
cypher_ref.md loaded into the query agent's context
stages/ readers.py any format -> text: PDF, txt, Markdown, docx, pptx, HTML
extract.py documents -> clean text (+ doc-type guess)
pagemap.py page/line for every character offset
analyze.py spans, claims, risks, classify, explain, extras
linker.py evidence linking: pair -> judge -> weighted candidates
score.py claim/document ranking, KG health
load_neo4j.py accepted decisions -> Neo4j (only what a human accepted)
export_rdf.py the same, as RDF for SHACL validation
ontology/ OWL/RDFS TBox + SHACL shapes (PROV-O / SKOS / DC)
review/ questions.py the queue: everything outstanding, ordered
rank.py uncertainty x impact
triage.py near-duplicate grouping
bundle.py carry one decision across papers
batch.py a settled slice, behind a spot-check
validate.py runs the checks in validators/
validators/ free · local · model · network tiers
review.py terminal review, same files as the UI
learn/ learn.py fits and calibrates the verifier
bandit.py proposes changes
replay.py evaluates a proposal counterfactually first
finetune.py exports training data
eval_*.py score the linker and agreement against gold
web/ ui.py HTTP server, request auth, v1 dispatch
api.py route table: path -> handler, scope, action, spec
ui.html the entire front end, one file, no build step
vendor/ vis-network 9.1.9 (MIT/Apache-2.0), not from a CDN
accounts/ auth.py email/password, sessions, idle timeout
users.py CLI: accounts and API tokens
apikeys.py per-user tokens, scopes, revocation
policy.py the rules engine
principals.py what a rule is evaluated about
ops/ health.py can this install actually work?
capacity.py will this document finish here, and when?
archive.py copy the irreplaceable 11 MB somewhere safe
recover.py get review decisions back
migrate_*.py move an existing install onto a new layout
Modules import by package — from core import plib, from toolbox import llm — so
an import says where a thing lives. There is still no install step: scripts are
run directly (python3 pipeline/stages/extract.py --topic mytopic) and put
pipeline/ on sys.path themselves. Heavy dependencies stay lazy and run under
uv run --with X; the rest is standard library.
Derived data never lands among the source: a fresh clone writes to oem-data/, and
OEM_DATA_ROOT overrides it.
Setup, the extension points, and the rules that will get a PR sent back: CONTRIBUTING.md.
Model choices aren't vibes: every extraction approach was benchmarked against a
hand-labelled gold standard (benchmark/ holds the methodology and scoring code; per-item
reports make every number inspectable).
Working prototype, verified end-to-end: extraction → review → evidence linking → Neo4j → ranking → querying. See docs/09-roadmap.md for what's next and known limits. License: to be decided — all rights reserved until then.