FinVerify

The verification layer for financial AI.

Deterministic correction for AI-generated numbers in the browser, in your backend, in your terminal.

Live Demo • Docs • Paper • Model • Discussions

Ecosystem · Extension · SDK · Benchmark · Features · Architecture · Pipeline · Core Components · Quick Start · Results · Research · Contributing

🔒 Deterministic	🧾 Auditable	🌱 Open Source
Rule-based correction, not another model guess	Every correction logged — rule, input, output	Apache 2.0, actively developed, open to contributors

FinVerify Ecosystem

FinVerify isn't a single backend anymore it's a monorepo of interoperable components that all share the same verification core (the DVL). Start here, then go deeper via each component's own README.

Component	What it does
🧩 finverify-extension	Chrome extension providing inline financial verification inside AI chat UIs
🖥 finverify-terminal	Backend services — REST API, WebSocket server — plus the terminal UI and market dashboard
📦 finverify-sdk	Official Python SDK (`pip install finverify-sdk`) for integrating FinVerify into your own applications
📊 finverify-bench	Benchmark suite and evaluation harness for deterministic financial verification
🔬 research	Papers, notebooks, experiments, and reproducibility assets

New here? Jump to Quick Start to run any of these locally, or Repository Structure for the full layout.

Why FinVerify

LLMs answering financial questions are often directionally right and numerically wrong — a decimal point misplaced, a percentage reported as a raw fraction, a sign flipped. In a regulated or capital-allocation context, that's not a rounding error. It's a liability.

Most fixes reach for more prompting. FinVerify reaches for a rule engine instead: the Deterministic Verification Layer (DVL). Scale, sign, and magnitude errors are mechanically distinct from reasoning errors — they need a rule, not another model call.

Underneath the DVL sits a Numeric Canonicalizer that parses raw numeric tokens into an unambiguous, Decimal-based representation before any correction rule runs, and a Constraint Engine that checks whether multiple claims are consistent with each other (e.g. does GrossProfit actually equal Revenue − COGS) using a dependency graph and dimensional analysis, not just single-number correction.

Traditional AI Workflow	FinVerify
Trust the output	Verify the output
Probabilistic	Deterministic
Hidden reasoning	Auditable corrections
Fix errors with better prompts	Fix errors with rules
Black box	Transparent, logged, reproducible

Built for


🛠️ Developers	Shipping AI products that surface financial numbers and need an auditable correction layer
🔬 Researchers	Studying numerical hallucination, who need a reproducible, ground-truth-free method
📊 Analysts	Using AI chat assistants for financial analysis who want a deterministic second check

Actively maintained · Apache 2.0 · Discussions enabled · Extension in active development

TypeScript React Next.js Python FastAPI Mistral-7B (QLoRA) HuggingFace Playwright GitHub Actions

Chrome Extension

FinVerify's flagship surface. It verifies numbers in AI chat output inline, without leaving the page.

_{Default popup state, provider connected.}

Capability	Description
Inline verification	Numerical claims in a chat response run through the DVL as you read
Trust badges	Each verified number gets a HIGH / MEDIUM / LOW badge from the Trust Engine
Verification report	Expand a badge to see the correction rule, the original value, and the corrected value
Provider Adapters	New chat surfaces can be added without touching the DVL

_{A HIGH / MEDIUM / LOW badge rendered next to an AI chat answer.}

_{Expanded badge showing the correction rule, original value, and corrected value.}

Built as a monorepo workspace (@finverify/core shared package) with separate build targets: content and background scripts as IIFE bundles, popup as an ES module. Playwright end-to-end tests against local chat-UI fixtures are in progress, alongside the existing unit test suite.

Python SDK

The official Python client for FinVerify — for developers who want DVL verification inside their own applications, without going through the extension or terminal UI.

pip install finverify-sdk

Capability	Description
Sync + async clients	`FinVerify` and `AsyncFinVerify`, identical public surface
Offline deterministic verification	`verify_local()` runs the DVL correction rules in-process, no network call
Batch verification	Verify multiple claims in one call
Typed models	Dataclass response models, full type hints, `py.typed` marker
Automatic retries	Exponential backoff with jitter on 429/5xx, honoring `Retry-After`

See finverify-sdk/README.md for the full API and finverify-sdk/CHANGELOG.md for release notes.

Benchmark

finverify-bench is the evaluation side of FinVerify: a benchmark suite and harness for measuring deterministic financial verification, independent of any single model.

Reproducible evaluation harness for FinQA-derived and synthetic samples
Ground-truth-blind DVL scoring — corrections never see the answer key
Documented benchmark methodology in BENCHMARK_DESIGN.md

See finverify-bench/README.md to run the harness yourself.

Features

Category	Highlights
Verification	DVL — deterministic scale, sign, and magnitude correction · Numeric Canonicalizer — Decimal-based numeric token parsing shared by DVL and the parser · Constraint Engine — dependency-graph + dimensional-analysis consistency checks across multiple claims · Batch Verification API — one shared constraint pass across a batch of claims · Trust Engine — delta-based confidence scoring
Browser Extension	Inline trust badges and verification reports · Provider Adapter architecture · Monorepo workspace
Backend	FastAPI REST + WebSocket API · Live market data verified through the DVL · SEC EDGAR & earnings-transcript ingestion · RAG pipeline (Pinecone + fallback)
Research	FinVerifyBench — synthetic diagnostic benchmark · Reproducible FinQA evaluation harness · Published ablation study
Developer Experience	Standalone SDK (`pip install finverify-sdk`) · Terminal UI · CI pipelines for backend and SDK
Open Source	Apache 2.0 · CONTRIBUTING guide, Code of Conduct, Security policy · Issues triaged by label

Architecture

End-to-end

flowchart TD
    A[Browser: AI chat page] --> B[Provider Adapter]
    B --> C[DVL]
    C --> D[Backend: FastAPI]
    D --> E[Trust Engine]
    E --> F["UI (extension badge / terminal / dashboard)"]

Backend detail — single-claim pipeline

flowchart TD
    A[User Query] --> B{Query Classifier}
    B -->|advisory| C[LLM Only] --> D[Unverified Response]
    B -->|numerical| E["LLM Inference (Mistral-7B + QLoRA)"]
    E --> F["Numeric Canonicalizer: token → Decimal + unit"]
    F --> G["DVL Pipeline: scale → sign → magnitude + audit log"]
    G --> H["Trust Engine (delta-based)"]
    H --> I[Verified Output + correction log]

Multi-claim pipeline — Constraint Engine

flowchart TD
    A["Batch of claims (POST /v1/verify/batch)"] --> B["verify() per claim (DVL)"]
    B --> C["Formula Parser + concepts.yaml"]
    C --> D["Constraint Graph (dependency order, cycle detection)"]
    D --> E["Dimensional Analysis (Currency / % / Ratio / PerShare / …)"]
    E --> F["Constraint Verifier (tolerance-based comparison)"]
    F --> G["BatchVerifyResponse: per-claim results + shared violations"]

Why it matters — every surface (extension, terminal, API) calls the same DVL for single-claim correction. The Constraint Engine adds a second, independent check across claims: does GrossProfit actually equal Revenue − COGS, not just "is this one number formatted correctly."

TODO (unverified in-repo): the backend currently contains two constraint-checking code paths — backend/fcg/constraint_engine.py (older) and backend/core/financial/constraints/ (newer, described above). Both have live test suites. This README describes the newer constraints/ module since it's the one wired into verify_batch(); the relationship between the two, and whether fcg/ is being deprecated, isn't documented in the repo and should be clarified rather than assumed.

This diagram intentionally omits ingestion and RAG subsystems — see Repository Structure for those.

Verification Pipeline

The full flow a claim goes through, end to end:

Input — a claim arrives either from LLM output (extension, terminal query) or as a direct API call (/verify, /v1/verify/batch).
Claim Extraction — the numeric assertion and its associated concept (e.g. "gross margin") are pulled out of the surrounding text.
Numeric Canonicalization — the raw numeric token is parsed into a structured, unambiguous form (Decimal value + unit), rejecting ambiguous input rather than guessing.
DVL — scale, sign, and magnitude correction rules run against the canonicalized value, with every correction logged.
Constraint Verification (multi-claim only) — if two or more related claims are present, they're checked against each other via the dependency graph and dimensional analysis, producing Violation or INDETERMINATE results rather than silently passing.
Trust Engine — a delta-based confidence score is computed from how much correction was needed.
Output — a VerificationResult (or BatchVerifyResponse for batches) containing the corrected value, the trust score, and the full audit trail.

Core Components

Numeric Canonicalizer (backend/numeric/canonicalizer.py) — the single source of truth for turning a raw numeric token into a Decimal value with an explicit unit. Deliberately refuses to guess on ambiguous input (e.g. locale-ambiguous grouping, unclear scale words) rather than silently picking an interpretation. Lives outside core/ specifically to avoid a circular import with app.dvl.

Constraint Engine (backend/core/financial/constraints/) — a formula parser, a dependency graph (Kahn's algorithm, explicit cycle reporting), and a tolerance-based verifier that together check whether multiple financial claims are mutually consistent, independent of whether any single claim's number is "correct" in isolation.

Formula Engine — the sole evaluator of parsed equations; the constraint parser deliberately only parses (produces an intermediate representation) and never evaluates, keeping evaluation logic in one place.

Trust Engine — computes a delta-based HIGH / MEDIUM / LOW confidence score from how much a claim's raw value had to be corrected.

Transcript Ingestion (backend/ingestion/transcripts.py) — extracts and verifies numerical claims from earnings-call transcripts.

Financial Constraint Graph — see the TODO above: this term currently refers ambiguously to either backend/fcg/constraint_engine.py (older) or the dependency graph inside backend/core/financial/constraints/graph.py (newer). Not yet resolved in-repo.

Quick Start

🖥 Backend

Runs the FastAPI verification service.

git clone https://github.com/aadityat23/finverify-llm.git
cd finverify-llm/finverify-terminal/backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env        # fill in HF_TOKEN
uvicorn app.main:app --reload --port 8000

🖼 Frontend

Runs the terminal and market dashboard UI.

cd finverify-llm/finverify-terminal/frontend
npm install
cp .env.local.example .env.local
npm run dev    # http://localhost:3000

📦 SDK

Installs the standalone Python SDK for local, offline verification.

pip install finverify-sdk

For local development against this repo instead:

cd finverify-llm/finverify-sdk
pip install -e ".[dev]"

See finverify-sdk/README.md to use it against a hosted API instead of local, offline verification.

🧩 Chrome Extension

Builds the browser extension for inline verification.

cd finverify-llm/finverify-extension
npm install
npm run build

Load it via chrome://extensions → Load unpacked.

Verify the backend is running:

curl http://localhost:8000/health

curl -X POST http://localhost:8000/verify \
  -H "Content-Type: application/json" \
  -d '{"question": "What was the profit margin?", "raw_number": 0.1240}'

curl http://localhost:8000/market/quotes?symbols=AAPL,TSLA

Showcase

_{Query flow in the terminal-style UI.}

_{Watchlist with verified metric cards and sparklines.}

Screenshots are an open contribution — see Contributing.

Results

FinQA dev set, n=873, 95% bootstrap CI:

Configuration	Accuracy	95% CI	Δ
Baseline (no context)	1.00%	[0.4, 1.9]	—
+Document Context	24.00%	[21.2, 26.9]	+23.0pp
+DVL v1	32.00%	[29.0, 35.1]	+8.0pp
+QLoRA Fine-tuning	38.50%	[35.4, 41.7]	+6.5pp
+DVL v2 (final)	42.61%	[39.5, 45.7]	+4.1pp

Negative results: CoT prompting −9.0pp, CoT fine-tuning −12.0pp, cross-doc RAG −7.5pp.

At 42.61%, this is 5.4pp behind GPT-3.5 (no CoT, 48.0%) — using a model 25x smaller, no proprietary compute, and fully deterministic, auditable output.

The DVL only fires on formatting-level errors, not reasoning errors — see the error taxonomy below.

Error taxonomy (n=539 remaining failures)

Error type	Count	%
Reasoning (close, <50% rel.)	210	39.0%
Reasoning (far, >50% rel.)	184	34.1%
Magnitude	66	12.2%
Order-of-magnitude	62	11.4%
Sign	9	1.6%
Scale	4	0.8%

73.1% of remaining failures are reasoning errors, not correctable by the DVL. 0% are formatting or extraction failures after fine-tuning.

Repository Structure

finverify-llm/
├── README.md                    # this file
├── docs/                        # cross-component documentation, images
├── artifacts/                   # build artifacts, exported reports
├── finverify-extension/         # Chrome Extension (monorepo)
│   ├── packages/
│   │   └── core/                 # @finverify/core — shared verification client
│   ├── content/                  # content script (IIFE build)
│   ├── background/                # background script (IIFE build)
│   └── popup/                    # popup UI (ESM build)
├── finverify-terminal/           # backend services + terminal/dashboard UI
│   ├── backend/
│   │   ├── app/
│   │   │   ├── main.py          # FastAPI app and route definitions
│   │   │   ├── dvl.py           # Deterministic Verification Layer
│   │   │   ├── router.py        # numerical vs advisory query classifier
│   │   │   ├── parser.py        # numeric extraction from LLM text
│   │   │   ├── market.py        # yfinance wrapper, DVL-verified metrics
│   │   │   └── models.py        # request/response schemas
│   │   ├── numeric/             # Numeric Canonicalizer (Decimal-based token parsing)
│   │   ├── core/
│   │   │   ├── engine.py        # verify(), verify_batch()
│   │   │   ├── math_engine/     # DVL rule engine
│   │   │   └── financial/
│   │   │       └── constraints/ # Formula Parser, Constraint Graph, Dimensional Analysis, Verifier
│   │   ├── fcg/                 # TODO: older constraint-checking module — see note above on
│   │   │                        # its relationship to core/financial/constraints/, unresolved in-repo
│   │   ├── ingestion/           # SEC EDGAR and earnings-transcript ingestion
│   │   ├── rag/                 # retrieval pipeline (Pinecone + fallback search)
│   │   └── evals/               # cross-model evaluation harness
│   └── frontend/
│       ├── app/                 # Next.js pages: terminal, market, metrics
│       ├── components/          # TrustScore, DVLReport, VerificationLog, etc.
│       └── lib/                 # API client, offline DVL fallback, history
├── finverify-sdk/                # standalone `pip install finverify-sdk` package
│   └── finverify/                # SDK source — sync/async clients, typed models
├── finverify-bench/               # benchmark suite and evaluation harness
│   ├── BENCHMARK_DESIGN.md        # methodology and construction notes
│   └── DVL_mapping/               # ground-truth-blind DVL evaluation mapping
└── research/                     # papers, notebooks, experiments, reproducibility assets

Component reference (click to expand)

Component	Path	Purpose
Chrome Extension core	`finverify-extension/packages/core`	Shared verification client used across content/background/popup
DVL engine	`backend/app/dvl.py`	Scale/sign/magnitude correction with audit logging
Numeric Canonicalizer	`backend/numeric/canonicalizer.py`	Decimal-based numeric token parsing shared by DVL and the parser
Constraint Engine	`backend/core/financial/constraints/`	Formula parsing, dependency graph, dimensional analysis, tolerance-based multi-claim verification
Batch Verification	`backend/core/engine.py` (`verify_batch`) · `POST /v1/verify/batch`	One shared constraint pass across a batch of claims
Query classifier	`backend/app/router.py`	Routes numerical vs advisory queries
Market layer	`backend/app/market.py`	Live yfinance data, DVL-verified financial metrics
~~Financial Constraint Graph~~ (legacy)	`backend/fcg/constraint_engine.py`	Older multi-number accounting-identity checker; TODO — relationship to the newer Constraint Engine above is not documented in-repo
SEC EDGAR ingestion	`backend/ingestion/sec_edgar.py`	XBRL/fallback ingestion of 10-K/10-Q fundamentals
Earnings transcript verification	`backend/ingestion/transcripts.py`	Regex extraction and DVL verification of earnings-call claims
RAG pipeline	`backend/rag/pipeline.py`	Pinecone vector + keyword-overlap fallback retrieval
WebSocket server	`backend/app/main.py`	Real-time market data push (5s interval)
Terminal UI	`frontend/app/page.tsx`	Terminal-style query interface, three-panel layout
Market mode	`frontend/app/market/page.tsx`	Live watchlist, verified metric cards, sparklines
Metrics dashboard	`frontend/app/metrics/page.tsx`	Paper results, ablation study, error taxonomy
Python SDK	`finverify-sdk/finverify/`	Sync/async client, typed models, `verify_local()` offline mode
Benchmark suite	`finverify-bench/`	FinVerifyBench dataset, DVL evaluation mapping, design docs

Test suite size: finverify-terminal/backend/tests/ currently defines 224 test functions across 19 files (largest: test_constraint_engine.py with 37, test_numeric_canonicalizer.py with 23). This count was taken directly from the test files, not from a CI run — TODO: confirm the actual passing count from a real pytest run in CI, since the backend's heavier dependencies (torch, transformers) weren't installed for this audit.

API Reference

Method	Path	Description
POST	`/query`	LLM inference + DVL verification
POST	`/verify`	DVL-only verification, no LLM call
GET	`/health`	Health check
GET	`/market/quotes?symbols=AAPL,TSLA`	Live stock quotes
GET	`/market/indices`	S&P 500, NASDAQ, VIX
GET	`/market/verified-metrics?symbol=AAPL&metric=profit_margin`	DVL-verified metric
GET	`/market/all-metrics?symbol=AAPL`	All five metrics for a symbol
POST, GET	`/v1/fcg/*`	FCG endpoints: verify, normalize, list constraints
POST, GET	`/v1/rag/*`	RAG endpoints: query, stats, seed
GET, POST, DELETE	`/v1/history/*`	User query-history persistence
WS	`/ws/market`	Real-time market data stream

/v1/fundamentals/{ticker}, /v1/earnings/{ticker}, and /v1/ingest/* are also exposed, for on-demand SEC and transcript ingestion. Endpoint-by-endpoint documentation is an open contribution — see Contributing.

Research


Paper	Modular Verification Outperforms Chain-of-Thought Reasoning in Small Financial LLMs: A Systematic Ablation Study on Numerical Hallucination Reduction
Submitted to	FinNLP @ EMNLP 2026 / IEEE Access
Author	Aaditya Thokal, Universal College of Engineering, Mumbai — aaditya.thokal24@gmail.com
Model	aadi2026/finverify-lora — Mistral-7B + QLoRA, trained on 2,000 FinQA examples
Dataset	FinQA dev set (n=873); FinVerifyBench isolates formatting-level errors from reasoning errors

Roadmap

Tracked through GitHub Milestones — here's where things stand, and where help is most useful.

Core Infrastructure

✅ FastAPI backend, WebSocket market stream
✅ CI pipelines for backend and SDK
🔧 API stability & docs for ingestion routes

Verification Engine

✅ DVL, Trust Engine
✅ Numeric Canonicalizer
✅ Constraint Engine (formula parser, dependency graph, dimensional analysis)
✅ Batch Verification API
🔧 Reconciling backend/fcg/ (legacy) with the newer core/financial/constraints/ module
📋 Verification methods beyond scale/sign/magnitude

Browser Extension

✅ Core, monorepo restructure, Provider Adapters
🔧 Playwright E2E against chat-UI fixtures
📋 Additional Provider Adapters

Developer Experience & Research

✅ Standalone SDK, Terminal UI
🔧 Expanding backend test coverage
📋 Broader model evaluation, deployment tooling

✅ Completed 🔧 In progress 📋 Planned

Contributing

FinVerify is a young project with a lot of open surface area — there's a meaningful way to contribute regardless of your background.

Start with CONTRIBUTING.md for setup and workflow, and CODE_OF_CONDUCT.md for community guidelines.

Label	Good for
`good first issue`	Self-contained, no deep internals required
`help wanted`	Open tasks looking for a contributor
`research`	Benchmark design, ablations, evaluation methodology
`backend`	FastAPI, DVL, ingestion, RAG
`frontend`	Next.js terminal and dashboard
`extension`	Chrome Extension, Provider Adapters, Playwright E2E
`documentation`	Endpoint docs, guides, screenshots

First open-source contribution? good first issue is the place to start.

Community


🌐 Website	Live Demo
💬 Discussions	GitHub Discussions — design questions, feedback, "is this worth doing" conversations
🐛 Issues	GitHub Issues — bugs and tracked work
📄 Contributing guide	CONTRIBUTING.md
👥 Contributors	CONTRIBUTORS.md

FinVerify was created and is maintained by Aaditya Thokal, University of Mumbai/ IITM.

License

Apache License 2.0 — see LICENSE.

If FinVerify is useful to you, consider starring the repository. ⭐

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

FinVerify

The verification layer for financial AI.

FinVerify Ecosystem

Why FinVerify

Chrome Extension

Python SDK

Benchmark

Features

Architecture

Verification Pipeline

Core Components

Quick Start

🖥 Backend

🖼 Frontend

📦 SDK

🧩 Chrome Extension

Showcase

Results

Error taxonomy (n=539 remaining failures)

Repository Structure

API Reference

Research

Roadmap

Contributing

Community

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 119 Commits
.github/workflows		.github/workflows
artifacts		artifacts
docs/images		docs/images
finverify-bench		finverify-bench
finverify-extension		finverify-extension
finverify-sdk		finverify-sdk
finverify-terminal		finverify-terminal
planning		planning
research		research
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
CONTRIBUTORS.md		CONTRIBUTORS.md
IMPLEMENTATION_REPORT.md		IMPLEMENTATION_REPORT.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
package-lock.json		package-lock.json
package.json		package.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

FinVerify

The verification layer for financial AI.

FinVerify Ecosystem

Why FinVerify

Chrome Extension

Python SDK

Benchmark

Features

Architecture

Verification Pipeline

Core Components

Quick Start

🖥 Backend

🖼 Frontend

📦 SDK

🧩 Chrome Extension

Showcase

Results

Error taxonomy (n=539 remaining failures)

Repository Structure

API Reference

Research

Roadmap

Contributing

Community

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages