Skip to content

Releases: robertoecf/OpenFinData

Release list

v0.3.1 — Fix 5 bugs from Codex adversarial review

v0.3.1 — Fix 5 bugs from Codex adversarial review Latest
Latest

Choose a tag to compare

View all tags
@robertoecf robertoecf released this 29 Apr 12:14
v0.3.1
41474e7

Patch release. v0.3.0 shipped with a critical DoS bug in /registry/lookup that we caught only after the release, via adversarial review by an external LLM (gpt-5.4 xhigh via tmux). All five issues are fixed here, with regression tests for each.

What was broken in v0.3.0

# Bug Severity What happened
1 FTS5 reserved words crashed the endpoint 🔴 Critical GET /registry/lookup?q=OR returned 500 Internal Server Error (sqlite3 syntax error). Public DoS.
2 FTS5 syntax injection 🟠 High q=PETR4 NOT VALE3 returned PETROBRAS — user could inject FTS5 operators to steer results.
3 Prefix / fragment search didn't work 🟠 High q=PETROBR returned 0 hits despite the contract claiming "company-name fragment" / "prefix searches".
4 Labeled identifiers stopped resolving 🟡 Medium q="CNPJ: 33.000.167/0001-01", q="FIP 05886", q="CVM 9512" all returned 0.
5 cdb series in BCB SGS catalog was secretly historical 🟢 Low Last observation 2012-12; users picking findata bcb get cdb got ancient data with no warning.

How they were fixed

A single rewrite of _normalize_query in src/findata/registry/store.py:

_FTS5_RESERVED = frozenset({"AND", "OR", "NOT", "NEAR"})

def _escape_token(t: str) -> str:
    if t in _FTS5_RESERVED:
        return f'"{t}"'   # phrase-quoted literal — no operator interpretation
    return f"{t}*"        # prefix wildcard — fragment / prefix matching

Plus a third "digits-only joined" variant in the OR'd query, so labels like CNPJ: don't contaminate the joined form.

For Bug 5: cdb in SERIES_CATALOG now self-documents as "Taxa média de CDB pré-fixado (histórico, descontinuado em 2012)".

Regressions added

test_lookup_fts5_reserved_word_does_not_crash      # Bug 1
test_lookup_fts_injection_is_neutralized           # Bug 2
test_lookup_prefix_match_finds_full_word           # Bug 3
test_lookup_labeled_cnpj_resolves                  # Bug 4
test_normalize_query_reserved_word_gets_quoted     # unit
test_normalize_query_injection_attempt_neutralized # unit
test_normalize_query_labeled_cnpj_includes_digits_only_variant
+ 2 unit updates for changed normalize behavior

168 → 176 tests. Ruff + mypy --strict clean. Pre-push hook validated.

Upgrade

pip install -U findata-br==0.3.1

Adversarial review takeaway

The five bugs were caught only because a separate model reviewed the diff after the release. Doing this review before v0.3.0 shipped would have saved the patch release — the workflow now treats Codex review as a release gate, not a follow-up.

🤖 Generated with Claude Code

Assets 2
Loading

v0.3.0 — Cross-source CNPJ resolver

v0.3.0 — Cross-source CNPJ resolver

Choose a tag to compare

View all tags
@robertoecf robertoecf released this 28 Apr 23:10
v0.3.0
78224ec

Sprint 5 — Cross-source CNPJ resolver via embedded registry

Um catálogo SQLite FTS5 (~50k entidades) embarcado direto no wheel. Uma única chamada MATCH resolve qualquer formato de identificador brasileiro:

from findata.registry import lookup

await lookup("33000167000101")         # CNPJ raw
await lookup("33.000.167/0001-01")     # CNPJ com máscara
await lookup("PETR4")                  # ticker B3
await lookup("9512")                   # cod_cvm
await lookup("petrobras")              # nome (fuzzy via BM25)

O Entity.rank (BM25) discrimina match exato (~ -10) de match difuso (~ -2), útil pra agentes / consumers decidirem confiança.

Highlights
Entidades indexadas ~50.000 (CVM companies + funds + SUSEP, com tickers B3)
Tamanho do wheel 4.2 MB compactado (28 MB descompactado)
Build time ~5 segundos contra BCB/CVM/SUSEP/B3 ao vivo
Tests 117 → 168 (+51 novos)
REST routes 73 → 75 (+2: /registry/lookup, /registry/meta)
MCP tools Auto-expostos via fastapi-mcp

Sprint 5.5 (paralelo)

  • BCB SGS catalog expandido de 17 → 71 séries nomeadas: inflação detalhada (livres / monitorados / serviços / bens / núcleos), agregados monetários (M1–M4), crédito (saldo SFN, inadimplência, spread), setor externo (balança, conta corrente, IDP, reservas), setor público (DBGG, primário, juros nominais), confiança (ICC FGV, ICEI CNI), e mais.
  • ANBIMA IMA história: get_ima_history(family, start, end) — fetch via formulário "Série Histórica", validado contra 14 anos de dados.

CLI

findata registry lookup 33000167000101    # CNPJ raw
findata registry lookup PETR4             # ticker B3
findata registry lookup "porto seguro"    # nome
findata registry meta                     # build sha + counts

REST

curl 'http://localhost:8000/registry/lookup?q=33000167000101'
curl 'http://localhost:8000/registry/lookup?q=PETR4&limit=5'
curl 'http://localhost:8000/registry/meta'

Arquitetura

  • 1 tabela FTS5 (entities com searchable + payload UNINDEXED). Simplicidade brutal — minimalista vs. as 5 tabelas que esbocei na primeira iteração.
  • (spaced) OR (joined) na normalização da query — CNPJ com ou sem máscara resolvem o mesmo, multi-word names também.
  • Wheel-embedded SQLite atualizada semanalmente via CI cron (.github/workflows/rebuild-registry.yml), abre PR se conteúdo mudou (human-in-the-loop, nunca auto-merge).

Limitações conhecidas (v1)

  • B3 ticker enrichment via match exato de nome falha quando B3 escreve nome_ativo colado ("ITAUUNIBANCO") e CVM separa ("ITAU UNIBANCO"). v2: fuzzy match.
  • CVM funds dataset dominado por CANCELADA (46.566 de 46.810); upstream parece ser legacy. Filtro disponível via extra.situacao no payload.
  • Mesma legal entity pode aparecer com múltiplos cod_cvm, gerando duplicatas. v2: dedupe on CNPJ at build time.

Quickstart

pip install findata-br==0.3.0
findata registry lookup 33000167000101

🤖 Generated with Claude Code

Loading

v0.2.0 — 4-sprint catalog expansion

v0.2.0 — 4-sprint catalog expansion

Choose a tag to compare

View all tags
@robertoecf robertoecf released this 28 Apr 14:44
v0.2.0
750ef37

Major expansion of the public-data catalog across 4 sprints, each adversarially reviewed by an external LLM (gpt-5.4 xhigh via tmux). 14 real bugs caught and fixed before release.

What's new

  • Sprint 1 — CVM IPE (fatos relevantes / comunicados), CVM FCA (B3-ticker → CNPJ resolver), B3 COTAHIST (oficial diário desde 1986), B3 índices (IBOV, IBrX, SMLL, IDIV, IFIX + 14 sectoriais)
  • Sprint 2 — CVM FII (Fundos Imobiliários), CVM FIDC (Direitos Creditórios), CVM FIP (Participações)
  • Sprint 3 — Tesouro SICONFI (RREO/RGF, 5,598 entidades), Receita Federal arrecadação por UF (45 tributos × meses desde 2000)
  • Sprint 4 — ANEEL leilões (geração + transmissão desde 1999), SUSEP empresas (233 entidades supervisionadas)

Stack

  • 10 fontes públicas (era 6): BCB, CVM, B3, IBGE, IPEA, Tesouro, ANBIMA, Receita, ANEEL, SUSEP
  • 117 testes unitários (era 57)
  • 73 rotas REST (era 35) com MCP auto-mount em /mcp
  • CLI com 10 subcomandos
  • ICP-Brasil chain trust via integração truststore
  • ruff + ruff-format + mypy --strict clean

Adversarial reviews

Sprint Bugs caught Examples
1 4 FATCOT 1000x off · indices pagination · NBSP not stripped · COTAHIST RAM bomb
2 3 FII pre-2021 schema · FIP quarter=5 silent expansion
3 4 SICONFI co_anexo→no_anexo · uf="BR" wrong · BR-thousand decimals
4 3 ANEEL/SUSEP CP1252 encoding · search whitespace strip

Quickstart

pip install findata-br
findata serve  # REST + MCP at :8000
findata cvm ipe 33.000.167/0001-01 -y 2024 --categoria "Fato Relevante"
findata b3 cotahist PETR4 -y 2024 -m 3
findata aneel leiloes --fonte Eólica -y 2024

Full changelog

See CHANGELOG.md.

Loading