Skip to content
BranchesTags

Repository files navigation

FINCH

Rust CLI for scraping public X surfaces with session cookies.

Contents

Overview

finch is a Rust CLI that turns noisy X web/GraphQL payloads into stable command outputs for:

  • search
  • profile surfaces
  • tweet reads
  • list reads
  • article reads
  • exact batch hydration

At a high level, it works like this:

flowchart LR
    A["CLI Command"] --> B["Planner"]
    B --> C{"Cache Fresh?"}
    C -->|Yes| D["Normalized Output"]
    C -->|No| E["Cookie / Session Router"]
    E --> F["Registry-Driven Request"]
    F --> G["Normalize Payload"]
    G --> H{"Needs Hydration?"}
    H -->|Yes| I["Batch Hydrate Entities"]
    H -->|No| J["Pagination / Stop Rules"]
    I --> J
    J --> K["Persist Cache / Cursor / Health"]
    K --> D
Loading

Architecture

flowchart TB
    subgraph CLI["CLI Layer"]
        MAIN["main.rs"]
        CMD["commands.rs"]
        OUT["output.rs"]
    end

    subgraph Runtime["Runtime Layer"]
        PLAN["planner.rs"]
        CACHE["cache.rs"]
        STATE["state.rs"]
        COOKIES["cookies.rs"]
        TRANSPORT["transport.rs"]
    end

    subgraph Data["Data / Parsing Layer"]
        REG["registry.rs"]
        NORM["normalize.rs"]
        CFG["config.rs"]
    end

    MAIN --> CMD
    CMD --> PLAN
    CMD --> CACHE
    CMD --> STATE
    CMD --> COOKIES
    CMD --> TRANSPORT
    CMD --> REG
    CMD --> NORM
    CMD --> OUT
    CMD --> CFG
Loading

Request strategy

The CLI is built around four runtime ideas:

Area Strategy
Session routing Pick the healthiest cookie per endpoint family instead of random rotation
Cache Cache-first unless --live, with dynamic TTL by tier, pressure, confidence, and depth
Hydration Discover on timeline/search surfaces, then batch-hydrate entities
Pagination Adaptive continuation based on target, yield, duplication, cursor presence, and pressure

Cache and TTL overview

Finch does not use one flat cache timer. It uses cache tiers plus dynamic TTL adjustment.

Cache tiers

Tier Typical use Default base TTL
frontier hot first-page collection reads 45s
surface_page deeper paginated reads 300s
entity single stable entities 3600s
derived derived views such as people-search shaped results 300s
failure thin, empty, or suspicious payloads 20s

Dynamic TTL inputs

The final TTL is adjusted using these default rules:

Signal Rule Default effect
thin / empty result result is thin or returned count is 0 multiply TTL by 0.45
low fill ratio returned/requested ratio is < 0.25 multiply TTL by 1.6
overshoot / full page returned/requested ratio is > 1.0 multiply TTL by 0.85
page depth >= 2 deeper pagination page multiply TTL by 1.8
page depth >= 4 very deep pagination page multiply TTL by 3.0
medium pressure family pressure >= 0.25 multiply TTL by 1.3
high pressure family pressure >= 0.5 multiply TTL by 1.8
severe pressure family pressure >= 0.75 multiply TTL by 2.5
low confidence confidence < 0.4 multiply TTL by 0.4
medium confidence confidence < 0.7 multiply TTL by 0.75
search family search result family multiply TTL by 0.8
article entity article entity cache multiply TTL by 2.0
list entity list entity cache multiply TTL by 3.0
deep family thread/edit-history family multiply TTL by 1.4

Then the final TTL is clamped between:

  • minimum: 10s
  • maximum: 86400s | family type | family-specific multipliers |

TTL flow

flowchart LR
    A["Base TTL by Cache Tier"] --> B["Apply Result Quality Multipliers"]
    B --> C["Apply Page Depth Multiplier"]
    C --> D["Apply Family Pressure Multiplier"]
    D --> E["Apply Confidence Multiplier"]
    E --> F["Apply Family-Specific Multiplier"]
    F --> G["Clamp Between Min/Max TTL"]
Loading

Pagination strategy

Finch keeps fetching collection pages while they are still useful.

Stop rules

Rule Meaning
target reached requested limit has been met
source exhausted no cursor / no more content
duplicate frontier repeated frontier or too many duplicates
low-yield stop consecutive pages add too little new value
pressure stop runtime pressure says stop live pagination

Bonus behavior

Collection commands are target-based by default.

If the final productive page takes:

  • 48 -> 63

then Finch keeps the full 63.

Use:

--strict-limit

if you want exact trimming.

Runtime policy config

The main tuning file is:

~/.finch/config.json

Key sections:

Section What it controls
transport timeout and user-agent
cache.tier_ttl_s base TTL per cache tier
hydration default hydrate batch size
pagination.search search hard caps
pagination.collection collection hard caps
pagination.yield_rules low-yield detection
pagination.thresholds duplicate and pressure stop thresholds
routing.base_concurrency concurrency defaults by family

What is safe to change

These are good user-tunable knobs:

Area Safe examples
hydrate batch size 200, 400, 600
frontier TTL 30-120 seconds
entity TTL 1800-21600 seconds
search hard caps 20-60
duplicate ratio stop 0.7-0.9
pressure stop 0.8-0.95
low-yield streak limit 2-4

What should generally stay in code, not config

These are intentionally not treated as user policy knobs:

Area Reason
response parsing correctness logic
normalization rules output contract stability
endpoint wiring protocol correctness
cache key structure cache integrity
author-filter enforcement correctness for search semantics

Example policy edits

Make Finch more conservative

{
  "hydration": {
    "default_batch_size": 200
  },
  "pagination": {
    "search": {
      "broad_hard_cap": 25,
      "filtered_hard_cap": 25,
      "author_hard_cap": 20
    },
    "thresholds": {
      "duplicate_ratio_stop": 0.75,
      "pressure_stop": 0.85
    }
  }
}

Make Finch more aggressive

{
  "hydration": {
    "default_batch_size": 600
  },
  "pagination": {
    "search": {
      "broad_hard_cap": 50,
      "filtered_hard_cap": 40,
      "author_hard_cap": 35
    },
    "yield_rules": {
      "minimum_new_items_floor": 2,
      "minimum_new_items_ratio": 0.005,
      "low_yield_streak_limit": 3
    }
  }
}

Docs

Install

zsh

git clone https://github.com/KEYURBODAR/finch-cli.git
cd finch-cli

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
echo 'source "$HOME/.cargo/env"' >> ~/.zshrc
source "$HOME/.cargo/env"
source ~/.zshrc

cargo install --path .

mkdir -p ~/.finch
cp config.example.json ~/.finch/config.json
cp registry.phase1.json ~/.finch/registry.json
cp cookies.example.json ~/.finch/cookies.json

finch status

bash

git clone https://github.com/KEYURBODAR/finch-cli.git
cd finch-cli

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
echo 'source "$HOME/.cargo/env"' >> ~/.bashrc
source "$HOME/.cargo/env"
source ~/.bashrc

cargo install --path .

mkdir -p ~/.finch
cp config.example.json ~/.finch/config.json
cp registry.phase1.json ~/.finch/registry.json
cp cookies.example.json ~/.finch/cookies.json

finch status

Configure cookies

Edit:

nano ~/.finch/cookies.json

Put real values in this format:

[
  {
    "id": "ck_01",
    "auth_token": "YOUR_AUTH_TOKEN",
    "ct0": "YOUR_CT0"
  }
]

Runtime policy tuning

~/.finch/config.json now controls the main runtime policy knobs.

You can change things like:

  • cache TTLs
  • default hydrate batch size
  • pagination hard caps
  • low-yield stop thresholds
  • duplicate frontier stop threshold
  • pressure stop threshold

Start from:

cp config.example.json ~/.finch/config.json

Then edit:

nano ~/.finch/config.json

Verify

finch status
finch search posts --query 'from:naval ai' --dry-run --output json

Alternative install

Install directly from GitHub:

zsh

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
echo 'source "$HOME/.cargo/env"' >> ~/.zshrc
source "$HOME/.cargo/env"
source ~/.zshrc

cargo install --git https://github.com/KEYURBODAR/finch-cli.git

mkdir -p ~/.finch
curl -L https://raw.githubusercontent.com/KEYURBODAR/finch-cli/main/config.example.json -o ~/.finch/config.json
curl -L https://raw.githubusercontent.com/KEYURBODAR/finch-cli/main/registry.phase1.json -o ~/.finch/registry.json
curl -L https://raw.githubusercontent.com/KEYURBODAR/finch-cli/main/cookies.example.json -o ~/.finch/cookies.json

finch status

Stack

  • Rust
  • clap
  • tokio
  • reqwest
  • serde
  • rusqlite
  • sha2
  • regex
  • time
  • csv
  • tracing

Features

  • search posts
  • search users
  • user resolve / user get
  • user posts / user media / user articles / user highlights
  • tweet get / tweet thread / tweet edits
  • article get
  • list resolve / list get / list posts / list members
  • batch tweets / batch users / batch handles
  • cache stats / cache clear
  • cookies health
  • health / health --watch
  • output formats: table, json, csv, md

Examples

  • finch status
  • finch search posts --query 'from:naval ai' --limit 5 --live --output json
  • finch search users --query 'openai' --limit 5 --strict-limit --live --output json
  • finch user resolve @naval --live --output json
  • finch user posts @injective --limit 20 --live --output json
  • finch user media @injective --limit 20 --live --output json
  • finch user articles @injective --limit 20 --live --output json
  • finch user highlights @injective --limit 20 --live --output json
  • finch article get 2032323059714306048 --live --output json
  • finch list resolve @pmarca/ai-founders --live --output json
  • finch tweet get 2031702075307176248 --output json
  • finch tweet thread 2031702075307176248 --output json
  • finch tweet edits 2031702075307176248 --output json
  • finch batch users --ids users.txt --output json
  • finch batch tweets --ids tweets.txt --output json

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages