logcompressor

Intelligent log compression with millisecond search — built in Rust.

---

What is it?

logcompressor is a command-line tool that compresses text logs with industry-leading ratios (typically 75–90% space savings) while remaining searchable without full decompression. It ships as a single, dependency-free binary that runs anywhere your shell does.

Point it at a folder of logs, give it a JSON config describing your log shape, and you get:

• 🗜️ Compact storage — pattern-aware compression on top of a Zstd dictionary.
• ⚡ Sub-second search — bloom-filtered indexes skip irrelevant blocks entirely.
• 🌐 Built-in web viewer — pivot, filter and inspect matches directly in the browser.
• 🧩 Universal tokenizer — works on any text log (CloudWatch, nginx, k8s, Postgres, JVM, custom) without per-format adapters.
• 🔒 Bounded memory — handles 100GB+ files with a fixed memory ceiling.

Why use it?

Feature	logcompressor	gzip / zstd plain	grep + tail
Compression ratio on real logs	~85%	~60%	n/a
Search without decompressing	✅	❌	✅
Block-level skip (time / level)	✅	❌	❌
JSON output for pipelines	✅	❌	partial
Web UI for triage	✅	❌	❌
Bounded memory on huge files	✅	✅	✅

Download

Every release publishes signed, ready-to-run binaries on the GitHub Releases page. Pick the command that matches your machine and paste it into a terminal — it downloads the binary, drops it on your PATH, and you're ready to go.

macOS (Apple Silicon — M1/M2/M3/M4)

mkdir -p ~/.local/bin && \
  curl -L -o ~/.local/bin/logcompressor \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/logcompressor-macos-arm64 && \
  curl -L -o ~/.local/bin/global.dict \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/global.dict && \
  chmod +x ~/.local/bin/logcompressor && \
  xattr -d com.apple.quarantine ~/.local/bin/logcompressor 2>/dev/null; \
  echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && \
  exec $SHELL
logcompressor --help

macOS (Intel)

mkdir -p ~/.local/bin && \
  curl -L -o ~/.local/bin/logcompressor \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/logcompressor-macos-x86_64 && \
  curl -L -o ~/.local/bin/global.dict \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/global.dict && \
  chmod +x ~/.local/bin/logcompressor && \
  xattr -d com.apple.quarantine ~/.local/bin/logcompressor 2>/dev/null; \
  echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc && \
  exec $SHELL
logcompressor --help

Linux (x86_64)

mkdir -p ~/.local/bin && \
  curl -L -o ~/.local/bin/logcompressor \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/logcompressor-linux-x86_64 && \
  curl -L -o ~/.local/bin/global.dict \
    https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/global.dict && \
  chmod +x ~/.local/bin/logcompressor && \
  echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && \
  exec $SHELL
logcompressor --help

Windows (PowerShell)

$InstallDir = "$env:LOCALAPPDATA\Programs\logcompressor"
New-Item -ItemType Directory -Force -Path $InstallDir | Out-Null
Invoke-WebRequest -Uri "https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/logcompressor-windows-x86_64.exe" -OutFile "$InstallDir\logcompressor.exe"
Invoke-WebRequest -Uri "https://github.com/InigoRomero/logzstd-Log-Compressor/releases/latest/download/global.dict" -OutFile "$InstallDir\global.dict"
$user = [Environment]::GetEnvironmentVariable("Path", "User")
if ($user -notlike "*$InstallDir*") { [Environment]::SetEnvironmentVariable("Path", "$user;$InstallDir", "User") }
Write-Host "Installed to $InstallDir. Open a new terminal and run: logcompressor --help"

The CLI looks for global.dict next to its own executable, so both files must live in the same folder.

For a step-by-step walkthrough (uninstall, troubleshooting, custom install paths) see INSTALL.md.

Quick start

# 1) Point the CLI at the config that describes your log shape
logcompressor config myapp

# 2) Compress
logcompressor compress --input logs/app.log

# 3) Search
logcompressor search -F "Session expired"
logcompressor search "user_id: 1234" --from 2026-05-01 --to 2026-05-12
logcompressor search "." --level ERROR --level WARN

# 4) Open the web viewer
logcompressor serve --port 8081

A real-world example with AWS Lambda CloudWatch logs (1.1 MB, 10k records):

$ logcompressor compress --input logs/aws.logs --level 5
Detected 5 log templates.

Total original size:     1.12 MB | Compressed:     0.29 MB | Space savings: 73.80%
Compression finished in 14.68ms

Configuration in 30 seconds

logcompressor needs a JSON file describing where the timestamp lives in each line and which named groups to extract:

{
  "output": "logs/compressed/myapp",
  "parse_rules": {
    "pattern": "^(?<timestamp>\\S+) \\[(?<level>[A-Z]+)\\] (?<message>.*)$",
    "timestamp_format": "YYYY-MM-DDTHH:mm:ss"
  },
  "columns": [
    { "key": "timestamp", "label": "Date",    "type": "datetime" },
    { "key": "level",     "label": "Level",   "type": "string", "filter": ["DEBUG", "INFO", "WARN", "ERROR"] },
    { "key": "message",   "label": "Message", "type": "text" }
  ],
  "compression": {
    "include_metadata_keys": ["timestamp", "level"],
    "text_field": "message"
  }
}

That's it. The tokenizer handles everything else automatically — decimals, IPs, UUIDs, request IDs, currency amounts, paths — using a universal rule: if a token contains a digit, it is a variable.

Need more control? See the tutorial on writing a config from scratch.

Documentation

Topic	Where
First-time install	INSTALL.md
Hands-on tutorial (recommended for new users)	TUTORIAL.md
Release notes	CHANGELOG.md

Troubleshooting

The CLI complains about a missing global.dict

The dictionary must sit next to the executable, not next to your config. Copy it once after installing:

cp global.dict $(dirname "$(which logcompressor)")/

Web viewer shows "Found N events" but rows are empty

Your parse_rules.pattern does not match the raw lines. The web viewer caches the config on startup, so after editing config.json you must restart logcompressor serve. Validate with:

curl -s 'http://127.0.0.1:8081/api/search?q=.&page_size=1' | jq '.hits[0].metadata'

If metadata is {}, the regex still doesn't match. Adjust and restart.

Compression reports many "Rotated N pattern parts"

The tokenizer is producing too many distinct templates, typically because your log has identifier-like tokens without digits (e.g. req-abcDEF). Add a custom variable rule to your config:

"parse_rules": {
  "pattern": "...",
  "variables": [
    { "name": "request_id", "regex": "req-[A-Za-z0-9]+" }
  ]
}

Paths get resolved relative to the wrong directory

--input and config.output are resolved against the current working directory. Either always run from the repo root, or use absolute paths in both.

Performance

Measured on a M2 MacBook Pro (single-threaded compress, 4 threads search):

Input	Size	Ratio	Compress time	Search latency
Application log (synthetic)	5 GB	89.6%	35 s	5.7 s (full scan, 2.5M hits)
AWS Lambda CloudWatch export	1.1 MB	73.8%	15 ms	3 ms (6 hits)
AWS Lambda CloudWatch export	1.1 MB	73.8%	15 ms	5 ms (2k hits)

Memory stays bounded regardless of input size thanks to the streaming compressor and the LRU-evicted encoder pool.

How it works

logcompressor uses a streaming, single-pass design tailored to pattern-rich text logs:

1. Parse the timestamp out of each line using the regex you provide.
2. Tokenize the rest with configurable delimiters; any token containing at least one digit (or matching a custom regex) becomes a {} placeholder.
3. Group lines by the resulting template hash — Duration: 12.09 ms and Duration: 19.01 ms collapse to the same group.
4. Stream-compress each group through Zstd with a shared dictionary, writing a <hash>_<run-id>.txt.lzstd block and a small .zidx sidecar containing min/max timestamp, level, and metadata samples.
5. Search consults .zidx files first to skip entire blocks that cannot possibly match, then streams matches out — yielding sub-second latency on multi-gigabyte archives.

A bounded LRU pool of 512 live encoders guarantees memory stays flat even if a log produces thousands of distinct templates.

License

logcompressor is distributed under the EULA included with this release. See LICENSE for the full terms.

Acknowledgements

• The Zstandard team for the compression core.
• The clap, rayon, regex and ahash crates that make Rust CLIs a joy to build.