Groktobench — Hermes Agent Readiness Protocol (HARP)

Groktobench runs 15 synthetic probes against a clean Hermes Agent container. It records raw session exports, per-probe scores, a composite HARP report, and the run environment needed to diagnose a model's performance.

Quick Start

Requirements: Python 3.11+, Docker, a locally built hermes-agent:latest base image, and an environment file accepted by your model provider.

git clone https://github.com/groktopus/groktobench.git
cd groktobench

# Build the Hermes base image once.
git clone https://github.com/NousResearch/hermes-agent.git /tmp/hermes-agent
docker build -t hermes-agent:latest /tmp/hermes-agent

# Configure the model under test. The env file contains the provider API key.
export GROKTOBENCH_PROVIDER=openai-compatible
export GROKTOBENCH_MODEL="your-local-model"
export GROKTOBENCH_BASE_URL="http://your-local-model-gateway:8000/v1"
export GROKTOBENCH_ENV_FILE="$HOME/.hermes/.env"

# Initialize clean local fixtures, then build, start, execute, score, and report.
python3 scripts/fixtures.py setup --offline
python3 scripts/run-benchmark.py --start

The command prints the generated report path. Its run directory defaults to /tmp/groktobench/harp-<timestamp>/ and contains:

Path	Contents
sessions/	Raw Hermes session exports, one per probe
scores/	Per-probe score JSON and diagnostics
logs/	Hermes stdout and stderr for each probe
report.json	HARP aggregate and phase breakdown
reproducibility-manifest.json	Resolved non-secret configuration, content hashes, container/image metadata, and invocation details
environment.json	Backward-compatible copy of the reproducibility manifest
manifest.json	The exact versioned suite definition used

Pass --output-dir <path> to keep artifacts elsewhere. Use --dry-run to validate the manifest and create an artifact skeleton without Docker.

Use --trials 3 or higher for a calibrated diagnostic run. Repeated attempts are stored in attempts.jsonl; the report adds per-capability success rates, variation, confidence intervals, provider telemetry when available, and failed assertion evidence. Official baseline and held-out rotation policy is in references/evaluation-policy.md.

Suite Definition

probes/manifest.json is the canonical, ordered suite definition. It lists eight skill-recognition probes, five fidelity probes, and two workflow-chain probes. The runner exclusively uses this manifest; legacy or unlisted files cannot run accidentally.

Scoring

HARP combines the three phases into a 0–100 result:

HARP = ((P1 x 2.5) + (P2 x 2.0) + (P3 x 2.0)) / 86 x 100

Detailed criteria are in references/scoring-rubric.md. A score is a diagnostic for the current benchmark revision, model configuration, and runtime environment, not a general model ranking.

Configuration

profiles/benchmarker/config.yaml is the versioned minimal profile template. It contains no credentials. Every run requires GROKTOBENCH_PROVIDER, GROKTOBENCH_MODEL, and GROKTOBENCH_BASE_URL; the latter must be an HTTP(S) OpenAI-compatible endpoint. GROKTOBENCH_ENV_FILE identifies the dotenv API key source and is required with --start. Optional GROKTOBENCH_MODEL_PARAMETERS must be a JSON object without credential fields. The generated manifest records resolved non-secret values, never dotenv contents, and report.json links to it.

Development

Run the smoke test without Docker:

python3 -m unittest discover -s tests

All scripts use Python's standard library. No benchmark data contains real projects or personal data.

Offline Fixtures

The benchmark has a versioned, local fixture boundary for arXiv research and GitHub issues. docker compose puts Hermes and groktobench-fixtures on an internal-only Docker network: no probe can reach a production arXiv or GitHub endpoint. The fixture service is available to configured tools at http://groktobench-fixtures:8080/api; its supported endpoints are /search?query=... (or arXiv-style /query?search_query=...) and /papers/<arxiv-id>.

The image installs a narrow gh shim. It accepts only gh issue create, records the supplied title and body in local fixture state, and returns a synthetic github.local URL. All other gh commands fail rather than contacting GitHub. Versioned papers live in fixtures/arxiv-papers.json; the expected citation IDs for workflow probes are in fixtures/expected-facts.json.

Use the lifecycle interface before each independent suite run:

# Creates /tmp/groktobench-fixtures with an empty issue ledger.
python3 scripts/fixtures.py setup --offline

# Clears recorded issues while keeping the same versioned fixture data.
python3 scripts/fixtures.py reset --offline

# Inspect or remove the disposable state directory.
python3 scripts/fixtures.py status --offline
python3 scripts/fixtures.py cleanup --offline

Set GROKTOBENCH_FIXTURE_STATE_DIR to use another state directory, and pass the same path to --state-dir for lifecycle commands. The internal network requires a model provider reachable within that boundary (for example, a local model gateway) when running a real model evaluation.