AgentForge

AgentForge is an AI agent evaluation and orchestration platform for software teams.

Most AI demos stop at "the model answered something." Real engineering teams need a harder question answered before they trust an agent in production:

Is this agent accurate, safe, fast, and affordable enough for the workflow it is supposed to handle?

AgentForge turns that question into a measurable workflow. Teams define agent tasks, version prompts, attach evaluation cases, run scorecards, and compare accuracy, latency, cost, and failure patterns over time.

Target Users

• Engineering teams building AI agents for internal operations.
• Product teams comparing prompt and model behavior before release.
• Platform teams that need repeatable AI evaluation instead of one-off demos.
• Developers learning production-grade AI engineering practices.

Current Slice

This first version includes:

• A TypeScript monorepo.
• Shared domain model for agent tasks, prompt versions, eval cases, eval runs, score, cost, and latency.
• Deterministic scoring logic with tests.
• Fastify API exposing a seeded AgentForge workspace.
• PostgreSQL-backed repository with schema bootstrap and in-memory fallback.
• React dashboard summarizing agent quality, failures, cost, and latency.
• Recruiter-facing architecture and roadmap documentation.

Architecture

apps/
  api/        Fastify API for tasks, prompt versions, cases, and run summaries
  web/        React dashboard for evaluation visibility
packages/
  domain/     Shared types, seed data, and scoring logic
docs/
  architecture.md

The API uses a repository boundary so storage can evolve without changing the public domain model. Docker Compose runs PostgreSQL by default; local API development without DATABASE_URL falls back to seeded in-memory data.

Tech Stack

• TypeScript
• Fastify
• React
• Vite
• Vitest
• PostgreSQL

Local Development

npm install
npm run dev

The API runs on http://localhost:4317. The web app runs on http://localhost:5173.

If Node is not installed locally, run the stack with Docker:

docker compose up

Validation

npm run typecheck
npm test
npm run build

Docker-based validation:

docker run --rm -v "$PWD":/workspace -w /workspace node:20-bookworm \
  bash -lc "npm install && npm run typecheck && npm test && npm run build"

Recruiter Summary

AgentForge demonstrates practical AI engineering beyond chatbot demos: evaluation design, prompt versioning, scoring, observability signals, API design, and product thinking around reliability and cost.

Roadmap

• Add model/provider adapters for OpenAI-compatible APIs.
• Add authenticated workspaces and role-based access.
• Add run history charts and prompt comparison views.
• Add CI with test and build gates.
• Add exportable evaluation reports.
• Add Docker Compose for one-command local review.