KernelSense is a lightweight, high-fidelity, open-source evaluation dashboard designed to benchmark, audit, and analyze Microsoft Semantic Kernel agents. It enables developers to evaluate agent accuracy, map intent execution paths, analyze token expenditure, and isolate model failure modes side-by-side across different Large Language Models (such as gpt-4o and gpt-4o-mini).
Developed specifically for financial services (FSI) use cases, KernelSense includes native plugins for account extraction and risk grading, simulating real-world decision-making matrices under high-performance benchmarks.
KernelSense utilizes a split-layer architecture with a highly-responsive Next.js frontend communicating with a robust FastAPI backend. The backend orchestrates test vectors through Microsoft Semantic Kernel and captures execution metrics.
┌────────────────────────────────────────────────────────┐
│ KernelSense Dashboard │
│ (Next.js & Tailwind CSS v4) │
└───────────────────────────┬────────────────────────────┘
│
│ HTTP POST /api/v1/run-matrix-evals
▼
┌────────────────────────────────────────────────────────┐
│ FastAPI Backend Core │
│ (Uvicorn HTTP Gateway) │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Analytical Evaluation Engine │
│ (app/evaluator/matrix_engine.py) │
└─────────────────────┬─────────────┬────────────────────┘
│ │
┌─────────────┘ └─────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ GPT-4o │ │ GPT-4o-mini │
│ (High Cost/ │ │ (Low Cost/ │
│ High Acc) │ │ Med Acc) │
└──────┬───────┘ └──────┬───────┘
│ │
└───────────────────┬─────────────────────┘
▼
┌────────────────────────────────────────────────────────┐
│ Microsoft Semantic Kernel │
│ - Native Plugin Function Registration │
│ - Custom FinanceRiskPlugins Class │
│ - Automated Function Choice Behavior │
└────────────────────────────────────────────────────────┘
- Multi-Model Evaluation Matrix: Simultaneously benchmark identical prompt vectors against multiple models (e.g.
gpt-4ovs.gpt-4o-mini) to measure accuracy trade-offs. - Token Cost Analysis: Live cost calculation based on token expenditure utilizing customizable per-1k-token pricing models.
- Failure Mode Analytics: Automatic categorization of non-passing test cases into structured failure types:
- Tool Selection Error: The agent failed to invoke any tools.
- Incorrect Tool Selected: The agent invoked the wrong plugin or function.
- Output Mismatch / Hallucination: The agent successfully called the tool, but the output text failed to satisfy expected deterministic FSI assertions.
- Premium Dashboard UI: Dark-mode themed visual interface featuring:
- Accuracy Contrast Cards comparing overall percentage performance.
- Financial Metric Cards showcasing real-time cost differentials.
- Enterprise Cloud Savings Indicator indicating the cost-efficiency gains (upwards of ~97%) of shifting to mini models.
- Comparison Table Layout with unified sub-rows presenting model actions and status pills side-by-side.
KernelSense/
├── backend/
│ ├── app/
│ │ ├── agents/
│ │ │ ├── plugins.py # Custom FSI Native plugins (FinanceRiskPlugins)
│ │ │ └── sk_agent.py # Microsoft Semantic Kernel setup & completions
│ │ ├── evaluator/
│ │ │ ├── engine.py # Single model evaluation runner
│ │ │ ├── matrix_engine.py # Multi-model comparison execution loop
│ │ │ └── schemas.py # Pydantic v2 data models for response structures
│ │ └── main.py # FastAPI app routing gateway & middlewares
│ ├── requirements.txt # Fully pinned backend dependencies
│ └── venv/ # Python virtual environment
├── frontend/
│ ├── public/ # Public assets & favicons
│ ├── src/
│ │ └── app/
│ │ ├── globals.css # Core styles & Tailwind v4 configurations
│ │ ├── layout.tsx # Root Next.js layout metadata
│ │ └── page.tsx # Responsive UI Dashboard source
│ ├── package.json # Frontend script definitions
│ └── tsconfig.json # Strict TypeScript rules
└── README.md # Public documentation root asset
KernelSense requires Azure OpenAI configurations to communicate with LLM services. Create a .env file in the backend/ directory or export them in your shell:
# Azure OpenAI Orchestration Variables
export AZURE_OPENAI_ENDPOINT="https://your-deployment-name.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-azure-api-key-here"
export AZURE_OPENAI_DEPLOYMENT_NAME="your-llm-deployment-name-here"Note: For validation and offline testing, the system automatically falls back to deterministic, high-fidelity mock telemetry when live credentials are not present.
Follow these steps to run KernelSense in your localized development hosting environment.
Prerequisites: Python 3.10+ installed.
# Navigate to the backend directory
cd backend
# Initialize and activate virtual environment
python3 -m venv venv
source venv/bin/activate
# Install the pinned requirements
pip install -r requirements.txt
# Start the FastAPI Uvicorn server
uvicorn app.main:app --host 127.0.0.1 --port 8000 --reloadThe backend API documentation will be available at http://127.0.0.1:8000/docs.
Prerequisites: Node.js 18+ and npm installed.
# Open a new terminal window and navigate to the frontend directory
cd frontend
# Install packages
npm install
# Start the Next.js development server
npm run devOpen http://localhost:3000 in your web browser to access the live KernelSense Matrix dashboard!
To run strict evaluation sweep validations:
- Frontend Typecheck: Run
npx tsc --noEmitinsidefrontend/to confirm strict TypeScript compiling. - Frontend Build: Run
npm run buildinsidefrontend/to compile optimized statically-generated Next.js pages. - Backend Endpoint Check: Execute
curl -X POST http://127.0.0.1:8000/api/v1/run-matrix-evalsto confirm that Pydantic models serialize accurately.
Distributed under the MIT License. See LICENSE for more information.