Adds Fumadocs coverage for the Usage feature

.gitignore

@@ -111,4 +111,9 @@ frontend/dist/
 # Enterprise license private keys
 enterprise/keys/*.pem
 
+# Celery Beat persistent schedule (SQLite + WAL sidecars)
+celerybeat-schedule
+celerybeat-schedule-*
+celerybeat-schedule.*
+
 #

README.md

@@ -54,8 +54,21 @@ There are two ways to run the application:
 
    This will automatically:
    - Pull pre-built images from GitHub Container Registry (no build required!)
-   - Start all services (database, Redis, API, worker)
+   - Start all services: `db`, `redis`, `api`, `media`, `worker`, `beat`, `worker-imports`, `worker-usage`
    - Run database migrations automatically on startup
+
+   | Service | Purpose |
+   |---------|---------|
+   | `db` | PostgreSQL |
+   | `redis` | Redis (Celery broker + usage counters) |
+   | `api` | HTTP API + frontend |
+   | `media` | Live voice WebSocket media server |
+   | `worker` | Celery: `celery` (evaluator cron dispatch), `audio-metrics` queues |
+   | `beat` | Celery Beat scheduler + `platform` queue worker (alerts, FX, OSS prune) — **single replica** |
+   | `worker-imports` | Celery: `imports`, `diarization`, `eval-control`, `evaluations` |
+   | `worker-usage` | Celery: `usage` queue (flush Redis counters + cost recompute) |
+
+   **Usage costs:** token/cost rollups stay stale without `beat`, `worker-usage`, and default `worker` (evaluator crons; or `eai start-all`).
 
    **Using a specific version:**
    ```bash
@@ -137,20 +150,29 @@ docker compose up -d
      url: "redis://localhost:6379/0"
    ```
 
-4. **Start the application and worker**
+4. **Start the application and workers**
+
+   **Infra only (optional):** if Postgres/Redis run in Docker but the app runs locally:
+   ```bash
+   docker compose up -d db redis
+   ```
 
-   **Option A: Start both together (Recommended)**
+   **Option A: Start everything together (Recommended)**
    ```bash
    eai start-all --config config.yml
    ```
 
-   This single command will:
-   - Start the API server
-   - Start the Celery worker (for background task processing)
-   - Run database migrations automatically
-   - Build the frontend (if needed)
+   This single command spawns:
+   - API server (uvicorn)
+   - Telephony media server (`media` port, default 8001)
+   - Celery worker (`celery`, `audio-metrics`)
+   - Celery worker (`imports`, `diarization`, `eval-control`, `evaluations`)
+   - Celery worker (`usage` — flush + cost recompute)
+   - Celery Beat (platform schedules: usage flush, alerts, FX refresh, OSS prune)
+
+   It also runs database migrations and builds the frontend when needed.
 
-   Press `Ctrl+C` to stop both services together.
+   Press `Ctrl+C` to stop all processes.
 
    **Option B: Start separately (for advanced use)**
 
@@ -163,6 +185,11 @@ docker compose up -d
    ```bash
    eai worker --config config.yml
    ```
+
+   For platform periodic tasks (usage flush, alerts, etc.), start Celery Beat in a separate terminal (single replica):
+   ```bash
+   eai beat --config config.yml
+   ```
 
    Or use the Celery command directly:
    ```bash
@@ -247,7 +274,7 @@ make test-docker-db TEST_DB_HOST=localhost TEST_DB_PORT=5432 TEST_DB_NAME=effici
 
 ### Start Application and Worker Together (Recommended)
 ```bash
-# Start both app and worker with default config.yml
+# Start API + all workers with default config.yml
 eai start-all
 
 # Start with custom config
@@ -264,9 +291,17 @@ eai start-all --no-reload --no-build-frontend
 
 # Customize worker log level
 eai start-all --worker-loglevel debug
+
+# Skip dedicated workers (not recommended for production)
+eai start-all --no-imports-worker
+eai start-all --no-usage-worker
+eai start-all --no-telephony-worker
+
+# Tune usage worker concurrency (default: 4, thread pool)
+eai start-all --usage-worker-concurrency 8
 ```
 
-**Note:** This is the recommended way to run EfficientAI. It starts both the API server and Celery worker in a single command. Press `Ctrl+C` to stop both services.
+**Note:** This is the recommended local-dev workflow. One command spawns the API, telephony media server, three Celery workers (`celery,audio-metrics` · `imports,…` · `usage`), and Celery Beat. Press `Ctrl+C` to stop all processes. For Docker deployments, use `docker compose up -d` instead (separate containers per role; see Quick Start).
 
 ### Start Application Only
 ```bash
@@ -340,7 +375,50 @@ eai worker --loglevel debug
 celery -A app.workers.celery_app worker --loglevel=info
 ```
 
-**Note:** The worker is required for processing background tasks (transcription, evaluation, etc.). If you use `eai start-all`, the worker starts automatically. Only use this command if you need to run the worker separately.
+**Note:** Workers are required for background tasks (transcription, evaluation, usage cost flush, etc.). If you use `eai start-all`, they start automatically. Only use `eai worker` if you need to run a worker separately (e.g. `eai worker --queues usage` for the usage queue only).
+
+### Usage Pricing Ops
+Manage model pricing rates and backfill stored usage costs on `llm_usage_daily` rollups. Requires `beat`, `worker-usage`, and default `worker` (or `eai start-all`).
+
+```bash
+# Upsert model_pricing_rates from app/config/models.json
+eai usage seed-rates --config config.yml
+
+# Compare models.json pricing vs Postgres
+eai usage diff-rates --config config.yml
+
+# Backfill costs in-process (all orgs; use after migrate or catalog change)
+eai usage recompute --config config.yml --sync
+
+# Async recompute via usage queue (requires --organization-id)
+eai usage recompute --config config.yml --organization-id <org-uuid>
+
+# Optional scopes: --model, --usage-kind, --start-date, --end-date
+
+# Optional: fetch LiteLLM prices into pricing_catalog.json
+eai usage sync-litellm --local
+eai usage sync-litellm --local --write-models
+```
+
+**After migrations or catalog changes:**
+```bash
+eai migrate
+eai usage seed-rates --config config.yml
+eai usage recompute --config config.yml --sync
+```
+
+**Flush / Usage UI tuning** — set in `.env` (see `env.example`):
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `USAGE_FLUSH_BUCKET_BATCH_SIZE` | `500` | Buckets per DB transaction |
+| `USAGE_FLUSH_MAX_BATCHES_PER_RUN` | `30` | Batches per flush tick (≤ **15,000** buckets/run) |
+| `USAGE_FLUSH_BEAT_SECONDS` | `120` | Celery Beat flush interval (~2 min lag vs Redis) |
+| `USAGE_FLUSH_LOCK_TTL_SECONDS` | `300` | Per-org flush lock TTL |
+| `USAGE_READ_CACHE_TTL_SECONDS` | `90` | Redis cache TTL for usage summary/breakdown/filters |
+| `CRON_DISPATCH_INTERVAL_SECONDS` | `30` | Evaluator cron dispatcher tick (default worker) |
+
+Usage UI reads Postgres only (summary/breakdown/filters); Redis counters flush on the Celery Beat schedule (~2 min eventual consistency). If Redis backlog grows, lower `USAGE_FLUSH_BEAT_SECONDS` or raise `USAGE_FLUSH_MAX_BATCHES_PER_RUN`.
 
 ### Generate Config File
 ```bash
@@ -455,6 +533,10 @@ POSTGRES_PASSWORD=password
 POSTGRES_DB=efficientai
 SECRET_KEY=your-secret-key-here
 
+# Usage cost flush (see README "Usage Pricing Ops"; full list in env.example)
+# USAGE_FLUSH_BEAT_SECONDS=120
+# USAGE_FLUSH_MAX_BATCHES_PER_RUN=30
+
 # Optional: GCS blob storage (also set storage.blob_provider: gcs in config.yml)
 BLOB_STORAGE_PROVIDER=gcs
 GCS_BUCKET_NAME=your-gcs-bucket

app/api/v1/api.py

@@ -47,6 +47,8 @@
     dashboard,
     llm_gateway,
     platform_admin,
+    org_usage,
+    usage_pricing,
 )
 
 api_router = APIRouter()
@@ -97,3 +99,5 @@
 api_router.include_router(dashboard.router)
 api_router.include_router(llm_gateway.router)
 api_router.include_router(platform_admin.router)
+api_router.include_router(org_usage.router)
+api_router.include_router(usage_pricing.router)