LLM Message Proxy

A tiny OpenAI-compatible proxy for local/open-weight LLM backends. It accepts standard /v1/chat/completions requests, reorders any system or developer messages to the top of the messages array, and forwards the normalized request to the upstream LLM endpoint.

Table of Contents

• Why
• Features
• Requirements
• Quick Start
• Configuration
• Run
• Test
• Usage
  • Non-streaming chat completion
  • Streaming chat completion
  • List models
  • Health check
• How It Works
• Request Logging
• Error Handling
• Endpoints
• Troubleshooting
• License

Why

Some harnesses send the system message after user messages. Chat templates for many local models expect system instructions first, so this proxy ensures consistent behavior without changing the client.

Features

• OpenAI-compatible /v1/chat/completions endpoint
• Moves system and developer messages to the top of messages
• Preserves order within system/developer messages and within other messages
• Supports streaming (stream: true) and non-streaming responses
• Passes through /v1/models from the upstream
• Returns upstream errors with their original status code and body
• Request logging with endpoint, model, message count, and reorder flag
• Graceful handling when messages is missing or not an array
• CORS enabled for browser/Electron-based clients
• Unit tests for the message normalization logic

Requirements

• Node.js >= 18.0.0
• npm (or pnpm/yarn)

Quick Start

# 1. Install dependencies
npm install

# 2. Create your environment file from the template
cp .env.example .env

# 3. Edit .env with your upstream LLM details
#    See the Configuration section below.

# 4. Start the server
npm start

Configuration

The .env.example file contains empty keys for you to fill in:

PORT=
UPSTREAM_BASE_URL=
UPSTREAM_API_KEY=
DEFAULT_MODEL=
REQUEST_TIMEOUT_MS=
DEBUG_PROXY=

Copy it to .env and set your values:

cp .env.example .env

Example .env

PORT=9090
UPSTREAM_BASE_URL=http://tailnetproxy.net:9090/v1
UPSTREAM_API_KEY=dummy
DEFAULT_MODEL=Qwen3.5-122B-A10B-FP8
REQUEST_TIMEOUT_MS=300000
DEBUG_PROXY=false

Variable	Required	Default	Description
PORT	No	3001	Port the proxy listens on
UPSTREAM_BASE_URL	Yes	—	Base URL of the upstream OpenAI-compatible endpoint
UPSTREAM_API_KEY	Yes	—	API key sent to the upstream in the Authorization header
DEFAULT_MODEL	No	—	Fallback model when the request does not specify one
REQUEST_TIMEOUT_MS	No	300000	Upstream request timeout in milliseconds
DEBUG_PROXY	No	false	Log message role sequences for each request

Note: The proxy uses UPSTREAM_API_KEY to authenticate to the upstream. It does not forward the client's own Authorization header.

Run

Start the server:

npm start

For development with auto-reload (Node.js >= 18):

npm run dev

You should see:

LLM proxy listening on http://localhost:3001

Test

Run the unit tests for the message normalization logic:

npm test

Usage

Non-streaming chat completion

curl -X POST http://localhost:3001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-122B-A10B-FP8",
    "messages": [
      { "role": "user", "content": "Say hi." },
      { "role": "system", "content": "You are a helpful assistant." }
    ],
    "stream": false
  }'

The proxy forwards the request to the upstream with the system message moved to the top:

{
  "model": "Qwen3.5-122B-A10B-FP8",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Say hi." }
  ],
  "stream": false
}

Streaming chat completion

Set stream: true to receive a Server-Sent Events (SSE) stream just like the OpenAI API:

curl -X POST http://localhost:3001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-122B-A10B-FP8",
    "messages": [
      { "role": "user", "content": "Count to three." },
      { "role": "system", "content": "You are a helpful assistant." }
    ],
    "stream": true
  }'

List models

curl http://localhost:3001/v1/models

This is passed through to ${UPSTREAM_BASE_URL}/models.

Health check

curl http://localhost:3001/health

Response:

{ "status": "ok" }

How It Works

1. The client sends a standard OpenAI chat completion request.
2. The proxy clones the request body so the original object is never mutated.
3. The messages array is normalized:
   • All messages with role: "system" or role: "developer" are collected.
   • All other messages are collected.
   • The two groups are concatenated: system/developer first, then the rest.
   • Order within each group is preserved.
   • Consecutive leading system/developer messages are merged into a single system message, joined by blank lines. Some local backends only accept one system message at the beginning.
4. The normalized request is forwarded to the upstream LLM.
5. The upstream response (JSON or streaming) is passed back to the client.

Request Logging

Each call to /v1/chat/completions writes a single JSON log line to stdout:

{
  "timestamp": "2026-06-20T19:12:22.242Z",
  "endpoint": "/v1/chat/completions",
  "model": "Qwen3.5-122B-A10B-FP8",
  "messageCount": 2,
  "reordered": true,
  "merged": false
}

• reordered is true when at least one system/developer message was originally positioned after a non-system message.
• merged is true when multiple leading system/developer messages were collapsed into a single system message.

Error Handling

• If messages is missing or not an array, the proxy returns 400 Bad Request without contacting the upstream.
• If the upstream returns an error, the proxy forwards the upstream status code and body unchanged.
• If the upstream times out, the proxy returns 504 Gateway Timeout.
• If the proxy cannot reach the upstream, it returns 502 Bad Gateway.

Example error response:

{
  "error": {
    "message": "`messages` is missing or not an array",
    "type": "invalid_request_error",
    "param": "messages"
  }
}

Endpoints

Method	Path	Description
GET	/health	Health check
GET	/v1/models	Pass-through to upstream /models
POST	/v1/chat/completions	Normalize messages and forward upstream

Troubleshooting

System message must be at the beginning. — status 400, invalid_request_error

This is the exact error this proxy was built to fix.

Cause: The client sent a system or developer message after a user message (or sent multiple system messages), and the upstream chat template only allows one system message at the very beginning.

Fix: The proxy normalizes every request by moving all system/developer messages to the top and merging consecutive ones into a single system prompt. No client changes are needed.

Example before:

[
  { "role": "user", "content": "Say hi." },
  { "role": "system", "content": "You are a helpful assistant." }
]

Example after proxy normalization:

[
  { "role": "system", "content": "You are a helpful assistant." },
  { "role": "user", "content": "Say hi." }
]

Not found — status 404 from upstream

Cause: UPSTREAM_BASE_URL is missing the /v1 path.

Fix: Set it to the full OpenAI-compatible base path, e.g.:

UPSTREAM_BASE_URL=http://spark.tail93d320.ts.net:9090/v1

502 Bad Gateway

Cause: The proxy cannot reach the upstream host.

Fix: Verify the host, port, and network path from the proxy machine to the upstream LLM server.

504 Gateway Timeout

Cause: The upstream did not respond within REQUEST_TIMEOUT_MS.

Fix: Increase REQUEST_TIMEOUT_MS or check that the upstream is healthy.

Streaming responses are empty or cut off

Make sure the upstream actually supports streaming and that no intermediary is buffering the response. The proxy passes through the upstream's Content-Type header as-is.

The system message is not moved to the top

Only messages with role: "system" or role: "developer" are reordered. Check the role casing and spelling in the request payload.

License

MIT