AI Gateway MLflow chat-completions omits finish_reason on natural stop (breaks inkling in Pi)

[dgokeeffe]

Summary

The AI Gateway MLflow chat-completions endpoint (/ai-gateway/mlflow/v1/chat/completions) omits the OpenAI-spec-required finish_reason field on the terminal chunk of a streaming response when a model completes naturally (i.e. not truncated by max_tokens). The stream ends with only data: [DONE].

This breaks any OpenAI-compatible streaming client that (correctly, per spec) requires a finish_reason before the stream terminates. Concretely it makes databricks-inkling unusable through the Pi coding agent, which errors Stream ended without finish_reason.

Affected model

• databricks-inkling (a reasoning model — emits reasoning_content deltas, then content deltas)

Non-reasoning endpoints on the same route (e.g. databricks-glm-5-2) do send finish_reason and work correctly, so this appears specific to inkling's serving stack.

Reproduction (natural completion → NO finish_reason)

curl -sN "$WS/ai-gateway/mlflow/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"model":"databricks-inkling","max_tokens":4000,"stream":true,
       "messages":[{"role":"user","content":"reply with exactly the two letters: ok"}]}'

Observed tail of the stream:

data: {"id":"chatcmpl-...","choices":[{"delta":{"role":"assistant","content":"ok"},"index":0,"logprobs":null}], ...}

data: [DONE]

grep -c finish_reason over the full stream returns 0 — no chunk carries it.

Contrast: truncated completion DOES send finish_reason

With a small budget the model is cut off and the gateway does emit finish_reason:"length":

curl -sN ... -d '{"model":"databricks-inkling","max_tokens":32,"stream":true, ...}'
...
data: {"...","choices":[{"delta":{"...","reasoning_content":" friendly"},"finish_reason":"length","index":0,...}], ...}
data: [DONE]

So the field is wired up — it's just missing on the natural stop path (should be finish_reason:"stop").

Expected

Per the OpenAI Chat Completions streaming spec, the final content chunk (or a final empty-delta chunk) before [DONE] should carry finish_reason:"stop" on normal completion.

Impact

• databricks-inkling cannot be used by spec-compliant streaming clients (Pi, and likely others).
• The corresponding ucode work exposing MLflow chat-completions-only models to Pi (closes ucode pi automatic model detection #204) leaves inkling selectable but non-functional until this is fixed. Once the gateway emits finish_reason, inkling works with no further ucode change.

Environment

• Workspace: AI Gateway v2, /ai-gateway/mlflow/v1/chat/completions
• Client: Pi 0.80.6 (@earendil-works/pi-ai openai-completions parser throws at the missing finish_reason)

[dgokeeffe]

ucode PR #216 ships a client-side workaround: a small local SSE-repair proxy in Pi's launch path that injects the missing finish_reason:"stop" before [DONE] when the gateway omits it, so databricks-inkling is usable through Pi today. It's a no-op for well-behaved models and is intended to be removed once the gateway emits finish_reason on natural completion — this issue tracks the proper fix at the source.

[why6689]

This is a good compatibility catch. In streaming chat-completions, the terminal chunk needs to be explicit enough for downstream clients to distinguish natural completion from truncation, disconnect, or malformed stream.

A regression test I’d add:

• normal natural stop emits a final chunk with finish_reason="stop",
• max token truncation emits finish_reason="length",
• client disconnect or upstream error does not look like a clean stop,
• non-streaming and streaming responses expose equivalent completion semantics.

I work on Routerra, an OpenAI-compatible gateway, and missing terminal finish_reason is one of those small protocol gaps that can break otherwise compatible clients. If useful for testing, here is my Routerra link:
https://routerra.ai/register?aff=Uzwx