Feature Request: Allow Assistant Messages to Act as Generation Prefix (Prefill)

[fonfonya]

This issue proposes adding support for assistant prefill: allowing an assistant message to be provided as a prefix that the model should continue generating from, rather than treating it as a completed assistant turn.

This capability enables deterministic continuation, structured output anchoring, and server-controlled tool or schema-guided generation.

Current Behavior

When passing messages like:

messages = [
    {"role": "user", "content": "Write a short apology email to a customer for a delayed shipment."},
    {"role": "assistant", "content": "Hi John,\n\nI'm sorry for the delay with your order. "}
]

the model server currently serializes the assistant message as a completed turn (constexpr bool add_generation_prompt = true;) and then starts a new assistant turn:

Pipeline input text: <|im_start|>user
Write a short apology email to a customer for a delayed shipment.<|im_end|>
<|im_start|>assistant
Hi John,

I'm sorry for the delay with your order. <|im_end|>
<|im_start|>assistant

As a result, the provided assistant content cannot be used as the active generation prefix.

Expected Behavior

The assistant message should be treated as a partial prefix, and generation should continue immediately after it:

<|im_start|>user
Write a short apology email to a customer for a delayed shipment.<|im_end|>
<|im_start|>assistant
Hi John,

I'm sorry for the delay with your order. 

Other Use Cases

Structured Output Prefill

messages = [
    {
        "role": "user",
        "content": "Is the customer satisfied? Respond in JSON with fields \"reasoning\" and \"answer\"."
    },
    {
        "role": "assistant",
        # Prefills the JSON shape and anchors a concise reasoning style.
        "content": '{\n  "reasoning": "Based on the user\'s tone and wording, '
    }
]

This allows the server to enforce output structure while still letting the model complete the response naturally.

Tool-Guided Generation

messages = [
    {
        "role": "user",
        "content": "Tell the user that their package is delayed by 2 days."
    },
    {
        "role": "assistant",
        # Tool name and immutable arguments are injected by the system.
        # The model only needs to generate the remaining message content.
        "content": '{"tool": "send_notification", "args": {"user_id": "uid_541", "message": "'
    }
]

This pattern enables server-controlled tool configuration, while allowing the model to complete the remaining schema-constrained fields.

[mzegla]

Would configuring add_generation_prompt in the request be the solution here?

Also, regarding mentioned use cases, note that for structured output in general we have response_format parameter:
https://github.com/openvinotoolkit/model_server/tree/main/demos/continuous_batching/structured_output

Tool guided generation is strictly connected to agentic cases and it's generally implemented for a few models that we support in terms of tool calling:
https://github.com/openvinotoolkit/model_server/blob/main/docs/llm/reference.md#output-parsing-settings
https://github.com/openvinotoolkit/model_server/tree/main/demos/continuous_batching/agentic_ai

[fonfonya]

Thanks for the clarification.

For general assistant prefilling, enabling add_generation_prompt can be helpful. It appears that frameworks such as transformers and vLLM expose two separate controls:

• whether to add a new generation prompt (add_generation_prompt), and
• whether the final assistant message should be treated as an active prefix (continue_final_message).

For example, using the transformers Tokenizer for Qwen3/Qwen3-4B-Instruct-2507:

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=True,
    continue_final_message=False,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm<|im_end|>
# <|im_start|>assistant

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=False,
    continue_final_message=True,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm

tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "Who are you?"},
        {"role": "assistant", "content": "I'm"}
    ],
    tokenize=False,
    add_generation_prompt=False,
    continue_final_message=False,
)
# <|im_start|>user
# Who are you?<|im_end|>
# <|im_start|>assistant
# I'm<|im_end|>

The transformers documentation also notes the following behavior:

TextGenerationPipeline sets add_generation_prompt to True by default to start a new message. However, if the final message in the chat has the assistant role, it assumes the message is a prefill and switches to continue_final_message=True. This is because most models don’t support multiple consecutive assistant messages. To override this behavior, explicitly pass the continue_final_message argument to the pipeline.

---

Regarding structured model outputs and tool-guided generation: while response_format and existing agentic tooling cover many scenarios, there are cases where part of the JSON prefix, or the tool name and a subset of arguments, are already determined by the server. Being able to prefill those prefixes and let the model continue generation from them may reduce latency (when the prefix has non-trivial length), while avoiding the need for the model to generate all content from scratch.

In this sense, assistant prefill can be seen as a general primitive that complements existing structured output and tool-calling mechanisms.

References:

• transformers chat templating: https://huggingface.co/docs/transformers/main/chat_templating#addgenerationprompt
• vLLM OpenAI-compatible chat API: https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#chat-api

[mzegla]

Makes sense. I think it's worth looking into.

@Wovchena, @apaniukov I know add_generation_prompt is already configurable, but maybe continue_final_message is also something that could be useful on GenAI level? Just an idea.

[apaniukov]

The continue_final_message feature is in the backlog. The actual implementation is quite simple, so external contributions to OpenVINO GenAI are welcome in that regard.

[exzile]

Following up on your suggestion — I've implemented add_generation_prompt as a request parameter and would like to contribute it.

Shape: optional add_generation_prompt (bool, default true) on /v3/chat/completions, matching HF transformers and vLLM. When false, the trailing generation prompt is not rendered.

What it touches:

• Parsed and validated in the request handler, stored on the request struct.
• Honored at every chat-template application site: the MINJA path (LLM + VLM continuous batching, and legacy) and the Python-Jinja path (read from the request body and passed into the template render).
• Replaces the previously hardcoded add_generation_prompt = true in those servables.
• Tests for default (generation prompt added) and false (omitted, assistant message preserved).

Verified end-to-end on the MINJA path with HuggingFaceTB/SmolLM2-360M-Instruct:

• default → prompt ends with <|im_start|>assistant
• add_generation_prompt: false → that trailing generation prompt is omitted

Scope note: this implements add_generation_prompt only. Full assistant prefill — continuing from the final assistant message without closing it (continue_final_message in transformers/vLLM) — is a separate control. The genai C++ apply_chat_template doesn't currently expose it, so I've left it as a follow-up unless you'd like me to pursue it too.

If the field name/placement looks good I'll open a PR.

[exzile]

Opened #4331 with this change.