This document defines the schema for WebSocket messages exchanged between the client and the NeMo Agent Toolkit server. Its primary purpose is to guide users on how to interact with the NeMo Agent Toolkit server via WebSocket connection. Users can reliably send and receive data while ensuring compatibility with the web server’s expected format. Additionally, this schema provides flexibility for users to build and customize their own user interface by defining how different message types should be handled, displayed, and processed. With a clear understanding of the message structure, developers can seamlessly integrate their customized user interfaces with the NeMo Agent Toolkit server.
The message schema described below facilitates transactional interactions with the NeMo Agent Toolkit server. The messages follow a
structured JSON format to ensure consistency in communication and can be categorized into two main types: User Messages
and System Messages. User messages are sent from the client to the server. System messages are sent from the server
to the client.
type: Defines the category of the message.- Possible values:
auth_messageauth_response_messageuser_messagesystem_intermediate_messagesystem_response_messagesystem_interaction_messageuser_interaction_messageobservability_trace_messageerror_message
- Possible values:
schema_type: Defines the response schema for a given workflowid: A unique identifier for the message.- Purpose: Used for tracking, referencing, and updating messages.
conversation_id: A unique identifier used to associate all messages and interactions with a specific conversation session.- Purpose: Groups-related messages within the same conversation/chat feed.
parent_id: Links a message to its originating message.- Optional: Used for responses, updates, or continuations of earlier messages.
content: Stores the main data of the message.- Format: String for text messages and array for contents which can have attachments such as image, audio and videos. See above example.
- Attachments support OpenAI compatible chat objects such as (Default, Image, Audio, and Streaming)
status: Indicates the processing state of the message.- Possible values:
in_progress,completed,failed. - Optional: Typically used for system messages.
- Possible values:
timestamp: Captures when the message was created or updated.- Format: ISO 8601 (e.g.,
2025-01-13T10:00:00Z).
- Format: ISO 8601 (e.g.,
user: Stores user information - OPTIONAL- name: User name
- email: User email
- other info: Any other information
error: Error information object withcode(string, see Error types),message(string), anddetails(string)schema_version: schema version -OPTIONAL
To resume an active workflow, reconnect using the conversation_id query parameter for that workflow and an identity credential that resolves to the same user who started the workflow. The server restores the workflow state only when both values match. A connection with another identity, or no identity, does not receive the existing workflow state or the pending Human-in-the-Loop prompt for that workflow.
The server accepts the following identity credentials:
- A
nat-sessioncookie or?session=query parameter. - A JWT Bearer token.
- An API key supplied as a Bearer token or
X-API-Keyheader. - HTTP Basic credentials.
Clients can also send a JWT, API key, or Basic credentials through an auth_message. Send the message before expecting restoration; restoration occurs after authentication succeeds.
By default, all listed identity credential methods are accepted. To restrict WebSocket identity credentials, set accepted_identity_credentials in the FastAPI front-end configuration:
general:
front_end:
_type: fastapi
accepted_identity_credentials:
- session_cookie
- jwtSupported values are session_cookie, jwt, api_key, and basic. The api_key value covers both Bearer API keys and the X-API-Key header. An empty list rejects every supplied identity credential. When a client supplies a disabled credential method, the server returns an authentication error and does not restore workflow state.
JWT signature and claim verification is optional. Define one or more named JWT authentication providers, then select them with identity_authentication in the FastAPI front-end configuration. Each selected provider verifies JWT tokens from its configured issuer before the server resolves the user identity:
authentication:
corporate_jwt:
_type: jwt
issuer_url: https://identity.example.com
jwks_uri: https://identity.example.com/.well-known/jwks.json
audience: nemo-agent-toolkit
scopes:
- workflow:resume
general:
front_end:
_type: fastapi
accepted_identity_credentials:
- jwt
identity_authentication:
- corporate_jwtEach JWT provider requires issuer_url, jwks_uri, and audience. It can also require scopes and configure timeout and leeway. Add another named provider and select it in identity_authentication to accept JWT tokens from another issuer. The issuer claim selects the matching provider; an unknown issuer or a failed signature, time, audience, or scope check returns an authentication error and does not restore workflow state. If identity_authentication is omitted, JWT tokens retain the existing decode-only behavior. Configuring identity_authentication while excluding jwt from accepted_identity_credentials is invalid.
For local testing or a deployment where an authenticating reverse proxy is the only service that can reach nat serve, the FastAPI front end can instead trust an upstream identity header:
general:
front_end:
_type: fastapi
identity_header: X-User-IDThis setting is secure only if the proxy authenticates every connection, overwrites any client-supplied value, and the server is in an isolated network environment. When enabled, the header is authoritative: the connection is rejected if it is missing, empty, or repeated,
and an auth_message cannot replace the resolved identity. identity_header cannot be combined with
accepted_identity_credentials or identity_authentication. Do not use this technique if nat serve is reachable by untrusted clients.
This message allows clients to authenticate over a WebSocket connection when header-based or
cookie-based authentication is not feasible (e.g., browser WebSocket APIs that do not support custom headers).
The server validates the credentials, resolves a user identity, and associates it with the current session.
The server responds with an auth_response_message in both cases — with status: "success" and the resolved
user_id on success, or status: "error" with structured error details on failure.
{
"type": "auth_message",
"payload": {
"method": "jwt",
"token": "<jwt-token>"
}
}{
"type": "auth_message",
"payload": {
"method": "api_key",
"token": "<api-key>"
}
}{
"type": "auth_message",
"payload": {
"method": "basic",
"username": "<username>",
"password": "<password>"
}
}Declares how the UI presents the OAuth 2.0 login page. Unlike the identity methods above, it carries no
credential and receives no auth_response_message. The UI sends it once when the WebSocket opens;
mode is "redirect" (default) or "popup".
{
"type": "auth_message",
"payload": {
"method": "oauth_mode_preference",
"mode": "redirect"
}
}The server responds to an auth_message with an auth_response_message indicating success (with the resolved
user_id) or failure (with structured error details).
{
"type": "auth_response_message",
"status": "success",
"user_id": "5a3f8e2b-1c4d-5e6f-7a8b-9c0d1e2f3a4b",
"payload": null,
"timestamp": "2025-01-13T10:00:00Z"
}{
"type": "auth_response_message",
"status": "error",
"user_id": null,
"payload": {
"code": "user_auth_error",
"message": "Authentication failed",
"details": "Could not resolve user identity from auth payload (method=jwt)"
},
"timestamp": "2025-01-13T10:00:00Z"
}Definition: This message is used to send text content to a running workflow. The entire chat history between the user
and assistant is persisted in the message history and only the last user message in the list will be processed by the
running workflow.
{
"type": "user_message",
"schema_type": "string",
"id": "string",
"conversation_id": "string",
"content": {
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Hello, how are you?"
}
]
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "im good"
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "solve this question"
}
]
}
]
},
"timestamp": "string",
"user": {
"name": "string",
"email": "string"
},
"error": {
"code": "string",
"message": "string",
"details": "string"
},
"schema_version": "string"
}Definition: This message contains the response content from the human in the loop interaction.
{
"type": "user_interaction_message",
"id": "string",
"thread_id": "string",
"parent_id": "string",
"conversation_id": "string",
"content": {
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Yes continue processing sensitive information"
}
]
}
]
},
"timestamp": "string",
"user": {
"name": "string",
"email": "string"
},
"schema_version": "string"
}Definition: This message contains the intermediate step content from a running workflow.
{
"type": "system_intermediate_message",
"id": "step_789",
"thread_id": "thread_456",
"parent_id": "id from user message",
"intermediate_parent_id": "default",
"conversation_id": "string",
"content": {
"name": "name of the step - example Query rephrasal",
"payload": "Step information, it can be json or code block or it can be plain text"
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:01Z"
}Definition: This message contains the final response content from a running workflow.
{
"type": "system_response_message",
"id": "token_001",
"thread_id": "thread_456",
"parent_id": "id from user message",
"conversation_id": "string",
"content": {
"text": "Response token can be json, code block or plain text"
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:02Z"
}Definition: This message sends various types of error content to the client. The content object matches the Error model: code is one of unknown_error, workflow_error, invalid_message, invalid_message_type, invalid_user_message_content, invalid_data_content, user_auth_error; message and details are strings.
{
"type": "error_message",
"id": "token_001",
"thread_id": "thread_456",
"parent_id": "id from user message",
"conversation_id": "string",
"content": {
"code": "workflow_error",
"message": "The provided email format is invalid.",
"details": "ValidationError"
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:02Z"
}System Human Interaction messages are sent from the server to the client containing Human Prompt content.
Each interaction prompt content object supports the following optional fields:
timeout: Timeout in seconds for the prompt. Defaults tonull(no timeout). When set, the frontend should display a countdown timer. If the user does not respond within the specified duration, the frontend should dismiss the prompt and display theerrormessage. The server also enforces this timeout and raises aTimeoutErrorto the workflow. The value is set per-prompt by the workflow code. See the Interactive Workflows Guide for details.error: Error message to display on the prompt if the timeout expires or another error occurs. Defaults to"This prompt is no longer available.".
{
"type": "system_interaction_message",
"id": "interaction_303",
"thread_id": "thread_456",
"parent_id": "id from user message",
"conversation_id": "string",
"content": {
"input_type": "text",
"text": "Hello, how are you today?",
"placeholder": "Ask anything.",
"required": true,
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}{
"type": "system_interaction_message",
"id": "interaction_303",
"thread_id": "thread_456",
"parent_id": "id from user message",
"conversation_id": "string",
"content": {
"input_type": "text",
"text": "Hello, how are you today?",
"placeholder": "Ask anything.",
"required": true,
"timeout": 300,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}{
"type": "system_interaction_message",
"id": "interaction_304",
"thread_id": "thread_456",
"parent_id": "msg_123",
"conversation_id": "string",
"content": {
"input_type": "binary_choice",
"text": "Should I continue or cancel?",
"options": [{
"id": "continue",
"label": "Continue",
"value": "continue",
}, {
"id": "cancel",
"label": "Cancel",
"value": "cancel",
}],
"required": true,
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}{
"type": "system_interaction_message",
"id": "interaction_305",
"thread_id": "thread_456",
"parent_id": "msg_123",
"conversation_id": "string",
"content": {
"input_type": "radio",
"text": "I'll send you updates about the analysis progress. Please select your preferred notification method:",
"options": [
{
"id": "email",
"label": "Email",
"value": "email",
"description": "Receive notifications via email"
},
{
"id": "sms",
"label": "SMS",
"value": "sms",
"description": "Receive notifications via SMS"
},
{
"id": "push",
"label": "Push Notification",
"value": "push",
"description": "Receive notifications via push"
}
],
"required": true,
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}{
"type": "system_interaction_message",
"id": "interaction_306",
"thread_id": "thread_456",
"parent_id": "msg_123",
"conversation_id": "string",
"content": {
"input_type": "checkbox",
"text": "The analysis will take approximately 30 minutes to complete. Select all notification methods you'd like to enable:",
"options": [
{
"id": "email",
"label": "Email",
"value": "email",
"description": "Receive notifications via email"
},
{
"id": "sms",
"label": "SMS",
"value": "sms",
"description": "Receive notifications via SMS"
},
{
"id": "push",
"label": "Push Notification",
"value": "push",
"description": "Receive notifications via push"
}
],
"required": true,
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}{
"type": "system_interaction_message",
"id": "interaction_307",
"thread_id": "thread_456",
"parent_id": "msg_123",
"conversation_id": "string",
"content": {
"input_type": "dropdown",
"text": "I'll send you updates about the analysis progress. Please select your preferred notification method:",
"options": [
{
"id": "email",
"label": "Email",
"value": "email",
"description": "Receive notifications via email"
},
{
"id": "sms",
"label": "SMS",
"value": "sms",
"description": "Receive notifications via SMS"
},
{
"id": "push",
"label": "Push Notification",
"value": "push",
"description": "Receive notifications via push"
}
],
"required": true,
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}Sent by the server when the user must complete an OAuth 2.0 login. The content.text field holds the
authorization URL for the UI to open — as a popup or same-tab redirect per the
OAuth Mode Preference Message. It has no options and expects no reply.
{
"type": "system_interaction_message",
"id": "interaction_308",
"thread_id": "thread_456",
"parent_id": "msg_123",
"conversation_id": "string",
"content": {
"input_type": "oauth_consent",
"text": "https://auth.example.com/oauth/authorize?client_id=...&redirect_uri=...&state=...",
"timeout": null,
"error": "This prompt is no longer available."
},
"status": "in_progress",
"timestamp": "2025-01-13T10:00:03Z"
}Definition: This message contains the observability trace ID for tracking requests across services.
{
"type": "observability_trace_message",
"id": "trace_001",
"parent_id": "id from user message",
"conversation_id": "string",
"content": {
"observability_trace_id": "019a9f4d-072a-77b0-aff1-262550329c13"
},
"timestamp": "2025-01-20T10:00:00Z"
}