@inferencesh/sdk — ai inference api for javascript & typescript

official javascript/typescript sdk for inference.sh — the ai agent runtime for serverless ai inference.

run ai models, build ai agents, and deploy generative ai applications with a simple api. access models including flux, stable diffusion, llms (claude, gpt, gemini), video generation (veo, seedance), and more.

Installation

npm install @inferencesh/sdk
# or
yarn add @inferencesh/sdk
# or
pnpm add @inferencesh/sdk

Getting an API Key

Get your API key from the inference.sh dashboard.

Quick Start

import { inference, TaskStatusCompleted } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Run a task and wait for the result
const result = await client.tasks.run({
  app: 'your-app',
  input: {
    prompt: 'Hello, world!'
  }
});

if (result.status === TaskStatusCompleted) {
  console.log(result.output);
}

Usage

Basic Usage

import { inference, TaskStatusCompleted } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Wait for result (default behavior)
const result = await client.tasks.run({
  app: 'my-app',
  input: { prompt: 'Generate something amazing' }
});

if (result.status === TaskStatusCompleted) {
  console.log('Output:', result.output);
}

With Setup Parameters

Setup parameters configure the app instance (e.g., model selection). Workers with matching setup are "warm" and skip setup:

const result = await client.tasks.run({
  app: 'my-app',
  setup: { model: 'schnell' },  // Setup parameters
  input: { prompt: 'hello' }
});

Fire and Forget

// Get task info immediately without waiting
const task = await client.tasks.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  { wait: false }
);

console.log('Task ID:', task.id);
console.log('Status:', task.status);

Real-time Status Updates

By default, the client streams task progress over NDJSON (/tasks/{id}/stream) and invokes onUpdate as the task changes. Use onPartialUpdate when you only need specific fields from a partial stream payload:

const result = await client.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  {
    onUpdate: (update) => {
      console.log('Status:', update.status);
      console.log('Progress:', update.logs);
    },
    onPartialUpdate: (update, fields) => {
      console.log('Changed fields:', fields, update.status);
    },
  }
);

Streaming vs Polling

SSE/NDJSON streaming is the default. For edge runtimes that cannot keep long-lived connections open (Convex actions, Cloudflare Workers, etc.), disable streaming and use lightweight status polling instead:

const client = inference({
  apiKey: 'your-api-key',
  stream: false,           // poll /tasks/{id}/status instead of streaming
  pollIntervalMs: 2000,    // default: 2000
});

// Per-call override
const result = await client.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  { stream: false, onUpdate: (u) => console.log(u.status) }
);

In polling mode, the SDK checks /tasks/{id}/status and fetches the full task when the status changes. If that fetch fails after a status transition, run() rejects with the underlying error.

Batch Processing

async function processImages(images: string[]) {
  const results = [];

  for (const image of images) {
    const result = await client.tasks.run({
      app: 'image-processor',
      input: { image }
    }, {
      onUpdate: (update) => console.log(`Processing: ${update.status}`)
    });

    results.push(result);
  }

  return results;
}

File Upload

// Upload from base64
const file = await client.files.upload('data:image/png;base64,...', {
  filename: 'image.png',
  contentType: 'image/png'
});

// Use the uploaded file in a task
const result = await client.tasks.run({
  app: 'image-app',
  input: { image: file.uri }
});

Cancel a Task

const task = await client.tasks.run(
  { app: 'long-running-app', input: {} },
  { wait: false }
);

// Cancel if needed
await client.tasks.cancel(task.id);

Stream Functions (Live Sockets)

A stream function keeps a socket open with its caller for the life of the task: frames go both ways until the caller closes or the app returns. client.live starts the task and dials its socket; the run response carries where to dial (task.socket).

const { task, session } = await client.live(
  { app: 'infsh/voice-loop', function: 'stream', input: { effect: 'robot' } },
  {
    onState: (state) => console.log(state), // connecting → waiting → live → ended
    onBinary: (pcm) => speaker.write(new Int16Array(pcm)),
    onPatch: (patch) => console.log(patch), // e.g. { frames: 120 }
  }
);

session.sendBinary(micFrame);          // one item of the input's binary live field
session.sendPatch({ effect: 'echo' }); // change an ordinary input while it runs
session.close();                       // the function returns and the task completes
await session.ended;

The session is waiting until the app's first frame (a cold start can take a minute) and gives up if the task ends before then. It dials again with a fresh credential when the relay restarts under it. client.sockets.open(taskOrId, handlers) reconnects to a running task's socket, e.g. after a page reload.

What a function's socket carries is in its schemas: a live field is {"type": "array", "format": "stream", "items": ...}. splitLiveSchema(schema) separates the ordinary fields (the request body) from the live ones, and pcmFormat(field.media) reads the sample rate of a PCM audio field.

On Node 18–21 there is no global WebSocket: pass one from the ws package as { webSocket: WebSocket }.

Sessions (Stateful Execution)

Sessions allow you to maintain state across multiple task invocations. The worker stays warm between calls, preserving loaded models and in-memory state.

// Start a new session
const result = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'hello' },
  session: 'new'
});

const sessionId = result.session_id;
console.log('Session ID:', sessionId);

// Continue the session with another call
const result2 = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'remember what I said?' },
  session: sessionId
});

Custom Session Timeout

By default, sessions expire after 60 seconds of inactivity. You can customize this with session_timeout (1-3600 seconds):

// Create a session with 5-minute idle timeout
const result = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'hello' },
  session: 'new',
  session_timeout: 300  // 5 minutes
});

// Session stays alive for 5 minutes after each call

Notes:

session_timeout is only valid when session: 'new'
Minimum timeout: 1 second
Maximum timeout: 3600 seconds (1 hour)
Each successful call resets the idle timer

Session management API

Manage sessions directly without running a task:

// Inspect a session
const info = await client.sessions.get(sessionId);
console.log(info.status, info.expires_at, info.call_count);

// List active sessions
const sessions = await client.sessions.list();

// Extend idle timeout without a task call (sliding window)
await client.sessions.keepalive(sessionId);

// Release the worker immediately
await client.sessions.end(sessionId);

Session errors

import {
  SessionNotFoundError,
  SessionExpiredError,
  SessionEndedError,
} from '@inferencesh/sdk';

try {
  await client.tasks.run({
    app: 'my-stateful-app',
    input: { prompt: 'hello' },
    session: sessionId,
  });
} catch (error) {
  if (
    error instanceof SessionNotFoundError ||
    error instanceof SessionExpiredError ||
    error instanceof SessionEndedError
  ) {
    // Start a new session and retry
    const result = await client.tasks.run({
      app: 'my-stateful-app',
      input: { prompt: 'hello' },
      session: 'new',
    });
  } else {
    throw error;
  }
}

For complete session documentation including error handling, best practices, and advanced patterns, see the Sessions Developer Guide.

Agent Chat

Chat with AI agents using client.agents.create().

Using a Template Agent

Use an existing agent from your workspace by its namespace/name@shortid:

import { inference } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Create agent from template
const agent = client.agents.create('my-org/assistant@abc123');

// Send a message with streaming
await agent.sendMessage('Hello!', {
  onMessage: (msg) => {
    if (msg.content) {
      for (const c of msg.content) {
        if (c.type === 'text' && c.text) {
          process.stdout.write(c.text);
        }
      }
    }
  }
});

// Clean up
agent.disconnect();

Creating an Ad-Hoc Agent

Create agents on-the-fly without saving to your workspace:

import { inference, tool, string } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Create ad-hoc agent (config uses API field names: core_app, system_prompt)
const weatherTool = tool('get_weather')
  .describe('Get current weather')
  .param('city', string('City name'))
  .build();

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@abc123' },
  system_prompt: 'You are a helpful assistant.',
  tools: [weatherTool], // only schemas are sent to the API; handlers stay client-side
});

await agent.sendMessage('What is the weather in Paris?', {
  onMessage: (msg) => console.log(msg),
  onToolCall: async (call) => {
    const result = await runMyClientTool(call.name, call.args);
    await agent.submitToolResult(call.id, result);
  },
});

For multi-turn chats, the SDK opens the chat stream before sending the next message so updates are not missed. Use stopChat() to cancel in-flight generation (POST /chats/{id}/stop), and reset() to clear the current chat and start fresh.

Tool builder

Use the fluent builders to define AgentTool schemas. Client tools (tool) run in your app via onToolCall; server-side tools run on inference.sh.

Builder	Runs on	Description
`tool(name)`	Client	Local handler; only the schema is sent to the API
`appTool(name, appRef)`	Server	Invoke another inference app
`agentTool(name, agentRef)`	Server	Delegate to a sub-agent
`httpTool(name, url)` / `callTool(name, url)`	Server	HTTP request with credential injection (preferred over `webhookTool`)
`webhookTool(name, url)`	Server	Unsigned webhook (legacy; use `httpTool` for new tools)
`mcpTool(name, credentialId, toolName)`	Server	Call a tool through a connected MCP credential
`internalTools()`	Server	Built-in plan, memory, and widget tools

import {
  inference,
  tool,
  appTool,
  httpTool,
  mcpTool,
  internalTools,
  string,
  CredentialProviderGoogle,
} from '@inferencesh/sdk';

const clientTool = tool('get_weather')
  .describe('Get current weather')
  .param('city', string('City name'))
  .build();

// HTTP tool authenticated with a connected account (token injected server-side)
const gmailSend = httpTool('gmail_send', 'https://gmail.googleapis.com/gmail/v1/users/me/messages/send')
  .describe('Send an email via Gmail')
  .method('POST')
  .auth({ credential: CredentialProviderGoogle, credentialId: 'your-credential-id' })
  .build();

// API key or bearer auth
const fetchData = httpTool('fetch', 'https://api.example.com/data')
  .method('GET')
  .auth({ apiKey: 'YOUR_KEY', header: 'X-API-Key' }) // default header: X-API-Key
  .header('Accept', 'application/json')
  .build();

const bearerFetch = httpTool('bearer_fetch', 'https://api.example.com')
  .auth({ bearer: 'YOUR_TOKEN' })
  .build();

const imageGen = appTool('generate_image', 'infsh/flux-schnell@abc123')
  .param('prompt', string('Image description'))
  .requireApproval()
  .build();

const mcpSearch = mcpTool('notion_search', 'your-mcp-credential-id', 'search')
  .describe('Search Notion pages')
  .param('query', string('Search query'))
  .build();

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@latest' },
  system_prompt: 'You are helpful.',
  tools: [clientTool, gmailSend, imageGen, mcpSearch],
  internal_tools: internalTools().memory().build(),
});

callTool is an alias for httpTool. Run npx tsx examples/tool-builder.ts for more schema examples (no API key required).

Lifecycle hooks

Use lifecycleHook() to attach hooks to agent events. Hooks can call a webhook or delegate to another agent as a task:

import { inference, lifecycleHook, HookEventTurnStart, HookEventToolCall } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

const agent = client.agents.create({
  core_app: { ref: 'openrouter/claude-sonnet-4@abc' },
  hooks: [
    lifecycleHook(HookEventTurnStart)
      .webhook('https://example.com/on-turn')
      .timeout(10)
      .build(),
    lifecycleHook(HookEventToolCall)
      .task('my-org/approval-agent')
      .async(true)
      .build(),
  ],
});

Queued messages

sendMessage automatically queues when the agent is busy processing a previous message. Queued messages appear with a queued status until the agent picks them up.

To cancel a queued message before it is processed, use cancelMessage on the React provider or chats.cancelMessage on the API client:

// Via the API client
await client.chats.cancelMessage(messageId);

File attachments

Pass files in sendMessage options. Blob values are uploaded first; objects with a uri (already uploaded via client.files.upload) are attached as-is:

const uploaded = await client.files.upload(imageBlob, {
  filename: 'photo.png',
  contentType: 'image/png',
});

await agent.sendMessage('Describe this image', {
  files: [imageBlob, uploaded], // Blob uploads; FileDTO reuses uri
  onMessage: (msg) => console.log(msg),
});

Structured Output

Use output_schema to get structured JSON responses:

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@latest' },
  output_schema: {
    type: 'object',
    properties: {
      summary: { type: 'string' },
      sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
      confidence: { type: 'number' },
    },
    required: ['summary', 'sentiment', 'confidence'],
  },
  internal_tools: { finish: true },
});

const output = await agent.run('Analyze: Great product!');

agent.run() sends a message with polling (no SSE), waits until the chat is idle, and returns chat.output (parsed finish-tool result, or null if none).

Agent Methods

Method	Description
`sendMessage(text, options?)`	Send a message; streams or polls until idle when callbacks or `stream: false`
`run(text, options?)`	Send and return structured `chat.output` (always uses polling)
`getChat(chatId?)`	Get the current or specified chat (`chat_messages` on the returned chat)
`stopChat()`	Stop generation for the current chat (no-op if no active chat)
`submitToolResult(toolId, resultOrAction)`	Submit result for a client tool (string or `{action, form_data}`)
`startStreaming(options?)`	Manually attach to `/chats/{id}/stream` for the current chat
`disconnect()`	Stop active stream/poll connections
`reset()`	Disconnect and clear chat state so the next message starts a new chat

Messages that arrive through a channel integration (Slack, Telegram, a wearable) can carry channel_context so the reply is routed back the same way. The API records it on the chat the first time it sees it, so later turns inherit it:

await agent.sendMessage(text, {
  channel_context: { channel_type: 'slack', channel_metadata: { channel: 'C123', thread_ts: '1700000000.000100' } },
});

API Reference

`inference(config)`

Creates a new inference client.

Parameter	Type	Required	Description
`config.apiKey`	`string`	Yes	Your inference.sh API key
`config.baseUrl`	`string`	No	Custom API URL (default: `https://api.inference.sh`)
`config.stream`	`boolean`	No	Use NDJSON streaming (`true`, default) or status polling (`false`)
`config.pollIntervalMs`	`number`	No	Poll interval when `stream: false` (default: `2000`)
`config.proxyUrl`	`string`	No	Proxy base URL for frontend apps (keeps API keys server-side)

`client.tasks.run(params, options?)`

Runs a task on inference.sh.

Parameters:

Parameter	Type	Required	Description
`params.app`	`string`	Yes	App identifier (e.g., `'username/app-name'`)
`params.input`	`object`	Yes	Input parameters for the app
`params.setup`	`object`	No	Setup parameters (affects worker warmth/scheduling)
`params.infra`	`string`	No	Infrastructure: `'cloud'` or `'private'`
`params.variant`	`string`	No	App variant to use
`params.session`	`string`	No	Session ID or `'new'` to start a new session
`params.session_timeout`	`number`	No	Session timeout in seconds (1-3600, only with `session: 'new'`)

Options:

Option	Type	Default	Description
`wait`	`boolean`	`true`	Wait for task completion
`stream`	`boolean`	client default	Use NDJSON streaming or status polling
`pollIntervalMs`	`number`	client default	Poll interval when `stream: false`
`onUpdate`	`function`	-	Callback for task updates (full fetch on status change when polling)
`onPartialUpdate`	`function`	-	Callback for partial NDJSON stream updates `(task, fields)`
`maxReconnects`	`number`	`5`	Max poll retries when `stream: false`

`client.tasks.get(taskId)`

Gets a task by ID.

`client.tasks.cancel(taskId)`

Cancels a running task.

`client.files.upload(data, options?)`

Uploads a file to inference.sh.

Parameters:

Parameter	Type	Description
`data`	`string \| Blob`	Base64 string, data URI, or Blob
`options.filename`	`string`	Filename
`options.contentType`	`string`	MIME type
`options.public`	`boolean`	Make file publicly accessible

`client.agents.create(templateOrConfig)` / `client.agent(...)`

Creates an agent instance from a template or ad-hoc configuration. client.agent(...) is an alias for client.agents.create(...).

sendMessage options: onMessage, onChat, onToolCall, files, stream, pollIntervalMs. Client tools with status awaiting_input are dispatched once per invocation ID via onToolCall.

Template mode:

const agent = client.agents.create('namespace/name@version');

Ad-hoc mode:

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@abc123' },
  system_prompt: 'You are helpful.',
  tools: [...],
});

`client.sessions`

Method	HTTP	Description
`get(sessionId)`	`GET /sessions/{id}`	Session metadata (`status`, `expires_at`, `call_count`, …)
`list()`	`GET /sessions`	All sessions (empty array if none)
`keepalive(sessionId)`	`POST /sessions/{id}/keepalive`	Reset idle expiration
`end(sessionId)`	`DELETE /sessions/{id}`	End session and release worker

`client.sockets`

Method	HTTP	Description
`open(taskOrId, handlers?, options?)`	—	Dial a stream task's socket; returns a `LiveSession`
`get(socketId)`	`GET /sockets/{id}`	The socket and what is known of its life
`forTask(taskId)`	`POST /sockets/list`	The task's socket, or null
`access(socketId)`	`POST /sockets/{id}/access`	A fresh credential for the caller's end
`delete(socketId)`	`DELETE /sockets/{id}`	Delete the record

client.live(params, handlers?, options?) is tasks.run(params, { wait: false }) followed by sockets.open.

Task Status Constants

import {
  TaskStatusQueued,
  TaskStatusRunning,
  TaskStatusCompleted,
  TaskStatusFailed,
  TaskStatusCancelled
} from '@inferencesh/sdk';

if (task.status === TaskStatusCompleted) {
  console.log('Done!');
}

Credential Constants

CredentialDTO fields (provider, type, status) use typed string unions exported as constants:

import type { CredentialDTO } from '@inferencesh/sdk';
import {
  CredentialProviderGoogle,
  CredentialTypeOAuth,
  CredentialStatusConnected,
  CredentialStatusDisconnected,
  CredentialStatusExpired,
  CredentialStatusError,
  RequirementTypeCredential,
  isRequirementsNotMetException,
} from '@inferencesh/sdk';

function isGoogleConnected(credential: CredentialDTO): boolean {
  return (
    credential.provider === CredentialProviderGoogle &&
    credential.status === CredentialStatusConnected
  );
}

// HTTP 412 when an app requires a missing secret, credential, or scope
try {
  await client.run({ app: 'my-app', input: {} });
} catch (error) {
  if (isRequirementsNotMetException(error)) {
    for (const req of error.errors) {
      if (req.type === RequirementTypeCredential && req.action?.provider === CredentialProviderGoogle) {
        // User must connect Google — see https://inference.sh/docs/extend/credentials
      }
    }
  }
}

Constant group	Values
`CredentialProvider*`	`google`, `slack`, `notion`, `github`, `x`, `microsoft`, `salesforce`, `discord`, `gcp`, `mcp`, `reddit`
`CredentialType*`	`service_account`, `oauth`, `api_key`, `wif`, `mcp`
`CredentialStatus*`	`connected`, `disconnected`, `expired`, `error`

Instance Status Constants

Use when working with engine instance APIs (InstanceDTO.status):

import {
  InstanceStatusCreating,
  InstanceStatusPendingProvider,
  InstanceStatusPending,
  InstanceStatusActive,
  InstanceStatusError,
  InstanceStatusDeleting,
  InstanceStatusDeleted,
} from '@inferencesh/sdk';

Tool Parameter Types

When building AgentTool schemas manually (outside the tool builder), use ToolParamType* for JSON Schema type fields:

import {
  ToolParamTypeObject,
  ToolParamTypeString,
  ToolParamTypeInteger,
  ToolParamTypeNumber,
  ToolParamTypeBoolean,
  ToolParamTypeArray,
  ToolParamTypeNull,
} from '@inferencesh/sdk';

const schema = {
  type: ToolParamTypeObject,
  properties: {
    city: { type: ToolParamTypeString, description: 'City name' },
  },
  required: ['city'],
};

The fluent tool builder (string(), number(), object(), …) infers these types automatically.

TypeScript Support

This SDK is written in TypeScript and includes full type definitions. All types are exported:

import type {
  Task,
  ApiAppRunRequest,
  RunOptions,
  CredentialDTO,
  AgentTool,
} from '@inferencesh/sdk';

Requirements

Node.js 18.0.0 or higher
Modern browsers with fetch support

resources

documentation — getting started guides and api reference
blog — tutorials on ai agents, image generation, and more
app store — browse ai models
discord — community support
github — open source projects

Name		Name	Last commit message	Last commit date
.github		.github
examples		examples
scripts		scripts
src		src
.gitattributes		.gitattributes
.gitignore		.gitignore
.mise.toml		.mise.toml
.prettierrc		.prettierrc
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
eslint.config.js		eslint.config.js
jest.config.cjs		jest.config.cjs
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation

@inferencesh/sdk — ai inference api for javascript & typescript

Installation

Getting an API Key

Quick Start

Usage

Basic Usage

With Setup Parameters

Fire and Forget

Real-time Status Updates

Streaming vs Polling

Batch Processing

File Upload

Cancel a Task

Stream Functions (Live Sockets)

Sessions (Stateful Execution)

Custom Session Timeout

Session management API

Session errors

Agent Chat

Using a Template Agent

Creating an Ad-Hoc Agent

Tool builder

Lifecycle hooks

Queued messages

File attachments

Structured Output

Agent Methods

API Reference

inference(config)

client.tasks.run(params, options?)

client.tasks.get(taskId)

client.tasks.cancel(taskId)

client.files.upload(data, options?)

client.agents.create(templateOrConfig) / client.agent(...)

client.sessions

client.sockets

Task Status Constants

Credential Constants

Instance Status Constants

Tool Parameter Types

TypeScript Support

Requirements

resources

license

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages