Streaming responses

With "stream": true the reply arrives as server-sent events while the model writes it, instead of as one JSON body at the end. Use it for anything a person watches: the first words appear much sooner, and long answers do not run into idle timeouts.

Streaming costs the same as a normal request. You pay for the same tokens.

OpenAI format

Set stream: true on chat completions. Add stream_options: {"include_usage": true} to get the token counts in one extra chunk at the end.

curl -N https://api.oneaiapikey.com/v1/chat/completions \
  -H "Authorization: Bearer $RELAYKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-terra",
    "stream": true,
    "stream_options": {"include_usage": true},
    "messages": [{"role": "user", "content": "Write a haiku about latency."}]
  }'

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.oneaiapikey.com/v1", api_key=os.environ["RELAYKIT_API_KEY"])

stream = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[{"role": "user", "content": "Write a haiku about latency."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # last chunk, choices is empty
        print("\n", chunk.usage)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.oneaiapikey.com/v1", apiKey: process.env.RELAYKIT_API_KEY });

const stream = await client.chat.completions.create({
  model: "gpt-5.6-terra",
  messages: [{ role: "user", content: "Write a haiku about latency." }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content;
  if (text) process.stdout.write(text);
  if (chunk.usage) console.log("\n", chunk.usage); // last chunk, choices is empty
}

On the wire, each event is a data: line with one JSON chunk. The text is in choices[0].delta.content; the last content chunk carries finish_reason; the usage chunk has an empty choices array; data: [DONE] ends the stream.

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"gpt-5.6-terra","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"gpt-5.6-terra","choices":[{"index":0,"delta":{"content":"Packets"}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"gpt-5.6-terra","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","model":"gpt-5.6-terra","choices":[],"usage":{"prompt_tokens":14,"completion_tokens":19,"total_tokens":33}}

data: [DONE]

Reasoning models may stream their thinking first, as delta.reasoning_content, before the answer arrives in delta.content. See Reasoning models.

Anthropic format

Set "stream": true on /v1/messages. The events are Anthropic’s own: message_start, then content_block_start, content_block_delta (text in delta.text), content_block_stop for each block, message_delta with stop_reason and the output token count, and message_stop. The Anthropic SDKs assemble them for you.

curl -N https://api.oneaiapikey.com/v1/messages \
  -H "x-api-key: $RELAYKIT_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5-5",
    "max_tokens": 512,
    "stream": true,
    "messages": [{"role": "user", "content": "Write a haiku about latency."}]
  }'

import os
import anthropic

client = anthropic.Anthropic(base_url="https://api.oneaiapikey.com", api_key=os.environ["RELAYKIT_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-5-5",
    max_tokens=512,
    messages=[{"role": "user", "content": "Write a haiku about latency."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
print("\n", final.usage)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ baseURL: "https://api.oneaiapikey.com", apiKey: process.env.RELAYKIT_API_KEY });

const stream = client.messages.stream({
  model: "claude-sonnet-5-5",
  max_tokens: 512,
  messages: [{ role: "user", content: "Write a haiku about latency." }],
});
stream.on("text", (text) => process.stdout.write(text));
const final = await stream.finalMessage();
console.log("\n", final.usage);

Usage is always included in this format: input tokens in message_start, output tokens in message_delta.

Tool calls in a stream

When a model calls a tool while streaming, the arguments arrive in pieces. In the OpenAI format they are in delta.tool_calls[i].function.arguments, keyed by index; in the Anthropic format they are input_json_delta events. Join the pieces and parse the JSON only after the call is complete (finish_reason: "tool_calls" or content_block_stop). The SDK helpers above do this for you. See Tool calling.

Timeouts and disconnects

• Idle timeout. The gateway ends a stream if the provider sends nothing for 300 seconds. Very long thinking phases on reasoning models are the usual cause; a lower reasoning_effort or a smaller max_tokens helps.
• Client timeouts. Set your HTTP client’s read timeout above the longest gap you expect between chunks, not above the total response time. A stream can run for many minutes while chunks keep arriving.
• If you disconnect, the gateway closes the request to the provider. You are billed for the tokens produced up to that point, not for the full max_tokens.
• Proxies and buffering. If you relay the stream through your own server, turn off response buffering (for nginx: proxy_buffering off) or the text arrives in one lump at the end.

Errors during a stream

Errors that happen before the first byte (bad key, unknown model, low balance) come back as a normal JSON error with a 4xx or 5xx status; see Errors. Once the stream has started the status is already 200, so a later failure shows up as an error object inside a data: event, or the stream ends without finish_reason / message_stop. Treat a stream that ends without one of those as incomplete, and retry it if your app needs the full answer.

Related

• Chat completions and Anthropic Messages: the request fields.
• Rate limits and retries: when to retry, and how.
• Performance metrics: first-token time and tokens per second.