Streaming vs Non-Streaming LLM Responses: Timeouts, UX and SSE in Python and Node

In short:

• A non-streaming request sends nothing until the whole answer is ready. For long outputs that can mean minutes of silence on the connection, and many proxies close idle connections after around 100 seconds.
• Streaming (stream: true) sends tokens as they are generated over Server-Sent Events (SSE). The connection stays active, users see output immediately, and long generations complete reliably.
• On Relaykit, use streaming for anything that may produce long output: code generation, documents, agent steps. Non-streaming is fine for short, bounded answers.
• Use the official SDKs’ streaming helpers. If you must parse SSE yourself, buffer partial lines and handle the end-of-stream signal for your format.

What actually happens in a non-streaming request

When you call a chat endpoint without streaming, the model generates the entire response on the server, and only then is the full JSON body returned. Generation speed varies by model and load, but output is produced token by token, so a long answer simply takes longer. A few hundred tokens usually arrive quickly. A long file, a detailed report or a large refactor can keep the connection silent for minutes.

That silence is the problem. Between your code and the model there are usually several network hops: your own reverse proxy, a CDN or load balancer, the gateway, and the upstream provider. Many of these close a connection that has received no response for a while. Cloudflare is a well-known example: when an origin does not respond within about 100 seconds, it returns HTTP 524. Requests through our gateway are subject to the same kind of limit, so a long non-streaming request can fail after roughly 100 seconds even though the model was still working. Worse, you have waited all that time and have nothing to show for it.

Client libraries also have their own timeouts. The official OpenAI and Anthropic SDKs default to generous timeouts (10 minutes), and the Anthropic SDKs go further: for non-streaming requests with very large max_tokens, they raise an error and point you to streaming instead.

Why streaming fixes it

With stream: true, the server responds immediately with Content-Type: text/event-stream and sends a small event for every few tokens. The connection is never idle long enough to trip a timeout, so a ten-minute generation completes just like a ten-second one.

The user-experience benefit is just as important. What users notice is time to first token, not total time. A response that starts appearing within a second or two and then flows feels fast, even when it takes the same total time as a non-streamed response that appears all at once after a long wait. Streaming also lets users stop a bad answer early, which saves output tokens.

When non-streaming is fine

Streaming is not always worth the extra code. Non-streaming is a good fit for:

• Short, bounded outputs: classification, extraction, short answers, with a modest max_tokens.
• Background jobs where no one is watching and responses are short.
• Structured JSON you can only use once it is complete, if outputs are small. For large JSON, you can still stream and simply accumulate the text before parsing.

A practical rule: if the response can be long enough to take more than a minute, stream it.

How SSE works in 30 seconds

Server-Sent Events is a plain-text format over a normal HTTP response. Each event is a few lines like event: name and data: payload, and a blank line ends the event:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

data: {"choices":[{"index":0,"delta":{"content":" world"}}]}

data: [DONE]

The first event is Anthropic format (named events, ending with message_stop). The last two are OpenAI format (unnamed data: events, ending with the literal [DONE]). Our post on OpenAI vs Anthropic API formats lists every event type.

Streaming in Python

The official SDKs handle the parsing. With the OpenAI SDK, iterate over the stream and read delta.content:

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.oneaiapikey.com/v1", api_key=os.environ["LLM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=4000,
    messages=[{"role": "user", "content": "Write a README for a small CLI todo app."}],
    stream=True,
    stream_options={"include_usage": True},   # usage arrives in the last chunk
)

usage = None
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        usage = chunk.usage
print("\n", usage)

With the Anthropic SDK, the messages.stream() helper yields plain text and gives you the assembled final message, including usage and stop reason, at the end:

import os
import anthropic

client = anthropic.Anthropic(base_url="https://api.oneaiapikey.com", api_key=os.environ["LLM_API_KEY"])

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=4000,
    messages=[{"role": "user", "content": "Write a README for a small CLI todo app."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()

print("\n", final.stop_reason, final.usage)

Always check the stop reason at the end. length (OpenAI) or max_tokens (Anthropic) means the output was cut by your token cap, not finished.

Streaming in Node.js

The Node SDKs mirror the Python ones. Streams are async iterables:

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.oneaiapikey.com/v1", apiKey: process.env.LLM_API_KEY });

const stream = await client.chat.completions.create({
  model: "claude-sonnet-5",
  max_tokens: 4000,
  messages: [{ role: "user", content: "Write a README for a small CLI todo app." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const text = chunk.choices[0]?.delta?.content;
  if (text) process.stdout.write(text);
  if (chunk.usage) console.log("\n", chunk.usage);
}

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ baseURL: "https://api.oneaiapikey.com", apiKey: process.env.LLM_API_KEY });

const stream = client.messages.stream({
  model: "claude-sonnet-5",
  max_tokens: 4000,
  messages: [{ role: "user", content: "Write a README for a small CLI todo app." }],
});

stream.on("text", (text) => process.stdout.write(text));
const final = await stream.finalMessage();
console.log("\n", final.stop_reason, final.usage);

Parsing SSE without an SDK

Sometimes you cannot use an SDK: an edge runtime, a minimal container, or a language without one. The one rule to remember is that network chunks do not line up with events. A chunk can end halfway through a line, so you must buffer until you see the blank line that ends an event. The examples below read these environment variables:

export LLM_BASE_URL=https://api.oneaiapikey.com
export LLM_API_KEY=sk-your-key-here
export LLM_MODEL=claude-sonnet-5

// Node 18+: raw fetch, OpenAI format
async function streamChat(prompt, onText) {
  const res = await fetch(process.env.LLM_BASE_URL + "/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.LLM_API_KEY,
    },
    body: JSON.stringify({
      model: process.env.LLM_MODEL,
      max_tokens: 4000,
      stream: true,
      messages: [{ role: "user", content: prompt }],
    }),
  });
  if (!res.ok) throw new Error("HTTP " + res.status + ": " + (await res.text()));

  const decoder = new TextDecoder();
  let buffer = "";
  for await (const bytes of res.body) {
    buffer += decoder.decode(bytes, { stream: true }).replace(/\r\n/g, "\n");
    let end;
    while ((end = buffer.indexOf("\n\n")) !== -1) {
      const event = buffer.slice(0, end);
      buffer = buffer.slice(end + 2);
      for (const line of event.split("\n")) {
        if (!line.startsWith("data:")) continue;
        const data = line.slice(5).trim();
        if (data === "[DONE]") return;
        const json = JSON.parse(data);
        const text = json.choices?.[0]?.delta?.content;
        if (text) onText(text);
      }
    }
  }
}

await streamChat("Explain SSE in two sentences.", (t) => process.stdout.write(t));

For the Anthropic format the loop is the same, but you read the event: line too: append delta.text from content_block_delta events, stop at message_stop, and throw on an error event.

Streaming to a browser, safely

Never call the API directly from browser JavaScript: anyone can open developer tools and copy your key. Instead, your server calls the API and relays the stream to the browser. A minimal Node relay, with no framework:

import http from "node:http";

http.createServer(async (req, res) => {
  if (req.method !== "POST" || req.url !== "/api/chat") {
    res.writeHead(404).end();
    return;
  }
  let body = "";
  for await (const part of req) body += part;
  const { prompt } = JSON.parse(body);

  // Stop the upstream request if the user closes the tab.
  const abort = new AbortController();
  res.on("close", () => abort.abort());

  const upstream = await fetch(process.env.LLM_BASE_URL + "/v1/chat/completions", {
    method: "POST",
    signal: abort.signal,
    headers: { "Content-Type": "application/json", Authorization: "Bearer " + process.env.LLM_API_KEY },
    body: JSON.stringify({
      model: process.env.LLM_MODEL,
      max_tokens: 2000,
      stream: true,
      messages: [{ role: "user", content: String(prompt).slice(0, 8000) }],
    }),
  });

  res.writeHead(upstream.ok ? 200 : 502, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no", // tell nginx not to buffer the stream
  });
  try {
    for await (const chunk of upstream.body) res.write(chunk);
  } catch {
    // client disconnected or upstream failed mid-stream
  }
  res.end();
}).listen(3000);

A few details make this production-worthy:

• Disable buffering in every proxy in front of it. If nginx or another proxy buffers the response, the browser receives everything at once and you lose the benefit. The X-Accel-Buffering: no header handles nginx; other proxies have equivalent settings.
• Forward aborts. When the user closes the tab, abort the upstream request so the model can stop generating output nobody will read.
• Validate and limit input (the slice above is a crude example) and add your own authentication and rate limits. Our API key security guide covers this.

In the browser, EventSource only supports GET requests, so use fetch and read response.body with getReader(), reusing the same buffering logic as the raw parser above.

Error handling while streaming

• Errors before the first byte arrive as a normal HTTP error status with a JSON body. Retrying with backoff is safe here.
• Errors mid-stream can arrive as an error event (Anthropic format) or as an abruptly closed connection. Do not automatically retry a half-finished generation without telling the user. You would pay for the first attempt’s output twice.
• Check the final stop reason. A stream that ends with a length or max_tokens stop is complete at the protocol level but truncated in content.

Checklist

1. Use stream: true for anything that may run longer than a minute.
2. Prefer SDK stream helpers; buffer partial lines if you parse SSE yourself.
3. Request usage in the stream (stream_options.include_usage in OpenAI format) if you track costs.
4. Relay streams through your server; keep keys off the client.
5. Turn off proxy buffering and forward client aborts.

Ready to try it? The quickstart has copy-paste examples, and the models page lists every model you can stream from.