A chat interface feels fast when the first words show up within a second, even if the full answer takes twenty. To get that, your server has to pass the model’s stream through to the browser as it arrives. Server-Sent Events (SSE) is the simplest way to do it: plain HTTP, one direction, text only, and supported by every browser and proxy.
This tutorial builds the full path: browser → your server → Relaykit → model, and back again, one token at a time. There is a Node server, a Python (FastAPI) server and a small browser client. The examples use {haiku.id} through the OpenAI-compatible endpoint at {SITE.apiBase}/v1. Any model on the models page works the same way.
Why your server sits in the middle
You could call the API from the browser directly. Don’t. Any key that reaches the browser can be read by anyone who opens developer tools, and it will be used up within hours. Your server holds the key, checks who the user is and what they may spend, and passes only the text through. Our API key security guide covers this in more detail.
The setup has two hops, and both are streams:
- Upstream: your server calls the API with
stream: trueand receives SSE chunks in the OpenAI format. - Downstream: your server sends its own, simpler SSE events to the browser:
token,usage,doneanderror.
Don’t forward the raw upstream bytes. Re-emitting your own events lets you hide provider details, drop fields the browser does not need, and keep the same browser code if you later switch API formats.
What an SSE event looks like
An SSE response has the content type text/event-stream. Each event is a few lines of text followed by a blank line:
event: tokengives the event a name (optional).data: ...carries the payload. We always send one line of JSON, so newlines inside the model’s text can’t break the format.- A line starting with a colon is a comment. We send one every 15 seconds as a heartbeat.
That’s the whole protocol.
Step 1: the server key
Set the key in the server’s environment. Create one in the console and give it a spend limit, so a bug can only cost a known amount.
export LLM_API_KEY=sk-your-key-here # server only, never in browser codeStep 2a: a Node server
This uses only node:http and the official openai package. It works the same way inside Express or Fastify: write the same headers and call res.write for each event.
// server.mjs (Node 20+, npm install openai)
import http from "node:http";
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.oneaiapikey.com/v1",
apiKey: process.env.LLM_API_KEY,
});
function send(res, event, data) {
res.write("event: " + event + "\ndata: " + JSON.stringify(data) + "\n\n");
}
const server = http.createServer(async (req, res) => {
if (req.method !== "POST" || req.url !== "/api/chat") {
res.writeHead(404).end();
return;
}
let body = "";
for await (const chunk of req) body += chunk;
let message;
try {
message = JSON.parse(body).message;
} catch {}
if (typeof message !== "string" || message.length === 0 || message.length > 8000) {
res.writeHead(400).end("message must be 1-8000 characters");
return;
}
res.writeHead(200, {
"Content-Type": "text/event-stream; charset=utf-8",
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no", // tells nginx not to buffer this response
});
// If the browser goes away, cancel the upstream request so you stop paying for tokens.
const upstream = new AbortController();
res.on("close", () => upstream.abort());
// A comment line every 15 s keeps idle proxies from closing the connection.
const heartbeat = setInterval(() => res.write(": ping\n\n"), 15000);
try {
const stream = await client.chat.completions.create(
{
model: "claude-haiku-4-5",
max_tokens: 1024,
stream: true,
stream_options: { include_usage: true },
messages: [{ role: "user", content: message }],
},
{ signal: upstream.signal },
);
for await (const chunk of stream) {
const text = chunk.choices[0]?.delta?.content;
if (text) send(res, "token", { text });
if (chunk.usage) send(res, "usage", chunk.usage);
}
send(res, "done", {});
} catch (err) {
if (!upstream.signal.aborted) {
console.error(err);
send(res, "error", { message: "The model request failed. Please try again." });
}
} finally {
clearInterval(heartbeat);
res.end();
}
});
server.listen(8787, () => console.log("http://localhost:8787"));Three details matter more than they seem:
res.on("close")aborts the upstream request. If a user closes the tab halfway through a long answer, the model would otherwise keep generating, and you would keep paying for tokens nobody reads. This listener stops it.max_tokensis set. Each request has a known worst-case cost.- Errors become an event, not a crash. Once the 200 status and headers are sent, you can’t change the status code. The only way to report a failure is in the stream itself. The browser gets a short, generic message. The full error stays in your server logs.
Step 2b: a Python server with FastAPI
The same design with AsyncOpenAI and StreamingResponse:
# app.py (pip install fastapi uvicorn openai) run: uvicorn app:app --port 8787
import json
import os
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
from pydantic import BaseModel, Field
client = AsyncOpenAI(base_url="https://api.oneaiapikey.com/v1", api_key=os.environ["LLM_API_KEY"])
app = FastAPI()
class ChatIn(BaseModel):
message: str = Field(min_length=1, max_length=8000)
def sse(event: str, data) -> str:
return f"event: {event}\ndata: {json.dumps(data)}\n\n"
@app.post("/api/chat")
async def chat(body: ChatIn, request: Request):
async def events():
stream = None
try:
stream = await client.chat.completions.create(
model="claude-haiku-4-5",
max_tokens=1024,
stream=True,
stream_options={"include_usage": True},
messages=[{"role": "user", "content": body.message}],
)
async for chunk in stream:
if await request.is_disconnected():
break # tab closed: stop reading and close upstream below
if chunk.choices and chunk.choices[0].delta.content:
yield sse("token", {"text": chunk.choices[0].delta.content})
if chunk.usage:
yield sse("usage", chunk.usage.model_dump())
yield sse("done", {})
except Exception:
yield sse("error", {"message": "The model request failed. Please try again."})
finally:
if stream is not None:
await stream.close()
return StreamingResponse(
events(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache, no-transform", "X-Accel-Buffering": "no"},
)request.is_disconnected() is how FastAPI notices a closed tab. When the loop exits, stream.close() closes the upstream connection, and generation stops. Use the async client here. A synchronous client inside an async route blocks the event loop, and one slow stream would delay every other user.
Step 3: the browser client
The browser has a built-in EventSource class for SSE, but it can only make GET requests and can’t send a JSON body or custom headers. You could put the prompt in the URL, but URLs end up in server logs, proxy logs and browser history. So we use fetch with POST and read the response body as a stream. It is about forty lines:
// chat.js — runs in the browser. No API key here.
const output = document.querySelector("#output");
let current = null; // AbortController for the request in flight
export async function ask(message) {
current?.abort();
current = new AbortController();
output.textContent = "";
const res = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ message }),
signal: current.signal,
});
if (!res.ok || !res.body) throw new Error("HTTP " + res.status);
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value.replace(/\r\n/g, "\n");
let end;
while ((end = buffer.indexOf("\n\n")) !== -1) {
handleEvent(buffer.slice(0, end));
buffer = buffer.slice(end + 2);
}
}
} catch (err) {
if (err.name !== "AbortError") throw err;
}
}
export function stop() {
current?.abort(); // the server sees the closed connection and cancels upstream
}
function handleEvent(block) {
let event = "message";
const data = [];
for (const line of block.split("\n")) {
if (line.startsWith(":")) continue; // heartbeat comment
if (line.startsWith("event:")) event = line.slice(6).trim();
else if (line.startsWith("data:")) data.push(line.slice(5).trimStart());
}
if (data.length === 0) return;
const payload = JSON.parse(data.join("\n"));
if (event === "token") output.append(payload.text); // a text node, never innerHTML
else if (event === "error") output.append("\n[" + payload.message + "]");
else if (event === "usage") console.debug("tokens", payload);
}Some notes on the client:
- Buffering matters. A network chunk can end in the middle of an event, or hold three events at once. The loop keeps the remainder in
bufferand only parses complete events, those ending in a blank line. output.append(text)adds a text node. Model output is untrusted input. Never put it intoinnerHTML. If you render Markdown, run it through a sanitiser after each update.stop()closes the connection. The server notices and cancels the upstream request, so a Stop button also saves money.
EventSource is still a good fit when the stream can be fetched with GET. For example, POST the prompt to create a job, then open EventSource("/api/jobs/123/stream"). It also reconnects on its own, which is useful for long-running jobs.
Step 4: make sure nothing buffers the stream
If tokens show up all at once at the end, something between the server and the browser is buffering. The usual causes are:
- Reverse proxies. nginx buffers by default. The
X-Accel-Buffering: noheader turns that off per response, or you can set it in the config. Caddy needsflush_interval -1. - Compression middleware. Gzip middleware often waits for a full block before writing. Exclude
text/event-streamfrom compression. - Serverless platforms and CDNs. Some only support streaming on certain runtimes. Check your platform’s docs before you assume streaming works there.
# nginx: location block that proxies your app
location /api/chat {
proxy_pass http://127.0.0.1:8787;
proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 300s;
}
# Caddy: flush every write immediately
example.com {
reverse_proxy /api/chat 127.0.0.1:8787 {
flush_interval -1
}
}Step 5: test it
Start either server and run curl -N -X POST localhost:8787/api/chat -H "Content-Type: application/json" -d '...' with a message in the body. The -N flag turns off curl’s own buffering, so you should see event: token lines arrive one by one. Then try the cases that go wrong in production. Press Ctrl+C in the middle of a stream and check that your usage log shows a short completion, not a full one. Send an empty message and check for a 400. Disconnect your network and check that the page shows a clear error.
Checklist
- The API key lives only in the server environment.
max_tokensis set on every request, and the key has a spend limit.- Closing the tab or pressing Stop cancels the upstream request.
- The server sends
token,usage,doneanderrorevents, with JSON on a singledata:line. - The client parses complete events only and never uses
innerHTMLfor model text. - Proxies and compression are set not to buffer
text/event-stream.
For background on why streaming helps with timeouts, and how the raw upstream SSE looks in both the OpenAI and Anthropic formats, see streaming vs non-streaming responses.