LLM APIs come with their own vocabulary, and pricing pages assume you know it. This glossary explains the terms you’ll meet most often when you build on an API like Relaykit: what each one means, why it matters for your bill or your code, and the mistake it most often causes.
Tokens and cost
Token. The unit a model reads and writes. A token is a piece of text: often a whole short word, sometimes part of a longer word, a space plus a word, or a punctuation mark. As a rough rule for English prose, one token is about four characters, or three quarters of a word. Code, numbers, and many languages other than English use more tokens for the same amount of text. Common mistake: estimating cost from word counts and being 30–50% off on code-heavy prompts.
Tokenizer. The algorithm that splits text into tokens. Each model family has its own, so the same prompt can be a different number of tokens on Claude, GPT or Grok. Only the usage numbers in the API response are exact. Estimates, including our token estimator, are approximations.
Input tokens (also prompt tokens). Everything you send: the system prompt, the whole conversation so far, tool definitions, documents and images. In a chat app, every new message resends the full history, so input tokens grow with each turn.
Output tokens (also completion tokens). Everything the model generates, including reasoning tokens (see below). Output is usually priced several times higher than input. For Claude Sonnet 5 it is $11 against $2.20 per 1M tokens.
Per-1M-token rate. How prices are quoted: dollars per million tokens, with separate rates for input, output and cached input. A request’s cost is input tokens × input rate + output tokens × output rate, each divided by one million. The rates for every model we serve are on the models page.
Worked example. A coding-assistant request to Claude Sonnet 5 with 12,000 input tokens and 1,500 output tokens costs $0.0264 for input plus $0.0165 for output, which is $0.0429. If 10,000 of those input tokens are a cache hit (see below), the same request costs $0.0231.
Usage object. The part of every API response that reports the tokens you were billed for. It’s called usage in both API formats, but the field names differ. Log it. It’s the only accurate source for cost per request, per user or per feature.
Context and limits
Context window. The maximum number of tokens a model can consider in one request: input and output together. If a model has a 200K context window and you send 190K tokens of input, at most 10K are left for the answer. Common mistake: treating the context window as memory. The model remembers nothing between requests. Anything it should know, you send again each time.
Max output tokens (max_tokens, or max_completion_tokens for some OpenAI reasoning models). A cap you set on how many tokens the model may generate in one response. It is required in the Anthropic format and optional in the OpenAI format. It is your per-request cost ceiling. When it is reached, the response stops mid-sentence, and the API tells you so (stop_reason: "max_tokens" or finish_reason: "length").
Reasoning tokens (also thinking tokens). Tokens a reasoning model generates while working through a problem before it answers. They are billed as output, even when the API hides their content from you, and they count toward max_tokens. Common mistake: setting a small max_tokens on a reasoning model and getting an empty answer because the budget was spent on thinking.
Effort. A setting on newer reasoning models (often low, medium, high) that controls how much they think before answering. More effort usually means better answers on hard problems, plus more output tokens and more waiting.
Rate limit. A cap on how many requests or tokens you can send per minute. When you go over it, the API returns HTTP 429. The right response is to wait and retry with increasing delays. The errors reference explains which status codes are worth retrying.
Caching
Prompt caching (also cached input). When many requests start with the same long prefix, such as a system prompt, a document or a set of tool definitions, the provider can keep that prefix processed for a short time. Later requests that start with the identical prefix read it from the cache. Cached input is billed at a much lower rate: $0.22 instead of $2.20 per 1M tokens for Claude Sonnet 5. Common mistake: putting something that changes on every request, like a timestamp or the user’s name, at the start of the prompt. That breaks the identical prefix, so nothing is ever read from the cache.
Cache write and cache read. In the Anthropic format you mark the cacheable part yourself. The first request writes the cache and later ones read it. The usage object reports both (cache_creation_input_tokens, cache_read_input_tokens). With OpenAI models, caching happens automatically for long prompts, and hits appear in prompt_tokens_details.cached_tokens.
Cache lifetime (TTL). How long a cached prefix lasts without being used, typically a few minutes. Caching saves the most under steady traffic and little for one request an hour.
Streaming
Streaming. Getting the response a few tokens at a time while it’s generated, instead of all at once at the end. Total time and cost stay the same, but users see text within a second or two, and long requests avoid idle timeouts. You turn it on with stream: true.
Server-Sent Events (SSE). The plain-HTTP format streaming responses use: a long-lived response of text/event-stream content, made of small text events separated by blank lines. The SDKs parse it for you. If you pass a stream on to a browser, our SSE tutorial shows how.
Time to first token (TTFT). How long it takes from sending a request until the first token arrives. In a chat interface, it’s the delay users notice.
Endpoints, formats and keys
Endpoint. One specific URL path that does one job, such as /v1/chat/completions (chat in the OpenAI format) or /v1/messages (chat in the Anthropic format).
Base URL. The part of the address that comes before the endpoint path. SDKs add the path to it. Changing the base URL is how you point an existing app at a different, compatible provider without changing your code. For Relaykit, the OpenAI SDKs use {SITE.apiBase}/v1 and the Anthropic SDKs use {SITE.apiBase} (they add /v1/messages themselves). Common mistake: a missing or doubled /v1, which shows up as a 404. See switching an OpenAI SDK app to a different base URL.
OpenAI-compatible / Anthropic-compatible. An API that accepts the same requests and returns the same response shapes as OpenAI’s or Anthropic’s own API, so their SDKs and the tools built on them work unchanged. “Compatible” covers the common features. Provider-specific extras may only work in a model’s native format. We compare the two in OpenAI vs Anthropic compatible APIs.
Gateway. A service that sits between your code and one or more model providers. It gives you one endpoint, key and bill for several model families. Relaykit is a gateway.
Model ID. The exact string you put in the model field, for example {sonnet.id}. IDs are case-sensitive and not shared between providers. The list of IDs your key can call is returned by GET /v1/models.
API key. A secret string that identifies your account and authorises spending. OpenAI-format clients send it as Authorization: Bearer .... Anthropic-format clients send it as x-api-key: .... Keep keys on servers, never in browser code or repositories. Our guides to API key security and one key per project go further.
Spend limit. A cap on how much one API key can spend in total. When the key reaches it, the API rejects that key’s requests (HTTP 403) until you raise the limit. Your other keys keep working.
Prepaid credits. Money added to your account before use. Each request’s cost is taken from the balance. When the balance reaches zero, requests are rejected rather than billed later, so the most you can spend is what you have loaded.
System prompt. Standing instructions that apply to the whole conversation, such as the role, tone or rules. In the OpenAI format it is a message with the system role. In the Anthropic format it is a separate top-level system field. It’s sent, and billed as input, on every request, which makes it a natural candidate for prompt caching.
Tool calling (also function calling). You describe functions the model may use. Instead of answering directly, the model can reply with a request to call one, with arguments. Your code runs the function and sends the result back.
Embeddings. Lists of numbers that represent the meaning of a piece of text, used for search and for finding similar documents. They come from dedicated embedding models through their own endpoint, /v1/embeddings, not from chat models.
If a term you need isn’t here, tell us and we’ll add it.