# Performance metrics explained

> What first-token time, total duration and tokens per second measure, where the gateway takes each one, what changes them, and how to read them in the console.

Source: https://oneaiapikey.com/docs/performance

Three numbers describe how fast a model answers: how long until the first token arrives, how long the whole request takes, and how quickly tokens come once they start. The gateway records the total duration of every request and the first-token time of every streamed one; the third number follows from them. This page explains each one and what moves it. It contains no benchmark figures: speed depends on the model, the prompt and the provider’s load at that moment, so measure your own traffic.

## The three numbers

| Metric | What it measures | Where it is taken |
| --- | --- | --- |
| **First-token time** | From the gateway receiving your request to the first chunk of the answer arriving from the provider. | At the gateway, in milliseconds. Streamed requests only. |
| **Total duration** | From the gateway receiving your request to the end of the answer. | At the gateway, in whole seconds. Every request. |
| **Tokens per second** | Output tokens divided by the time spent producing them, that is total duration minus first-token time. | Calculated from the two above and the output token count. |

First-token time is what a person in a chat feels as “waiting”. Tokens per second is how fast the text then fills in. Total duration is what a batch job or an agent loop pays for each call.

> **Streamed requests only:** A non-streamed request returns nothing until the answer is complete, so it has no separate first-token time. Its usage record shows the total duration only. Send `"stream": true` if you want both numbers.

## Read them in the console

- **Usage averages.** Above the chart, the [usage page](https://oneaiapikey.com/console/usage) shows the **Success rate**, **Avg first token** and **Output speed** for the selected range. First token and output speed average the newest streamed requests in the range (up to 100), and success rate compares billed requests with failed attempts.
- **Usage page.** Each request on the [usage page](https://oneaiapikey.com/console/usage) shows its duration, and for streamed requests its first-token time in seconds. Requests that finished in under a second show `<1 s`.
- **CSV export.** The export has a `first_token_ms` column (empty for non-streamed requests) and a `duration_s` column, so you can work out medians and percentiles in a spreadsheet.
- **Playground.** The [playground](https://oneaiapikey.com/console/playground) shows the first-token time measured in your browser. It includes the network between you and the gateway, so it is a little higher than the gateway’s figure for the same request.

## What makes them rise or fall

**First-token time** grows with:

- **Prompt length.** The model reads the whole prompt before it writes anything, so a long conversation history or a large document takes longer to start.
- **Reasoning.** [Reasoning models](https://oneaiapikey.com/docs/reasoning) think before they answer. Depending on the model, the first chunk is either thinking text or arrives only after the thinking is done. A lower `reasoning_effort` starts the answer sooner.
- **Provider load.** At busy times a provider may queue requests before running them.
- **Cache misses.** With [prompt caching](https://oneaiapikey.com/docs/prompt-caching), a repeated prefix can be read from the provider’s cache instead of being processed again, which can shorten the wait on long prompts.

**Total duration** grows mostly with the number of **output tokens**. A model that writes twice as much takes roughly twice as long once it has started. Capping `max_tokens` and asking for shorter answers is the most reliable way to cut it.

**Tokens per second** depends mainly on the **model**: smaller and faster tiers generate quicker than flagship models, and the same model can vary with provider load.

Tool-using agents add one more factor: each tool call is a separate request, so an agent turn that calls five tools is six round trips. Their durations add up.

## Read them carefully

- **Compare medians, not single requests.** One slow request says little. Export a day of usage and look at the median and the slowest 5 %.
- **Ignore tokens per second on short requests.** Duration is recorded in whole seconds, so for a request that takes one or two seconds the figure is coarse. It becomes meaningful once requests run for several seconds.
- **Compare like with like.** A 20,000-token prompt starts slower than a 200-token one on any model. Compare models on the same prompt.
- **Count your own network.** The gateway’s figures start when the request reaches it. Your users also wait for the trip from their device to your server and on to the gateway.

## Measure it yourself

This script streams one answer and prints first-token time, total time and tokens per second as your code sees them, including the network:

Python:

```python
import os, time
from openai import OpenAI

client = OpenAI(base_url="https://api.oneaiapikey.com/v1", api_key=os.environ["RELAYKIT_API_KEY"])

start = time.perf_counter()
first = None
usage = None
stream = client.chat.completions.create(
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Explain DNS in one paragraph."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if first is None and chunk.choices and chunk.choices[0].delta.content:
        first = time.perf_counter()  # first visible text
    if chunk.usage:
        usage = chunk.usage
end = time.perf_counter()
first = first or end  # no visible text (for example, only reasoning)

print(f"first token: {(first - start) * 1000:.0f} ms")
print(f"total:       {end - start:.2f} s")
if usage and end > first:
    print(f"tokens/s:    {usage.completion_tokens / (end - first):.1f}")
```

Node:

```js
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.oneaiapikey.com/v1", apiKey: process.env.RELAYKIT_API_KEY });

const start = performance.now();
let first = 0;
let usage;
const stream = await client.chat.completions.create({
  model: "claude-haiku-4-5",
  messages: [{ role: "user", content: "Explain DNS in one paragraph." }],
  stream: true,
  stream_options: { include_usage: true },
});
for await (const chunk of stream) {
  if (!first && chunk.choices[0]?.delta?.content) first = performance.now(); // first visible text
  if (chunk.usage) usage = chunk.usage;
}
const end = performance.now();
first ||= end; // no visible text (for example, only reasoning)

console.log(`first token: ${Math.round(first - start)} ms`);
console.log(`total:       ${((end - start) / 1000).toFixed(2)} s`);
if (usage && end > first) console.log(`tokens/s:    ${(usage.completion_tokens / ((end - first) / 1000)).toFixed(1)}`);
```

Run it a few times with your real prompt and model, and compare with the figures on the usage page.

## Related

- [Streaming](https://oneaiapikey.com/docs/streaming): how to stream in both formats and get usage in the last chunk.
- [Rate limits and retries](https://oneaiapikey.com/docs/rate-limits): timeouts for long requests.
- [Routing and privacy](https://oneaiapikey.com/docs/routing-privacy): what the usage record contains.