# Rate limits and retries

> Where 429 errors come from, how many requests you can run at once, and how to retry with backoff and cap concurrency so bursts and long jobs finish reliably.

Source: https://oneaiapikey.com/docs/rate-limits

## Where limits come from

- **The gateway** does not cap how many requests per minute your account or key sends to the API. What limits your usage is your balance and each key’s spend limit.
- **Model providers** throttle their own capacity. When a model is busy, the provider answers `429` or `503`, and the gateway passes that on to you unchanged. This is the usual source of a `429`.
- **Per-model limits** are off by default and can be switched on if a model needs protecting. A request over such a limit gets `429` with the message “Rate limit reached: at most N requests per M minutes”, which tells you the exact numbers.

The gateway does not retry a failed request on another route for you. What the provider answered is what you get, so your code decides whether to retry.

## Retry the right errors

| Status | Retry? | Why |
| --- | --- | --- |
| `429` | Yes, with backoff | Temporary: the model or provider is busy. Honour `Retry-After` if present. |
| `500`, `502`, `503` | Yes, with backoff | Usually short provider problems. |
| Network error, connection reset | Yes | Nothing reached the model, or the stream broke. |
| Timeout without a response | Carefully | The model may have finished and been billed. See [Errors](https://oneaiapikey.com/docs/errors#timeouts). |
| `400`, `401`, `403`, `404`, `413`, `501` | No | The same request fails the same way. Fix the request, key or balance. |

Back off exponentially (1 s, 2 s, 4 s, 8 s, with random jitter) and give up after about five tries. The [Errors](https://oneaiapikey.com/docs/errors#retries) page has a hand-written retry loop if you are not using an SDK.

## Cap concurrency

For batch jobs, the most reliable pattern is a fixed number of requests in flight plus the SDK’s built-in retries. Start at around 8 parallel requests and raise it while errors stay rare.

Python:

```python
import asyncio, os
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.oneaiapikey.com/v1",
    api_key=os.environ["RELAYKIT_API_KEY"],
    max_retries=5,          # the SDK retries 429, 5xx and network errors with backoff
    timeout=120,
)
limit = asyncio.Semaphore(8)   # at most 8 requests in flight

async def summarise(text: str) -> str:
    async with limit:
        reply = await client.chat.completions.create(
            model="claude-haiku-4-5",
            max_tokens=300,
            messages=[{"role": "user", "content": f"Summarise in two sentences:\n\n{text}"}],
        )
        return reply.choices[0].message.content

async def main(texts):
    return await asyncio.gather(*(summarise(t) for t in texts))
```

Node:

```js
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.oneaiapikey.com/v1",
  apiKey: process.env.RELAYKIT_API_KEY,
  maxRetries: 5,      // the SDK retries 429, 5xx and network errors with backoff
  timeout: 120_000,
});

// Run at most 8 requests at a time.
async function mapLimited(items, limit, fn) {
  const results = new Array(items.length);
  let next = 0;
  const worker = async () => {
    while (next < items.length) {
      const i = next++;
      results[i] = await fn(items[i]);
    }
  };
  await Promise.all(Array.from({ length: limit }, worker));
  return results;
}

const summaries = await mapLimited(texts, 8, async (text) => {
  const reply = await client.chat.completions.create({
    model: "claude-haiku-4-5",
    max_tokens: 300,
    messages: [{ role: "user", content: `Summarise in two sentences:\n\n${text}` }],
  });
  return reply.choices[0].message.content;
});
```

> **SDK retries are on by default:** The official OpenAI and Anthropic SDKs already retry `429`, `5xx` and connection errors twice with backoff. Raise `max_retries` / `maxRetries` for batch work rather than wrapping a second retry loop around theirs, which multiplies the attempts.

## Keep long requests alive

- Stream long outputs. The gateway ends a stream after 300 seconds without data, and a non-streamed answer arrives only at the end, so proxies and clients in between may give up first. See [Streaming](https://oneaiapikey.com/docs/streaming#timeouts-and-disconnects).
- Set your client’s timeout to a few minutes for reasoning models.
- A request your client abandons is still billed for the tokens the model produced before it stopped.

## When one model keeps failing

If a model returns `5xx` or `429` for several minutes, switch to a sibling model from the [models table](https://oneaiapikey.com/docs/models#current-models). Every model uses the same API, so it is a change of the `model` string. Keep the fallback list in your config so you can change it without a deploy.

## Related

- [Errors](https://oneaiapikey.com/docs/errors): every status code and message.
- [Cost control](https://oneaiapikey.com/docs/cost-control): per-key spend limits.
- [Routing and privacy](https://oneaiapikey.com/docs/routing-privacy#retries-and-failover): the gateway makes one attempt per request.