Rate limits and retries

Where limits come from

• The gateway does not cap how many requests per minute your account or key sends to the API. What limits your usage is your balance and each key’s spend limit.
• Model providers throttle their own capacity. When a model is busy, the provider answers 429 or 503, and the gateway passes that on to you unchanged. This is the usual source of a 429.
• Per-model limits are off by default and can be switched on if a model needs protecting. A request over such a limit gets 429 with the message “Rate limit reached: at most N requests per M minutes”, which tells you the exact numbers.

The gateway does not retry a failed request on another route for you. What the provider answered is what you get, so your code decides whether to retry.

Retry the right errors

Back off exponentially (1 s, 2 s, 4 s, 8 s, with random jitter) and give up after about five tries. The Errors page has a hand-written retry loop if you are not using an SDK.

Cap concurrency

For batch jobs, the most reliable pattern is a fixed number of requests in flight plus the SDK’s built-in retries. Start at around 8 parallel requests and raise it while errors stay rare.

import asyncio, os
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.oneaiapikey.com/v1",
    api_key=os.environ["RELAYKIT_API_KEY"],
    max_retries=5,          # the SDK retries 429, 5xx and network errors with backoff
    timeout=120,
)
limit = asyncio.Semaphore(8)   # at most 8 requests in flight

async def summarise(text: str) -> str:
    async with limit:
        reply = await client.chat.completions.create(
            model="claude-haiku-4-5",
            max_tokens=300,
            messages=[{"role": "user", "content": f"Summarise in two sentences:\n\n{text}"}],
        )
        return reply.choices[0].message.content

async def main(texts):
    return await asyncio.gather(*(summarise(t) for t in texts))

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.oneaiapikey.com/v1",
  apiKey: process.env.RELAYKIT_API_KEY,
  maxRetries: 5,      // the SDK retries 429, 5xx and network errors with backoff
  timeout: 120_000,
});

// Run at most 8 requests at a time.
async function mapLimited(items, limit, fn) {
  const results = new Array(items.length);
  let next = 0;
  const worker = async () => {
    while (next < items.length) {
      const i = next++;
      results[i] = await fn(items[i]);
    }
  };
  await Promise.all(Array.from({ length: limit }, worker));
  return results;
}

const summaries = await mapLimited(texts, 8, async (text) => {
  const reply = await client.chat.completions.create({
    model: "claude-haiku-4-5",
    max_tokens: 300,
    messages: [{ role: "user", content: `Summarise in two sentences:\n\n${text}` }],
  });
  return reply.choices[0].message.content;
});

Keep long requests alive

• Stream long outputs. The gateway ends a stream after 300 seconds without data, and a non-streamed answer arrives only at the end, so proxies and clients in between may give up first. See Streaming.
• Set your client’s timeout to a few minutes for reasoning models.
• A request your client abandons is still billed for the tokens the model produced before it stopped.

When one model keeps failing

If a model returns 5xx or 429 for several minutes, switch to a sibling model from the models table. Every model uses the same API, so it is a change of the model string. Keep the fallback list in your config so you can change it without a deploy.

Related

• Errors: every status code and message.
• Cost control: per-key spend limits.
• Routing and privacy: the gateway makes one attempt per request.