Vision and files

Vision models read images (and some read PDFs) as part of the prompt. You send them as content parts next to the text, either as a public URL or inline as base64.

Models that accept images

Models not in this list are text only: an image part sent to them fails or is ignored, depending on the provider.

Send an image

In the OpenAI format, content becomes a list of parts. An image is an image_url part whose url is either an https:// link or a data: URL with base64 data.

curl https://api.oneaiapikey.com/v1/chat/completions \
  -H "Authorization: Bearer $RELAYKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-5-5",
    "max_tokens": 500,
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this picture? One sentence."},
        {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/3/3f/Fronalpstock_big.jpg"}}
      ]
    }]
  }'

import base64, os
from openai import OpenAI

client = OpenAI(base_url="https://api.oneaiapikey.com/v1", api_key=os.environ["RELAYKIT_API_KEY"])

with open("receipt.png", "rb") as f:
    data_url = "data:image/png;base64," + base64.b64encode(f.read()).decode()

reply = client.chat.completions.create(
    model="claude-opus-5-5",
    max_tokens=500,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "List the items and the total on this receipt."},
            {"type": "image_url", "image_url": {"url": data_url}},
        ],
    }],
)
print(reply.choices[0].message.content)

import { readFile } from "node:fs/promises";
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://api.oneaiapikey.com/v1", apiKey: process.env.RELAYKIT_API_KEY });
const dataUrl = "data:image/png;base64," + (await readFile("receipt.png")).toString("base64");

const reply = await client.chat.completions.create({
  model: "claude-opus-5-5",
  max_tokens: 500,
  messages: [{
    role: "user",
    content: [
      { type: "text", text: "List the items and the total on this receipt." },
      { type: "image_url", image_url: { url: dataUrl } },
    ],
  }],
});
console.log(reply.choices[0].message.content);

• By URL: the image must be publicly reachable without cookies or a login. For Claude models the gateway downloads it and sends it inline; for other models the provider fetches it.
• As base64: works for private files and is the more reliable choice. Use the right MIME type (image/png, image/jpeg, image/webp, image/gif).
• detail (low, high, auto) is passed to models that support it.
• Several images can go in one message. Put the question after the images for best results.

PDFs

Claude models read PDFs, page text and images together. In the OpenAI format, send a file part with a base64 data:application/pdf URL; the gateway passes it to Claude as a document block.

with open("contract.pdf", "rb") as f:
    pdf_data = "data:application/pdf;base64," + base64.b64encode(f.read()).decode()

reply = client.chat.completions.create(
    model="claude-sonnet-5-5",
    max_tokens=1000,
    messages=[{
        "role": "user",
        "content": [
            {"type": "file", "file": {"filename": "contract.pdf", "file_data": pdf_data}},
            {"type": "text", "text": "Summarise the termination clauses."},
        ],
    }],
)

file_id references do not work: the gateway does not offer a Files API. Always send the file data. Other providers’ PDF support varies; for them, extract the text yourself and send it as text.

Anthropic format

On /v1/messages, use Anthropic’s image and document blocks with a base64 or url source.

message = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=500,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": png_base64}},
            {"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": pdf_base64}},
            {"type": "text", "text": "Does the chart in the image match the numbers in the PDF?"},
        ],
    }],
)

Cost and limits

• Images and PDF pages are billed as input tokens at the model’s normal input rate. How many tokens an image uses depends on its size and on the model; the provider counts them and the count is in usage.prompt_tokens.
• Shrink large images before sending. A phone photo at full resolution costs more and rarely reads better than one scaled to about 1,500 px on the long side.
• A request body can be up to 128 MB at the gateway. Providers apply their own, smaller limits on image size, image count and PDF pages, and reject requests that go over them.
• Images sent again in later turns are billed again. Drop them from the history once the model has answered about them.

Related

• Models: context windows and prices.
• Cost control: keeping input tokens down.