Skip to main content

Routing & Providers

Provider resolution​

The provider is determined only from the gateway URL's first path segment - there is no auto-detection based on request shape or headers. A request to /openai/... is routed to the OpenAI adapter, /anthropic/... to the Anthropic adapter, and so on. Each provider has one dedicated adapter responsible for building the upstream URL, translating requests/responses if needed, and building upstream headers.

The model itself is determined entirely by the client - from the request body's model field, or from the URL path for Gemini (/models/{model}:generateContent). The gateway never selects a model or routes based on it.

Transparent proxy vs. cross-protocol translation​

When the client's protocol matches the upstream provider's native protocol, the gateway is a transparent proxy: the raw request buffer is forwarded byte-for-byte, with no JSON re-serialization, and the raw response bytes are streamed back unchanged. Metrics (tokens, model) are extracted from the same bytes without altering what the client receives.

When the client sends one protocol shape and the target provider expects another (e.g. an OpenAI-format client calling a Gemini or Anthropic upstream), the gateway detects the incoming protocol and translates:

  • Incoming protocol detection - the gateway inspects the request body shape to determine which client protocol it's speaking (with a special case for Anthropic-shaped requests).
  • Request translation - the client's body is converted into the shape the upstream protocol expects.
  • Response translation - the upstream's native response (buffered or streamed chunk-by-chunk) is converted back into the client's expected format.

If the detected protocol isn't supported by the target provider's route, the gateway returns 400 naming the supported protocols.

Streaming vs. buffered responses​

The gateway inspects the upstream Content-Type to decide whether to stream: text/event-stream (OpenAI/Anthropic SSE), application/x-ndjson / application/jsonl (Ollama), or a JSON body where the client explicitly requested stream: true. Streaming responses are relayed line-by-line as they arrive; buffered responses are read in full, then optionally translated and sent.

An upstream error response is always routed through the buffered-response error path even if its content-type looks like a streaming type - this avoids silently dropping a non-SSE-formatted error body (e.g. Gemini can return a 400 with a streaming content-type but a plain JSON error payload).

Metadata fetches​

Some provider calls are treated as internal metadata fetches, exempt from protocol validation and never turned into an observation: Ollama's POST .../api/show, and Gemini's GET .../models/... calls.

Path handling​

For OpenAI-protocol requests, common version prefixes are stripped before being re-added by the adapter, so the same client code works whether the upstream needs /v1/chat/completions, /openai/v1/chat/completions (Groq-style), or /api/v1/chat/completions (OpenRouter-style).