Skip to main content

Providers and streaming

Which providers work, on which integration path, and how streaming behaves on each.

Native provider SDKs​

With keeping your own SDK — configure() plus instrument() — the SDK rewrites outbound requests for these 15 provider origins. Your client stays stock: no baseURL, no custom headers.

ProviderOrigin reroutedGateway route
OpenAIapi.openai.com/openai
Anthropicapi.anthropic.com/anthropic
Groqapi.groq.com/groq
Cerebrasapi.cerebras.ai/cerebras
Google Geminigenerativelanguage.googleapis.com/gemini
Togetherapi.together.xyz/together
Fireworksapi.fireworks.ai/fireworks
DeepSeekapi.deepseek.com/deepseek
xAIapi.x.ai/xai
Mistralapi.mistral.ai/mistral
Cohereapi.cohere.com/cohere
Hugging Facerouter.huggingface.co/huggingface
OpenRouteropenrouter.ai/openrouter
TokenRouterapi.tokenrouter.com/tokenrouter
Ollamaollama.com/ollama

A request to any other host passes through untouched — the SDK doesn't intercept traffic it doesn't recognise, and doesn't attach your trace context to third-party hosts your application happens to call.

Credential remapping​

Provider SDKs put their API key in different headers. The SDK moves whichever one it finds into a dedicated provider-key header and adds your gateway key.

This matters most for Authorization: Bearer. The gateway reads that header as the gateway key, so leaving a provider key there would make the gateway reject the request as an invalid credential. The SDK handles the swap for you.

Examples​

import * as observra from "observra-sdk-node";
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import Groq from "groq-sdk";
import Cerebras from "@cerebras/cerebras_cloud_sdk";
import { Mistral } from "@mistralai/mistralai";

observra.configure({ serviceName: "my-app" });
await observra.instrument();

// Every one of these is stock construction, straight from the provider's own docs.
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });
const cerebras = new Cerebras({ apiKey: process.env.CEREBRAS_API_KEY });
const mistral = new Mistral({ apiKey: process.env.MISTRAL_API_KEY });

Cerebras uses an OpenAI-compatible API. Its stock SDK is routed from api.cerebras.ai to /cerebras, just like the other native SDKs above:

const res = await cerebras.chat.completions.create({
  model: "llama-3.3-70b",
  messages: [{ role: "user", content: "Hello" }],
});

SDK provider clients​

With the SDK's own clients, the SDK ships a client per provider. No instrument(), no fetch patching.

OpenAI-protocol providers​

Eighteen providers speak the OpenAI protocol at the gateway boundary, so they share one client shape:

const client = new observra.Groq({ apiKey: process.env.GROQ_API_KEY });

const res = await client.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "Hello" }],
});

Swap the class name for any of:

OpenAI · Groq · Cerebras · Azure · Ollama · OpenRouter · TokenRouter · Together · Fireworks · DeepSeek · XAI · Mistral · NIM · LMStudio · Cohere · HuggingFace

The gateway translates to each provider's real wire format, so the shape you write is identical across all of them.

:::caution Vertex and Bedrock are not usable yet Both are exported, but they need extra per-request headers that the SDK's transport cannot pass through yet — a Vertex endpoint, and AWS access key/region/session token. Treat these two exports as placeholders. To use Vertex or Bedrock today, call the gateway routes directly; see Routing and providers. :::

Anthropic​

Anthropic's native message shape:

const anthropic = new observra.Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const res = await anthropic.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 256,
  messages: [{ role: "user", content: "Hello" }],
});

Gemini​

Gemini's native generate-content shape:

const gemini = new observra.Gemini({ apiKey: process.env.GEMINI_API_KEY });

const res = await gemini.models.generateContent({
  model: "gemini-2.0-flash",
  contents: "Hello",
});

Response types​

These clients return the gateway's response body directly, typed as unknown — the SDK doesn't ship a copy of every provider's response schema. Cast to whatever shape you expect:

const res = (await client.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "Hello" }],
})) as { choices: Array<{ message: { content: string } }> };

console.log(res.choices[0].message.content);

If you want full provider types, use keeping your own SDK — your provider's own SDK keeps its own types, and the SDK never touches them.

Streaming​

With your own SDK​

Streaming works exactly as your provider SDK documents it. Nothing changes:

const stream = await groq.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "Count to five." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

With the SDK's clients​

Pass stream: true and the return type narrows to an async iterable:

const stream = await client.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "Count to five." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = (chunk as { choices?: Array<{ delta?: { content?: string } }> })
    .choices?.[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
}

Chunks are parsed from the gateway's SSE response, including across network read boundaries — a chunk split mid-frame is buffered and reassembled, not dropped.

Streaming and guardrails​

Output guardrails cannot be enforced on a streamed response. By the time the SDK could inspect the text, your code already has it. Streamed output is scanned only to annotate the trace, never to alter or withhold what you received.

Input guardrails work normally on streamed calls — the prompt is fully known before anything is sent.

If you need enforced output guardrails, don't stream. See Guardrails.

Next​