Fallback

Automatically retry failed requests with backup models. When your primary provider returns a 5xx error, rate limit (429), or times out — the SDK silently tries the next model in your chain. Your application code doesn't change.

import { createRuntime } from '@yourgpt/llm-sdk';
import { createFallbackChain } from '@yourgpt/llm-sdk/fallback';
import { createOpenAI } from '@yourgpt/llm-sdk/openai';
import { createAnthropic } from '@yourgpt/llm-sdk/anthropic';

const chain = createFallbackChain({
  models: [
    createOpenAI({ apiKey: '...' }).languageModel('gpt-5.4'),
    createAnthropic({ apiKey: '...' }).languageModel('claude-haiku-4-5'),
  ],
});

const runtime = createRuntime({ adapter: chain });

Installation

npm install @yourgpt/llm-sdk openai @anthropic-ai/sdk

Basic Usage

Pass createFallbackChain() as the adapter in createRuntime(). The rest of your server code — streaming, tools, sessions — stays exactly the same.

server.ts

import { createRuntime } from '@yourgpt/llm-sdk';
import { createFallbackChain } from '@yourgpt/llm-sdk/fallback';
import { createOpenAI } from '@yourgpt/llm-sdk/openai';
import { createAnthropic } from '@yourgpt/llm-sdk/anthropic';

const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const runtime = createRuntime({
  adapter: createFallbackChain({
    models: [
      openai.languageModel('gpt-5.4'),            // tried first
      anthropic.languageModel('claude-haiku-4-5'), // tried if OpenAI fails
    ],
  }),
  systemPrompt: 'You are a helpful assistant.',
});

// Use exactly as normal — no other changes
app.post('/api/chat', async (req, res) => {
  await runtime.stream(req.body).pipeToResponse(res);
});

What Triggers Fallback

Error type	Triggers fallback?
`5xx` server errors	✅ Yes
`429` rate limit	✅ Yes
Network timeout / connection refused	✅ Yes
`4xx` client errors (bad key, bad request)	No by default; a narrow `fallbackErrors` policy can advance on a known compatibility error

Once content has started streaming to the client, fallback is not attempted. You cannot restart a stream mid-flight.

Routing Strategies

Control which model is tried first on each request.

Always tries models in the order defined. First model handles all traffic until it fails.

createFallbackChain({
  models: [primaryModel, backupModel],
  strategy: 'priority', // default — can be omitted
});

Distributes load evenly. Request 1 starts at model A, request 2 starts at model B, and so on. If the starting model fails, the chain falls through to the next one as usual.

createFallbackChain({
  models: [openaiModel, anthropicModel],
  strategy: 'round-robin',
});

Multi-instance deployments: Round-robin state is in-memory by default and resets on restart. For shared state across instances, plug in a custom store (Redis, Upstash, etc.) via the store option.

Per-Model Retries

Retry the same model before moving to the next one. Useful for transient errors like brief rate limits or flaky connections.

createFallbackChain({
  models: [openaiModel, anthropicModel],
  retries: 2,                  // retry each model up to 2 times
  retryDelay: 500,             // base delay: 500ms
  retryBackoff: 'exponential', // 500ms → 1000ms → 2000ms (default)
  onRetry: ({ model, retryAttempt, maxRetries, delayMs, error }) => {
    console.warn(`[retry] ${model} attempt ${retryAttempt}/${maxRetries} — waiting ${delayMs}ms`);
  },
});

Backoff options:

`retryBackoff`	Pattern (retryDelay=500)
`exponential` (default)	500ms → 1000ms → 2000ms
`fixed`	500ms → 500ms → 500ms

With retries: 2, each model gets 3 total attempts (1 initial + 2 retries) before the chain moves to the next model.

Observability Callbacks

Two callbacks give you visibility into what the chain is doing.

createFallbackChain({
  models: [openaiModel, anthropicModel, googleModel],

  // Fires on each per-model retry (before the wait delay)
  onRetry: ({ model, provider, error, retryAttempt, maxRetries, delayMs }) => {
    console.warn(`[retry] ${provider}/${model} — attempt ${retryAttempt}/${maxRetries}`);
    metrics.increment('llm.retry', { provider, model });
  },

  // Fires when a model is abandoned and the next one is tried
  onFallback: ({ attemptedModel, nextModel, error, attempt }) => {
    console.warn(`[fallback] ${attemptedModel} → ${nextModel}: ${error.message}`);
    metrics.increment('llm.fallback', { from: attemptedModel, to: nextModel });
  },
});

Handling Full Failure

When every model in the chain fails, FallbackExhaustedError is thrown. It includes a per-model breakdown of what failed.

import { FallbackExhaustedError } from '@yourgpt/llm-sdk/fallback';

try {
  await runtime.stream(req.body).pipeToResponse(res);
} catch (err) {
  if (err instanceof FallbackExhaustedError) {
    // Per-model breakdown
    for (const f of err.failures) {
      console.error(
        `${f.provider}/${f.model} failed after ${f.retriesAttempted} retries: ${f.error.message}`
      );
    }
    res.status(503).json({ error: 'All models unavailable. Try again later.' });
  }
}

Compatibility fallback without retries

Available in LLM SDK 2.6.0-beta.2. Use fallbackErrors for a known non-retryable compatibility error. It advances to the next model without retrying the same invalid request. The default retry rules still handle rate limits, server failures, and network errors.

import { createRuntime } from '@yourgpt/llm-sdk';
import { createFallbackChain } from '@yourgpt/llm-sdk/fallback';
import { createOpenAI } from '@yourgpt/llm-sdk/openai';
import { createAnthropic } from '@yourgpt/llm-sdk/anthropic';

const runtime = createRuntime({
  adapter: createFallbackChain({
    models: [
      createOpenAI({ apiKey: process.env.OPENAI_API_KEY }).languageModel('gpt-6-luna'),
      createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY }).languageModel('claude-sonnet-4-5'),
    ],
    retries: 1,
    fallbackErrors: (error: unknown) => {
      if (!error || typeof error !== 'object') return false;
      const e = error as { status?: number; code?: string; param?: string };
      return e.status === 400 && e.code === 'unsupported_parameter' &&
        ['temperature', 'max_tokens', 'max_completion_tokens'].includes(e.param ?? '');
    },
  }),
});

Match the actual structured error shape returned by your provider. Do not enable every 400 or authentication error: many represent invalid application input. OpenAI parameter normalization should prevent the known temperature problem before fallback is needed.

retryableErrors changes which errors receive retries and can advance after those retries. Keep it for transient errors; fallbackErrors is the separate predicate for advancing without retries. Abort requests do not fall back, and a stream is not replayed once text or tool-call events have been emitted.

For the backend's TRAINING_CONFLICT workflow, the SDK supplies these controls but does not choose the application's decision. Validate the returned decision/schema, and keep an explicit application error path if all models fail or output is invalid. Do not silently treat an API or parsing failure as a successful respond decision. Backend adoption and end-to-end guardrail evaluation remain tracked in #122.

Shared Routing Store (Multi-Instance)

For round-robin to work correctly across multiple server instances or serverless functions, plug in a shared store.

import { createFallbackChain, type RoutingStore } from '@yourgpt/llm-sdk/fallback';

// Implement RoutingStore with any backend — Redis, Upstash, Cloudflare KV, etc.
// The SDK ships the interface. You own the implementation.
const redisStore: RoutingStore = {
  async get(key) {
    const val = await redis.get(key);
    return val ? Number(val) : undefined;
  },
  async set(key, value) {
    await redis.set(key, String(value));
  },
};

createFallbackChain({
  models: [openaiModel, anthropicModel],
  strategy: 'round-robin',
  store: redisStore,
});

The default MemoryRoutingStore is zero-config and works for single-process apps. No store configuration is needed unless you run multiple instances.

With Tools

Tools work transparently across fallback providers. Define your tools once — whichever provider handles the request formats them natively.

const runtime = createRuntime({
  adapter: createFallbackChain({
    models: [openaiModel, anthropicModel],
  }),
  tools: [
    {
      name: 'get_weather',
      description: 'Get current weather for a city',
      location: 'server',
      inputSchema: {
        type: 'object',
        properties: { city: { type: 'string' } },
        required: ['city'],
      },
      handler: async ({ city }) => fetchWeather(city),
    },
  ],
});

OpenAI receives tools as function-calling JSON. Anthropic receives them as tool_use blocks. Your handler always runs on your server regardless of which provider responded.

Full Configuration Reference

import { createFallbackChain } from '@yourgpt/llm-sdk/fallback';

createFallbackChain({
  // Required: adapters to try in order
  models: LLMAdapter[],

  // Routing strategy (default: 'priority')
  strategy?: 'priority' | 'round-robin',

  // Pluggable store for round-robin state (default: MemoryRoutingStore)
  store?: RoutingStore,

  // Retries per model before moving to next (default: 0)
  retries?: number,

  // Base delay between retries in ms (default: 500)
  retryDelay?: number,

  // Backoff strategy (default: 'exponential')
  retryBackoff?: 'exponential' | 'fixed',

  // Called on each per-model retry attempt
  onRetry?: (info: RetryInfo) => void,

  // Called when a model is abandoned and next one is tried
  onFallback?: (info: FallbackInfo) => void,

  // Transient errors: retry, then advance after retries
  retryableErrors?: (error: unknown) => boolean,

  // Non-retryable compatibility errors: advance without retrying
  fallbackErrors?: (error: unknown) => boolean,
})

Next Steps

OpenAI — Configure your primary provider
Anthropic — Add a Claude fallback
Server Storage — Persist sessions alongside fallback chains

Fallback

On this page