LLM Inference Provider Landscape

The market for LLM API access is a three-layer vertical supply chain: model creators (Anthropic, OpenAI, Google, Moonshot, DeepSeek) train the weights; inference providers (Together, Fireworks, Groq, DeepInfra, Novita, Baseten, the hyperscaler clouds) operate the GPU clusters that serve them; and aggregators / API routers (OpenRouter, Vercel AI Gateway, Martian, LiteLLM-cloud) resell unified access across many providers without owning any GPUs.¹²

Knowing which layer you’re talking to — and which layer your fallback talks to — is the difference between real redundancy and stacked raccoons.

The three layers

1. Creators / first-party APIs. Closed models (GPT, Claude, Gemini) are served only by the creator or contractually authorized partners (Azure for OpenAI, AWS/GCP for Anthropic). The execution layer is not broadly deployable.³
2. Independent inference providers. Open-weight models (Llama, Qwen, DeepSeek, Kimi, GLM) can be served by anyone with GPU capacity and a serving stack (vLLM etc.). These firms secure their own compute and compete on price, latency, throughput, and reliability — the company controlling the API endpoint also controls the hardware.⁴
3. Aggregators / routers. A proxy layer that forwards requests to upstream providers; it never touches a GPU on your behalf. OpenRouter is the canonical example: 400+ models from 60+ providers behind one OpenAI-compatible endpoint, revenue from a ~5.5% credit-purchase fee / 5% BYOK fee rather than token markup.⁵

Market structure facts (Demirer, Fradkin, Tadelis & Peng, Dec 2025)

• Inference providers grew from 27 (Nov 2024) to 90 (Dec 2025); entry is driven almost entirely by open-source hosting.⁶
• 14 open-source models are served by 10+ providers each; a few by 20+. Closed models stay single- or dual-provider.⁷
• Open-source models price ~90% below comparably capable closed models; heterogeneity persists within tiers.⁸
• OpenRouter alone served 100+ trillion tokens in 2025 — the aggregator layer is now a real market institution, not a niche tool.⁹

How an aggregator actually routes (OpenRouter mechanics)

Default strategy is price-weighted load balancing with an uptime filter: exclude providers with significant outages in the last ~30 seconds, then select among the cheapest candidates weighted by the inverse square of price, keeping the rest as fallbacks. Setting sort (price/throughput/latency) or order (explicit provider list) disables load balancing. The provider object also supports only/ignore lists, allow_fallbacks, require_parameters, quantization filters, max_price, percentile-based latency/throughput preferences (rolling 5-minute windows), ZDR and data-collection restrictions.¹⁰

Two consequences worth noting: routers improve tail reliability, never median latency (proxy hops are additive; rerouting around degraded endpoints is a p95 win), and a router can only bind itself — a zero-retention DPA with the router says nothing about the upstream provider that actually processes your tokens.¹¹

The correlated-failure trap (why this page exists)

A fallback chain is only as independent as its layers. If your primary and fallback are both aggregators fronting the same upstream creator endpoint, an upstream outage takes the whole chain — “three raccoons in a trench coat reselling the same API.” This is not hypothetical: on 2026-08-07 a cron job here failed when the primary gateway (Nous) and the fallback gateway (OpenCode Go) both 503’d on kimi-k3 within minutes — consistent with a shared upstream (Moonshot or a common router layer) being dark, while other models on the same gateway stayed healthy.

The design rule: each fallback tier should move down the independence gradient — same model via a different aggregator (weakest) → same model via an independent hoster (removes the routing layer, keeps the weights dependency) → different model family / different creator (strongest; survives creator-endpoint failure). Creator first-party APIs (e.g. DeepSeek direct) collapse layers 1–2 and remove all raccoons. See common-mode-failure for the general reliability principle: redundancy only buys reliability in proportion to failure independence — an assumption that fails empirically more often than designers expect.

How to tell a router from a provider (Fireworks’ field guide): docs saying “our clusters / our GPUs” = provider; “access 200+ models from leading providers” = router. DPA “sub-processor” language = router tell. Status pages listing GPU regions = provider. ASN lookup on the endpoint IP; consistent 20–80ms latency overhead vs. a known direct endpoint = proxy-hop signature.¹²

Application note: Hermes fallback chains

Hermes supports a top-level fallback_providers: list (tried in order on rate limits, server errors, auth failures) plus per-auxiliary fallback_chain: entries; first-class providers include Novita (NOVITA_API_KEY), Fireworks, Kimi/Moonshot direct, z.ai, MiniMax, Qwen, and others.¹³ A 2026-08-07 source review of hermes_cli/runtime_provider.py confirmed that api_mode on a fallback entry is resolved as URL-detection → provider overlay → literal chat_completions default — so for OpenAI-compatible aggregator endpoints (opencode-go, Novita) an explicit api_mode: chat_completions line is redundant.

The chain discussed that morning (not yet applied, pending Nathan’s Novita key): primary Nous → Novita (independent hoster; moonshotai/kimi-k3 $3/$15, deepseek/deepseek-v4-flash $0.14/$0.28) → OpenCode Go. Rationale: if Nous and OpenCode both route through OpenRouter, Novita is the tier that removes the shared routing layer; the next failure then discriminates between “aggregator layer failed” and “creator endpoint failed.”

Connections

• deepseek — the open-weights pricing floor whose first-party API is the archetypal raccoon-free fallback
• common-mode-failure — the general principle this market keeps re-teaching: correlated failure defeats nominal redundancy
• intelligence-as-tool — the inference market is literally the market for intelligence-as-a-deployable-capability: tokens as metered cognition
• filter-bubble — epistemic homogeneity as the cognitive analog of a single-upstream dependency

Sources

• 2025
• 2026 — Inference Providers vs. API Routers: where do tokens come from?
• Provider Routing — OpenRouter Docs
• With an API key (pay-per-token)

Footnotes

1. 2025 ↩

2. 2026 — Inference Providers vs. API Routers: where do tokens come from? ↩

3. 2026 — Inference Providers vs. API Routers: where do tokens come from? ↩

4. 2026 — Inference Providers vs. API Routers: where do tokens come from? ↩

5. 2025 ↩

6. 2025 ↩

7. 2025 ↩

8. 2025 ↩

9. 2025 ↩

10. Provider Routing — OpenRouter Docs ↩

11. 2026 — Inference Providers vs. API Routers: where do tokens come from? ↩

12. 2026 — Inference Providers vs. API Routers: where do tokens come from? ↩

13. With an API key (pay-per-token) ↩