A load balancer sits in front of a pool of backend resources and decides which one should handle each incoming request. Its purpose is to prevent any single node from becoming a bottleneck or a single point of failure: it tracks which backends are healthy, routes traffic away from those that are not, and spreads load according to a chosen policy. In conventional web infrastructure the pool is a set of application servers. In AI infrastructure the same pattern applies to model endpoints—self-hosted GPU replicas, hosted provider APIs, or a mix of both—where balancing decisions must account for rate limits, token cost, and wildly variable request duration rather than simple connection counts.
The common policies are round robin (each backend in turn), least connections (whichever node is currently least busy), weighted distribution (proportional to declared capacity), and hash-based routing (a consistent key such as session or user ID always maps to the same backend). Weighted and least-connections policies dominate in practice because real backend pools are rarely homogeneous.
Assumptions that hold for web requests break down. Response times vary by orders of magnitude depending on output length, so "least connections" is a poor proxy for actual load—queued tokens or GPU memory pressure are better signals. Backends are constrained by provider rate limits and token quotas rather than CPU. Costs differ per endpoint, making the cheapest healthy route a legitimate balancing criterion. And requests are expensive enough that failing over on error, rather than returning a 503, is usually worth the added latency.
A load balancer answers "which backend gets this request?" A gateway additionally handles authentication, rate limiting, request transformation, caching, and observability. An AI Gateway is the AI-specific form of the latter: it load balances across model providers while also enforcing spend caps, applying fallback chains when a provider degrades, caching repeated prompts, and logging every call for audit.
Only alongside health checking. A balancer that keeps routing to a failed backend distributes errors rather than traffic. Active health probes, circuit breaking after repeated failures, and automatic reinstatement once a node recovers are what convert distribution into genuine resilience.
Shakudo provisions load balancing as part of the platform layer inside your own VPC or on-premise cluster, so scaling and failover for both application services and model endpoints are managed without bespoke infrastructure work. Traffic across self-hosted models and external providers is routed through the AI Gateway, giving teams cost-aware routing, automatic failover, and a single observability surface—while inference and the data feeding it remain entirely within your governance boundary.
