Routing AI Jobs Across GPU Clusters: Five Lessons From Production

Orchestra exists because one GPU is never enough and one provider is never reliable enough. Routing AI workloads across clusters and APIs looks like a scheduling problem; in production it is a failure-handling problem wearing a scheduler's clothes.

Lesson 1 — Health is a spectrum, not a boolean

Providers rarely go down; they go slow, then flaky, then down. A router that only knows up/down will send jobs into the flaky window. Track rolling latency and error rates per destination and start shifting traffic at degradation, not at death.

Lesson 2 — Every job needs a cost ceiling

Retries are the silent budget killer. A job that fails over across three providers with exponential prompts can cost five times its estimate. Attach a spend ceiling to the job, not the provider, and stop when it is reached.

Lesson 3 — Queues beat immediate failure

When every destination is saturated, the right answer is usually 'wait 90 seconds', not 'error'. A visible queue with priorities turns capacity incidents into latency blips.

Lesson 4 — Route by capability, not just price

The cheapest model that cannot do function-calling is not cheap — it is a retry loop. Capability tags on destinations (context length, tool use, vision, region) prevent an entire class of doomed dispatches.

Lesson 5 — Log the routing decision itself

When a customer asks why their job ran where it ran, 'the algorithm chose' is not an answer. Persist the decision inputs — health, price, capability, queue depth at dispatch time. Auditability is a feature, and in regulated shops it is the feature.

The architecture behind the lessons

An orchestration layer for AI workloads has a simple outline. Jobs arrive with a description of what they need: a model capability, a context length, a latency target, a budget and sometimes a data-residency constraint. A catalogue describes the destinations available — self-hosted GPU clusters, cloud instances and external model APIs — along with their capabilities, prices and current health. The router matches jobs to destinations, a queue absorbs the moments when nothing suitable is free, and a ledger records every decision and its cost.

Each of those pieces is easy to build in a toy version. The difficulty comes from the real world: providers degrade without warning, prices change, models behave differently from their documentation and customers expect predictable latency anyway. The five lessons above are what remains after those collisions.

Measuring health without fooling yourself

Health measurements need to reflect what users experience. A destination that answers a lightweight health check instantly may still be failing real requests or taking a minute to produce the first token. The most reliable signals come from production traffic itself: rolling error rates, the share of requests that time out, latency percentiles rather than averages, and the time to first token for streaming responses. Synthetic probes are still useful when traffic is light, but they should exercise the same path as real jobs.

Smoothing matters too. Reacting to a single slow request makes the router flap between destinations; waiting for an hour of data makes it blind. Short rolling windows with gentle weighting — and a minimum amount of traffic before a destination is judged — keep decisions stable without ignoring genuine degradation.

Retries, idempotency and budgets in practice

Retries are necessary, but every retry should be a deliberate choice. Classify failures first: a rate-limit response means "wait and try again here", a timeout may mean "try elsewhere", and a validation error means "do not retry at all, the request itself is wrong". Use backoff with jitter so that many jobs do not retry in lockstep, and cap the total number of attempts.

Make jobs idempotent wherever possible, so that a retry after an ambiguous failure cannot produce duplicate side effects such as two charges, two emails or two database writes. And track spend per job as it accumulates, not only at the end. When the ceiling is reached, the job should stop and report clearly why, rather than quietly continuing to consume budget across providers.

Fair queuing and priorities

A queue only helps if it is fair. Without care, a single customer submitting thousands of batch jobs can starve everyone else. Separate queues by priority class — interactive requests, standard work and bulk processing — and within each class share capacity between tenants so that no one monopolizes it. Give interactive work a short maximum wait and let batch work absorb the delays. Expose queue position and estimated wait to users; a visible wait feels very different from a mysterious delay.

Capability matching beyond the obvious

Capability tags start with the obvious — context length, tool use, image input — but production adds subtler ones. Some models follow structured-output formats more reliably than others. Some regions are required for certain customers. Some destinations support long-running streaming connections while others cut them off. Encoding these properties explicitly lets the router exclude destinations that would technically accept a job but predictably fail it. The catalogue should be versioned and reviewed when providers announce changes, because a capability that disappears silently is a source of mysterious failures.

Self-hosted clusters versus external APIs

Running your own GPUs and calling external APIs are complementary rather than competing choices. Owned or reserved hardware offers predictable cost at steady utilization, full control over data and the ability to run open models tuned for your tasks. External APIs offer access to frontier models, elastic capacity and no hardware to maintain. A sensible orchestration policy uses owned capacity for the steady base load and sensitive data, and external APIs for peaks, specialized capabilities and work that is allowed to leave your environment.

The policy must respect data rules strictly. If a job is marked as containing sensitive data, the router should never move it to an external API during a failover, even if that means waiting in the queue. A slower answer is acceptable; an unauthorized data transfer is not.

Observability for routing decisions

Logging each decision is the foundation, but useful observability goes further. Dashboards should show, per destination, traffic share, error rates, latency and spend over time, alongside the reasons traffic moved: degradation, price, capacity or policy. Traces that follow a single job through queueing, dispatch, retries and completion make difficult cases explainable in minutes. And alerts should fire on symptoms users feel — rising queue times, falling success rates, spend growing faster than traffic — rather than on every individual provider hiccup.

A checklist before production

Before trusting an orchestration layer with real workloads, confirm a few things. Every destination has health measured from real traffic. Every job carries a budget and a retry limit. Failures are classified and handled differently. Sensitive jobs cannot be routed to unapproved destinations. Queues have priorities and per-tenant fairness. And every routing decision can be reconstructed afterwards from the logs. With those in place, outages become routine events instead of emergencies.

Cost accounting that finance can trust

Routing decisions affect money, so the ledger should be precise enough for finance to rely on. Record the cost of every attempt, not only the successful one, and attribute it to the job, the tenant and the workflow that launched it. Keep the provider's own usage figures alongside your estimates and reconcile them regularly, because pricing and token counting differ subtly between providers. Showing teams their own spend per workflow encourages better habits than any central policy: people optimize what they can see.

Testing failure on purpose

The only way to know that failover works is to exercise it. In a staging environment, simulate the situations that occur in production: a provider that returns errors for a few minutes, one that becomes very slow, a rate limit that suddenly halves, a region that disappears entirely. Watch how the router behaves, how queues grow, whether budgets hold and whether the logs explain what happened. Repeat these exercises after significant changes to the routing logic. Teams that rehearse failure discover their surprises on a quiet afternoon instead of during a customer's most important batch job.

Keeping humans in control

Automation should route the routine work, but people need clear levers for the exceptional moments. Provide a way to drain a destination manually during maintenance, to pin a critical workload to a specific provider, to pause bulk queues when interactive traffic needs everything, and to change priorities without a deployment. Every manual override should be logged with who made it and why, and should expire automatically unless renewed, so that temporary decisions do not quietly become permanent configuration.

What we would tell a team starting today

Start with fewer destinations than you think you need and measure them well. Put budgets, retry limits and data rules into the job definition from the first day, because adding them later means auditing every workflow. Prefer simple, explainable routing rules over clever ones until the logs prove you need more. Orchestration earns trust by being predictable; sophistication can come later.