Concurrency & overflow

LunaRoute measures capacity in concurrent inference requests: one request actively processing at a time. Your plan sets how many requests run with priority scheduling, plus overflow capacity for additional requests. A request beyond both parks — it waits rather than failing immediately. A full queue or an expired park deadline is rejected with 429; see What happens when you exceed your concurrency.

Priority concurrency

One concurrent request means one inference request actively processing. Requests waiting in a queue do not count as running requests. One agent can make several requests at once, so size your plan for requests, not agents.

Priority concurrency gives your included requests preferential access to available capacity. It does not reserve a GPU or guarantee latency, generation speed, or uninterrupted availability. Any performance commitments belong in your Enterprise agreement or SLA.

Background scheduling

Requests run with priority scheduling by default — a human is waiting. Work that can wait (batch evals, indexing, long jobs) should use the -background form of the model id:

{ "model": "glm-5.3-background" }

Background requests are scheduled at a lower fair-share priority, so they fill in around priority traffic. The dialect and parameters are unchanged — only scheduling differs.

Overflow capacity is an entitlement. If your plan or organization has none, a -background request runs in the priority pool instead. Inference responses (chat, messages, responses) also carry X-LunaRoute-Warning: no-background-entitlement; image routes downgrade the same way but do not set that header.

Heavy sustained use doesn’t trigger usage-based blocking or overage billing. Under load, work runs at a lower fair-share scheduling priority and may wait longer to start.

Overflow is automatic

Keep sending normal requests when your priority concurrency is busy. LunaRoute uses available overflow capacity automatically; there is no special model name to choose. Automatic overflow applies to managed text generation — chat completions, legacy completions, messages, and responses — that use default scheduling. Image generation currently waits for priority capacity rather than using overflow, and embeddings and System One requests draw on their own limits.

Overflow is on by default for organizations created from September 28, 2026. Organizations created earlier keep whatever they had, and can enable it for the workspace from the dashboard’s Settings page — until then, extra requests wait as described under Queues and limits.

A request that overflows runs at lower priority and may take longer, and it counts as overflow usage at background weight toward fair-use. Token prices are unchanged. An overflowed response carries X-LunaRoute-Service-Class: background and X-LunaRoute-Warning: overflowed-to-background, so a client can tell a spill from a request it marked background itself. If no overflow capacity is free, the request waits for a priority slot as usual.

Requests that set the scheduler tier explicitly — for example X-LunaRoute-Scheduler: realtime — never overflow, and neither does a request that already names the -background model suffix; only requests that left both at their default are eligible.

Queues and limits

Overflow is included in your fixed price for included inference, but acceptance is not unlimited. Plan capacity, queue limits, client timeouts, and service timeouts still apply. A full queue can reject a request immediately; a request can also time out before capacity becomes available. Excess requests park by default:

Priority requests park for up to 60 seconds.
Requests scheduled with -background park for their plan’s background window, capped at 240 seconds.
A queue that is completely full rejects immediately.

If a request is rejected — or the park deadline expires — the response is:

HTTP/1.1 429 Too Many Requests
Retry-After: 60

{ "error": { "code": "CONCURRENT_REQUEST_LIMIT_EXCEEDED", "message": "…" } }

Because the priority park deadline is exactly 60 seconds, a client that times out at 60 seconds may be racing the gate rather than hitting a separate timeout. Give clients a timeout comfortably above 60 seconds, plus enough time for generation, and size a parallel worker pool no larger than your priority concurrency allowance. For predictable waits, keep sustained concurrency within that allowance; honor Retry-After and use bounded retries with backoff for bursts. The 429 codes per surface and their Retry-After values are in Errors & rate limits.

Your capacity

Your plan or Enterprise agreement defines your concurrency allowance: a number of priority concurrent requests plus capacity for additional overflow requests. For example Max includes 4 priority concurrent requests and capacity for up to 8 additional overflow requests. Those extra requests depend on available capacity; they are not another guaranteed priority allowance.

Embedding capacity is its own entitlement and may be zero on plans that do not include embeddings. System One capacity is likewise an organization-level entitlement: five concurrent requests on every plan unless an override changes them.

The authoritative, override-inclusive values for your organization are shown on the Metrics page of the dashboard. Organization admin overrides always win over the plan default. Teams adds per-member controls and expandable capacity bundles.

Personal Agent is a separate offer: one concurrent request scheduled as capacity becomes available, with its own model scope and no priority scheduling.

The dashboard is not the only way to read these values. GET /v1/capacity returns your organization’s effective configuration — priority concurrent requests, overflow capacity, and whether automatic overflow is enabled — together with live occupancy and a daily history of how requests were admitted: served immediately, served after waiting, or refused. See Capacity & admission.

Managing concurrency per member

On plans that carry the entitlement, organization owners and admins can cap how many concurrent requests an individual member — or service user — consumes. The two classes cap independently: limiting someone’s overflow requests leaves their priority concurrency untouched.

Set caps from the Members page of the dashboard. Each member row has Priority requests and Overflow requests fields, and the page shows the organization allowance a member inherits by default (Org concurrency: N priority · M overflow).

Value	Meaning
empty	Inherit the organization allowance — the default.
`0`	Blocked from that class: requests are refused immediately with `429`, they do not park.
`n`	A ceiling, not a reservation — at most `n` of the member’s requests in flight in that class; unused headroom stays available to the rest of the organization.

A member at their cap parks like an organization at its allowance: the request waits for a slot in the member’s own class before it contends for the organization allowance. A cap change applies to requests arriving within about a minute.

Three classes deliberately have no per-member caps: embedding, System One, and BYOK requests are organization-level only.

Platform admins can set or clear member caps for any organization, regardless of its plan.

What happens when you exceed your concurrency

Excess requests park, or are rejected, as described under Queues and limits. The short version: priority requests wait up to 60 seconds for a slot, -background requests wait for their plan’s window (capped at 240 seconds), and a full queue rejects immediately with 429 and Retry-After: 60.

The refusal code is the same in every case — CONCURRENT_REQUEST_LIMIT_EXCEEDED — whether the gate refuses immediately or the request gives up waiting.

Bring your own key

Requests routed to your provider key use a flat platform-level concurrency cap (not your plan’s allowance). An “unlimited” setting bypasses the cap. Everything else — parking, Retry-After — behaves the same. Routed this way, the upstream provider’s own data retention policy applies — see Zero data retention.