Aquifer

Keep the machines alive until reinforcements arrive.

A circuit breaker protects a backend by refusing work fast and leaving callers to retry. Aquifer holds the work, gives every tenant a turn, and releases it at the pace the machines advertise. Below: the idea you can play with, then what the benchmarks measured.

Fail fast, or hold and pace

Illustration, not benchmark data

Both sides get the same traffic and lose the same machine at the same moment. Watch the bar under each tenant: it shows how long that tenant's oldest job has been waiting.

Circuit breaker: fail fast
Completed
0
Refused
0
Sends per job
1.0
Slowest small tenant
0 s
Requests per second, last 35 seconds
What the breaker tells callers
HTTP/1.1 503 Service Unavailable
(retry on your own timer)
Where its limit lives
circuit_breakers:
  thresholds:
  - max_requests: 6   # written in config
To change it, someone edits the config and rolls it out, or runs a control plane that pushes it. The backend itself has no say.
Aquifer: hold and pace
Completed
0
Refused
0
Sends per job
1.0
Slowest small tenant
0 s
Requests per second, last 35 seconds
Last headers the machines sent back

        
        
No deploy. The next response carries the new number and Aquifer follows it.

A proxy has to guess from the outside: fixed limits, or latency it measures after the damage is done. So when capacity drops, the traffic hitting it goes up, because every refusal comes back as a retry. Aquifer is told. The machines put the rate they can take on every response (X-Aqueduct-Rps), it moves up and down with their real capacity, and what Aquifer releases follows it. Drag the slider under the Aquifer side to change the rate yourself: that is the backend sending a different header, with nothing redeployed.

The big tenant waits longer on the right. An equal turn gives it less than the share it can grab by volume on the left. That cost is real, and the benchmarks below show it too.

Answer on the same connection for as long as it can

How it works

A webhook gateway usually takes the event, answers at once, and delivers later. Aquifer starts one step earlier: while the target is healthy the caller gets the real answer on the same connection, with no queue in the way. It only becomes a queue when the target says it is struggling, and the target says so with headers on its own responses.

Target healthy

Straight through

POST /proxy tries the target directly. A healthy target never sees a queue.

POST /proxy
← 200 OK
  the target's own answer
Target under pressure

The target sets the pace

The target answers with its limits. Aquifer honors a lower pace at once and recovers gradually.

← 429 Too Many Requests
  X-Aqueduct-Rps: 6
  X-Aqueduct-Max-Concurrent: 4
  X-Aqueduct-Max-Backlog: 3600
Work is queued

Same connection, now a line

The caller's connection becomes a stream showing its place in line. Hang up and the result still arrives by webhook.

event: queued
event: position (3 ahead)
event: dispatching
event: completed (200)
In review, PR #133

Too many callers holding on

When held connections near the open-file limit, Aquifer tells the caller when to come back and the caller polls then. Used in the late-capacity benchmark; not merged to main yet.

come back when told,
not on a backoff timer

GPU servers that cannot send these headers are read through their ORCA load report (vLLM, Triton). Headers and events as documented in docs/pages/pacing.md and streaming.md.

When help comes late, how long does the unluckiest caller wait?

Real run data

Traffic runs at three times what the backend can do. Capacity grows from 8 workers to 32 some minutes in, the way an autoscaler catches up. Callers retry until their job is done. Both front doors finished every job. With help 10 minutes away, the longest wait was about 11 minutes behind Aquifer and 24 behind Envoy.

Help arrives after 3 min
3.1
15.6
Help arrives after 5 min
5.3
18.2
Help arrives after 10 min
10.9
23.9

Source: aqueduct-benchmarks, envoy/results/unb-* (two rounds per setting, midpoints shown). Envoy with limits sized for the capacity after scaling.

A retry storm, replayed

Real run data

Four services sit in a chain and each one retries three times, so one refusal at the bottom can become 27 calls. Fifteen tenants push up to three times what the backend can do for 40 minutes. The backend goes down for a minute, then its capacity doubles. Same backend, same load, two front doors.

2.2×

The same backend finished about 2.2 times as many requests behind Aquifer. It did nearly the same amount of work both times. Behind Envoy, 58% of that work was redoing jobs it had already finished.

43:20 / 43:20

Source: aqueduct-benchmarks, envoy/results/wild-envoy-wild-1 and wild-aquifer-wild-1 (10-second samples). Chart lines are a 30-second rolling average.

One runtime for every kind of traffic

Illustration, not benchmark data

Webhooks arriving, agents calling tools over MCP, model calls headed to a GPU and calls to your own backend all pass through the same queue. Each target says how fast it wants work, and Aquifer releases at that pace. Send a burst, change a target's pace, then switch Aquifer off and watch the same traffic.