# Why Aquifer > What Aquifer is, the problem it solves, and where it sits next to a gateway, Canalis and Inlet. ![How engineers respond to congestion: pacing and backpressure everywhere else in infrastructure, versus retrying everything immediately at the API layer](https://rjpruitt16.github.io/aquifer/images/how-engineers-respond-to-congestion.jpg) **TL;DR — why not Hookdeck or Envoy?** Envoy's circuit breaking protects an overloaded service by rejecting excess work, and a rejected request is gone: every client retries on its own and they compete for whatever capacity frees up. Aquifer stores the excess durably and releases it at the rate the backend says it can take right now (see [Dynamic Pacing](https://rjpruitt16.github.io/aquifer/pacing.md)). Circuit breaking says "stop." Aquifer says "wait here, and I'll tell you when to go." Hookdeck does something similar for inbound webhooks, as a hosted service billed by usage. Aquifer is open source and runs on your own machines like Envoy, so scaling it costs only the machines you add, and it paces outbound API calls, agent tool calls and WebSocket sessions too. Results come back however suits the caller: synchronously on the same connection ([`POST /proxy`](https://rjpruitt16.github.io/aquifer/api.md#post-proxy)), over a WebSocket that replays anything missed after a reconnect ([`GET /websocket`](https://rjpruitt16.github.io/aquifer/api.md#get-websocket)), or by durable webhook. You can run it with both: Envoy in front as the gateway, Aquifer behind it holding the work, and Hookdeck delivering its webhooks into Aquifer. **See it move.** [Aquifer in Motion](https://rjpruitt16.github.io/aquifer/interactive/aquifer-in-motion.html) is an interactive page: take a machine offline and watch a circuit breaker and Aquifer handle the same traffic, change the pace from the backend with a header, then replay the Envoy benchmark runs. **Not going to read all this?** Point your coding agent at the [Aquifer pilot skill](https://github.com/rjpruitt16/aquifer/blob/main/skills/aquifer-pilot/SKILL.md). It evaluates Aquifer against your own workload in a shadow run on a branch, then shows you the numbers. **Increase your rate limit without DDoSing your backend.** Distributed agents call tools and APIs in bursts. Your backend gets overwhelmed on inbound. Your app gets 429s on outbound. One slow dependency takes everything else down with it, and the retries agents fire off while they wait only make it worse: [wasted utilization and higher cost](https://rahmipruitt.me/content/gpu-retry-tax/) on one end, [outages reactive autoscaling alone can't prevent](https://rahmipruitt.me/content/github-outage-reactive-scaling/) on the other. The usual answer to that is more hardware: double the fleet, triple the GPU budget, eat the bill. It works, but it's expensive, it leaves capacity idle most of the time, and it doesn't actually fix the burst, just buys enough headroom to survive the next one: ![How teams handle traffic spikes: overprovisioning the fleet versus queueing the burst and pacing the flow while capacity catches up](https://rjpruitt16.github.io/aquifer/images/how-teams-handle-traffic-spikes.png) Aquifer gives those agents a coordination layer: a self-hosted load balancer that absorbs the burst, queues requests durably (Pebble by default, or SQLite, see below), and releases them at a rate you configure, or a slower one, if the destination service asks for it. What's usually behind that backend can't scale instantly either: a GPU, a database, a CI runner. Aquifer buys time for more of it to come online; it's overkill if that ceiling is fixed for good. Exposed through pluggable adapters (an MCP server for agent tool-calling, a plain HTTP API, or an A2A/Agent2Agent-protocol agent), with cryptographic agent identity via the L8 protocol for trustless webhook delivery. **Benchmarked:** 10x traffic spikes accepted with zero errors (this measures intake), 30/30 jobs surviving a `kill -9` (with per-write sync, `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0`), and clean `429` admission shedding under sustained overload, including a real GPU under load, where a fixed rate cut peak backend queue depth from 449 to 8 waiting requests, and the ORCA fallback signal held the KV cache in a 22–79% cycle once the rate ceiling was raised past what vLLM could take. See [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md) for throughput ceilings, crash recovery, memory behavior, capacity by machine size, and the [GPU/vLLM run](https://rjpruitt16.github.io/aquifer/benchmarks.md#9-gpu-inference-and-the-retry-tax-runpodvllm). Longer, end-to-end tests of Aquifer in front of real open-source systems, starting with Hatchet, are in [aqueduct-benchmarks](https://github.com/rjpruitt16/aqueduct-benchmarks). ![Traditional load balancing collapsing under a spike, round-robin flickering faster as instances die, versus Aqueduct pacing that keeps the fleet stable while an autoscaler brings real capacity online](https://rjpruitt16.github.io/aquifer/images/fleet-degradation.gif) **Terms used in these docs:** the **origin** is whoever sends a job to Aquifer: your app, an agent, a gateway, or Inlet. The **target API** is the service Aquifer sends it on to, whose capacity it protects. Some names in the code, config and headers still say "upstream" for the target API, such as `AQUIFER_MAX_PENDING_PER_UPSTREAM` and `X-Aqueduct-Upstream-URL`. They mean the same thing and keep working. **Aquifer is not an API gateway.** It does not provide general end-user authentication, authorization policy, WAF rules, CORS policy, or arbitrary public routing. Keep it on a private network behind a trusted front door that decides who the caller is and which target is allowed. Aquifer durably queues the accepted work and controls how quickly it reaches that target. ``` API / agent callers → API gateway or Canalis → Aquifer → target API webhook providers → Inlet → Aquifer → webhook handlers ``` Use **[Canalis](https://github.com/rjpruitt16/canalis-rs)** as the trusted front door for API or agent traffic when you need JWT verification, sticky tenant placement, and fleet-wide capacity coordination; it can also sit behind an existing gateway. Use **[Inlet](https://github.com/rjpruitt16/inlet)** for provider webhooks: it verifies provider signatures and queues accepted deliveries into Aquifer. Aquifer is the capacity-aware load balancer behind either one: it persists jobs, enforces fairness, and paces dispatch to the target API. --- # Quick start > Run Aquifer from the binary, Docker Compose, Fly.io, AWS, or a Canalis fleet. **Binary** ```bash go install github.com/rjpruitt16/aquifer/cmd/aquifer@latest aquifer ``` Then send it a job and watch it go through the line: ```bash curl -s localhost:8080/jobs -H 'Content-Type: application/json' -d '{ "user_id": "me", "idempotent_key": "first-job", "url": "https://httpbin.org/post", "method": "POST", "body": "{\"hello\":\"aquifer\"}" }' # {"job_id":"c22c473d…","status":"queued"} curl -N localhost:8080/jobs//stream # event: queued # event: completed (data carries the target API's response_status and body) ``` **Docker Compose** ```bash docker compose build --pull --no-cache docker compose up --detach curl http://127.0.0.1:8080/ready ``` Compose installs the latest checksum-verified release, persists `/data`, and listens only on loopback by default. See the [Docker Compose guide](https://github.com/rjpruitt16/aquifer/blob/main/examples/docker-compose/README.md) for version pinning and destination allowlisting. **Fly.io** ```bash ./examples/fly/deploy-quickstart.sh ``` The [Fly.io quick start](https://github.com/rjpruitt16/aquifer/blob/main/examples/fly/README.md) deploys the latest checksum-verified release to a private one-CPU Machine with an encrypted persistent volume. Use `fly proxy` for local evaluation; Aquifer is not exposed publicly by default. **AWS test node** ```bash export AWS_REGION=us-east-2 ./examples/aws/deploy-quickstart.sh ``` The [AWS quick start](https://github.com/rjpruitt16/aquifer/blob/main/examples/aws/README.md) deploys the latest release to a CIDR-restricted one-vCPU EC2 instance with encrypted local storage. It is intended for evaluation, not production. **Canalis fleet** ```bash docker compose -f examples/canalis/compose.yaml up --detach --build ``` The [Canalis + Aquifer quick start](https://github.com/rjpruitt16/aquifer/blob/main/examples/canalis/README.md) runs two independently persisted Aquifer nodes behind Canalis. The same guide includes tested two-host AWS and private Fly.io deployment paths. Cross-platform release binaries (linux/darwin, amd64/arm64) are attached to every [GitHub Release](https://github.com/rjpruitt16/aquifer/releases); no Go toolchain required if you'd rather grab one directly. --- # How it works > The job lifecycle and the delivery guarantee (at least once). 1. Client submits a job through an adapter (MCP tool or HTTP endpoint) and moves on 2. Aquifer persists it to disk (Pebble by default), surviving crashes and re-dispatching on restart 3. A per-target-API worker dispatches at your configured RPS with jitter 4. On completion Aquifer POSTs your webhook with the response body and status, or skips it if you [stayed on the line](https://rjpruitt16.github.io/aquifer/streaming.md) and got the result there 5. The target API can adjust the rate live via `X-Aqueduct-*` response headers **Delivery semantics:** Aquifer provides at-least-once dispatch and webhook delivery, not exactly-once execution. If Aquifer crashes after a dispatch succeeds but before it records that completion, the recovered job dispatches to the target API again on restart, so it's not just the webhook that can repeat, the target API call itself can. Make both your target API endpoint and your webhook handler idempotent on `job_id` (or `idempotent_key`) anywhere duplicate execution isn't safe, the same contract Stripe and GitHub webhooks already ask of you. --- # Use cases > MCP tools, protecting an HTTP API, webhook intake, outbound pacing, the /proxy reverse proxy, WebSockets and cross-region redirect. **MCP tools — coordinate distributed agents** ``` agents / MCP clients → aquifer_enqueue_job → Aquifer queue → target API ``` Agents call Aquifer as an MCP server instead of racing each other directly against the same backend or external API. Aquifer returns a job id immediately, dispatches the request at a controlled rate, and delivers the result to your webhook. **HTTP API — protect your API** ``` agents / clients → POST /jobs to Aquifer → your backend (at controlled RPS) ``` Agents hammering your API over HTTP? Aquifer queues their requests and drains them to your backend at a pace it can handle. Your backend returns `X-Aqueduct-Rps` headers to signal how fast it wants traffic in real time. **Webhook ingestion — keep noisy senders from monopolizing processing** ``` webhook providers → your verification endpoint → POST /jobs to Aquifer → webhook handlers ``` Verify each provider's signature at your edge, map the sender or customer to `user_id`, and use the provider's event ID as the `idempotent_key`. Aquifer absorbs bursts into its persisted queue and paces work into your handlers. With AccountQueue mode enabled, one noisy sender gets its own queue; under shared backlog pressure, fair admission increasingly rejects that oversized queue while leaving room for quieter senders. Aquifer does not replace provider authentication or translate arbitrary webhook formats. For supported providers, **[Inlet](https://github.com/rjpruitt16/inlet)** is the ready-made public front door; otherwise, use your own verification endpoint or custom adapter. For high-throughput intake, keep the default batched Pebble flush and resubmit the same event after a short jittered delay before acknowledging the provider. Acknowledge only when the resubmission returns `200` with `"duplicate": true`; a `201` means the earlier job was gone and was just recreated, so wait and confirm again. Both submissions need the same `user_id` and `idempotent_key` and must reach the same Aquifer owner. See the [confirmation best practice](https://rjpruitt16.github.io/aquifer/api.md#best-practice-confirm-batched-webhook-intake). Use `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0` when every accepted event requires a strict disk-sync guarantee. **Outbound — a durable checkpoint in front of a rate-limited resource** ``` your app → POST /jobs to Aquifer → database / CI runner / OpenAI / Stripe / any rate-limited API ``` Calling something with its own capacity limit (a database read replica, a CI runner, a third-party API)? Aquifer queues the calls durably and dispatches them at your configured rate, so a burst from your own side never becomes the thing that takes the target API down. Works especially well closed-loop: if the target API already speaks `X-Aqueduct-*` headers, it can tell Aquifer to back off in real time instead of you guessing a static rate. **Capacity-aware reverse proxy — pace and route behind your gateway** ``` your users → trusted gateway → POST /proxy to Aquifer → your resources ``` Point your trusted gateway at Aquifer, then point Aquifer at your resources. Aquifer tries the request directly first (a healthy resource sees no queue at all) and only falls back to durable queuing when something's actually overloaded, on the same connection, staying in queue mode until that domain's backlog is genuinely drained (not just until a cooldown timer expires). See [`POST /proxy`](https://rjpruitt16.github.io/aquifer/api.md#post-proxy) for the details, including the header a target API can use to request queuing proactively. **WebSocket proxy — durable replay and paced reconnects** ``` clients → trusted gateway → GET /websocket on Aquifer → WebSocket backend ``` Aquifer can persist an ordered WebSocket transcript in Valkey, replay missed backend events after a client reconnects, and pace new or replacement target API connections. Authentication remains the gateway's job. See [`GET /websocket`](https://rjpruitt16.github.io/aquifer/api.md#get-websocket) for the protocol and limits. **Multigenerational search — keep every lineage alive** ``` search coordinator → POST /jobs to Aquifer (user_id = lineage) → your worker → model API / evaluator ``` Evolutionary and tree searches (program search, prompt or agent optimization, red-teaming) fan out a generation of model and evaluator calls, keep the winners, and fan out again. Give each lineage one stable `user_id` (and the same API key, since each `user_id` and key pair is its own queue): a lineage that suddenly spawns ten times the children gets the same turns as the rest instead of starving them, the model API's `429`s pace the whole search instead of setting off a retry storm, and a crashed coordinator resumes from the persisted queue instead of losing the generation. Put a reference in the job (a lineage and generation id, or a prompt-cache key) and have your worker build the full model request; to call the model API directly, submit the complete request instead. Request bodies are capped at 1MB and stored results at 64KB by default (`AQUIFER_MAX_BODY_BYTES`, `AQUIFER_RESULT_MAX_BYTES`): check `body_truncated`, and store large outputs elsewhere with a reference in the result. Keep each child's submission state in the coordinator: a child refused with `429` under backlog pressure wasn't queued, so resubmit it after `Retry-After` with the same `user_id` and `idempotent_key`, and set `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0` if a crash must not lose accepted children. Have your worker return the model API's `429` and `Retry-After` as they are, not wrapped in a `200`, so Aquifer slows down, and treat a child that still fails after its retries (`max_retries`) as a result the coordinator must handle. Not benchmarked yet. **Sleep through partial outages.** On Fly.io, set `AQUIFER_FLY_REGIONS` and that same overload signal drives real cross-region redirect: other regions Aquifer is deployed to get tried live, over Fly's private network, before this instance ever falls back to its own local queue, nearest-first by measured latency, deterministic enough that two callers racing the same job converge on the same region instead of each chasing their own nearest option. One region degrading routes around itself instead of paging you at 3am. If every known region is down too, that's a real fleet-wide problem, and Aquifer says so with a `429` and a long `Retry-After` rather than quietly queueing it somewhere and hoping. See [`POST /proxy`](https://rjpruitt16.github.io/aquifer/api.md#post-proxy)'s "Cross-region redirect" section for the full mechanics, including the one honestly-documented tradeoff it doesn't try to hide. In all of these, the target API can lower the dispatch pace via response headers. See [Dynamic Pacing](https://rjpruitt16.github.io/aquifer/pacing.md) for how the ceiling, backoff, and recovery actually work. Long-term protocol goal: if more services emit `X-Aqueduct-*`, agents can respond to capacity signals instead of independently guessing retry and concurrency behavior. Aquifer works today without ecosystem adoption; broader protocol adoption is the longer-term goal. --- # Dynamic pacing > How the target API sets the pace with X-Aqueduct-* headers: per-tenant queues, fair admission, sizing the backlog budget, slow start and the ORCA fallback. **Terminology**: Aquifer is this implementation; Aqueduct is the implementation-agnostic header protocol (`X-Aqueduct-*`) it speaks, so other services could speak it too. The target API controls pace at runtime via response headers (`X-Aqueduct-Rps`, `X-Aqueduct-Max-Concurrent`, and per-tenant queue isolation), and Aquifer honors a lower pace immediately, recovering gradually once pressure clears. For backends that can't speak Aqueduct directly, Aquifer also reads the real open [ORCA](https://github.com/cncf/xds/blob/main/xds/data/orca/v3/orca_load_report.proto) standard as a fallback signal: vLLM and Triton/TensorRT-LLM both work today, verified against their actual source. ![Traditional circuit breaking rejecting requests outright, forcing clients to retry independently into a thundering herd on reopen, versus Aqueduct's per-client queues pacing release via X-Aqueduct-Rps until the backend catches up](https://rjpruitt16.github.io/aquifer/images/circuit-breaking-vs-queue.gif) | Header (preferred) | Alias | Effect | |------------------------------|-------------------------------|------------------------------------------| | `X-Aqueduct-Rps` | `X-Aquifer-Rps` | Set this tenant's rate and share; the highest one advertised is the target's total | | `X-Aqueduct-Max-Concurrent` | `X-Aquifer-Max-Concurrent` | Reduce max in-flight requests | | `X-Aqueduct-Account-Queue` | `X-Aquifer-Account-Queue` | `enabled` / `disabled`: one queue per tenant, or one shared queue (has the last say) | | `X-Aqueduct-Slow-Start` | `X-Aquifer-Slow-Start` | `true`: new queues ramp up instead of firing at full rate immediately | | `X-Aqueduct-Max-Backlog` | `X-Aquifer-Max-Backlog` | Set the shared backlog budget used by fair admission | Aquifer reads both namespaces, preferring `X-Aqueduct-*` when both are present. By default each `(user_id, api_key)` pair gets its own independently paced queue at every target API, so one tenant's burst, or one tenant's broken endpoint retrying, can't slow down another: a failed attempt gives its concurrency slot back and waits out its backoff in that tenant's own queue. Who decides, in order: 1. **The target API.** Answering with `X-Aqueduct-Account-Queue: disabled` (or `false`) puts everyone sending to it in one shared queue, for example an API that needs one global order. `enabled` (or `true`) gives everyone their own. Aquifer remembers the choice per target API. 2. **The node owner.** `AQUIFER_ACCOUNT_QUEUE=on` (the default) or `off`. 3. **The origin** (whoever sends the job), with the same header on its request, can ask for `enabled` when the node owner chose `off`, but can never turn isolation off. The mode is per target API, so one origin switching it off would take isolation away from every other tenant. Each queue's pace still stays inside the target API's actual budget: before dispatching, every tenant queue takes a slot from one shared pace per target API, set to the configured `rps` (or a pool's live capacity), so the target never sees more than that in total, however many tenants are active. The queues still take turns for those slots, so isolation never means an unbounded copy of the full rate per tenant. `max_concurrent` is still per queue: with N active tenants, up to N × `max_concurrent` requests can be in flight. When that shared backlog crosses 70% of `X-Aqueduct-Max-Backlog`, queues holding more than their current fair share receive progressively more `429` responses while smaller queues continue entering, even once the budget is full (up to 125% of it). One active queue can still use the whole budget. See [API.md](https://rjpruitt16.github.io/aquifer/api.md#fair-queue-admission) for the formula, response headers, and scope. **Size `X-Aqueduct-Max-Backlog` in seconds of work, not as a job count.** Pick how long Aquifer should be able to hold work for the target API, at least the longest outage you want callers to ride out, and send `rate × seconds`, where `rate` is the peak rate jobs for that target API arrive at this node, across all tenants: during an outage, intake fills the budget, not dispatch. The budget is per node. A budget that's too small turns an outage into a wall of `429`s: in the [Hatchet outage benchmark](https://github.com/rjpruitt16/aqueduct-benchmarks), a 60-second budget at 20 rps (1,200 jobs) meant a 5-minute database outage refused about 25,000 submissions, and small tenants' recovery p99 was 107 s, against 3.8 s behind Envoy. A 600-second budget let the same small tenants through with no refusals at all, at a 5.5 s p99. If the advertised rate falls during an outage, keep the budget from falling with it below what Aquifer already holds; otherwise every queue is over budget at once. A backend can lower RPS at any time via these headers when it's under pressure; Aquifer honors the lower pace immediately and recovers gradually toward the configured ceiling once pressure clears. With `X-Aqueduct-Slow-Start: true`, a new queue starts at a low floor rate instead of its full configured rate and climbs toward that ceiling using the same gradual-recovery mechanism above, rather than firing at full speed from its very first dispatch. This applies per domain: since a queue's first-ever dispatch has no prior response to read the signal from, the setting takes effect on the *next* new queue created for that domain once any response has carried it, not the request that carried the header itself, and not retroactively for queues already running. Use the pacing headers for intentional backpressure. A `5xx` response is treated as a failed dispatch attempt and, for pool members, lowers that member's reputation. If a service is alive but overloaded, prefer `429` and/or `X-Aqueduct-Rps` / `X-Aqueduct-Max-Concurrent` so Aquifer slows down without interpreting the member as broken. **ORCA fallback for backends that can't speak Aqueduct directly.** Some backends already report load in a different, real open standard: ORCA (Open Request Cost Aggregation), the gRPC/Envoy ecosystem's convention for backends to report utilization. Aquifer sends `endpoint-load-metrics-format: text` on every dispatch (the request-side opt-in both verified backends require; there's no server startup flag for this), and a backend that understands it replies with an `endpoint-load-metrics` header carrying a KV-cache utilization fraction. If a response carries no `X-Aqueduct-Rps`/`X-Aquifer-Rps`, Aquifer reads this header as a fallback and paces down as utilization rises: full configured rate below 70%, 2 RPS at 70-90%, 0.5 RPS at 90-97%, 0.25 RPS above that, never dropping to zero, same pacing-down-gracefully philosophy as everywhere else. An explicit `X-Aqueduct-Rps` always wins if present; this only fires when the backend hasn't opted into speaking Aqueduct's own headers. Two backends verified directly against their own source, not assumed: **vLLM** (`vllm/entrypoints/serve/utils/orca_metrics.py`, metric name `kv_cache_usage_perc`, case-insensitive opt-in) and **Triton/TensorRT-LLM** (`src/orca_http.cc`, metric name `kv_cache_utilization`, case-sensitive lowercase-only opt-in; Aquifer sends lowercase specifically so both work). Aquifer tries both known metric names, so whichever backend you're running, this works without any configuration. ## Dynamic priority Most queues that support priority read it from a static file: a tier per customer, or a list of user IDs that go first. That works until the important user changes faster than someone can edit the file. Aquifer has no priority setting. Every tenant takes equal turns until the target says otherwise, and the target says otherwise on any response, with the same `X-Aqueduct-Rps` header it already uses to set its pace. **The target sets its total.** A target's queues together never go past the highest `X-Aqueduct-Rps` it is currently advertising to any active tenant (one that dispatched in the last 3 seconds). The configured rate applies only while it advertises nothing, so a backend can run faster or slower than its config on headers alone. A backend that stamps one number on every response gets that number in total, not per tenant. A pool's total is its live capacity, which its members advertise. **It sets each tenant's share the same way.** When several tenants are backlogged, the total is split in proportion to the rate the target last sent each one. Advertise `600` to one tenant, `60` to a second and `6` to a third, and the total is 600/s, split 90%, 9% and 1%. Tenants advertised the same rate take equal turns. A tenant that has never been sent a rate, or whose last one a failure cleared, is weighted by its own current pace instead, which failures halve and slow start lowers, so with no headers at all, healthy tenants take equal turns and a failing one takes fewer. Rates have a floor of 0.5 requests a second, per tenant and in total: a target advertising less still gets 0.5, so a struggling target always sees enough traffic to show it has recovered. A tenant that asks for less than its share keeps its own pace, and the rest goes to the others. **That allows deliberately unfair processing when it matters.** Fair turns are the right default, but some moments aren't fair: - A flood warning goes out and one county's alert jobs have to reach phones before anything else does. - A security team is responding to a breach, and its log queries and key rotations can't wait behind routine scans. - A hospital's monitoring alerts arrive during a surge, and a patient's alert can't sit behind billing exports. In each case the backend already knows which tenant is in trouble, often before any human does. It advertises that tenant a much higher rate, say `1000` next to `10` for each of ten others, and from the next response on that tenant gets 1000/1100 of the target, about 91%, while everyone else slows down but keeps moving. When the emergency ends, the backend advertises that tenant its normal rate again: from its next response the turns are back to normal, and the total falls to the highest rate still advertised. Send the lower rate rather than dropping the header, because an advertised rate stays in force until the target sends a new one or a failure clears it; a response without the header keeps it. A tenant that goes idle stops counting toward the total after 3 seconds. Nothing was configured, and nobody had to deploy anything. **A real recovery, done by hand.** On May 20, 2026, Railway was recovering a backed-up build queue after a cloud outage. To keep paid builds moving, they paused free, trial and hobby builds, resumed them, and repeated that four times over about ten hours, while a hobby customer called it unfair ([status page](https://status.railway.com/incident/KVZ1Z8GY)). With Aquifer, the build service would advertise a high rate to paid workspaces and a low one to the rest, raising it as the backlog drained until every workspace was back to equal turns, with no switches to flip and no one watching. The rate a client submits never counts: only the target's responses can set pace or priority, so a tenant can't make itself the emergency. ### More than one node Nodes don't coordinate, so each one applies the rates it's sent on its own: three nodes each advertised `20` send up to 60/s in total. To hold the whole cluster to a budget, the target divides it by the number of nodes sending to it, counting the distinct `X-Aqueduct-Node-Id` values it has seen in the last few seconds, and scales every rate it advertises by that share: a budget of 20/s across three nodes means advertising each node's top tenant `6.7`, with the others in proportion. When a node comes online the count goes up and every node slows down on its next response; when one crashes its id stops arriving, and the rest speed back up. The split is even, so a node with little work leaves part of its share unused. Three things keep the count honest: - Each node needs its own `AQUIFER_NODE_ID` (by default the Fly machine id or host name). Nodes sharing an id count as one and each send the whole budget. - Count ids only from Aquifer nodes you trust, over a private network or authenticated requests; otherwise a caller could forge ids and push your real nodes' rates down. - A new rate reaches each queue on its next response, and a node that hasn't been sent one runs at its configured rate. So the total can briefly overshoot after a node joins or restarts; set configured rates to the per-node share, and put a limiter in front of the target if the budget must never be exceeded. --- # Streaming > Stay on the line over SSE or /proxy, or hang up and get a webhook; when the webhook is skipped. After submitting a job you choose how to get the result. Stay on the line with `GET /jobs/:id/stream` (or call `POST /proxy`, which streams automatically when it has to queue), and watch your job move through the line in real time. Or hang up whenever you like: the result still arrives at your `webhook_url`, and it's stored for `GET /jobs/:id` too. ``` event: queued data: {"job_id":"abc123","status":"queued"} event: position data: {"job_id":"abc123","position":3} event: dispatching data: {"job_id":"abc123","attempt":1} event: completed data: {"job_id":"abc123","response_status":200,"body":"..."} ``` The webhook is only a fallback. If you stayed on the line and received the final event (or a `/proxy` direct response), Aquifer doesn't send it, so you never pay for the same result twice. If you hung up first, or never streamed, the webhook goes out as usual. A client that disappears without closing its connection (a dropped network, a sleeping laptop) can't be told apart from one that's still listening, so it may miss both; that's what the stored result on `GET /jobs/:id` is for. --- # Pools > Load balancing across registered members, weighted by declared capacity and live reputation. Instead of dispatching to a fixed `url`, a job can target a named **pool**: a group of registered service instances Aquifer picks from at dispatch time, weighted by declared capacity and live reputation. Useful when you have several interchangeable backends instead of one fixed endpoint, and it grows or shrinks automatically as members register, degrade, or drop out, with no need to reconfigure Aquifer as your fleet autoscales. **Registering a member:** ```bash curl -X POST https://your-aquifer/pools/writers/members \ -d '{"member_id": "writer-1", "address": "http://10.0.1.5:8080", "capacity_rps": 20, "heartbeat_interval_seconds": 30}' ``` The same call is both initial registration and heartbeat: call it again periodically (at roughly your declared `heartbeat_interval_seconds`) to stay in the pool. Missing several consecutive expected heartbeats evicts a member. A member can register under more than one pool id. **Dispatching to a pool:** ```json { "user_id": "user-123", "idempotent_key": "job-1", "pool_id": "writers", "method": "POST", "webhook_url": "https://yourapp.com/webhooks/aquifer" } ``` `pool_id` and `url` are mutually exclusive: a job sets exactly one. **How a member gets picked:** proportional to `capacity_rps × reputation`, not equal-split round robin: a member declaring 100 RPS gets roughly 4x the dispatches of one declaring 25. The pool's aggregate ceiling is the live sum of every member's current effective rate, so it grows and shrinks automatically as members register, degrade, or drop out, with no need to reconfigure Aquifer as your fleet autoscales. **Reputation**: a dispatch failure halves a member's effective share; a successful dispatch nudges it back up, and heartbeats recover it more slowly after a restart. A member isn't evicted on one bad response, only once its reputation has stayed at the floor continuously, with no interrupting success, for a sustained window. This avoids flapping a member in and out of the pool over a single transient error. **Treat `5xx` carefully.** Aquifer interprets connection errors and `5xx` responses as reliability signals for the selected member. One `5xx` does not fail the job by itself: Aquifer records failure for that member and retries another member when possible. If every retry across the pool still ends in connection errors or `5xx`, the job is marked failed. That behavior is intentional for reliability, but it means application bugs that accidentally return `5xx` on a new code path can reduce that member's traffic share or eventually remove it from the pool. For overload, prefer explicit backpressure over generic server errors: | Situation | Recommended signal | |-----------|--------------------| | Instance is healthy but needs less traffic | `X-Aqueduct-Rps` or `X-Aqueduct-Max-Concurrent` | | Request should be retried later due to pressure | `429` with `Retry-After` | | Instance/code path is actually failing | `5xx` | Roll new members into a pool gradually. Start new versions with a conservative `capacity_rps`, send a small share of traffic first, watch `/health` reputation and your own error metrics, then raise capacity as confidence grows. Blue/green or canary rollout matters more here than with a blind round-robin balancer because Aquifer uses runtime failures as routing input. **Set `capacity_rps` conservatively, not at your true theoretical max.** Aquifer only learns a member died via a failed dispatch or a missed heartbeat, both of which lag the actual failure, so leaving headroom in what you declare gives real slack for that detection delay. Reputation decay is a second line of defense on top of this: a member that's silently struggling gets throttled down by observed failures even if its last-declared capacity was optimistic. **Watch the model locally:** Aquifer includes a local scenario harness that starts one fake backend server with multiple logical workers, registers them as pool members, and prints per-second traffic, failures, dynamic header values, and reputation. ```bash go run ./cmd/aquifer-scenario --scenario mixed --workers 10 --jobs 500 --duration 30s --rps 50 ``` Scenarios: `steady`, `weighted`, `flapping`, `backpressure`, `recovering`, `mixed`, and `harsh`. The `harsh` scenario penalizes sustained overload by adding latency, then `5xx`, then simulated crash windows. Add `--mode regular` to compare against a simple round-robin load balancer model that retries on `5xx` but ignores Aquifer reputation and dynamic pacing headers. Pool state isn't shared across Aquifer instances; see [Deployment model](https://rjpruitt16.github.io/aquifer/deployment.md) for how that constrains a given `pool_id` to one instance. `GET /health` reports every pool's current members, their declared capacity, and current reputation. --- # Partitioning > Static partitioning, regional cluster routing, drain mode and external registration. Running one instance for everything works fine until you have multiple tenants or multiple target APIs sharing it, and then one tenant's burst, or one target API's own rate limit, ends up affecting everyone else on that same instance. Two ways to split traffic apart so that doesn't happen, not mutually exclusive: **Static partitioning**: decided once, at deploy time. Dedicate one instance to a single protected resource (a CI runner, a database, a GPU, or a rate-limited external API you want to be nice to) so that resource only ever sees traffic paced the way you configured, up to whatever it can actually bear. Multiple tenants can safely share that same instance: turn on [account-queue isolation](https://rjpruitt16.github.io/aquifer/pacing.md) and each tenant gets their own independently-paced queue, so one tenant's burst doesn't starve another's, and the resource itself never sees more aggregate load than it's rated for. The mistake to avoid: pointing multiple *instances* at the *same* resource instead of routing everyone through this one pacing checkpoint, which just multiplies your total request rate against it. Same rule for pools: a given `pool_id` should belong to exactly one instance, since pool state isn't shared across instances. Optional regional cluster routing supports either a static rendezvous member list or Valkey-coordinated, load-aware rendezvous. The Valkey mode discovers Aquifer instances, keeps active users sticky while capacity is available, and releases idle assignments without requiring another control plane. See [API.md](https://rjpruitt16.github.io/aquifer/api.md#regional-cluster-routing) for failure semantics and configuration. **Dynamic partitioning (drain mode)**: off by default, for a more specific shape. Instead of deciding every assignment up front, an instance gets handed to one tenant at a time, absorbs and drains whatever burst that tenant sends, then frees itself up to be handed to a *different* tenant next, useful when you want dedicated capacity per user without hand-assigning it at deploy time. Aquifer can stream completed/failed job ledger events in acknowledged batches to a webhook or Valkey, and when idle for `AQUIFER_DRAIN_TIMER_SECONDS`, it flushes anything remaining before clearing local state and moving through an `active` → `draining` → `unassigned` state machine visible via `GET /health`. See **[DRAIN_MODE.md](https://rjpruitt16.github.io/aquifer/drain-mode.md)** for the full state machine, env vars, and payload shape. The two combine: a fleet can partition statically by target API domain, while individual instances within a partition cycle through tenants dynamically via drain mode. **External registration**: off by default, and orthogonal to the above. `AQUIFER_REGISTRY_URL` makes an instance periodically report its own listening port to an external control plane (deciding tenant assignment, scaling, etc. is entirely that service's job, not Aquifer's). See **[REGISTRATION.md](https://rjpruitt16.github.io/aquifer/registration.md)** for the env vars and ping payload shape. None of this needs instances sharing state centrally, and that's the normal shape for a load balancer, not a gap unique to Aquifer: nginx and HAProxy make local decisions the same way. Pure central rate limiting is a gateway-layer concern that composes in front of Aquifer if you want it, not something Aquifer needs to reinvent. What an instance does need is to be safe to hand off without knowing anything about the rest of the fleet, which is what [slow start](https://rjpruitt16.github.io/aquifer/pacing.md) is for: a freshly-assigned instance starts below its configured ceiling and creeps up, rather than assuming the ramp some other instance already earned. --- # Deployment > Sidecar, standalone, embedded or behind a Gateway API proxy; choosing a machine size. Aquifer runs four ways: as a **sidecar** alongside your app, as a **standalone service** multiple services point to, **embedded directly as a Go library** in your own process (see [Framework adapters](https://rjpruitt16.github.io/aquifer/adapters.md)), or as an **extension behind a Gateway API proxy** like Envoy Gateway in Kubernetes, where the proxy owns routing and TLS and Aquifer owns the queue behind it (see [examples/kubernetes](https://github.com/rjpruitt16/aquifer/blob/main/examples/kubernetes)). Each instance persists to its own volume (a Pebble directory by default, or a SQLite file when `AQUIFER_STORE_BACKEND=sqlite` or `DB_PATH` already holds a SQLite database), with no external database or coordination service to run. People run Aquifer in front of things like: internal coding platforms (GitLab, Forgejo), CI runners, database read replicas, and MCP servers, anywhere a burst of agent or service traffic needs to hit something that has its own capacity limit. See [Partitioning strategies](https://rjpruitt16.github.io/aquifer/partitioning.md) above for how to assign tenants to instances, statically or dynamically. See the [security warning](https://rjpruitt16.github.io/aquifer/api.md#post-jobs) under `POST /jobs`; the same untrusted-caller risk applies regardless of deployment shape. **Choosing a machine size:** with Pebble and the default 100ms flush, one Fly machine sustained about 1,000 jobs/s on performance-1x, 1,500 on 2x and 3,000 on 4x, end to end including webhooks. See [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md#12-capacity-per-machine-pebble-flyio-2026-10-07) for the method and caveats, and [section 7](https://rjpruitt16.github.io/aquifer/benchmarks.md#7-capacity-and-drain-time-sqlite-before-v0130) for earlier capacity work. --- # Integrations > Inlet, for provider webhook storms (GitHub, Shopify, Stripe, Anthropic). ## Inlet: webhook storms **[Inlet](https://github.com/rjpruitt16/inlet)** is the easy way to put provider webhooks in front of Aquifer without exposing Aquifer to the internet. It runs as an AWS Lambda (or a Cloudflare Worker). For each webhook it: 1. Verifies the provider's signature. 2. Durably queues the event before acknowledging the provider. 3. Pours the event into Aquifer's `POST /jobs`. Each event uses the sender as `user_id` and the provider's event ID as the idempotency key. Aquifer's fairness and pacing then keep one noisy shop or installation from swamping your handlers. | Provider | Default fairness identity | |---|---| | GitHub | installation, organization, repository owner, or sender | | Shopify | shop domain | | Stripe | event account | | Anthropic | workspace or organization | Deploy one Inlet stack per provider. See the [Inlet README](https://github.com/rjpruitt16/inlet#readme) for setup. --- # L8 protocol > Webhook delivery signed with public keys, optional payload encryption and request schemas. Traditional webhook security shares an HMAC secret between sender and receiver, stored in a database on both sides: something that can be stolen, logged accidentally, or forgotten during rotation, letting anyone forge deliveries forever once it leaks. Aquifer implements **L8 v0.2**, a lightweight challenge-response protocol that replaces the shared secret with public key cryptography: the receiver publishes a public key, a one-time handshake proves both sides own their private keys, and every delivery afterward carries a signature verified locally in microseconds, with no database lookup and no round-trip to any authority. Two additions in 0.2. A receiver that publishes an X25519 `encryption_public_key` gets its webhook bodies encrypted to that key, then signed, so a proxy or log in the middle sees only ciphertext. A target API that publishes `request_schemas` (JSON Schema per route) gets job bodies checked before dispatch: a mismatch is rejected with `422` instead of burning a target API call. Schema checks are off unless `AQUIFER_L8_SCHEMA_VALIDATION=true`. The full protocol rationale, wire format, and a reference receiver implementation live at the **[L8 spec](https://rjpruitt16.github.io/l8-protocol/)**, also served locally at `GET /l8-spec` for an agent/script with only network access to this instance. Set `L8_PRIVATE_KEY` for a stable identity across restarts, or let Aquifer auto-generate one on first start. --- # Reliability > The durable queue, crash and panic recovery, and job TTLs. Durable queue, automatic crash recovery, panic isolation per job. See [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md) for the numbers behind these claims. - **Durable queue**: jobs persist to the configured storage backend. With Pebble's default `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=100`, Aquifer answers before a write reaches disk, so a crash loses the jobs accepted since the last completed sync; set it to `0` for every write to wait for its own fsync (see [Configuration](https://rjpruitt16.github.io/aquifer/configuration.md)) - **Crash recovery**: queued jobs re-dispatched automatically on restart - **Panic recovery**: if a queue's loop panics, its unfinished jobs are re-enqueued from storage immediately, without waiting for a restart - **Stale job safety net**: in-flight jobs older than 5 min automatically reset to `queued` - **Per-job panic isolation**: a panic in one job marks it failed and delivers the webhook; the worker keeps running **Job TTLs:** | Status | TTL | |-------------|--------| | `queued` | 24 h | | `completed` | 30 min | | `failed` | 2 h | --- # API overview > The POST /jobs request and the warning about untrusted callers. ```http POST /jobs ``` ```json { "user_id": "user-123", "idempotent_key": "invoice-42-notify", "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "headers": { "Authorization": "Bearer sk-..." }, "body": "{\"model\":\"gpt-4o\",\"messages\":[...]}", "webhook_url": "https://yourapp.com/webhooks/aquifer" } ``` **Do not expose Aquifer directly to untrusted callers.** `url` is dispatched as a real HTTP request. If an arbitrary or untrusted party can set it, Aquifer becomes an open relay/SSRF vector, using Aquifer's own network position and identity to reach anything the machine can reach. The intended caller is **your own trusted backend or gateway code** dispatching to a destination it already knows about, not an agent, end user, or any other party choosing the destination itself. **[API.md](https://rjpruitt16.github.io/aquifer/api.md)** has the full reference: `GET /jobs/:id`, the SSE stream, `POST /proxy` (capacity-aware reverse-proxy mode, see [Use cases](https://rjpruitt16.github.io/aquifer/use-cases.md)), health/readiness and graceful shutdown, webhook payload shapes, and the autoscaling headers. --- # API reference > Every endpoint: jobs, /proxy, streams, WebSockets, health, metrics, webhooks, admission and cluster routing. ## POST /jobs ```json { "user_id": "user-123", "idempotent_key": "invoice-42-notify", "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "headers": { "Authorization": "Bearer sk-..." }, "body": "{\"model\":\"gpt-4o\",\"messages\":[...]}", "webhook_url": "https://yourapp.com/webhooks/aquifer", "execute_before": 1798761600000 } ``` `execute_before` is optional Unix time in milliseconds. Aquifer checks it immediately before dispatch and before every retry. A job that has not started its next attempt by that time is never sent target API; it becomes `failed` with reason `execution_deadline_exceeded`, remains available through the normal status/result retention window, and emits the normal failed SSE/webhook event. A job without `execute_before` that is still queued 24 hours after submission fails the same way, with reason `queued_ttl_exceeded`, instead of disappearing when its stored record expires. `webhook_url` is optional. Omit it when the caller only needs Aquifer to execute the job and will use `GET /jobs/{id}` or remote result storage if it needs the terminal result. When set, Aquifer delivers the normal completion webhook. **Do not expose Aquifer directly to untrusted callers.** `url` is dispatched as a real HTTP request — if an arbitrary or untrusted party can set it, Aquifer becomes an open relay/SSRF vector: it can be pointed at your internal network, cloud metadata endpoints (`169.254.169.254`), or anything else the machine Aquifer runs on can reach, using Aquifer's own network position and identity. The intended caller is **your own trusted backend or gateway code** dispatching to a destination it already knows about — not an agent, end user, or any other party choosing the destination itself. Run Aquifer on a private network, not bound to a public address, and put your own authorization and destination allow-listing in front if agents need to reach it indirectly. As a second, narrower layer on top of that — `AQUIFER_ALLOWED_URL_DOMAINS` (comma-separated hostnames, e.g. `api.openai.com,internal.yourapp.com`) restricts which domains `url` is allowed to target at all; a subdomain of an allowed entry is permitted (`api.example.com` matches an allowlisted `example.com`). Unset by default — unrestricted, unchanged behavior — this doesn't replace the network/authorization guidance above, it's what Aquifer itself can enforce regardless of who's calling it. Applies to `url`-routed jobs only; pool-routed jobs (`pool_id`) have no caller-supplied destination to check. Idempotent — duplicate `idempotent_key` per `user_id` returns the existing job. **201** new job queued · **200 + `"duplicate": true`** already exists **Shared idempotency scope.** Set `"idempotency_scope": "shared"` to dedup on `idempotent_key` alone, across every `user_id`. Many agents asking for the same resource then coalesce onto one target API call: the first request creates the job, later ones get `200 + "duplicate": true` with the same `job_id`. On `/proxy` they are streamed that job's result; with remote results enabled, `GET /results?idempotency_scope=shared&idempotent_key=...` serves the cached response to the whole pool. Followers do not get their own webhook. Requires `AQUIFER_SHARED_IDEMPOTENCY_ENABLED=true` (default `false`); otherwise `"shared"` is rejected with `400`. Anyone who knows a shared key can read its result, so use keys that name a public resource (`weather:sf:2026-10-05`), never one that carries a user's private data. In a cluster, shared requests are routed by the shared key instead of `user_id`, so every caller for that key reaches the same owner node and coalesces there; per-user requests keep routing by `user_id`. Independent instances that share only Valkey (not one cluster) can each run the job once before the first result lands; the last write to Valkey wins. **422** the target API's L8 `request_schemas` rejects the body (see [Request schemas](#request-schemas-l8-02)). ### Best practice: confirm batched webhook intake For high-throughput webhook intake, keep Pebble's default 100 ms batched flush and use Aquifer's idempotency as a confirmation retry. For Shopify, Stripe, GitHub and the other providers [Inlet](https://github.com/rjpruitt16/inlet) supports, use Inlet instead: it acknowledges the provider after its own durable enqueue. 1. Verify the provider signature, use its event ID as `idempotent_key`, and keep the provider request open. 2. Submit the event to Aquifer. Continue only after Aquifer returns **200** or **201**. 3. Wait with jitter, well inside the provider's response deadline (about 1 s for Shopify's 5 s), then submit the exact same request again. 4. A **200** response with `"duplicate": true` confirms that Aquifer still knows the accepted job. Return `2xx` to the provider. 5. If the confirmation returns **201**, the earlier job was absent and this request created its replacement. Wait and confirm again before acknowledging the provider. 6. If the intake process dies first, it never acknowledges the provider, so the provider's normal retry sends the event again. Both submissions must use the same `user_id` and `idempotent_key` and reach the same Aquifer owner. Aquifer's cluster routing provides that ownership for the same routing key; independent unclustered instances behind a random load balancer do not. Jitter prevents a burst of original webhooks from producing a second synchronized burst four seconds later. This adds one inexpensive idempotency lookup per event while preserving batched write throughput. It substantially narrows the default crash-loss window, but `duplicate:true` is not an fsync receipt. Set `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0` when every accepted event requires a strict disk-sync guarantee, or when the provider timeout is too short to complete the confirmation. ### Retries Connection errors, `5xx`, `408` and `429` are retried; any other response (including other `4xx`) is final. Set the limit per job with `"max_retries"` in the body or the `X-Aqueduct-Max-Retries` request header (the header wins): | Value | Behavior | |---|---| | _(unset)_ | 4 retries (5 attempts) | | `0` | no retries | | `1`-`100` | that many retries | | `-1` | retry until the job succeeds, stopping at `execute_before` if set, otherwise 24 hours after submission | ```bash curl -X POST localhost:8080/jobs -H "X-Aqueduct-Max-Retries: -1" -d '{ ... }' ``` `-1` is meant for deliveries that must eventually land, such as forwarding webhooks to a receiver that may be down for a while. It is never the default. How retries are paced: - **Per job:** the wait before each retry doubles (1 s, 2 s, 4 s, ...) up to `AQUIFER_RETRY_MAX_BACKOFF_SECONDS` (default `300`), with jitter. A `Retry-After` header on the failed response is honored instead. - **Per queue:** every retryable failure halves that account queue's dispatch rate (down to 0.5 rps); successes raise it again by 5% each. An `X-Aqueduct-Rps` on a successful response sets the rate directly, even above the configured rate, and that tenant's share of the target's total when tenants contend; the highest rate advertised to an active tenant is the target's total ([pacing](https://rjpruitt16.github.io/aquifer/pacing.md)). On a failed response it can only lower it, so a server that stamps the same rate on every response, `503`s included, still gets backed off while it's failing. - A job waiting out its backoff does not hold a concurrency slot, so other jobs in the same queue keep moving. The attempt count is persisted, so a restart doesn't reset it. **Webhook retries** are set separately, with `"webhook_max_retries"` in the body or the `X-Aqueduct-Webhook-Max-Retries` header (the header wins). It takes the same values as `max_retries`, applies only to delivering this job's webhook, and defaults to 10 retries (about 13 minutes with the default backoff, enough to ride out a receiver deploy; `max_retries` defaults to 4). The two are separate because they go to different places: a non-idempotent target API call can be tried once while its webhook keeps retrying through a receiver outage. ```bash curl -X POST localhost:8080/jobs -H "X-Aqueduct-Max-Retries: 0" -H "X-Aqueduct-Webhook-Max-Retries: -1" -d '{ ... }' ``` With `-1`, delivery keeps retrying for up to 24 hours after the job finished. The webhook delivery is stored like any job, so it survives a restart, and it skips admission control: the work it reports on was already accepted and done. If it still never lands, the result stays readable at `GET /jobs/{id}`. Each retry emits a `retrying` SSE event (`attempt`, `max_retries`, `retry_at`, `reason`, `response_status`). The final `failed` event and webhook report the last failure; a job that still had retries left when its deadline arrived reports `retry_window_exhausted: `. ### Fair queue admission Each target API has a soft shared backlog budget `B`, defaulting to `AQUIFER_MAX_PENDING_PER_UPSTREAM=10000`. The target API may update it at runtime with `X-Aqueduct-Max-Backlog` (or `X-Aquifer-Max-Backlog`); `0` disables this count-based limit. Dynamic values are learned from direct proxy responses and queued dispatch responses. With AccountQueue mode enabled (the default, see the README's pacing reference for who can change it), Aquifer gives idle capacity to whoever can use it while protecting smaller queues once the target API becomes congested. For a prospective request, let `q` be its queue's projected backlog, `Q` the projected target API backlog, and `N` the projected number of active account queues: ```text F = B / N pressure = clamp((Q - 0.70B) / 0.30B, 0, 1) excess = clamp((q - F) / F, 0, 1) P(429) = pressure * excess ``` A lone queue is never fairness-rejected and may use the whole budget. A queue at or below `F` is not fairness-rejected. Above 70% total pressure, queues above `F` receive progressively more `429` responses, and a queue at `2F` or more is refused every time once `Q` reaches `B`; when competitors drain, their capacity becomes borrowable again. If admitting a request would put `Q` above `B`, it is rejected deterministically unless its queue is at or below `F` and there are other active queues: those keep entering, so one large queue that fills the budget doesn't lock out small ones, until `Q` reaches `1.25B`, past which every request is rejected. If the target API lowers `B` with `X-Aqueduct-Max-Backlog` while Aquifer already holds more than that, the ceiling is 1.25 times the backlog held at that moment instead, until the backlog drains below the new `B`: queues above their share of the new budget are refused, and queues at or below it keep entering. Without AccountQueue mode, traffic remains one shared queue (`N=1`), so only the shared backlog ceiling applies. Duplicates, recovered work, and internal webhook deliveries bypass this admission decision; accepted jobs are never discarded by it. Accepted and fairness-rejected responses report `X-Aqueduct-Active-Queues`, `X-Aqueduct-Upstream-Backlog`, `X-Aqueduct-Queue-Backlog`, and `X-Aqueduct-Admission-Pressure` (with `X-Aquifer-*` aliases). Outbound dispatches report active queues and target API backlog to the backend alongside the existing load headers. These values cover the relevant target API on the serving node. `GET /health` reports node-wide local totals under `queues`; it is deliberately not presented as a fleet-wide count. Every response also carries `X-Aqueduct-Fullness`: how full this instance is, `0`–`100`. It's the most-constrained of the limits the instance already enforces: memory against `AQUIFER_MEMORY_LIMIT_MB`, the database against `AQUIFER_DB_MAX_BYTES`, and the fullest target API's backlog against its backlog budget (`AQUIFER_MAX_PENDING_PER_UPSTREAM` or the target API's own `X-Aqueduct-Max-Backlog`). Limits that are switched off don't count. Unlike `X-Aqueduct-Admission-Pressure`, which stays at `0` until a backlog reaches fair admission's 70% starting point, fullness rises from the first queued job, so whatever sits in front of the instance (a gateway like Canalis, or a client) can see it filling up before it starts pushing back, and decide for itself when to stop sending it new work. The same number is in the registration ping as `fullness` ([REGISTRATION.md](https://rjpruitt16.github.io/aquifer/registration.md)). Sampled at most every 250ms. Every response also carries the instance's load: `X-Aqueduct-Http-Users`, `X-Aqueduct-Http-Capacity`, `X-Aqueduct-WS-Sessions` and `X-Aqueduct-WS-Capacity`, the same numbers as the registration ping ([REGISTRATION.md](https://rjpruitt16.github.io/aquifer/registration.md#load-http-users-and-websocket-sessions)). ### Webhook backlog admission A user whose completion webhooks are piling up undelivered is consuming capacity faster than their receiver can absorb it. When other users are also active on the instance, new jobs from that user are rejected with **429** and `limit_reason: "webhook_backlog"` until they catch up: ```text W = AQUIFER_MAX_PENDING_WEBHOOKS_PER_USER (default 1000; 0 disables) w = the user's queued + in-flight webhook deliveries on this instance P(429) = clamp((w - W) / W, 0, 1) // 0 at or below W, certain at 2W ``` A user alone on the instance is never rejected for this, however large their backlog. The rule never drops accepted work or webhooks; it only slows new submissions. Counts are per instance, like `GET /health`. ### Inbound pacing (`X-Aqueduct-Rps`) Always on. Aquifer tells its own callers how fast to submit: every `/jobs` and `/proxy` response carries `X-Aqueduct-Rps` (alias `X-Aquifer-Rps`), the same header a target API uses to pace Aquifer. An Aquifer calling another Aquifer, or any client that honors the header, slows down before the server has to start returning `429`s. A caller that ignores the header loses nothing. Canalis honors it: it paces each tenant's hand-offs from its line by this header, so a burst of queued work reaches Aquifer at a rate its disk keeps up with instead of all at once. | Env var | Default | Description | |---|---:|---| | `AQUIFER_INBOUND_BENCHMARK_RPS` | `400` | Jobs per second you believe this machine can take end to end; the first guess at the knee | | `AQUIFER_INBOUND_START_PERCENT` | `40` | Where the advertised rate starts, as a percent of the benchmark | **How it moves.** Once a second the controller compares what callers sent against three signals: - **Insert latency.** The p95 time to durably store a job, against a healthy baseline (the lowest p95 seen, drifting up slowly). Above 1.5x the baseline plus 2ms means storage is saturated. - **Backlog growth.** Accepted work waiting to be dispatched. If it grows for 3 ticks in a row, Aquifer is accepting faster than it can dispatch, even when inserts still look fast. Intake and dispatch share the same storage writes, so this is usually the first signal to fire. - **Sync lag** (Pebble's default flush only). How long accepted jobs may have gone without reaching disk. Inserts don't wait for the disk, so their latency stays low however far behind it falls, and a crash loses everything since the last sync. Over 500ms (or two flush intervals, if `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS` is longer), the disk is behind. Admission starts refusing at `AQUIFER_MAX_SYNC_LAG_MS` (1s); slowing callers first keeps the lag from getting there. When any signal trips, the rate is cut by 30%, and the rate just under where it failed becomes the learned knee. Within one congestion episode it cuts once, then holds for up to 10 ticks while admitted work drains, so one slow patch doesn't drive the rate to the floor. While healthy, it grows 1.5x per tick below the knee and 5% per tick above it, and only when callers actually use at least 80% of what was advertised (unused headroom is never inflated). If callers send more than 1.5x the advertised rate, it holds instead of cutting: that is a client ignoring the header, and admission control handles it. The total is split evenly across active callers (any `user_id` seen in the last 3 seconds), so `X-Aqueduct-Rps` is a per-caller number. A direct `/proxy` response carries one value: the lower of Aquifer's and the target's own `X-Aqueduct-Rps`. **Tuning.** 1. **Measure the benchmark on the real machine.** Run your normal workload (or `make perf`, see [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md#11-per-job-overhead-make-perf)) on the same instance type and disk, and set `AQUIFER_INBOUND_BENCHMARK_RPS` to the sustained jobs/s where the backlog stays flat. Count whole jobs, not inserts: a job costs about 4 synced storage writes (accept, dispatch result, webhook), about 2 when the caller streams the result. On a Fly performance-1x with a volume, one client held about 170 jobs/s end to end, well under a 400 inserts/s micro-benchmark. 2. **Pick the start percent by how bursty callers are.** The default 40% ramps to the benchmark in about 3 seconds. Lower it (15 to 20) when one instance serves many callers and a cold start shouldn't flood it; raise it toward 80 for a single trusted caller that should get full speed right away. 3. **Watch `/health`.** With pacing on, `GET /health` reports `inbound`: ```json "inbound": { "advertised_rps": 168.2, "per_caller_rps": 56.1, "active_callers": 3, "inserts_per_sec": 171, "p95_insert_ms": 2.4, "baseline_insert_ms": 1.1, "benchmark_rps": 400, "threshold_rps": 181.0, "server_backlog": 212, "cuts": 4, "raises": 19, "last_adjustment": "raise" } ``` `threshold_rps` settling well below `benchmark_rps` means the benchmark is optimistic; lower it so the ramp stops overshooting. Repeated `cut: backlog growing` means dispatch, not intake, is the bottleneck: more target API capacity or more Aquifer instances (partition by `user_id`) helps; a faster disk alone won't. `hold: callers over advertised rate` means a caller is ignoring the header. 4. **Pacing is per instance.** Each node advertises its own capacity. To scale intake, add nodes and spread callers across them; one node's knee doesn't move because another node is idle. ## Regional cluster routing Cluster routing is optional HTTP-level partitioning for `POST /jobs` and `POST /proxy`. Callers may hit any Aquifer instance; the receiving node resolves an owner by `user_id`, forwards when necessary, and returns `X-Aquifer-Cluster-Owner` with the instance that handled the request. This changes where a user is processed, not how local account queues are grouped: `X-Aqueduct-Account-Queue` still controls whether a URL worker isolates `(user_id, api_key)` queues locally. Two providers are available: - `static` (default) builds a deterministic rendezvous ranking from `AQUIFER_CLUSTER_MEMBERS`. An unreachable peer is soft-pruned from that instance's local view and retried after the prune TTL. There is no shared assignment state. - `valkey` makes Aquifer instances heartbeat directly into a regional Valkey and atomically claims user assignments there. No separate control plane is required. Existing assignments stay sticky while their owner is active. For a new user, Aquifer walks the rendezvous ranking and chooses the first node below its active-user capacity. If every active node is full, it chooses the least-loaded node by `active_users / capacity` instead of rejecting work. When the owner can't be reached at all (connection refused, or the request never went out), the receiving node soft-prunes it and forwards to the next node in the ranking. When the owner received the whole request but didn't answer within 10 seconds, the receiving node does **not** fail over: the owner may already have accepted the job, and a second node would run it again. The caller gets `504` with `owner_id` and `Retry-After: 1`. Retrying with the same `idempotent_key` reaches the same owner, which returns the existing job if it accepted the first attempt. A slow owner costs its users some availability instead of running their jobs twice. Each forwarded request carries the time its sender will give up (`X-Aquifer-Cluster-Forward-Deadline`, Unix milliseconds). An owner that only gets to the request after that time, because it was stalled or frozen, answers `503` with `Retry-After: 1` and accepts nothing. Without this, an owner stalled longer than the instance TTL could lose the user to another node, which accepts the retry, and then run the original request too. The check compares clocks across nodes, so keep them NTP-synced; a peer on an older version that sends no deadline is still accepted. `AQUIFER_CLUSTER_MAX_ACTIVE_USERS` is therefore a soft placement target, not an admission-control ceiling. An active user is a distinct `user_id` with outstanding local work or a recently used assignment. Once its work reaches zero and it remains idle for `AQUIFER_CLUSTER_ASSIGNMENT_IDLE_SECONDS`, the owner releases it so that capacity becomes available again. A node holding more than 1.5× the cluster's fair share of active users (their total over the active nodes) also hands users back, on its regular state sync: only users with no work left on it, and only to a preferred node that's below its fair share. The user's next request is assigned afresh, starting with its preferred node. Without this, users that moved during a failure or drain stayed put as long as their traffic was steady. Configuration: | Env var | Default | Description | |---|---|---| | `AQUIFER_CLUSTER_ENABLED` | `false` | Enables HTTP cluster routing | | `AQUIFER_CLUSTER_PROVIDER` | `static` | `static` or `valkey` | | `AQUIFER_CLUSTER_SELF_ID` | _(required when enabled)_ | Stable id for this node | | `AQUIFER_CLUSTER_SELF_ADDR` | _(required when enabled)_ | Base HTTP URL other nodes use to reach this node | | `AQUIFER_CLUSTER_MEMBERS` | _(none)_ | Comma-separated `id=http://host:port` bootstrap/static members; self is added automatically | | `AQUIFER_CLUSTER_PRUNE_TTL_SECONDS` | `30` | How long this node excludes an unreachable peer before trying it again | | `AQUIFER_CLUSTER_VALKEY_URL` | `AQUIFER_VALKEY_URL` | Shared regional `redis://`, `rediss://`, `valkey://`, or `valkeys://` endpoint | | `AQUIFER_CLUSTER_VALKEY_PREFIX` | `aqueduct:` | Generic membership and assignment key prefix | | `AQUIFER_CLUSTER_MAX_ACTIVE_USERS` | `100` | Soft active-user placement capacity advertised by this instance | | `AQUIFER_CLUSTER_HEARTBEAT_SECONDS` | `5` | Membership refresh and assignment-renew interval | | `AQUIFER_CLUSTER_INSTANCE_TTL_SECONDS` | `15` | Time without a heartbeat before an instance is no longer assignable; forced to at least three heartbeat intervals | | `AQUIFER_CLUSTER_ASSIGNMENT_IDLE_SECONDS` | `300` | Completed user's idle window before its assignment is released | | `AQUIFER_CLUSTER_VALKEY_TIMEOUT_MS` | `50` | Per-operation coordination budget before local fallback | Example: ```bash AQUIFER_CLUSTER_ENABLED=true AQUIFER_CLUSTER_PROVIDER=valkey AQUIFER_CLUSTER_SELF_ID=aquifer-a AQUIFER_CLUSTER_SELF_ADDR=http://aquifer-a:8080 AQUIFER_CLUSTER_VALKEY_URL=valkey://valkey.internal:6379 AQUIFER_CLUSTER_MAX_ACTIVE_USERS=500 ``` The Valkey provider uses these generic keys: ```text aqueduct:instances aqueduct:instance: aqueduct:instance::assignments aqueduct:assignment: ``` Assignment and capacity selection happen in one Valkey Lua transaction using Valkey's clock, so concurrent healthy nodes converge on the first successful claim immediately rather than waiting for gossip. Heartbeats refresh membership; a graceful drain advertises `draining` before shutdown, and a crashed node becomes ineligible when its instance TTL expires. This mode is intended for Aquifer and Valkey instances in one region. The current multi-key transaction expects one Valkey primary/endpoint, not a sharded Redis Cluster keyspace. If Valkey exceeds the operation timeout or is unavailable, Aquifer keeps serving through its cached deterministic rendezvous view. That availability choice means two nodes with temporarily different views can both accept the same user. Healthy coordination converges again on the next successful assignment operation; dead-node membership normally converges within the instance TTL. Use the separate [Valkey remote-idempotency](#remote-idempotency) feature when duplicate execution across such a partition is unacceptable. This is not a distributed database. Job idempotency, queue contents, and pacing state remain local. A reassignment can therefore move a `user_id` to a node that does not have the previous owner's local history. Membership changes can also temporarily create duplicate account queues for the same target API URL on different nodes: old work may still be draining on the previous owner while new work moves to the new owner. That is expected during rebalance or a coordination outage and should be short-lived under normal TTL/drain cleanup. This availability-first mode does not provide a strict single-owner option. ## POST /proxy Capacity-aware reverse-proxy mode. Aquifer still belongs behind a trusted gateway; see [Use cases](https://rjpruitt16.github.io/aquifer/use-cases.md) for the deployment shape. Same request body as `POST /jobs`, same idempotency/admission rules, but tries the target API directly and synchronously first: - **Succeeds directly** (2xx, or any status not classified as overload — see below): the real target API's status, headers, and body are relayed back verbatim, on this same connection. The queue is never touched. A response the target API didn't mark as overload (a plain `500`, a `404`, anything unclassified) is a **success** in this sense too — relayed directly, not queued, since nothing said it shouldn't be. - **Fails, or the target API signals overload** (timeout, a classified status code, or an ORCA fallback threshold): falls back to the exact same durable-queue-and-delivery path `POST /jobs` uses — the connection seamlessly becomes the same SSE stream `GET /jobs/:id/stream` provides, rather than requiring a second call. The very first event on that stream is `event: proxy_fallback`, `data: {"job_id", "reason", "upstream_status"}` (status omitted when no real response was received — a skipped attempt or a timeout) — so a client with no server of its own to explain this any other way (a browser, an agent) sees explicitly why it's in the queue before the normal `queued`/`dispatching`/terminal sequence starts. `reason` is one of `upstream_overloaded`, `upstream_unreachable`, `domain_degraded` (breaker open or this domain's queue already has backlog), or `pool_routed`. **Which status codes count as overload is configurable, split into two kinds — queue locally, or also try cross-region redirect first (see below) — not lumped together the way "any 5xx" used to be.** Defaults are deliberately narrow: `429` → queue, `503` → reroute. Everything else (a plain `500`, `502`, `504`, `404`, anything not explicitly classified) is relayed to the caller as a normal, if unfortunate, direct response — not treated as overload at all. The reasoning: `429` is usually a *global* per-key/per-account rate limit, not a regional one — rerouting to a sibling region wouldn't help, since it hits the identical limit on the identical target API, so it stays queue-only. `503` and an ORCA overload signal are reroute-eligible by default, since they more plausibly indicate *this* region/instance specifically is struggling. Your own target API can configure its own sets via response headers, comma-separated, each entry either a literal code or an HTTP status class (`5xx` matches every `500`–`599`): | Header | Default | Effect | |---|---|---| | `X-Aqueduct-Queue-Codes` (or `X-Aquifer-Queue-Codes`) | `429` | Statuses that mean "queue this domain locally" | | `X-Aqueduct-Reroute-Codes` (or `X-Aquifer-Reroute-Codes`) | `503` | Statuses that mean "also try cross-region redirect first" | Due diligence is on you to say so if your target API uses something nonstandard — e.g. `X-Aqueduct-Reroute-Codes: 502,503,504` to widen reroute eligibility, or `X-Aqueduct-Reroute-Codes: 5xx` to sweep in every 5xx the way earlier versions of this feature did unconditionally. A domain that trips an overload signal has its direct attempts skipped entirely — anchored to the target API's own `Retry-After` header when it sends one (× a configurable safety multiplier, default 3) as the minimum cooldown, but direct attempts stay skipped for as long as the domain's queue actually has real backlog, even past that cooldown — a fixed timer alone doesn't know whether the traffic it caused has finished draining. Once both the cooldown has elapsed *and* the queue is genuinely empty, the next request is itself a real probe against the live target API. Which kind (queue vs. reroute) tripped the breaker is remembered for the whole cooldown — a domain breaker-tripped by a `429` stays queue-only on every retry during that window, not reroute-eligible just because *some* overload happened. **Routine local backlog — no breaker tripped at all, just a queue with real jobs in it — never triggers redirect on its own**, even with the feature configured. A domain paced at a low rps under a normal traffic burst has jobs sitting in queue as a matter of course; that's Aquifer working as designed, not a signal anything is regionally degraded. Redirect only ever gets tried alongside an actual tripped breaker or a failed/overloaded direct attempt. The target API can also proactively request queueing itself, on an otherwise-healthy response: `X-Aqueduct-Queue-Active: true` (or the product alias `X-Aquifer-Queue-Active`) trips the same breaker (always queue-kind, never reroute — matching its own name) for future requests to that domain — without discarding the response that already came back. Useful for "I'm nearing capacity, stop firing directly at me" ahead of an actual overload status. Pool-routed jobs (`pool_id` instead of `url`) always fall straight to queue+stream — there's no single canonical target API to try directly. ```bash curl -N -X POST http://localhost:8080/proxy -d '{ ... same shape as POST /jobs ... }' ``` ## GET /websocket WebSocket proxying with an ordered Valkey transcript, cursor replay, paced target API connection admission, and automatic target API reconnect. Aquifer does **not** authenticate callers or choose their destination. Put it behind a trusted gateway that authenticates the request and injects the target API URL. The endpoint enables automatically when `AQUIFER_VALKEY_URL` is configured; there is no separate feature flag to turn on. `AQUIFER_WS_ENABLED=false` remains an operational kill switch. Without a configured Valkey URL, the WebSocket subsystem stays inactive so ordinary HTTP/MCP installations do not acquire an external dependency. Active WebSocket sessions retry failed transcript reads and writes for `AQUIFER_WS_STORE_RETRY_SECONDS`, then fail closed if Valkey is still unavailable, because Aquifer cannot uphold record-before-forward and record-before-deliver without the stream. ### Handshake ```http GET /websocket?session_id=session-123&after=0-0 HTTP/1.1 Connection: Upgrade Upgrade: websocket Sec-WebSocket-Protocol: aqueduct.v1 Authorization: Bearer gateway-authenticated-identity X-Aqueduct-Upstream-URL: wss://backend.internal/socket ``` `session_id` identifies the session's transcript. `after` is the last Valkey stream ID the client has processed and defaults to `0-0`. Aquifer replays backend messages after that cursor before following live events. A cursor older than retained history is rejected with **409** instead of silently skipping data. The trusted `X-Aqueduct-Upstream-URL` must be an absolute `ws://` or `wss://` URL and is subject to `AQUIFER_ALLOWED_URL_DOMAINS`. Aquifer forwards end-to-end gateway headers such as `Authorization`, strips hop-by-hop and internal `X-Aqueduct-*`/`X-Aquifer-*` headers, and injects `X-Aqueduct-Session-ID` target API. ### Messages Every application message is a JSON `aqueduct.v1` envelope. Raw frame proxying is intentionally not supported. Client command: ```json {"type":"command","message_id":"command-42","payload":{"action":"start"}} ``` The `message_id` must be stable across client retries. Aquifer appends the command to Valkey before forwarding it and confirms that durable write with: ```json {"type":"command_recorded","message_id":"command-42","stream_id":"1798053731000-0"} ``` Backend acknowledgement and events: ```json {"type":"ack","message_id":"ack-42","caused_by":"command-42"} {"type":"event","message_id":"event-43","caused_by":"command-42","payload":{"state":"running"}} {"type":"event","message_id":"event-44","caused_by":"command-42","payload":{"state":"complete"}} ``` `caused_by` supports one command producing zero, one, or many events; it is not a one-request/one-response contract. Aquifer appends every backend `ack` and `event` before delivery, then adds its Valkey `stream_id` and target API `generation` to the client-facing envelope. Aquifer also sends non-durable control messages: ```json {"type":"status","state":"replaying"} {"type":"status","state":"replay_complete"} {"type":"status","state":"waiting","position":2} {"type":"status","state":"connecting"} {"type":"status","state":"connected","generation":7} {"type":"status","state":"reconnecting","retry_after_ms":914,"reason":"..."} {"type":"status","state":"server_draining","retry_after_ms":5000,"reason":"reconnect through the gateway"} ``` While a session waits, Aquifer emits another `waiting` status whenever its FIFO position changes (for example, `2` then `1`). These progress messages are live control state and are not appended to the durable transcript. On target API loss, Aquifer keeps the client connection open and reconnects with jittered exponential backoff. Commands are durably recorded, but v1 does **not** automatically replay a command after an ambiguous target API failure: Aquifer cannot know whether the backend acted before the connection disappeared. Clients may resend a command with the same `message_id`; backend actions must therefore be idempotent by `message_id`. Delivery of backend events is at least once when a client reconnects from its last acknowledged stream cursor. Each session transcript retains approximately the newest `AQUIFER_WS_STREAM_MAX_EVENTS` entries and expires `AQUIFER_WS_STREAM_TTL_SECONDS` after its last recorded entry. Every append refreshes that TTL. Expired history is treated as a replay gap when a client supplies a nonzero cursor, so deletion cannot silently skip events. #### What the transcript guarantees The transcript is a **replay window**, not a durable log. It is built for reconnects: a client that drops and comes back within the window gets everything it missed, in order. - **Guaranteed:** every message is recorded in Valkey before it is forwarded or delivered, and a client that reconnects with a cursor still inside the window receives every backend event after it, in order, at least once. - **Not guaranteed:** history older than the window. Once entries are trimmed (by count or TTL), a client whose cursor points before them gets **409** and must resynchronize from your application's own state. Aquifer never skips events silently, but it does not keep them forever. - **Valkey is the storage.** Durability is whatever your Valkey provides: persistence (AOF/RDB), replication, and its `maxmemory` policy. With an eviction policy, Valkey can drop whole transcripts under memory pressure, which also surfaces as **409**. Size `maxmemory` for sessions × `AQUIFER_WS_STREAM_MAX_EVENTS` × message size, or use `noeviction` and accept failed writes when it is full: new handshakes get **503**, a client whose command can't be recorded within `AQUIFER_WS_STORE_RETRY_SECONDS` is disconnected (it can reconnect from its cursor), and a backend whose event can't be recorded within it is disconnected and reconnected. If you need every message kept beyond the window (audit, analytics, replay from the beginning), turn on transcript export below, or record messages in your own durable store from the backend. #### Keeping transcripts longer: export Off by default. With `AQUIFER_WS_EXPORT_ENABLED=true`, Aquifer copies every recorded message to a receiver you run, such as [Inlet](https://github.com/rjpruitt16/inlet), which writes them to a Kafka topic. The Valkey transcript and its replay window don't change. - Each recorded message is also added to one shared export stream in Valkey (`{AQUIFER_WS_STREAM_PREFIX}export`), atomically with the transcript entry, so nothing recorded is missed. - A background exporter on every node reads that stream through one consumer group. It POSTs batches to `AQUIFER_WS_EXPORT_URL`, and acknowledges and deletes each batch from the export stream only after a **2xx**. A failed batch is retried with backoff. A node that stops leaves its batch for the next start, and another node takes over messages left unacknowledged for a minute. - Delivery is at least once. Drop repeats by `session_id` + `stream_id`. `stream_id` is the transcript entry's ID, the same cursor clients replay from. A retried transcript write can be recorded twice under two `stream_id`s; to drop those too, also dedupe by `session_id` + `direction` + `message_id`. - **Order is kept within a batch, not across batches.** A retried batch can arrive after a newer one, especially with several nodes. Order a session's messages by `stream_id`. - Batches are capped at `AQUIFER_WS_EXPORT_MAX_BATCH_BYTES`. A **413** halves the batch until it goes through. A single message the receiver still refuses with 413 is logged and skipped, so it can't block the messages behind it; it stays in its session transcript for the replay window. - The URL must be `https`, so the token and transcripts aren't sent in cleartext. `http` is allowed for loopback, or anywhere with `AQUIFER_WS_EXPORT_ALLOW_INSECURE=true` on a private network you trust. Redirects aren't followed. Otherwise export stays off and a warning is logged. - While the receiver is down, the export stream holds up to `AQUIFER_WS_EXPORT_MAX_PENDING` messages, approximately. Beyond that the oldest are dropped from export; they stay in their session transcripts for the replay window. The cap counts messages, not bytes: allow Valkey `maxmemory` for `AQUIFER_WS_EXPORT_MAX_PENDING` × average message size on top of the transcripts, or lower the cap. - A node that stops with nothing pending leaves the consumer group. Consumers of nodes that are gone are removed once they've been idle for an hour with nothing pending. - The two writes run in one script on two keys, so this needs one Valkey primary, not a sharded Redis Cluster, the same as cluster coordination. Batch body (`Content-Type: application/json`, `Authorization: Bearer ` when set): ```json { "schema": "aquifer.websocket_batch.v1", "node_id": "aquifer-iad-1", "events": [ { "session_id": "4d9c...", "stream_id": "1792541234567-0", "direction": "client", "type": "command", "message_id": "m-17", "caused_by": "", "generation": 3, "payload": {"action": "subscribe"}, "recorded_at": 1792541234567 } ] } ``` | Env var | Default | Description | |---|---|---| | `AQUIFER_WS_EXPORT_ENABLED` | `false` | Turns export on | | `AQUIFER_WS_EXPORT_URL` | _(none)_ | Receiver for batches; export stays off without it | | `AQUIFER_WS_EXPORT_TOKEN` | _(none)_ | Sent as a bearer token | | `AQUIFER_WS_EXPORT_BATCH` | `500` | Messages per batch | | `AQUIFER_WS_EXPORT_MAX_BATCH_BYTES` | `4194304` | Encoded size cap per batch (4 MiB, under Lambda's 6 MB request limit) | | `AQUIFER_WS_EXPORT_ALLOW_INSECURE` | `false` | Allow an `http` URL that isn't loopback | | `AQUIFER_WS_EXPORT_MAX_PENDING` | `1000000` | Approximate cap on messages waiting for export | | `AQUIFER_WS_EXPORT_PAYLOADS` | `true` | `false` exports everything but `payload` | Two clients may temporarily attach to the same `session_id` during an application-managed handoff; both follow the same durable backend event stream, and the client decides when to close the old socket. On `SIGTERM`, Aquifer sends `server_draining`, leaves the old socket open for `AQUIFER_WS_DRAIN_GRACE_SECONDS`, then closes it with WebSocket code **1012 Service Restart**. The client should open its replacement through the gateway using its last processed stream cursor. New handshakes receive **503** while the node drains. ### Capacity Each live session uses two file descriptors on the Aquifer host: the client socket and the backend socket. Size the process's open-file limit (`ulimit -n`) for about 2× `AQUIFER_WS_MAX_CLIENT_CONNECTIONS` plus headroom. Many container platforms default to 10,240, which tops out around 5,000 sessions. Sessions don't hold Valkey connections of their own. Every recorded backend event is also published on the session's notify channel, and each Aquifer instance follows its live sessions through `AQUIFER_WS_READER_SHARDS` shared pub/sub connections (default 8). A session reads its stream only when notified, so an idle session costs Aquifer and Valkey nothing. Correctness doesn't depend on notifications arriving: whenever a subscription is confirmed (including after a reconnect), the session re-reads from its own cursor. New sessions are also refused while the process is over `AQUIFER_MEMORY_LIMIT_MB`: `429` with `Retry-After` and `limit_reason: "memory"`, the same answer an HTTP job gets. Open sessions are not affected. Connection ceilings are local to one Aquifer process. `AQUIFER_WS_MAX_CLIENT_CONNECTIONS=1000` means that instance accepts at most 1,000 clients; it is not a fleet-wide semaphore. Ten identical instances can therefore admit up to 10,000 clients when the gateway distributes them. The target API can lower this instance's ceiling on connections to it, or its opening rate, during a successful handshake: ```http X-Aqueduct-WS-Max-Connections: 250 X-Aqueduct-WS-Connect-Rps: 20 ``` It can update either value on an established socket with `{"type":"aqueduct.capacity","max_connections":250,"connect_rps":20}`. Dynamic signals can only lower operator-configured ceilings, never raise them. `Retry-After` on a rejected target API handshake becomes the minimum reconnect delay. Opening starts at `AQUIFER_WS_SLOW_START_RPS` for each Aquifer process. Every successful target API handshake doubles the current ramp rate until `AQUIFER_WS_CONNECT_RPS` is reached; a failed handshake resets the ramp. An established socket disconnecting does not penalize unrelated connections by resetting the process-wide ramp. Backend capacity signals can still lower the effective rate at any point. Set `AQUIFER_WS_SLOW_START_RPS` equal to `AQUIFER_WS_CONNECT_RPS` to disable ramping. | Env var | Default | Description | |---|---:|---| | `AQUIFER_WS_ENABLED` | automatic | WebSockets are on when `AQUIFER_VALKEY_URL` is set; `false` explicitly disables them | | `AQUIFER_VALKEY_URL` | _(required)_ | Shared `redis://`, `rediss://`, `valkey://`, or `valkeys://` stream store | | `AQUIFER_WS_STREAM_PREFIX` | `aqueduct:ws:` | Stream key prefix; session IDs are SHA-256 hashed | | `AQUIFER_WS_STREAM_MAX_EVENTS` | `10000` | Approximate retained entries per session | | `AQUIFER_WS_STREAM_TTL_SECONDS` | `86400` | Sliding expiration after the last recorded session entry | | `AQUIFER_WS_READ_BATCH` | `100` | Maximum events fetched per stream read | | `AQUIFER_WS_READ_BLOCK_MS` | `1000` | Blocking-read interval for direct `ReadAfter` callers; live sessions are notified through pub/sub instead | | `AQUIFER_WS_READER_SHARDS` | `8` | Shared Valkey pub/sub connections per instance that tell live sessions when their stream has new events | | `AQUIFER_WS_MAX_MESSAGE_BYTES` | `1048576` | Maximum client or backend message size | | `AQUIFER_WS_HANDSHAKE_TIMEOUT_SECONDS` | `10` | Valkey check and target API handshake timeout | | `AQUIFER_WS_RECONNECT_MAX_SECONDS` | `30` | Maximum reconnect backoff before jitter | | `AQUIFER_WS_VALKEY_POOL_SIZE` | `64` | Valkey connections for transcript reads and writes (a `pool_size` in `AQUIFER_VALKEY_URL` wins) | | `AQUIFER_WS_STORE_RETRY_SECONDS` | `30` | How long a live session retries a failed transcript read or write before closing | | `AQUIFER_WS_MAX_CLIENT_CONNECTIONS` | `1000` | Client socket ceiling for this Aquifer instance | | `AQUIFER_WS_MAX_UPSTREAM_CONNECTIONS` | `1000` | Active backend socket ceiling for this instance | | `AQUIFER_WS_MAX_WAITING_CONNECTIONS` | `1000` | Local queue ceiling for sessions waiting on a target API slot | | `AQUIFER_WS_CONNECT_RPS` | `20` | Maximum target API connection openings per second on this instance | | `AQUIFER_WS_SLOW_START_RPS` | `1` | Initial and post-failure connection-opening rate before successful handshakes ramp it up | | `AQUIFER_WS_DRAIN_GRACE_SECONDS` | `5` | Handoff window between `server_draining` and close code 1012 on `SIGTERM`; `0` closes immediately | Handshake errors are returned before upgrade: **400** for invalid protocol/session/target API input, **409** for a replay gap, **429** for a local connection or waiting ceiling, and **503** when Valkey is unavailable or the node is draining. `GET /health` exposes this instance's client, waiting, active-upstream, configured, ramp, and effective limits under `websocket`. Measure a deployment with a controlled target API before raising the defaults: ```bash go run ./cmd/aquifer-websocket-bench \ -target ws://localhost:8080/websocket \ -upstream ws://localhost:6060/socket \ -connections 1000 -ramp 10s -hold 30s ``` This measures one Aquifer instance. Run it against each machine size and include the gateway, Valkey latency, file-descriptor limits, and the real backend in capacity planning. ## Cross-region redirect (Fly.io) If `AQUIFER_FLY_REGIONS` is set, an `upstream_unreachable` fallback, or an `upstream_overloaded`/`domain_degraded` fallback specifically classified as reroute-eligible (see `X-Aqueduct-Reroute-Codes` above — `503` by default, not every overload), tries other regions Aquifer is deployed to — live, over Fly's private network — before falling back to this instance's own local queue. Off by default; unset, `/proxy` behaves exactly as described above with zero change. When it triggers: every known-live region is tried for a fast direct success first, nearest (lowest measured round-trip time from the same health check that determines a region is live — Fly doesn't publish a region distance/latency table, so this doubles as the only real proximity signal available) first, except that two callers racing the same job always try one particular region first regardless of latency, so they tend to converge on the same region rather than each racing off after their own nearest option. If none can serve it directly, that same region is the one chosen to accept it into its own durable queue, and its live event stream is relayed back onto your original connection, so you see one continuous stream regardless of which region actually ends up handling the job. A reroute is never silent to the caller — same principle as `proxy_fallback` above: a client with no server of its own to explain this (a browser, an agent) shouldn't have to wonder why its connection is still open or where the response actually came from. - **Direct success via redirect:** the response carries an `X-Aquifer-Served-By-Region` header naming which region actually served it, alongside the relayed status/headers/body — purely additive, doesn't change anything you're already parsing. - **Queued on another region:** before relaying that region's own stream, the first region (the one the request arrived at) fires `event: rerouted`, `data: {"region"}` — so it arrives *before* that region's own `proxy_fallback`/`queued`/`dispatching` sequence, the same ordering `proxy_fallback` itself uses relative to `queued`. **A full queued-on-reroute sequence, start to finish** — this is what actually crosses the wire (Go's JSON encoder sorts map keys, so this is the literal byte-for-byte shape, not approximated). `position` fires every ~2s for as long as the job is genuinely still queued — real backpressure ticking down, not a fixed animation: ``` event: rerouted data: {"region":"ord"} event: proxy_fallback data: {"job_id":"b7e2...","reason":"domain_degraded"} event: queued data: {"job_id":"b7e2...","status":"queued"} event: position data: {"job_id":"b7e2...","position":3} event: position data: {"job_id":"b7e2...","position":2} event: position data: {"job_id":"b7e2...","position":1} event: dispatching data: {"job_id":"b7e2..."} event: completed data: {"body":"...","job_id":"b7e2...","response_status":200} ``` `job_id` throughout is the **redirect region's** ID, not the first region's — the first region deletes its own local row the instant redirect succeeds (`fallbackOutcome`), so there's no first-region ID for the client to have seen and compare against. `proxy_fallback` has no `upstream_status` field here because `domain_degraded` always passes status `0`; that field only appears on an `upstream_overloaded`/`upstream_unreachable` fallback, where a real (or attempted) target API response actually came back. Terminal event is `completed` on success or `failed` (`{"job_id","reason","response_status","body"}`) on failure — same shape `GET /jobs/:id/stream` uses, because this literally *is* that stream, just relayed from `target` through `origin`'s connection. If the original request set `webhook_url`, a webhook fires too — but separately, from the **redirect region** (whichever region actually completed the job), as its own independently-paced, durably-queued POST, not inline in this stream: ``` POST https://yourapp.com/webhooks/aquifer Content-Type: application/json {"job_id":"b7e2...","status":"completed","response_status":200,"body":"..."} ``` Both happen on success — the SSE `completed` event and the webhook — exactly the same contract every non-rerouted job already has (see [Webhooks](#webhooks) below). Reroute doesn't change that, it just changes which instance ends up firing it. If literally no known-live region can help either — none live at all, or every one tried and failed — the request is **rejected**, not queued locally: **429**, `Retry-After` set to `AQUIFER_REDIRECT_EXHAUSTED_RETRY_AFTER_SECONDS` (default 900 — a real regional outage, not a transient blip), `limit_reason: "redirect_exhausted"`, same response shape as an admission-control rejection. This is deliberate: queueing locally instead was never actually decided, so a caller (or its own retry/alerting logic) finds out the whole fleet is degraded rather than the request silently landing on one struggling instance's queue. A future per-deployment option to queue locally instead — plausible for an Aquifer instance dedicated to a single customer, where "queue and eventually deliver" might beat erroring — is a real possibility, just not the default and not built yet. This is separate from `AQUIFER_REDIRECT_GATE_COOLDOWN_SECONDS` (default 500), which is purely internal — how long this instance avoids re-running the whole candidate tour after finding nothing live, independent of what it tells the caller. **Honest limitation, not silently glossed over:** Aquifer's idempotency check remains per-instance (local SQLite/Pebble), unchanged by this feature. If the exact same `idempotent_key` is independently submitted to two different regions at nearly the same moment (a real scenario — a caller's own client retrying after a timeout can land on a different region via Fly's anycast), each region may independently begin its own redirect tour, and in rare cases the job could end up durably queued in two places. The deterministic region selection above narrows this window but does not close it. During cross-region redirect specifically, treat delivery as at-least-once, not exactly-once — standard practice for any webhook consumer, just worth calling out plainly here since it's a real, if narrow, exception to Aquifer's otherwise-exactly-once idempotency guarantee. ## GET /jobs/:id ```json { "job_id": "a3f9...", "status": "queued | completed | failed", "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "created_at": 1715000000000, "result": { "response_status": 200, "content_type": "application/json", "body": "{...}", "body_truncated": false } } ``` `result` appears once the job has finished and stays for the job's retention window (30 minutes after completion, 2 hours after failure). The body is capped at `AQUIFER_RESULT_MAX_BYTES` (default 65536). ## GET /jobs/:id/stream Server-Sent Events stream for live job updates: `queued` → `dispatching` → `completed` (`{"job_id","response_status","body"}`) or `failed` (`{"job_id","reason"}`), plus a `position` event every 2s while queued. Connecting late is safe — you'll receive synthetic catchup events for states you missed. If the job already finished, the stream sends its stored result as the `completed` or `failed` event (`{"job_id","status","response_status","content_type","body","body_truncated"}`; no `reason`, which isn't stored) and ends. SSE is a convenience, not the source of truth: the webhook fires regardless of whether the stream was ever open. ```bash curl -N http://localhost:8080/jobs//stream ``` ## GET /health ```json { "status": "ok", "l8_protocol": "0.1", "l8_public_key": "...", "admission": { "enabled": true, "memory_mb": 42, "memory_limit_mb": 400, "max_body_bytes": 1048576, "db_bytes": 81920, "db_max_bytes": 104857600, "retry_after_seconds": 5 } } ``` `admission.enabled` is `false` (with only that key present) when none of the `AQUIFER_*` admission env vars are set. With inbound pacing on, an `inbound` object is also present (see [Inbound pacing](#inbound-pacing-x-aqueduct-rps)). `GET /health` is a liveness endpoint. It remains **200** during graceful shutdown, with `status: "draining"`, so an orchestrator does not kill the process before accepted work has a chance to finish. ## GET /metrics Prometheus text format, on the same port as `/health`. The `aquifer` binary serves it by default. A program embedding Aquifer gets it by passing `aquifer.NewPrometheusMetrics()` as `RuntimeOptions.Metrics`; with any other adapter it's a 404. Like `/health`, everything is this node's own. | Metric | Type | Labels | What | |---|---|---|---| | `aquifer_jobs_total` | counter | `target`, `outcome` | Jobs that reached a terminal state: `completed`, `failed` or `expired` | | `aquifer_dispatch_attempts_total` | counter | `target`, `status_class` | Attempts at the target API, retries included: `2xx`, `4xx`, `429`, `5xx`, or `error` (no response) | | `aquifer_queue_wait_seconds` | histogram | `target` | Submission to first dispatch: time spent waiting in Aquifer | | `aquifer_job_duration_seconds` | histogram | `target`, `outcome` | Submission to terminal state: end to end | | `aquifer_admission_rejected_total` | counter | `reason` | Jobs refused with `429`: `memory`, `db_size`, `sync_lag`, `upstream_queue`, `queue_fair_share`, ... | | `aquifer_webhooks_total` | counter | `outcome` | Completion webhooks `delivered` (the receiver answered 2xx), or `failed`: refused with an answer Aquifer doesn't retry, such as `401`, or out of retries. They aren't counted in the job metrics above. | | `aquifer_active_queues` | gauge | `target` | Queues with work waiting (one per tenant with account queues on) | | `aquifer_backlog_jobs` | gauge | `target` | Jobs queued or in flight | | `aquifer_flow_rate_rps` | gauge | `target` | Rate Aquifer currently allows the target API: the active queues' rates after its pacing headers, summed and capped at the target's ceiling (configured `rps` or a pool's live capacity, never below the 0.5 rps floor; 0 for a pool with no live members). An upper bound; queues with little work send less | | `aquifer_fairness_jain` | gauge | `target` | [Jain's index](https://en.wikipedia.org/wiki/Fairness_measure) over queues with work waiting, by jobs each finished in about the last minute. `1` is perfectly even, `1/n` is one queue getting everything. Absent when nothing is waiting. | | `aquifer_node_info` | gauge | `node_id` | `1`, carrying the id sent as `X-Aqueduct-Node-Id` | | `aquifer_wal_sync_lag_seconds` | gauge | | Pebble's default flush only: how long accepted jobs may have gone without reaching disk. A crash loses them. Normally under about 0.15 | | `aquifer_wal_last_sync_seconds` | gauge | | Pebble's default flush only: how long the newest completed log sync took. Jobs accepted while a sync runs wait for the next one, so slow syncs are what make the lag grow | `target` is the target API's scheme and host (`https://api.example.com`), or `pool:`. The first 1,000 targets seen get their own label and the rest share `other`, since URL jobs can name any host. There are no tenant labels by default: tenant IDs are unbounded, and often customer data. For a pilot, `AQUIFER_METRICS_TENANT_LABELS=true` adds `aquifer_tenant_jobs_total` and `aquifer_tenant_job_duration_seconds` with a `tenant` (`user_id`) label. The first 1,000 tenants seen get their own label, and the rest share `other`. ## GET /ready Readiness for load balancers and fleet discovery. An active node returns **200** with `{"status":"ready"}`. A draining node returns **503**, `Retry-After`, and both `X-Aqueduct-Node-State: draining` and the compatibility alias `X-Aquifer-Node-State: draining`. Use `/ready`, not `/health`, for routing decisions. Fly region discovery uses `/ready`; cluster forwarding also soft-prunes a ranked owner when it returns the draining signal and tries the next owner. ## Graceful shutdown The built-in HTTP, MCP stdio, and A2A adapters handle `SIGINT`/`SIGTERM` through the same bounded lifecycle: 1. Mark the node draining, fail `/ready`, reject new jobs, proxy work, and WebSocket handshakes with **503**, and announce `state: "draining"` when external registration is enabled. 2. Keep the listener available for a short quiescence window so gateways and peers can observe the transition, then stop accepting connections. 3. Let already-accepted jobs and their completion webhooks finish. Notify active WebSockets, allow their handoff grace period, then close them with code 1012. 4. Flush the drain-event ledger once more. Events are acknowledged locally only after the configured sink confirms receipt, and periodic/final flushes are serialized to prevent duplicate concurrent batches. 5. Report `state: "offline"` to the external registry and exit. When the total deadline expires, Aquifer closes with durable queued work left for normal startup recovery. | Env var | Default | Description | |---|---:|---| | `AQUIFER_SHUTDOWN_TIMEOUT_SECONDS` | `30` | Total graceful-drain deadline; must be positive | | `AQUIFER_SHUTDOWN_QUIESCE_MS` | `500` | Time to expose draining readiness before stopping the listener; `0` disables the delay | | `AQUIFER_WS_DRAIN_GRACE_SECONDS` | `5` | WebSocket replacement window, capped at the shutdown timeout | The process termination grace configured by your orchestrator must exceed `AQUIFER_SHUTDOWN_TIMEOUT_SECONDS`; otherwise the orchestrator may send `SIGKILL` before Aquifer can finish or preserve the intended shutdown sequence. ## Request schemas (L8 0.2) A target API that implements L8 0.2 can publish a JSON Schema (draft 2020-12) per route in its `GET /.well-known/l8`: ```json { "protocol_version": "0.2", "public_key": "...", "challenge_endpoint": "/l8/challenge", "capabilities": ["signed_payloads", "request_schemas"], "schema_hash": "sha256:9c1e...", "request_schemas": { "POST /v1/chat/completions": { "type": "object", "required": ["model", "messages"] } } } ``` With `AQUIFER_L8_SCHEMA_VALIDATION=true`, Aquifer fetches this once per target API domain, caches it for up to 10 minutes, and checks `url`-routed job bodies whose method and path match a route. A mismatch never reaches the target API: ```json { "error": "request body does not match the schema the upstream advertises for POST /v1/chat/completions", "schema_route": "POST /v1/chat/completions", "schema_hash": "sha256:9c1e...", "schema_errors": "..." } ``` When the target API changes its contract it returns a new `X-Aqueduct-Schema-Hash` on any response. Aquifer then drops the cached schemas and the L8 trust for that domain, so the next request refetches metadata and re-runs the handshake. Routes without a schema, target APIs without L8, and metadata that fails to fetch or compile are all let through unchecked. External `$ref`s are never fetched. ### Webhooks are skipped for results you already received The completion webhook is a fallback, not a second copy. Aquifer never sends one for a result the caller already received: - If a stream (`GET /jobs/:id/stream` or a `/proxy` fallback stream) writes and flushes the final `completed`/`failed` event while its client is still connected, no webhook is sent. - A `/proxy` direct success works the same way: if the relayed response is written and flushed to a caller who is still connected, no webhook is sent. - If the client disconnects before the end (for example, gives up while waiting in line), or nobody was streaming, the webhook is sent as usual. - Aquifer waits up to `AQUIFER_STREAM_DELIVERY_WAIT_MS` (default 2000) for a stream's confirmation, and only when someone is actually streaming the job. A client that vanishes without closing its connection (a dropped network, a sleeping laptop) looks connected to the server, so it may miss both the final event and the webhook. That's why the result is stored: fetch it from `GET /jobs/:id`. ## Webhooks **Completed** ```json { "job_id": "a3f9...", "status": "completed", "response_status": 200, "body": "..." } ``` **Failed**: the completion webhook for a job that failed. The job's own target API call used up its retries (`max_retries`, 4 by default); this is not about retrying the webhook, which is covered below. ```json { "job_id": "a3f9...", "status": "failed", "reason": "connection refused" } ``` **Webhook delivery uses the same account-queue pacing as forward dispatch.** A webhook POST isn't fired immediately from the dispatch goroutine — it's enqueued as its own durable job, keyed by the webhook receiver's domain, and dispatched through the identical `AccountQueue`/`URLWorker` machinery described on the [Dynamic Pacing](https://rjpruitt16.github.io/aquifer/pacing.md) page. Practically, this means: - A webhook receiver can slow Aquifer down with the same `X-Aqueduct-Rps` / `X-Aqueduct-Max-Concurrent` response headers a real target API uses, instead of just getting hammered. - Delivery is crash-durable — a webhook still pending when the process restarts is recovered and retried, the same way a queued job is, rather than being lost with an in-memory retry loop. - Retries trigger on connection errors and `5xx`, `408` and `429` responses, the same retryable failures as forward dispatch. An unset `webhook_max_retries` means 10 retries, with backoff doubling from 1 s up to 300 s, about 13 minutes in total, enough to ride out a receiver deploy. - L8 signing (see [L8 protocol](https://rjpruitt16.github.io/aquifer/l8.md)) still applies exactly as before — trust is established and delivery is signed the same way, just from inside the paced dispatch path instead of a separate one-shot retry loop. - L8 encryption: when the receiver advertises `encrypted_payloads` and an X25519 `encryption_public_key`, the body is encrypted to that key (`x25519-hkdf-sha256-aes256gcm`) and the signature covers the ciphertext. The receiver verifies, then decrypts. See the [L8 spec](https://rjpruitt16.github.io/l8-protocol/) for the exact construction. Delivery is still at-least-once — see [Delivery semantics](https://rjpruitt16.github.io/aquifer/how-it-works.md). Drain mode's own ledger webhook is separate: it sends acknowledged batches from the local drain-event journal and deletes only the events confirmed by a `2xx` response. See [DRAIN_MODE.md](https://rjpruitt16.github.io/aquifer/drain-mode.md) for that payload contract. ## Remote idempotency Remote idempotency is optional and generic. When `AQUIFER_REMOTE_IDEMPOTENCY_ENABLED=true`, Aquifer still checks its local SQLite/Pebble store first. If the key is new locally, Aquifer performs a bounded lookup against Valkey before dispatching the job. A remote hit returns `duplicate:true` and deletes the speculative local row; a timeout or Valkey error falls back to local-only behavior so Aquifer stays on the hot path. Configuration: | Env var | Default | Description | |---|---|---| | `AQUIFER_REMOTE_IDEMPOTENCY_ENABLED` | `false` | Enables pre-dispatch Valkey duplicate lookup | | `AQUIFER_VALKEY_URL` | _(none)_ | `redis://` or `valkey://` URL | | `AQUIFER_REMOTE_IDEMPOTENCY_TIMEOUT_MS` | `25` | Lookup/write timeout budget | | `AQUIFER_REMOTE_IDEMPOTENCY_PREFIX` | `aqueduct:idempotency:` | Key prefix | | `AQUIFER_REMOTE_IDEMPOTENCY_TTL_SECONDS` | `7200` | TTL for entries written by Valkey drain sink | | `AQUIFER_REMOTE_RESULT_ENABLED` | `false` | Also writes a bounded terminal result snapshot to Valkey | | `AQUIFER_REMOTE_RESULT_PREFIX` | `aqueduct:result:` | Key prefix for result snapshots | | `AQUIFER_REMOTE_RESULT_MAX_BYTES` | `65536` | Maximum response-body bytes stored in a result snapshot; `0` stores metadata only | Key: ```txt {AQUIFER_REMOTE_IDEMPOTENCY_PREFIX}{sha256(user_id + ":" + idempotent_key)} ``` For `idempotency_scope: "shared"` jobs the hashed value is `"shared\0" + idempotent_key` instead, in both this key and the result key below. Value: ```json { "job_id": "a3f9...", "status": "completed", "recorded_at": 1798053731000, "source": "aquifer", "result_key": "aqueduct:result:..." } ``` `AQUIFER_DRAIN_SINK=valkey` uses the same prefix/value contract for completed and failed job records. When `AQUIFER_REMOTE_RESULT_ENABLED=true`, Aquifer also writes the terminal response snapshot to: ```txt {AQUIFER_REMOTE_RESULT_PREFIX}{sha256(user_id + ":" + idempotent_key)} ``` Value: ```json { "job_id": "a3f9...", "status": "completed", "response_status": 200, "content_type": "application/json", "body": "{\"ok\":true}", "body_truncated": false, "recorded_at": 1798053731000, "source": "aquifer" } ``` The result body is capped before writing to Valkey. This is meant for replaying or inspecting bounded API responses, not for storing large artifacts or unbounded streams. Use object storage or your own durable result store for large outputs. ## GET /results Retrieves a bounded remote result snapshot by the same logical idempotency key used to create the job: ```bash curl "http://localhost:8080/results?user_id=user-123&idempotent_key=invoice-42-notify" ``` **200** ```json { "job_id": "a3f9...", "status": "completed", "response_status": 200, "content_type": "application/json", "body": "{\"ok\":true}", "body_truncated": false, "recorded_at": 1798053731000, "source": "aquifer" } ``` Returns **404** if remote result recording is disabled, the result has expired, or the job has not reached a terminal state yet. For a shared-scope job, drop `user_id` and pass the scope (requires `AQUIFER_SHARED_IDEMPOTENCY_ENABLED=true`): ```bash curl "http://localhost:8080/results?idempotency_scope=shared&idempotent_key=weather:sf:2026-10-05" ``` ## Autoscaling Every dispatch carries these, with `X-Aquifer-*` aliases: | Header | Value | |---------------------------|----------------------------------------------------| | `X-Aqueduct-Total-Jobs` | Total jobs on this machine right now | | `X-Aqueduct-Queue-Depth` | Jobs waiting to be dispatched | | `X-Aqueduct-Flow-Rate` | Current dispatch rate (RPS) for this queue | | `X-Aqueduct-Active-Queues` | Queues on this node sending to this target API | | `X-Aqueduct-Upstream-Backlog` | Jobs on this node waiting for this target API | | `X-Aqueduct-Node-Id` | This node's id: count the distinct ids to know how many Aquifer nodes are sending to you | Pacing is per node, and nodes don't coordinate: on each node, a target's queues together send at most the highest rate you're advertising to any of them ([Dynamic priority](https://rjpruitt16.github.io/aquifer/pacing.md#dynamic-priority)). So when several nodes send to you, divide your budget by the number of nodes, not by their queues ([More than one node](https://rjpruitt16.github.io/aquifer/pacing.md#more-than-one-node)). `X-Aqueduct-Node-Id` tells the nodes apart. It's `AQUIFER_NODE_ID`, else `AQUIFER_CLUSTER_SELF_ID`, else `FLY_MACHINE_ID`, else the host name. Give every node a distinct id, and count ids only from nodes you trust: two nodes sharing an id count as one. Traditional load balancers and autoscalers rely on failure to start scaling — capacity only responds once something's already struggling. Aquifer treats resources as fluid, not fixed: it paces down as machines show strain and back up as capacity comes online, using the same signals it already reports on every response. --- # Configuration > Rate limits per target API in YAML, and the environment variables. Rate limits are set per target API hostname via a YAML file (`CONFIG_PATH`) and admission/runtime behavior via env vars. Set `CONFIG_PATH` to a YAML file to configure rate limits per target API hostname. `rps` is the host's total across all tenants. `max_concurrent` applies to each tenant's queue, so with N busy tenants up to N × `max_concurrent` requests can be in flight at the host. A key is matched against the job URL's host exactly, port included: for `http://your-backend.internal:8080`, the key is `your-backend.internal:8080`. A URL whose host matches no key gets `defaults`. ```yaml # aquifer.yml — copy from examples/aquifer.example.yml defaults: rps: 2 max_concurrent: 1 upstreams: api.openai.com: rps: 10 max_concurrent: 3 api.stripe.com: rps: 20 max_concurrent: 5 your-backend.internal: rps: 50 max_concurrent: 10 ``` | Env var | Default | Description | |---------------|--------------|--------------------------------| | `AQUIFER_ADAPTER` | `http` for binary, `mcp-stdio` in Docker image | Runtime adapter: `http`, `mcp-stdio`, or `a2a` | | `AQUIFER_A2A_PUBLIC_URL` | `http://localhost:$PORT` | A2A adapter only: externally-reachable base URL advertised in the Agent Card | | `PORT` | `8080` | HTTP listen port | | `DB_PATH` | `aquifer.db` | Storage path: a directory for Pebble (the default), or a file for SQLite | | `CONFIG_PATH` | _(none)_ | Path to rate limit config YAML | | `AQUIFER_STORE_BACKEND` | `pebble` | Storage engine: `pebble` (pure-Go LSM store, the faster backend; see [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md)) or `sqlite`. If `DB_PATH` is an existing SQLite file, Aquifer stays on SQLite so an upgrade doesn't orphan queued jobs | | `AQUIFER_PEBBLE_WAL_SYNC_INTERVAL_MS` | `5` | Pebble only: batches concurrent durable writes into fewer real fsyncs under load (Pebble's own group-commit); each caller still blocks until its own write is actually durable | | `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS` | `100` | Pebble only: answer before a write reaches disk and sync the log on this timer. A crash loses the jobs accepted since the start of the last completed sync: normally about this many milliseconds, but a sync slowed by heavy writes stretches it (seconds in the Canalis soak). `AQUIFER_MAX_SYNC_LAG_MS` and inbound pacing (`X-Aqueduct-Rps`, see [API.md](https://rjpruitt16.github.io/aquifer/api.md#inbound-pacing-x-aqueduct-rps)) keep it short; `aquifer_wal_sync_lag_seconds` shows it. On Fly it took one machine from about 150 to about 700 jobs/s end to end (see [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md#11-per-job-overhead-make-perf)). Set `0` to make every write wait for its own fsync: zero acknowledged-job loss, much lower throughput | | `AQUIFER_PEBBLE_CACHE_MB` | `64` | Pebble only: block cache size. Pebble's own default (8MB) is too small to keep recently written jobs in memory, so status updates read them back from disk | | `AQUIFER_DEBUG_ADDR` | unset | Serves Go's `net/http/pprof` (goroutine counts, heap, CPU profiles) on its own listener, e.g. `127.0.0.1:6060`. Off when unset. Bind it to loopback or a private network; it is not meant to be public | | `AQUIFER_MEMORY_LIMIT_MB` | 70% of the container's memory limit, or of the machine's memory | Reject new jobs with `429` once the memory the Go runtime is holding (mapped from the OS, minus what it has returned) exceeds this many MB. Between this and `AQUIFER_MEMORY_HARD_LIMIT_PERCENT` of it, only jobs for a target whose backlog isn't draining are refused, so one dead target can't close intake for the rest. It falls back below the limit once a spike's memory is returned, so intake reopens. `GET /health` shows it as `admission.memory_mb`, beside `memory_reserved_mb` (everything ever mapped). The default is logged at startup and is only a guess: if anything else runs on the machine (Valkey, a sidecar) or the machine is resized, set it yourself. `0` disables it, and then nothing is refused before the kernel's OOM killer acts. Off by default where memory can't be read (macOS). | | `AQUIFER_RECOVERY_PAGE_SIZE` | `1000` | On restart, each queue loads this many of its unfinished jobs before Aquifer starts serving; the rest stay on disk, counted in its backlog, and load a page at a time as it drains. Restart memory then doesn't grow with the backlog. Jobs stored by versions before this load all at once, as before, the first time. | | `AQUIFER_MEMORY_HARD_LIMIT_PERCENT` | `115` | Where memory refuses every new job, as a percent of `AQUIFER_MEMORY_LIMIT_MB`. `100` refuses everything at the limit, the behavior before this setting existed. | | `AQUIFER_MEMORY_STUCK_BACKLOG_SECONDS` | `60` | Over the memory limit, a target is refused when its backlog would take longer than this to drain at the rate its queues have been finishing jobs (sampled every 3 s). A target not yet measured, or with no backlog, isn't refused. `0` refuses every target. | | `AQUIFER_DEAD_TARGET_FAIL_AFTER_SECONDS` | `0` (off) | Once a target has had no successful answer for this many seconds (every attempt a connection error, `5xx`, `408` or `429`), jobs for it that have themselves waited this long fail with reason `target_unavailable` and their webhook, instead of waiting behind a target that is down. Newer jobs keep trying, so the target's first success ends it. Off by default because it fails work callers already handed over; `retry until done` callers expect it to wait. | | `AQUIFER_MAX_SYNC_LAG_MS` | `1000` | Reject new jobs with `429` (`sync_lag`) while accepted jobs may have gone this long without reaching disk. With the default Pebble flush (`AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=100`), an accepted job is written to disk within about 100ms; a crash loses what was accepted since the last sync, so this stops adding to that window when the machine is too busy to keep syncing. Jobs accepted just before still wait for the sync in progress, so the window can outlast this limit while one sync is slow. `0` disables it. `GET /health` shows `admission.sync_lag_ms`. | | `AQUIFER_MAX_BODY_BYTES` | `1048576` (1MB) | Reject oversized request bodies with `413` | | `AQUIFER_DB_MAX_BYTES` | 80% of the disk | Reject new jobs with `429` once the database exceeds this size. Unset, Aquifer sizes it at startup to 80% of the space the database can use (free space on its filesystem plus its current size), so it fits the volume without configuration. The startup log shows the value. Set a number to override, or `0` to disable. Falls back to 800MB where disk space can't be read | | `AQUIFER_MAX_PENDING_PER_UPSTREAM` | `10000` | Shared per-target-API backlog budget used by fair admission; `0` disables it | | `AQUIFER_ACCOUNT_QUEUE` | `on` | `on`: each tenant gets its own queue at every target API; `off`: tenants share one queue per target API. A target API's own `X-Aqueduct-Account-Queue` response header overrides this; a sender can only ask for `on` | | `AQUIFER_METRICS_TENANT_LABELS` | `false` | `true` adds per-tenant (`user_id`) series to `GET /metrics`, for the first 1,000 tenants seen (the rest share `other`). Off by default because tenant IDs are unbounded and often customer data. See [API.md](https://rjpruitt16.github.io/aquifer/api.md#get-metrics) | | `AQUIFER_NODE_ID` | `AQUIFER_CLUSTER_SELF_ID`, else `FLY_MACHINE_ID`, else the host name | This node's id, sent to target APIs as `X-Aqueduct-Node-Id` so a backend can count how many Aquifer nodes are sending to it | | `AQUIFER_RETRY_AFTER_SECONDS` | `5` | Base `Retry-After` value sent on `429` admission rejections | | `AQUIFER_IDLE_TIMEOUT_SECONDS` | `300` (5min) | How long a per-tenant/per-domain queue can sit idle before self-tearing-down; see [drain mode](https://rjpruitt16.github.io/aquifer/partitioning.md) for why this gates a real drain flush | | `AQUIFER_ALLOWED_URL_DOMAINS` | _(none, unrestricted)_ | Comma-separated hostnames `url`-routed jobs are permitted to target; see [`POST /jobs`](https://rjpruitt16.github.io/aquifer/api.md#post-jobs) | | `AQUIFER_FLY_REGIONS` | _(none, feature off)_ | Comma-separated Fly region codes this app is deployed to; enables `/proxy`'s cross-region redirect on Fly. See [`POST /proxy`](https://rjpruitt16.github.io/aquifer/api.md#post-proxy) | | `AQUIFER_FLY_POLL_INTERVAL_SECONDS` | `30` | How often to poll sibling regions over Fly's private network for liveness | | `AQUIFER_REDIRECT_GATE_COOLDOWN_SECONDS` | `500` | How long to stop attempting cross-region redirect after a tour finds no reachable region at all, before trying again: internal probe throttling, not what's told to the caller | | `AQUIFER_REDIRECT_EXHAUSTED_RETRY_AFTER_SECONDS` | `900` (15min) | `Retry-After` sent to the caller when cross-region redirect is configured but exhausted: no known-live region could serve or queue the request. Request is rejected (429), not queued locally. See [`POST /proxy`](https://rjpruitt16.github.io/aquifer/api.md#post-proxy) | Body-size, DB-size and memory admission are all on by default. Memory defaults to 70% of the container's limit (or the machine's memory); `AQUIFER_MEMORY_LIMIT_MB=0` turns it off. Retry-After backs off exponentially under sustained rejection (5s → 10s → 20s → 40s → capped at 60s, resets on the next allowed request). See [CONFIGURATION.md](https://rjpruitt16.github.io/aquifer/configuration-rationale.md) for the full rationale and [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md) for the numbers behind these defaults. --- # Configuration rationale > Why the admission defaults are what they are. Details behind the admission-control defaults and backoff behavior in the README's Configuration table. ## Why body-size and DB-size admission are on by default Aquifer protects itself from the traffic it's meant to be absorbing without needing to be told to. The defaults are sized off the infrastructure this project is actually benchmarked against (a single 512MB Fly.io instance with a 1GB volume — see [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md)); set an explicit `0` to disable a check entirely, or raise it for a bigger deployment. Memory is the exception: there's no safe one-size-fits-all default since it depends on your own deployment's memory budget, not Aquifer's disk usage, so it stays disabled until you set it — Aquifer logs a warning on startup if you haven't (benchmarked safe at 400MB on a 512MB instance, as a starting point). See [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md) for real numbers, including what happens under sustained load, a 10x burst, a memory ceiling, a mid-flight crash, multi-tenant fairness, and capacity/drain time by machine size. ## Retry-After backoff Retry-After backs off exponentially under sustained pressure. A single rejection returns your configured base value (default 5s). Each additional *consecutive* rejection — with no allowed request in between — doubles it: 5s → 10s → 20s → 40s → capped at 60s. The moment a request is allowed again, it resets to the base. This exists so that clients retrying into a sustained overload spread out over time instead of all hammering the same fixed 5-second ceiling forever, which is exactly the pattern that keeps an overloaded instance from ever catching up. ## Fair queue admission `AQUIFER_MAX_PENDING_PER_UPSTREAM` defaults to 10,000 queued plus in-flight jobs per target API. It is a shared safety budget, not a fixed per-user reservation: a lone active account queue may use all of it. Once total backlog exceeds 70%, an above-share queue receives probabilistic `429` responses according to the formula in [API.md](https://rjpruitt16.github.io/aquifer/api.md#fair-queue-admission). The target API can change the budget dynamically with `X-Aqueduct-Max-Backlog`; set either control to `0` to disable count-based fair admission. --- # Framework adapters > The HTTP, MCP stdio and A2A front doors. Aquifer has a framework-neutral core (idempotency, persistence, rate control, dispatch, SSE events, L8 signing, webhook delivery) with pluggable front doors: | Adapter | Env | Purpose | |---------|-----|---------| | HTTP | `AQUIFER_ADAPTER=http` | REST/SSE API on `PORT` (the default) | | MCP stdio | `AQUIFER_ADAPTER=mcp-stdio` | MCP server exposing Aquifer tools over stdio (the published Docker image's default) | | A2A | `AQUIFER_ADAPTER=a2a` | Agent2Agent protocol (v1.0) agent over JSON-RPC/HTTPS | **[ADAPTERS.md](https://rjpruitt16.github.io/aquifer/adapters-reference.md)** has the full reference for each built-in adapter (MCP tool list, A2A Agent Card details), plus how to write your own `FrameworkAdapter`, storage backend, or metrics adapter. --- # Adapters reference > The MCP tools, the A2A Agent Card, and writing your own adapter, storage backend or metrics adapter. Aquifer has a framework-neutral core — idempotency, persistence, rate control, dispatch, SSE events, L8 signing, webhook delivery — with pluggable front doors. This covers the three built-in adapters, then how to write your own. ## Built-in adapters | Adapter | Env | Purpose | |---------|-----|---------| | HTTP | `AQUIFER_ADAPTER=http` | REST/SSE API on `PORT` — see [API.md](https://rjpruitt16.github.io/aquifer/api.md) | | MCP stdio | `AQUIFER_ADAPTER=mcp-stdio` | MCP server exposing Aquifer tools over stdio | | A2A | `AQUIFER_ADAPTER=a2a` | Agent2Agent protocol (v1.0) agent over JSON-RPC/HTTPS on `PORT` | ### MCP stdio ```bash AQUIFER_ADAPTER=mcp-stdio aquifer ``` The published Docker image defaults to `AQUIFER_ADAPTER=mcp-stdio` so MCP directories such as Glama can start and introspect it directly. Set `AQUIFER_ADAPTER=http` when running Aquifer as an HTTP queue service. MCP tools: | Tool | Purpose | |------|---------| | `aquifer_enqueue_job` | Queue an HTTP request for durable, rate-controlled dispatch | | `aquifer_get_job` | Fetch job status and metadata | | `aquifer_health` | Return health and protocol metadata | | `aquifer_l8_metadata` | Return L8 public key metadata | | `aquifer_l8_challenge` | Answer an L8 challenge | MCP resource: `aquifer://jobs/{job_id}` reads current job status and metadata as JSON. The HTTP adapter remains the default for the binary, so existing deployments do not change. ### A2A Run as an A2A agent (JSON-RPC/HTTPS only for v1 — gRPC and REST bindings aren't wired up): ```bash AQUIFER_ADAPTER=a2a AQUIFER_A2A_PUBLIC_URL=http://localhost:8080 aquifer ``` `AQUIFER_A2A_PUBLIC_URL` is the externally-reachable base URL to advertise in the Agent Card — it defaults to `http://localhost:$PORT`, which is only correct for local use; set it explicitly behind any proxy or real deployment. The Agent Card is served at `/.well-known/agent-card.json` (the standard A2A convention); send a `SendMessage`/`SendStreamingMessage` request whose message contains a single data part shaped like `JobRequest` (`user_id`, `idempotent_key`, `url` or `pool_id`, `method`, `headers`, `body`) — the same structured-JSON shape MCP's `aquifer_enqueue_job` tool already takes. The target API response comes back as a task artifact. `CreateTaskPushNotificationConfig` is supported (backed by the SDK's SSRF-hardened HTTP sender); `CancelTask` deliberately returns an unsupported-operation error rather than a silent no-op, since Aquifer has no real job-cancellation mechanism yet. See [a2aadapter/a2a_adapter.go](https://github.com/rjpruitt16/aquifer/blob/main/a2aadapter/a2a_adapter.go) for the full translation between A2A's task model and Aquifer's `Enqueue`/`SubscribeJob`. ## Writing an adapter Adapter authors import Aquifer as a Go package, implement `FrameworkAdapter`, and pass the shared core into their framework. Built-in adapters are selected with `AQUIFER_ADAPTER`; third-party adapters normally ship as small custom binaries that call `aquifer.RunAdapter`. ```go package myframework import ( "context" "github.com/rjpruitt16/aquifer" ) type Adapter struct{} func (a *Adapter) Name() string { return "my-mcp-framework" } func (a *Adapter) Start(ctx context.Context, app *aquifer.Aquifer) error { // Register framework handlers that call: // app.Enqueue(req) // app.GetJob(jobID) // app.SubscribeJob(jobID) // app.Health() return nil } ``` Custom binaries can reuse Aquifer's runtime wiring: ```go package main import ( "context" "log" "os" "os/signal" "syscall" "github.com/rjpruitt16/aquifer" myadapter "github.com/you/your-adapter" ) func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() adapter := myadapter.New() log.Fatal(aquifer.RunAdapter(ctx, adapter, aquifer.RuntimeOptions{ DBPath: "aquifer.db", ConfigPath: "aquifer.yml", })) } ``` `RunAdapter` creates and recovers the runtime, rejects new work before canceling the adapter context, drains accepted jobs and WebSockets, performs the final acknowledged ledger flush, and enforces the configured shutdown deadline. An adapter's `Start` method must return when `ctx` is canceled. HTTP adapters can call `aquifer.ShutdownHTTPServer(server)` to get the same bounded listener shutdown used by the built-in HTTP and A2A adapters. The minimal call remains: ```go adapter := myadapter.New() log.Fatal(aquifer.RunAdapter(ctx, adapter, aquifer.RuntimeOptions{ DBPath: "aquifer.db", ConfigPath: "aquifer.yml", })) ``` See `examples/custom_adapter` for a complete compile-tested adapter binary. ## Writing a storage backend Persistence is also pluggable. Every core component (`Registry`, `AccountQueue`, `URLWorker`, `Aquifer` itself) is coded against the `JobStore` interface, not the concrete SQLite/Pebble types: ```go type JobStore interface { Path() string Close() error CheckOrInsert(job *Job) (string, bool) SetQueueKey(jobID, queueKey string) DeleteJob(jobID string) MarkInFlight(jobID string) RecoverInFlight(queueKey string) []*Job UpdateStatus(jobID string, status Status) Counts() StoreCounts GetJob(jobID string) *Job GetQueuedJobs() []*Job ListIdempotentKeys() []LedgerEntry ClearIdempotentKeys() ListDrainEvents(limit int) []DrainEvent AcknowledgeDrainEventsThrough(sequence int64) } ``` Implement it against your own backend (Postgres, rqlite, or anything else that can give you atomic check-and-set) and pass it via `RuntimeOptions.Store` — no need to bypass `NewRuntime`/`RunAdapter` or hand-wire the lower-level constructors. Three things a custom backend should be aware of: `CheckOrInsert` needs the same atomicity guarantee SQLite's `INSERT OR IGNORE` and Pebble's own store give it today (a non-atomic check-then-write reintroduces the exact idempotency race this project has already found and fixed twice — once in each language); drain mode expects final `completed`/`failed` status updates to become ordered `DrainEvent` rows that are deleted only after `AcknowledgeDrainEventsThrough`; and `AQUIFER_DB_MAX_BYTES` admission control does a local `os.Stat` on `DB_PATH`, which is meaningless for a networked backend — set it to `0` to disable that check if your store isn't a local file or directory. Remote idempotency is a separate extension point: ```go type RemoteIdempotency interface { Lookup(hash string) (RemoteIdempotencyEntry, bool) Record(events []DrainEvent) bool } ``` Pass one with `RuntimeOptions.RemoteIdempotency` if you want something other than the built-in Valkey implementation. Aquifer doesn't ship a Postgres or rqlite backend itself — this is documented as an extension point for anyone who wants multi-instance durability without local-disk-per-instance, not a promise one exists yet. ## Metrics adapter Aquifer emits lifecycle events through a pluggable metrics adapter — implement `MetricsAdapter` and pass it into `NewRegistry`: ```go type MetricsAdapter interface { JobQueued(userID, upstream string) JobDispatched(userID, upstream string) JobCompleted(userID, upstream string, durationMs int64) JobFailed(userID, upstream string, reason string) WebhookDelivered(url string, attempt int) WebhookFailed(url string, attempts int) QueueDepth(upstream string, depth int) FlowRate(upstream string, rps float64) } ``` Aquifer ships with `NoopMetricsAdapter`, so existing deployments do not change. --- # Drain mode > Handing an instance to one tenant at a time: the state machine, settings and ledger payloads. **Off by default.** A normal deployment (a single long-lived instance, or static domain/tenant partitioning as described in [Deployment](https://rjpruitt16.github.io/aquifer/deployment.md)) is completely unaffected unless you explicitly turn this on — no background watchdog runs, no added overhead, nothing about default behavior changes. Aquifer's idempotency store exists to dedupe retries while a burst is actively draining, not to be a permanent system of record. Drain mode is for a specific deployment pattern: instances get handed to a tenant, absorb and drain their burst, then get freed for reassignment to a different tenant. When enabled, Aquifer records completed/failed user jobs into a local drain-event journal. It can stream that journal in acknowledged batches while the instance is still active, and when the instance goes completely idle (no requests anywhere on the whole process, not just one tenant's queue) for `AQUIFER_DRAIN_TIMER_SECONDS`, it flushes any remaining events and only then clears its local ledger — making the instance safe to hand to someone else. **Aquifer does not decide who gets a freed instance next**, and does not retain the ledger itself beyond the next flush. That orchestration — durable long-term storage, and assigning tenants to instances — is entirely up to whatever service you build to receive this webhook. Aquifer only detects idle and hands off what it has. **[canalis-rs](https://github.com/rjpruitt16/canalis-rs) is a real example of that orchestrator, and of why draining to a genuinely stateless handoff is worth building.** A freed instance carries no memory of who it served last, so canalis-rs can hand it to any tenant currently waiting — durably queuing that tenant's work in Valkey if none is free yet — without needing this instance recreated or reconfigured first. Scaling the fleet becomes adding or removing interchangeable instances, not reprovisioning per-tenant ones. **State machine**, visible via `GET /health` (`"drain": {"state": "..."}`, only present when enabled): | State | Meaning | |---|---| | `active` | At least one target API has live work. Normal state, drain mode enabled or not. | | `draining` | Every target API has gone idle, but either the drain timer hasn't elapsed yet or a flush attempt is in flight/being retried. Not yet safe to hand off. | | `unassigned` | The ledger was flushed (or there was nothing to flush) and local state is clear — safe to hand off. Reverts to `active` the instant new work arrives. | `unassigned` is a status label, not an access gate — Aquifer keeps accepting new jobs in every state. Nothing stops a job from landing on an instance mid-handoff; if your orchestrator needs a hard guarantee that never happens, enforce it on your own end before routing traffic there. **Env vars:** | Var | Default | Notes | |---|---|---| | `AQUIFER_DRAIN_ENABLED` | `false` | The real gate — the other drain vars only matter when this is `true`. | | `AQUIFER_DRAIN_TIMER_SECONDS` | `45` | How long the whole instance must be idle before flushing. Deliberately separate from the per-tenant-queue self-GC timer below, which reclaims one queue's memory and has nothing to do with instance-wide handoff — but drain mode's own countdown only starts once every queue has already self-torn-down via that timer, so a real drain flush is gated by both. | | `AQUIFER_DRAIN_SINK` | `webhook` | `webhook` posts batch payloads to `AQUIFER_DRAIN_WEBHOOK_URL`; `valkey` writes idempotency entries directly to Valkey. | | `AQUIFER_DRAIN_WEBHOOK_URL` | *(none)* | Required when `AQUIFER_DRAIN_SINK=webhook` — if unset, drain mode logs a warning and stays off rather than flushing with nowhere to send it. | | `AQUIFER_DRAIN_BATCH_ENABLED` | `false` | When true, Aquifer periodically sends pending drain events before the final idle flush. | | `AQUIFER_DRAIN_BATCH_INTERVAL_SECONDS` | `60` | Periodic batch interval. Each wait gets a small jitter so a fleet does not synchronize batch sends against the same URL. | | `AQUIFER_DRAIN_BATCH_MAX_EVENTS` | `1000` | Maximum events sent in one webhook payload. | | `AQUIFER_VALKEY_URL` | *(none)* | Required when `AQUIFER_DRAIN_SINK=valkey`, and also used by remote idempotency lookup. Supports `redis://` and `valkey://` URLs. | | `AQUIFER_REMOTE_IDEMPOTENCY_PREFIX` | `aqueduct:idempotency:` | Key prefix for Valkey idempotency entries. | | `AQUIFER_REMOTE_IDEMPOTENCY_TTL_SECONDS` | `7200` | TTL for remote idempotency entries written to Valkey. | | `AQUIFER_REMOTE_RESULT_ENABLED` | `false` | Also writes bounded terminal result snapshots to Valkey. | | `AQUIFER_REMOTE_RESULT_PREFIX` | `aqueduct:result:` | Key prefix for terminal result snapshots. | | `AQUIFER_REMOTE_RESULT_MAX_BYTES` | `65536` | Maximum response-body bytes stored in each result snapshot; `0` stores metadata only. | | `AQUIFER_IDLE_TIMEOUT_SECONDS` | `300` (5min) | The per-tenant-queue self-GC timer itself. Exists mainly so contract tests don't have to burn 5+ real minutes to prove a real drain flush — leave this at the default in production. | **Webhook payloads:** Periodic batches use `event: "ledger_batch"`: ```json { "event": "ledger_batch", "batch_id": "12-48", "sequence_start": 12, "sequence_end": 48, "flushed_at": "2026-08-23T14:02:11Z", "ledger": [ { "sequence": 12, "idempotent_key_hash": "3fa9c1...", "job_id": "a3f9...", "status": "completed", "recorded_at": 1798053731000 } ] } ``` The final idle flush uses the same payload shape with `event: "instance_idle"`. ```json { "event": "instance_idle", "batch_id": "49-50", "sequence_start": 49, "sequence_end": 50, "flushed_at": "2026-08-23T14:02:11Z", "ledger": [ { "sequence": 49, "idempotent_key_hash": "3fa9c1...", "job_id": "a3f9...", "status": "completed", "recorded_at": 1798053731000 } ] } ``` Aquifer deletes drain events only after the webhook returns `2xx`. A failed delivery leaves the batch in local storage and retries it later. The downstream receiver should still treat `(instance_id or sender identity, sequence)` or `batch_id` as idempotent, because webhook delivery remains at-least-once across process/network failures. When `AQUIFER_DRAIN_SINK=valkey`, Aquifer skips the webhook payload and writes each event directly to: ```txt {AQUIFER_REMOTE_IDEMPOTENCY_PREFIX}{idempotent_key_hash} ``` The value is: ```json { "job_id": "a3f9...", "status": "completed", "recorded_at": 1798053731000, "source": "aquifer", "result_key": "aqueduct:result:..." } ``` Those Valkey writes are treated the same way as webhook delivery: only successful writes are acknowledged and deleted from Aquifer's local drain-event journal. If `AQUIFER_REMOTE_RESULT_ENABLED=true`, Aquifer also writes: ```txt {AQUIFER_REMOTE_RESULT_PREFIX}{idempotent_key_hash} ``` with the terminal response status, content type, body up to `AQUIFER_REMOTE_RESULT_MAX_BYTES`, and a `body_truncated` flag. The idempotency entry's `result_key` points at that snapshot. `idempotent_key_hash` is `sha256(user_id + ":" + idempotent_key)`, hex-encoded lowercase — the exact hash Aquifer already computes internally, never the plaintext key. A downstream consumer re-checking a key for a duplicate must hash it the same way. If you're also running [Spillway](https://github.com/rjpruitt16/spillway), its drain mode hashes the identical way — both systems share one hash-key namespace for the same `(user_id, idempotent_key)` pair, so a downstream consumer can hash lookups the same way regardless of which system a given ledger entry came from. --- # External registration > Reporting an instance to an external control plane: settings and payloads. **Off by default.** A normal deployment (a single long-lived instance, or static domain/tenant partitioning as described in [Deployment](https://rjpruitt16.github.io/aquifer/deployment.md)) is completely unaffected unless you explicitly turn this on — no background goroutine runs, no added overhead, nothing about default behavior changes. When `AQUIFER_REGISTRY_URL` is set, this instance periodically reports its own listening port to that URL — so an external control plane (Canalis, or anything else someone builds against the same contract) can assign tenants to it. Generic on purpose: Aquifer has no idea what's on the other end, and doesn't need to — it's a webhook target, delivered through the same retrying, L8-signed path drain mode's ledger flush already uses. **Only the port is reported, not a full address.** Whatever receives the ping already sees the real source IP on the connection itself — a self-reported address would just be redundant with that, and less trustworthy (Aquifer claiming its own reachable address is a weaker guarantee than the receiver's own connection telling it directly). **Aquifer does not decide who gets assigned where.** That orchestration — assignment, scaling, the whole control-plane layer — is entirely up to whatever service receives these pings. Aquifer only reports that it exists and is reachable. **Env vars:** | Var | Default | Notes | |---|---|---| | `AQUIFER_REGISTRY_URL` | *(none)* | The real gate — unset means no goroutine ever starts. | | `AQUIFER_REGISTRY_INTERVAL_SECONDS` | `15` | How often to ping. Pings once immediately on start too, so a freshly-booted instance doesn't sit unregistered for the first interval. | | `PORT` | `8080` | Already read elsewhere (the port Aquifer itself listens on) — reused here rather than introducing a second way to say the same thing. | | `AQUIFER_REGISTRY_RESERVED_FOR` | *(none)* | A `user_id` this instance is dedicated to. Reported as `reserved_for`; a control plane should route only that user here and never hand the instance to anyone else. | | `AQUIFER_REGISTRY_POOLS` | *(none)* | Comma-separated pools this instance joins, e.g. `gpu,eu`. Reported as `pools`; a control plane like Canalis only sends it requests for those pools. None means the `default` pool. Every ping carries the current list, so changing it (and restarting) is how an instance joins or leaves a pool. | | `AQUIFER_REGISTRY_ROLE` | *(none)* | What kind of traffic this instance is for, e.g. `http` or `ws` (lowercased). Reported as `role`; lets a control plane send WebSocket and HTTP traffic to instances dedicated to each. | **Ping payload:** ```json { "port": "8080", "state": "active", "reported_at": "2026-09-03T22:00:00Z", "reserved_for": "acme", "role": "http", "pools": ["gpu", "eu"], "drain_state": "unassigned", "fullness": 35, "http_users": 12, "http_capacity": 70, "ws_sessions": 100, "ws_capacity": 1000 } ``` `fullness` is how full this instance is, `0`–`100`: the same number as the `X-Aqueduct-Fullness` response header (see [API.md](https://rjpruitt16.github.io/aquifer/api.md)), so a control plane knows before sending an instance anything. ### Load: HTTP users and WebSocket sessions `http_users` is how many users have jobs or webhook deliveries still owed on this instance, and `ws_sessions` how many WebSocket clients are connected. `http_capacity` and `ws_capacity` are how many of each it can hold. A control plane like Canalis gives the instance no more HTTP users or sessions than that, counting each kind on its own, so open sessions never use up the room HTTP users need. Every response carries the same four numbers as `X-Aqueduct-Http-Users`, `X-Aqueduct-Http-Capacity`, `X-Aqueduct-WS-Sessions` and `X-Aqueduct-WS-Capacity`. Both kinds of traffic spend the same CPU, so the capacities come from one budget: what one kind isn't using, the other can have. At most every 5 seconds, Aquifer reads the CPU time the process used, and how many jobs and WebSocket messages it handled. Readings are averaged over about 30 seconds. The CPU in use is split between the two kinds by what each costs, measured in [benchmark.md](https://rjpruitt16.github.io/aquifer/benchmarks.md): about 750 jobs/s and 1,500 WebSocket round trips/s per CPU, which is 3,000 messages, since each round trip is a client message and a backend one. That gives the CPU one user or one session costs, and: ``` headroom = 80% of CPU - CPU in use http_capacity = http_users + headroom / CPU per user (at most 10,000) ws_capacity = the least of ws_sessions + headroom / CPU per session ws_sessions + memory left under AQUIFER_MEMORY_LIMIT_MB / 90 KB AQUIFER_WS_MAX_CLIENT_CONNECTIONS ``` Neither number is ever below what the instance already has. With nothing to measure yet, a user is taken to send 10 jobs/s and a session 1 message/s: an idle 1-CPU instance reports 60 users and 1,000 sessions, an idle 2-CPU instance 120 users. Spillway reports the same four numbers the same way, with its own costs. `reserved_for`, `role` and `pools` are only present when their env vars are set. `drain_state` is only present when drain mode is enabled ([DRAIN_MODE.md](https://rjpruitt16.github.io/aquifer/drain-mode.md)): it's drain mode's own `active` / `draining` / `unassigned`, kept separate from `state` because drain mode's "draining" means "finishing one tenant's burst", not "shutting down". Reaching `unassigned` sends a ping right away instead of waiting for the next heartbeat, since that's the control plane's cue to hand this instance to someone else. `state` is `active`, `draining`, or `offline`. Normal heartbeats report `active`. On `SIGTERM`, Aquifer immediately sends `draining` before its quiescence window so the control plane can stop assigning new work, then makes one best-effort `offline` report after accepted work and WebSockets have drained. Receivers should still expire nodes that stop heartbeating because abrupt termination cannot send either transition. --- # Benchmarks > Throughput, crash recovery, memory, capacity per machine size, and the GPU run. > Sections 1–7 ran before v0.13.0 (2026-10-05), when SQLite was the default store; Pebble is the default now. Sections 8 and 11 compare both, and 12–13 ran on Pebble. Real runs against a live deployment, not simulated numbers. Target: `aquifer-bench.fly.dev`, a single `shared-cpu-1x`/512MB Fly.io machine in `iad`, `AQUIFER_MEMORY_LIMIT_MB=400`, default pacing (2 RPS/1 concurrent per target API domain, no `CONFIG_PATH`). Load generated with [vegeta](https://github.com/tsenart/vegeta); scripts are in [`benchmark/`](https://github.com/rjpruitt16/aquifer/blob/main/benchmark). Scenarios 1-6 dispatch to a dummy target API (`postman-echo.com`) at the default 2 req/s ceiling — intentional, since Aquifer protects target APIs that haven't told it otherwise. A large batch queues for a while before draining; that's the configured ceiling working as intended, not a bottleneck. Production sets real per-domain rates via `CONFIG_PATH`. Scenario 7 deliberately raises the ceiling so machine size, not the default rate limit, is what's measured. --- ## 1. Sustained throughput (SQLite, before v0.13.0) Measures acceptance (`POST /jobs` → 201), not end-to-end dispatch — ingest and dispatch are decoupled by design. **50 req/s for 30s (1,500 requests):** ``` Requests 1500, rate 50.03, throughput 49.93 Latencies min 53.5ms, mean 69.8ms, p50 65.0ms, p90 72.2ms, p99 187.2ms, max 565.6ms Success 100.00% | Status 201:1500 ``` 100% acceptance, ~65ms median latency, no degradation over the run. --- ## 2. Burst absorption (retry-storm simulation) (SQLite, before v0.13.0) Baseline traffic, a 10x spike for 30s, then back to baseline. ``` Phase: baseline (10/s, 15s) — 150 req, 100% success, p99 194ms Phase: burst (100/s, 30s) — 3000 req, 100% success, p99 166ms Phase: recovery (10/s, 15s) — 150 req, 100% success, p99 189ms ``` 100% success through the entire 10x spike, no 5xx, no dropped connections. Recovery-phase latency matches baseline — the burst gets absorbed into the queue instead of hammering the target API. --- ## 3. Admission control under memory pressure (SQLite, before v0.13.0) Aquifer sheds load with clean `429`s (never 5xx, never a crash) once memory exceeds a configured ceiling. At the realistic setting (`AQUIFER_MEMORY_LIMIT_MB=400`), a 150 req/s, 45s hammer (6,750 requests, 100% accepted) never pushed the process past ~21MB resident — the ceiling was never exercised, which is itself a useful data point about baseline memory footprint. To demonstrate shedding, redeployed with `AQUIFER_MEMORY_LIMIT_MB=15` (deliberately below resting footprint): ``` 600 requests, 30.05 req/s → 0.00% success, 429:600 (all rejected) ``` Every rejected request gets a clean `429` with a structured body and `Retry-After`: ``` HTTP/2 429 retry-after: 5 {"current":21,"error":"admission rejected: memory at 21 exceeds limit 15","limit":15,"limit_reason":"memory"} ``` No crash, no hang, no partial writes. **One subtlety:** a rejected request's row is deleted — it was never durably accepted. The idempotency guarantee only covers jobs already accepted before the limit tripped, not ones that were shed. Verified by `TestAdmissionDuplicateStillSucceedsUnderPressure` and `TestAdmissionRejectedJobLeavesNoGhostRow`. --- ## 4. Crash recovery (SQLite, before v0.13.0) 30 jobs enqueued, machine `SIGKILL`'d ~3s in (mid-dispatch), restarted, polled for up to 60s: ``` enqueued: 30 | completed: 30 | failed: 0 | still queued: 0 | lost: 0 PASS: all 30 jobs reached a real terminal state ``` Every job persisted to SQLite before dispatch, so a kill mid-flight just means the recovery loop re-enqueues it on restart. --- ## 5. Multi-tenant fairness (`X-Aqueduct-Account-Queue`) (SQLite, before v0.13.0) Without isolation, every job hitting the same target API domain shares one dispatch queue — a noisy tenant can starve everyone behind it. Isolation per `(user_id, api_key)` is on by default (`AQUIFER_ACCOUNT_QUEUE`); a target API can switch it off for itself with `X-Aqueduct-Account-Queue: disabled`. `TestABrokenTenantDoesNotStallAHealthyOneOnTheSameHost` measures it: with a broken tenant retrying a 40-job burst, a healthy tenant on the same host finishes 20 jobs in ~0.2s with isolation and 0 of 20 in 5s without it. **Surfaced a real bug during benchmarking:** the isolation mechanism existed, but the handler that turns it on was never actually wired into the HTTP request path — every job silently shared one queue regardless of the header. Filed and fixed as [issue #5](https://github.com/rjpruitt16/aquifer/issues/5), covered by `TestAccountQueueHeaderIsolatesTenants` and `TestAccountQueueHeaderOmittedSharesQueue`. With the header now working — a noisy tenant flooding 100 concurrent jobs alongside a quiet tenant sending 5 steady jobs: ``` quiet jobs: 5s, 4s, 4s, 2s, 1s ``` completing on their own schedule regardless of the flood. Before the fix, the same scenario left the quiet tenant stuck for 37-52s each. --- ## 6. Retry-After backs off exponentially under sustained pressure (SQLite, before v0.13.0) A single `429` returns the base value (default 5s). Consecutive rejections double it, capped at 60s; the next allowed request resets it. Verified against a live instance held over its memory ceiling: ``` attempt 1-5: retry-after = 5, 10, 20, 40, 60 ``` This stops clients from hammering a fixed 5-second ceiling forever once an instance is genuinely overloaded, giving it room to drain. --- ## 7. Capacity and drain time (SQLite, before v0.13.0) Started as "what machine size fits a given traffic shape." Found something bigger: a real correctness/scaling bug in the SQLite layer, unrelated to machine size. **What the first pass showed:** `aquifer-bench` redeployed at 256/512/1024MB (1 vCPU held constant) — all three broke at the identical point: 100% success through 100 req/s, then ~68-73% at 200 req/s, with connection failures at only ~110MB memory use. A follow-up sweep across real CPU tiers (1/2/4 vCPU, paired memory) broke at the same point too: 68.3%, 68.3%, 69.5%. That ruled out memory and CPU as the bottleneck. **The real cause:** `store.go` had `db.SetMaxOpenConns(1)` — every request queued behind one SQLite connection regardless of machine size. Raising the pool size alone didn't fix it either: WAL/busy-timeout pragmas were being set via `db.Exec()` on whichever single connection served that call, so opening more connections meant most of them silently ran with SQLite's unsafe defaults and failed under concurrency. Fixed by moving the pragmas into the connection DSN itself, so every pooled connection gets them. That fix surfaced a third issue: `CheckOrInsert` checked for duplicates via `INSERT OR IGNORE` then a separate `SELECT` — non-atomic, so a raced `INSERT` could make the `SELECT` find nothing and report a **brand-new job as a duplicate with an empty job ID**, silently discarding it. Latent the whole time; only triggerable once the pool had more than one connection. Fixed by reading `INSERT`'s own `RowsAffected()` instead of a racy read-after-write. Covered by `TestStoreHandlesConcurrentInserts` (50 concurrent inserts, unique keys, none misreported). **After the fix**, same 1 vCPU/512MB instance: | Rate | Before fix | After fix | |------|-----------|-----------| | 25-100/s | 100% success | 100% success | | 200/s | ~68-73% success, connection refused | 74-100% success (variance), no refusals — remaining failures are slow, not refused | | 400/s | *(not tested — already broken at 200)* | 15.4% success, mean latency 28.6s, memory 107→264MB — a real, load-driven ceiling | **Drain time** (500 jobs against an idle instance) was identical across every configuration, before and after the fix: 75-79 seconds — governed by the configured dispatch pace, not CPU/RAM. **Takeaway:** a ceiling that doesn't move when you change the resource you'd expect to relieve it is a signal to check the code path, not the instance size. ### Checklist for picking a size - [ ] **Traffic sustains under ~200 req/s?** Any size works, including 256MB. - [ ] **Traffic sustains above ~200-300 req/s?** Re-run `benchmark/capacity_by_size.sh` against your own traffic shape first — verify the ceiling is actually hardware-bound, not code-level. - [ ] **Bursts followed by quiet periods?** A 500-job burst drains in ~75-79s at a 50 RPS dispatch pace, regardless of machine size — scale linearly against your own `CONFIG_PATH` rate. - [ ] **A ceiling looks identical no matter what you scale?** Treat it as a code-path question, not a bigger-machine question. - [ ] **Need per-tenant fairness under a shared target API?** It's on by default; check nobody set `AQUIFER_ACCOUNT_QUEUE=off` and that the target API doesn't answer `X-Aqueduct-Account-Queue: disabled`. --- ## 8. An alternative storage backend: Pebble Spillway (Elixir/Mnesia) showed a materially higher throughput ceiling than SQLite here — Mnesia writes memory-first with disk catching up in the background, SQLite is disk-first and single-writer. That raised the question: would a memory-first store change Aquifer's ceiling too, without giving up Go? Added `PebbleStore` ([cockroachdb/pebble](https://github.com/cockroachdb/pebble), pure Go, no CGo) behind the `JobStore` interface, opt-in via `AQUIFER_STORE_BACKEND=pebble` — default stays SQLite, additive not a rewrite. **A durability gap Pebble doesn't warn you about:** the assumption going in was that `WriteOptions.Sync: false` behaves like SQLite's `synchronous=NORMAL` (writes through to the OS, only relaxes fsync). Wrong — caught by the same write/`kill -9`/restart/read test used throughout this suite: ``` sync_write: {:atomic, :ok} # kill -9 immediately after, restart [JOB 1] WAL file stopped reading at offset 0; replayed 0 keys read err: pebble: not found ``` Unlike SQLite, `Sync: false` never writes through to the OS at all. `Sync: true` is required for any crash-durability guarantee; confirmed it survives the identical kill test once set. Pebble's `WALMinSyncInterval` (its own group-commit) batches concurrent syncs into fewer real fsyncs while still blocking each caller until its own write is durable — no silent-loss window. Configurable via `AQUIFER_PEBBLE_WAL_SYNC_INTERVAL_MS` (default 5ms). Verified: 30/30 jobs survive a real `kill -9` on Fly.io, matching SQLite and Mnesia. Idempotency also needed explicit handling: Pebble has no insert-if-absent primitive, so a naive `Get`-then-`Set` would reproduce the same race already fixed in SQLite and Mnesia. Guarded by a lock striped across 256 shards keyed by idempotent hash. Covered by `TestPebbleStoreHandlesConcurrentInserts`. **Throughput**, same tier, same ramp: | Rate | SQLite (post-fix) | Pebble | |------|--------------------|--------| | 50/s | 100% success, mean 69.6ms | 100% success, mean 72.7ms | | 200/s | 74-100% success (variance) | 100% success, mean 1.68s | | 400/s | 15.4% success — real ceiling | 100% success, mean 11.4s | | 600/s | *(already broken)* | 66.6% success | | 800/s | *(already broken)* | 40.0% success | | 1000/s | *(already broken)* | 17.2% success | Pebble roughly doubles the point where Aquifer stops reliably completing requests (~200-400 req/s vs. ~400-600 req/s). The failure mode differs too: SQLite fails outright past its ceiling; Pebble keeps succeeding well past its comfort zone, just slower — arguably the better mode for reliable eventual delivery, though it makes Pebble's "ceiling" fuzzier to define than SQLite's hard wall. **CPU scaling:** 4 vCPUs instead of 1, same rates — essentially no difference (400/s: 11.4s → 10.7s mean; 600/s: 66.6% → 60.4%; 800/s: 40.0% → 38.1%, all within noise). The opposite of Mnesia's result. Likely reason: Pebble's WAL/memtable write path is a single ordered pipeline per instance — faster than SQLite's B-tree, but still fundamentally serialized, unlike Mnesia's per-request independent BEAM processes. More cores help when the bottleneck is application concurrency; they don't when it's a storage engine's own internal serialization. **Takeaway:** Pebble is a legitimate, meaningfully faster alternative to SQLite for this workload (`AQUIFER_STORE_BACKEND=pebble`), not a magic unlock past ~600 req/s — that ceiling is the storage engine's serialized commit path. Horizontal partitioning remains the right lever past that point, for either backend. --- ## 9. GPU inference and the retry tax (RunPod/vLLM) Everything above dispatches to a dummy HTTP echo target API. This one targets a real GPU: `Qwen/Qwen2.5-1.5B-Instruct` on a single RunPod RTX PRO 4500 Blackwell running vLLM (`--max-num-seqs 48`, KV cache deliberately capped at 512MB so a realistic burst can actually pressure it), pacing driven entirely by the ORCA fallback signal (see [Dynamic pacing](https://rjpruitt16.github.io/aquifer/pacing.md)) — `kv_cache_usage_perc` on vLLM's own response headers, no `X-Aqueduct-*` config needed. Requests forced to `min_tokens=max_tokens=300` so each one holds GPU/KV-cache resources long enough to matter, not a one-token echo. **Same offered load (40 req/s for 30s), direct-to-vLLM vs. through Aquifer:** | | Direct | Through Aquifer | |---|---|---| | Client success | 100% | 100% | | Client mean latency | 11.2s | 1.0ms (durable enqueue, dispatch decoupled) | | Client p99 latency | 20.1s | 3.2ms | | Peak vLLM-side queue depth | 449 waiting | 8 waiting | | Peak `kv_cache_usage_perc` | 69.4% | 59.1% | Hitting vLLM directly at 40 req/s doesn't fail — vLLM queues everything internally rather than rejecting — but its own backlog balloons to 449 requests deep, and callers wait up to 20 seconds for a response that eventually succeeds. That's the retry tax: technically 100% success, but exactly the shape of slowness that makes a caller give up and retry, doubling load on a GPU that's already behind. Through Aquifer, the burst is durably absorbed in ~1ms per request and dispatched to vLLM at a controlled pace; vLLM's own internal queue barely builds. **Does the ORCA signal itself actually engage** (not just a static rate cap doing the work)? Loosened Aquifer's static ceiling above what vLLM can sustain (80 RPS / 120 concurrent vs. vLLM's real ~48-concurrent capacity) and pushed a sustained 60 req/s for 40s. `kv_cache_usage_perc` oscillated 22% → 79% → 22% → ... in a repeating cycle: cache crosses the 70% threshold, ORCA tells Aquifer to cut to 2 RPS, backlog drains, cache falls, Aquifer ramps back up (5%/dispatch recovery), cache climbs again. A real, self-regulating feedback loop against a live GPU, not a static number that happened to be tuned right. **Takeaway:** the durable-queue-plus-paced-dispatch model already prevents the worst of the retry tax on its own; ORCA adds automatic, no-config self-regulation on top for backends that report their own load, which vLLM already does. --- ## 10. GPU retry tax, measured in tokens (RunPod/vLLM) Section 9 above measures queue depth and latency. It doesn't measure the thing the retry-tax thesis actually claims: that a caller giving up and retrying makes the GPU do real, wasted generation work. This section closes that gap with two new numbers, computed from vLLM's own `/metrics` (`generation_tokens_total`) and a client harness that behaves like a real caller — times out and retries instead of waiting patiently forever (`benchmark/retry_tax_harness.py`, `benchmark/analyze_retry_tax.py`). **raw tokens/sec** — everything the GPU actually generated (ground truth, from vLLM's own counter), including generations for requests whose caller already gave up and retried. **useful tokens/sec** — tokens that actually reached a caller who was still listening. **wasted = raw − useful.** This is the number that proves or kills the "GPU waste" claim; raw throughput alone can't, since a GPU generating duplicates is still just as busy. Same instance as section 9 (RunPod RTX PRO 4500 Blackwell, `Qwen/Qwen2.5-1.5B-Instruct`, `--max-num-seqs 48`, 512MB KV-cache cap), offered load ramped 20 → 300 req/s over 60s — aggressive enough to actually saturate this GPU, unlike section 9's fixed 40 req/s: | | Direct, retrying client (5s timeout, 3 retries) | Through Aquifer, concurrency matched to vLLM's real limit | |---|---|---| | Logical requests | 9,537 | 9,585 | | Success rate | 0.9% | 25.2% | | **Wasted generation (raw − useful)** | **96.5%** | **22.0%** | | vLLM's own queue depth (peak) | 18,079 | **0** | Hitting vLLM directly with a client that actually retries (unlike section 9's patient client) is close to a worst case: 96.5% of everything the GPU generated was thrown away, spent on attempts whose caller had already given up and resubmitted. The GPU is fully busy the entire time — that busyness is exactly the trap: raw throughput alone would make this look fine. Through Aquifer, with its concurrency ceiling set at vLLM's real capacity (45, just under its `--max-num-seqs 48`), wasted generation drops to 22.0% and vLLM's own queue depth never leaves zero for the whole run — Aquifer's durable queue absorbs the backlog instead of vLLM's internal one taking the hit. **Worth knowing when setting that ceiling:** Aquifer's ORCA fallback (section 9) paces down based on `kv_cache_usage_perc` — memory pressure specifically. A deployment bottlenecked on scheduler concurrency rather than memory can be fully saturated while that signal stays quiet, since it isn't watching the thing that's actually full. Concretely, `--max-num-seqs 48` means kv-cache usage per concurrently-running sequence may never cross ORCA's 0.70 pace-down threshold even at full 48-slot occupancy, depending on sequence length and cache budget. Set a static `max_concurrent` at or below the backend's real capacity rather than relying on ORCA alone to catch a concurrency-driven ceiling — it's built to catch memory pressure, not slot exhaustion. Auto-detecting that real ceiling instead of requiring it to be set by hand is filed as a follow-up: [rjpruitt16/aquifer#11](https://github.com/rjpruitt16/aquifer/issues/11). --- ## 11. Per-job overhead (`make perf`) `make perf` runs two Go benchmarks in-process against both storage backends, with a local target API and webhook receiver that answer instantly and pacing set high enough never to wait. They measure Aquifer's own cost, so a change that slows the hot path shows up here before it shows up in production. Run it before and after a change on the same machine and compare. - **`BenchmarkJobLatency`** sends one job at a time and splits its trip into `accept_ms` (validate and durably store, i.e. `POST /jobs` returns), `queue_ms` (accepted until the target API receives the request) and `total_ms`. - **`BenchmarkPipelineThroughput`** has 8 concurrent callers submit jobs and reports `jobs/s` that make it all the way through: accepted, dispatched, completed and their webhook delivered. Baseline, 2026-10-06, Apple M3 Max laptop (macOS, 200 jobs per run): | Backend | accept | queue | total | pipeline | |---|---:|---:|---:|---:| | SQLite | 0.55 ms | 0.13 ms | 0.67 ms | 571 jobs/s | | Pebble, `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0` (sync every write) | 9.90 ms | 0.23 ms | 10.13 ms | 100 jobs/s | | Pebble, 100ms flush (default) | 0.04 ms | 0.07 ms | 0.11 ms | 11,107 jobs/s | | Pebble, `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=10` | 0.05 ms | 0.08 ms | 0.13 ms | 9,593 jobs/s | The flush-interval rows (2000 jobs each; the default is 100ms) acknowledge a write before it is on disk and sync the log on a timer, the same trade Spillway makes with its 100ms Mnesia flush: a crash can lose up to that window of accepted jobs. `pebble_flush_test.go` checks that a process killed after the window loses nothing. On a Mac most of the gain comes from skipping slow `fsync`s; expect a smaller gap on Linux, where a sync took about 0.3ms on a Fly volume. Read these as relative numbers, not capacity, and don't compare the two rows as like for like. SQLite runs in WAL mode with `synchronous=NORMAL`: a commit survives a process crash but isn't flushed to disk per write, so a power loss can drop the last few accepted jobs. Pebble with `AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0` syncs every write (zero acknowledged-job loss) with a 5ms group-commit window, and macOS `fsync` is far slower than Linux, so on a Mac a lone job waits out that window plus a full flush. On a Fly performance-1x volume, `fsync` measured about 0.3ms p50 and Pebble outran SQLite (see [section 8](#8-an-alternative-storage-backend-pebble)). Dispatch itself (`queue_ms`) is a fraction of a millisecond on both; the cost of a job is its storage writes. On Fly (performance-1x with a volume, every job also delivering a webhook), syncing every write sustained about 150 jobs/s end to end and the 100ms flush about 700, which is why 100ms is the default. --- ## 12. Capacity per machine (Pebble, Fly.io, 2026-10-07) How many jobs per second one Aquifer machine sustains end to end, measured on Fly in `iad`: - **Target:** one Aquifer machine with a volume and Pebble. - **Load generator:** a separate machine running [`benchmark/loadgen`](https://github.com/rjpruitt16/aquifer/blob/main/benchmark/loadgen/main.go). It ramps `POST /jobs` open-loop across 50 users and also serves as the target API and the webhook receiver, both answering instantly. - **Webhooks:** every job also delivers a webhook, so each job is one request in and two requests out. - **Config:** a high rate ceiling (`defaults: {rps: 100000, max_concurrent: 256}`) and the 100ms flush (`AQUIFER_PEBBLE_FLUSH_INTERVAL_MS`, the default). A step counts as sustained when Aquifer accepts what's offered and completes jobs as fast as it accepts them, with the backlog staying flat. | Machine | Sustained jobs/s | p99, submit to webhook | Notes | |---|---:|---:|---| | performance-1x, sync every write (`FLUSH_INTERVAL_MS=0`) | ~150 | 31 ms at 100/s | durable before acknowledging | | performance-1x | **~1,000** | ~0.6 s | one core is the limit | | performance-2x | **~1,500** | ~0.4 s | | | performance-4x | **~3,000**, sometimes 4,000 | 56 ms at 3,000 | 4,000 held on one run, shed on two | Past the ceiling Aquifer sheds with `429` rather than falling over. **What moved the ceiling.** Each stage was found by profiling Aquifer under load on Fly with `pprof`. | Stage | Changes | Result | |---|---|---| | 1 | 100ms batched WAL sync instead of syncing every write | 1x: ~150 → ~700 | | 2 | Cache admission readings (`ReadMemStats` and a walk of the Pebble directory ran on every submission); 64 MB Pebble block cache (`AQUIFER_PEBBLE_CACHE_MB`); no drain events unless drain mode is on; skip queue expiry scans until a deadline is due; one pooled HTTP client instead of a new one per request | 1x: ~700 → ~1,000, but 2x no faster (one core busy) | | 3 | Registry and worker locks cover only decisions, not store writes or queue handoffs; cleanup reads an expiry index instead of decoding every retained job | 2x: uses both cores | | 4 | Remember webhook receivers without L8 for 5 minutes instead of probing `/.well-known/l8` before every webhook over a fresh connection (`connect()` was 65% of CPU on 2x) | 2x: ~1,000 → ~1,500; 4x: ~3,000 | **Caveats.** - **These are best-case numbers.** With an instant target API they measure Aquifer's own cost per job. A slow target API doesn't change that cost, but it does mean more requests in flight at once. - **The 4x results vary.** Finished jobs are kept for 30 minutes, so back-to-back runs leave a larger store behind. A fresh store held 3,000/s cleanly; 4,000/s held on one run out of three. - **The load generator can be the limit.** On a performance-2x generator, 6,000/s produced client-side errors, so the 4x ceiling was probed with a performance-4x generator. **Reproduce:** build `benchmark/loadgen` for linux/amd64, run it on one machine, and point it at Aquifer over the private network: ```bash loadgen -target http://.internal:8080 -self http://.internal:8080 -rates 700,1000,1500,2000,3000,4000 -step 20s ``` --- ## 13. WebSocket sessions per machine (Fly.io, 2026-10-10) How many `aqueduct.v1` WebSocket sessions one machine holds, and how many messages it moves through them. Measured on Fly in `iad`: - **Target:** one performance-1x machine (1 CPU, 2 GB). - **Valkey:** on its own performance-1x machine. - **Backend:** a no-limits backend that acks each command and sends one event (`aquifer-websocket-bench -backend`). - **Client:** a performance-2x machine running [`cmd/aquifer-websocket-bench`](https://github.com/rjpruitt16/aquifer/blob/main/cmd/aquifer-websocket-bench/main.go). Run with the Valkey reader-pool fix (#33). Lab configs are in [`benchmark/websocket`](https://github.com/rjpruitt16/aquifer/blob/main/benchmark/websocket/README.md). Spillway was measured on the same lab for comparison. **Sessions held (idle):** | Sessions | Aquifer connected | Aquifer memory | Aquifer connect p50 / p99 | Spillway connected | Spillway memory | Spillway connect p50 / p99 | |---:|---:|---:|---:|---:|---:|---:| | 1,000 | 1,000 | 110 MB | 6 / 35 ms | 1,000 | 244 MB | 260 / 693 ms | | 5,000 | 5,000 | 493 MB | 7 / 351 ms | 5,000 | 498 MB | 750 ms / 6.2 s | | 10,000 | 10,000 | 991 MB | 13 ms / 1.0 s | 10,000 | 818 MB | 1.9 / 8.9 s | | 20,000 | ran out of memory near 15,000 | | | 20,000 | 1.16 GB | 5.2 / 44 s | - **Aquifer:** about 90 KB per session, mostly six goroutines and socket buffers. Once sessions close, live heap returns to baseline: after a forced GC it was 30 MB live with no sessions. - **Spillway:** about 50 KB per session, but every new session goes through one queue process, which makes opening slow at scale. **Messages:** each command is recorded durably, forwarded, acked, and answered with one event, which is also recorded before delivery. | Load | Aquifer events/s | Aquifer confirm p50 | Spillway events/s | Spillway confirm p50 | |---|---:|---:|---:|---:| | 100 sessions × 20/s | 2,000 (round trip p50 1.8 s) | 28 ms | 1,720 | 3.4 s | | 500 sessions × 10/s | ~1,300 | 9.9 s | 920–1,660 | 7.9–11.8 s | | 5,000 sessions × 1/s | ~1,000 | 11.4 s | 840 | 13.5 s | One 1-CPU Aquifer moves about **1,500–2,000 WebSocket round trips/s**. Past that, latency grows but the process stays up. **What limits it.** - **Aquifer is CPU-bound**, at 88% of its core: - 51% of CPU goes to each session's own blocking-read loop on Valkey; - Valkey sees about 17 operations per round trip, at about 50% of its core. A shared multi-stream reader and batched writes are the next step. The issue is filed separately. - **Spillway had no backpressure at the time.** Memory grew by about 5–10 MB/s under sustained overload, and in one run the process was killed out of memory. Fixed since in Spillway #27 and #28. **Configuration that matters.** - **File descriptors:** each Aquifer session uses three (client, backend, Valkey reader). Container platforms often default to 10,240 open files, which caps an instance near 3,000 sessions; raise `ulimit -n`. - **Valkey `maxclients`:** needs one connection per live session, across every Aquifer sharing it. - **Session ceiling:** `AQUIFER_WS_MAX_CLIENT_CONNECTIONS` defaults to 1,000, which is safe. Raise it no higher than memory allows (about 90 KB per session). Since this benchmark, new sessions are also refused with `429` (`limit_reason: memory`) while the process is over `AQUIFER_MEMORY_LIMIT_MB`. **Bugs this benchmark found:** - **Aquifer #33:** sessions starved the Valkey connection pool past ~10 per CPU. Command confirmations took 3.1 s at p50 on one CPU with 200 sessions; 0.34 ms after the fix. - **Spillway #23:** WebSocket target APIs on IPv6-only hosts, including Fly `.internal`, never connected. --- ## Reproducing these results ```bash cd benchmark ./throughput.sh 50 30s ./burst.sh 10 100 ./admission_degradation.sh 150 45s ./crash_recovery.sh 30 ./fairness.sh 100 # Capacity/drain by machine size -- much slower (multiple full redeploys), # run separately, not part of the GitHub Action's regular pass: ./capacity_by_size.sh "256 512 1024" 500 # GPU retry tax -- needs a real vLLM instance (RunPod or otherwise), # not part of the regular pass either: ./gpu_retry_tax.sh 40 40 300 # GPU retry tax measured in tokens -- open-loop ramp with a retrying client, # raw/useful/wasted tokens computed from vLLM's own /metrics. Needs # `pip install -r requirements.txt` first. python3 retry_tax_harness.py --mode direct --vllm-url \ --start-rate 20 --end-rate 300 --ramp-seconds 60 --client-timeout 5 --max-retries 3 --max-tokens 300 python3 retry_tax_harness.py --mode aquifer --vllm-url --aquifer-url \ --start-rate 20 --end-rate 300 --ramp-seconds 60 --max-tokens 300 python3 analyze_retry_tax.py --events --metrics ``` For Pebble instead of SQLite: deploy with `-e AQUIFER_STORE_BACKEND=pebble -e DB_PATH=/data/pebble` and run the same scripts unchanged.