Skip to content

Benchmarks

Sections 1–7 ran before v0.13.0 (2026-10-05), when SQLite was the default store; Pebble is the default now. Sections 8 and 11 compare both, and 12–13 ran on Pebble.

Real runs against a live deployment, not simulated numbers. Target: aquifer-bench.fly.dev, a single shared-cpu-1x/512MB Fly.io machine in iad, AQUIFER_MEMORY_LIMIT_MB=400, default pacing (2 RPS/1 concurrent per target API domain, no CONFIG_PATH). Load generated with vegeta; scripts are in benchmark/.

Scenarios 1-6 dispatch to a dummy target API (postman-echo.com) at the default 2 req/s ceiling — intentional, since Aquifer protects target APIs that haven’t told it otherwise. A large batch queues for a while before draining; that’s the configured ceiling working as intended, not a bottleneck. Production sets real per-domain rates via CONFIG_PATH. Scenario 7 deliberately raises the ceiling so machine size, not the default rate limit, is what’s measured.


1. Sustained throughput (SQLite, before v0.13.0)

Section titled “1. Sustained throughput (SQLite, before v0.13.0)”

Measures acceptance (POST /jobs → 201), not end-to-end dispatch — ingest and dispatch are decoupled by design.

50 req/s for 30s (1,500 requests):

Requests 1500, rate 50.03, throughput 49.93
Latencies min 53.5ms, mean 69.8ms, p50 65.0ms, p90 72.2ms, p99 187.2ms, max 565.6ms
Success 100.00% | Status 201:1500

100% acceptance, ~65ms median latency, no degradation over the run.


2. Burst absorption (retry-storm simulation) (SQLite, before v0.13.0)

Section titled “2. Burst absorption (retry-storm simulation) (SQLite, before v0.13.0)”

Baseline traffic, a 10x spike for 30s, then back to baseline.

Phase: baseline (10/s, 15s) — 150 req, 100% success, p99 194ms
Phase: burst (100/s, 30s) — 3000 req, 100% success, p99 166ms
Phase: recovery (10/s, 15s) — 150 req, 100% success, p99 189ms

100% success through the entire 10x spike, no 5xx, no dropped connections. Recovery-phase latency matches baseline — the burst gets absorbed into the queue instead of hammering the target API.


3. Admission control under memory pressure (SQLite, before v0.13.0)

Section titled “3. Admission control under memory pressure (SQLite, before v0.13.0)”

Aquifer sheds load with clean 429s (never 5xx, never a crash) once memory exceeds a configured ceiling.

At the realistic setting (AQUIFER_MEMORY_LIMIT_MB=400), a 150 req/s, 45s hammer (6,750 requests, 100% accepted) never pushed the process past ~21MB resident — the ceiling was never exercised, which is itself a useful data point about baseline memory footprint.

To demonstrate shedding, redeployed with AQUIFER_MEMORY_LIMIT_MB=15 (deliberately below resting footprint):

600 requests, 30.05 req/s → 0.00% success, 429:600 (all rejected)

Every rejected request gets a clean 429 with a structured body and Retry-After:

HTTP/2 429
retry-after: 5
{"current":21,"error":"admission rejected: memory at 21 exceeds limit 15","limit":15,"limit_reason":"memory"}

No crash, no hang, no partial writes.

One subtlety: a rejected request’s row is deleted — it was never durably accepted. The idempotency guarantee only covers jobs already accepted before the limit tripped, not ones that were shed. Verified by TestAdmissionDuplicateStillSucceedsUnderPressure and TestAdmissionRejectedJobLeavesNoGhostRow.


4. Crash recovery (SQLite, before v0.13.0)

Section titled “4. Crash recovery (SQLite, before v0.13.0)”

30 jobs enqueued, machine SIGKILL’d ~3s in (mid-dispatch), restarted, polled for up to 60s:

enqueued: 30 | completed: 30 | failed: 0 | still queued: 0 | lost: 0
PASS: all 30 jobs reached a real terminal state

Every job persisted to SQLite before dispatch, so a kill mid-flight just means the recovery loop re-enqueues it on restart.


5. Multi-tenant fairness (X-Aqueduct-Account-Queue) (SQLite, before v0.13.0)

Section titled “5. Multi-tenant fairness (X-Aqueduct-Account-Queue) (SQLite, before v0.13.0)”

Without isolation, every job hitting the same target API domain shares one dispatch queue — a noisy tenant can starve everyone behind it. Isolation per (user_id, api_key) is on by default (AQUIFER_ACCOUNT_QUEUE); a target API can switch it off for itself with X-Aqueduct-Account-Queue: disabled. TestABrokenTenantDoesNotStallAHealthyOneOnTheSameHost measures it: with a broken tenant retrying a 40-job burst, a healthy tenant on the same host finishes 20 jobs in ~0.2s with isolation and 0 of 20 in 5s without it.

Surfaced a real bug during benchmarking: the isolation mechanism existed, but the handler that turns it on was never actually wired into the HTTP request path — every job silently shared one queue regardless of the header. Filed and fixed as issue #5, covered by TestAccountQueueHeaderIsolatesTenants and TestAccountQueueHeaderOmittedSharesQueue.

With the header now working — a noisy tenant flooding 100 concurrent jobs alongside a quiet tenant sending 5 steady jobs:

quiet jobs: 5s, 4s, 4s, 2s, 1s

completing on their own schedule regardless of the flood. Before the fix, the same scenario left the quiet tenant stuck for 37-52s each.


6. Retry-After backs off exponentially under sustained pressure (SQLite, before v0.13.0)

Section titled “6. Retry-After backs off exponentially under sustained pressure (SQLite, before v0.13.0)”

A single 429 returns the base value (default 5s). Consecutive rejections double it, capped at 60s; the next allowed request resets it. Verified against a live instance held over its memory ceiling:

attempt 1-5: retry-after = 5, 10, 20, 40, 60

This stops clients from hammering a fixed 5-second ceiling forever once an instance is genuinely overloaded, giving it room to drain.


7. Capacity and drain time (SQLite, before v0.13.0)

Section titled “7. Capacity and drain time (SQLite, before v0.13.0)”

Started as “what machine size fits a given traffic shape.” Found something bigger: a real correctness/scaling bug in the SQLite layer, unrelated to machine size.

What the first pass showed: aquifer-bench redeployed at 256/512/1024MB (1 vCPU held constant) — all three broke at the identical point: 100% success through 100 req/s, then ~68-73% at 200 req/s, with connection failures at only ~110MB memory use. A follow-up sweep across real CPU tiers (1/2/4 vCPU, paired memory) broke at the same point too: 68.3%, 68.3%, 69.5%. That ruled out memory and CPU as the bottleneck.

The real cause: store.go had db.SetMaxOpenConns(1) — every request queued behind one SQLite connection regardless of machine size. Raising the pool size alone didn’t fix it either: WAL/busy-timeout pragmas were being set via db.Exec() on whichever single connection served that call, so opening more connections meant most of them silently ran with SQLite’s unsafe defaults and failed under concurrency. Fixed by moving the pragmas into the connection DSN itself, so every pooled connection gets them.

That fix surfaced a third issue: CheckOrInsert checked for duplicates via INSERT OR IGNORE then a separate SELECT — non-atomic, so a raced INSERT could make the SELECT find nothing and report a brand-new job as a duplicate with an empty job ID, silently discarding it. Latent the whole time; only triggerable once the pool had more than one connection. Fixed by reading INSERT’s own RowsAffected() instead of a racy read-after-write. Covered by TestStoreHandlesConcurrentInserts (50 concurrent inserts, unique keys, none misreported).

After the fix, same 1 vCPU/512MB instance:

Rate Before fix After fix
25-100/s 100% success 100% success
200/s ~68-73% success, connection refused 74-100% success (variance), no refusals — remaining failures are slow, not refused
400/s (not tested — already broken at 200) 15.4% success, mean latency 28.6s, memory 107→264MB — a real, load-driven ceiling

Drain time (500 jobs against an idle instance) was identical across every configuration, before and after the fix: 75-79 seconds — governed by the configured dispatch pace, not CPU/RAM.

Takeaway: a ceiling that doesn’t move when you change the resource you’d expect to relieve it is a signal to check the code path, not the instance size.

  • Traffic sustains under ~200 req/s? Any size works, including 256MB.
  • Traffic sustains above ~200-300 req/s? Re-run benchmark/capacity_by_size.sh against your own traffic shape first — verify the ceiling is actually hardware-bound, not code-level.
  • Bursts followed by quiet periods? A 500-job burst drains in ~75-79s at a 50 RPS dispatch pace, regardless of machine size — scale linearly against your own CONFIG_PATH rate.
  • A ceiling looks identical no matter what you scale? Treat it as a code-path question, not a bigger-machine question.
  • Need per-tenant fairness under a shared target API? It’s on by default; check nobody set AQUIFER_ACCOUNT_QUEUE=off and that the target API doesn’t answer X-Aqueduct-Account-Queue: disabled.

Spillway (Elixir/Mnesia) showed a materially higher throughput ceiling than SQLite here — Mnesia writes memory-first with disk catching up in the background, SQLite is disk-first and single-writer. That raised the question: would a memory-first store change Aquifer’s ceiling too, without giving up Go?

Added PebbleStore (cockroachdb/pebble, pure Go, no CGo) behind the JobStore interface, opt-in via AQUIFER_STORE_BACKEND=pebble — default stays SQLite, additive not a rewrite.

A durability gap Pebble doesn’t warn you about: the assumption going in was that WriteOptions.Sync: false behaves like SQLite’s synchronous=NORMAL (writes through to the OS, only relaxes fsync). Wrong — caught by the same write/kill -9/restart/read test used throughout this suite:

sync_write: {:atomic, :ok}
# kill -9 immediately after, restart
[JOB 1] WAL file stopped reading at offset 0; replayed 0 keys
read err: pebble: not found

Unlike SQLite, Sync: false never writes through to the OS at all. Sync: true is required for any crash-durability guarantee; confirmed it survives the identical kill test once set. Pebble’s WALMinSyncInterval (its own group-commit) batches concurrent syncs into fewer real fsyncs while still blocking each caller until its own write is durable — no silent-loss window. Configurable via AQUIFER_PEBBLE_WAL_SYNC_INTERVAL_MS (default 5ms). Verified: 30/30 jobs survive a real kill -9 on Fly.io, matching SQLite and Mnesia.

Idempotency also needed explicit handling: Pebble has no insert-if-absent primitive, so a naive Get-then-Set would reproduce the same race already fixed in SQLite and Mnesia. Guarded by a lock striped across 256 shards keyed by idempotent hash. Covered by TestPebbleStoreHandlesConcurrentInserts.

Throughput, same tier, same ramp:

Rate SQLite (post-fix) Pebble
50/s 100% success, mean 69.6ms 100% success, mean 72.7ms
200/s 74-100% success (variance) 100% success, mean 1.68s
400/s 15.4% success — real ceiling 100% success, mean 11.4s
600/s (already broken) 66.6% success
800/s (already broken) 40.0% success
1000/s (already broken) 17.2% success

Pebble roughly doubles the point where Aquifer stops reliably completing requests (~200-400 req/s vs. ~400-600 req/s). The failure mode differs too: SQLite fails outright past its ceiling; Pebble keeps succeeding well past its comfort zone, just slower — arguably the better mode for reliable eventual delivery, though it makes Pebble’s “ceiling” fuzzier to define than SQLite’s hard wall.

CPU scaling: 4 vCPUs instead of 1, same rates — essentially no difference (400/s: 11.4s → 10.7s mean; 600/s: 66.6% → 60.4%; 800/s: 40.0% → 38.1%, all within noise). The opposite of Mnesia’s result. Likely reason: Pebble’s WAL/memtable write path is a single ordered pipeline per instance — faster than SQLite’s B-tree, but still fundamentally serialized, unlike Mnesia’s per-request independent BEAM processes. More cores help when the bottleneck is application concurrency; they don’t when it’s a storage engine’s own internal serialization.

Takeaway: Pebble is a legitimate, meaningfully faster alternative to SQLite for this workload (AQUIFER_STORE_BACKEND=pebble), not a magic unlock past ~600 req/s — that ceiling is the storage engine’s serialized commit path. Horizontal partitioning remains the right lever past that point, for either backend.


9. GPU inference and the retry tax (RunPod/vLLM)

Section titled “9. GPU inference and the retry tax (RunPod/vLLM)”

Everything above dispatches to a dummy HTTP echo target API. This one targets a real GPU: Qwen/Qwen2.5-1.5B-Instruct on a single RunPod RTX PRO 4500 Blackwell running vLLM (--max-num-seqs 48, KV cache deliberately capped at 512MB so a realistic burst can actually pressure it), pacing driven entirely by the ORCA fallback signal (see Dynamic pacing) — kv_cache_usage_perc on vLLM’s own response headers, no X-Aqueduct-* config needed. Requests forced to min_tokens=max_tokens=300 so each one holds GPU/KV-cache resources long enough to matter, not a one-token echo.

Same offered load (40 req/s for 30s), direct-to-vLLM vs. through Aquifer:

Direct Through Aquifer
Client success 100% 100%
Client mean latency 11.2s 1.0ms (durable enqueue, dispatch decoupled)
Client p99 latency 20.1s 3.2ms
Peak vLLM-side queue depth 449 waiting 8 waiting
Peak kv_cache_usage_perc 69.4% 59.1%

Hitting vLLM directly at 40 req/s doesn’t fail — vLLM queues everything internally rather than rejecting — but its own backlog balloons to 449 requests deep, and callers wait up to 20 seconds for a response that eventually succeeds. That’s the retry tax: technically 100% success, but exactly the shape of slowness that makes a caller give up and retry, doubling load on a GPU that’s already behind. Through Aquifer, the burst is durably absorbed in ~1ms per request and dispatched to vLLM at a controlled pace; vLLM’s own internal queue barely builds.

Does the ORCA signal itself actually engage (not just a static rate cap doing the work)? Loosened Aquifer’s static ceiling above what vLLM can sustain (80 RPS / 120 concurrent vs. vLLM’s real ~48-concurrent capacity) and pushed a sustained 60 req/s for 40s. kv_cache_usage_perc oscillated 22% → 79% → 22% → … in a repeating cycle: cache crosses the 70% threshold, ORCA tells Aquifer to cut to 2 RPS, backlog drains, cache falls, Aquifer ramps back up (5%/dispatch recovery), cache climbs again. A real, self-regulating feedback loop against a live GPU, not a static number that happened to be tuned right.

Takeaway: the durable-queue-plus-paced-dispatch model already prevents the worst of the retry tax on its own; ORCA adds automatic, no-config self-regulation on top for backends that report their own load, which vLLM already does.


10. GPU retry tax, measured in tokens (RunPod/vLLM)

Section titled “10. GPU retry tax, measured in tokens (RunPod/vLLM)”

Section 9 above measures queue depth and latency. It doesn’t measure the thing the retry-tax thesis actually claims: that a caller giving up and retrying makes the GPU do real, wasted generation work. This section closes that gap with two new numbers, computed from vLLM’s own /metrics (generation_tokens_total) and a client harness that behaves like a real caller — times out and retries instead of waiting patiently forever (benchmark/retry_tax_harness.py, benchmark/analyze_retry_tax.py).

raw tokens/sec — everything the GPU actually generated (ground truth, from vLLM’s own counter), including generations for requests whose caller already gave up and retried. useful tokens/sec — tokens that actually reached a caller who was still listening. wasted = raw − useful. This is the number that proves or kills the “GPU waste” claim; raw throughput alone can’t, since a GPU generating duplicates is still just as busy.

Same instance as section 9 (RunPod RTX PRO 4500 Blackwell, Qwen/Qwen2.5-1.5B-Instruct, --max-num-seqs 48, 512MB KV-cache cap), offered load ramped 20 → 300 req/s over 60s — aggressive enough to actually saturate this GPU, unlike section 9’s fixed 40 req/s:

Direct, retrying client (5s timeout, 3 retries) Through Aquifer, concurrency matched to vLLM’s real limit
Logical requests 9,537 9,585
Success rate 0.9% 25.2%
Wasted generation (raw − useful) 96.5% 22.0%
vLLM’s own queue depth (peak) 18,079 0

Hitting vLLM directly with a client that actually retries (unlike section 9’s patient client) is close to a worst case: 96.5% of everything the GPU generated was thrown away, spent on attempts whose caller had already given up and resubmitted. The GPU is fully busy the entire time — that busyness is exactly the trap: raw throughput alone would make this look fine. Through Aquifer, with its concurrency ceiling set at vLLM’s real capacity (45, just under its --max-num-seqs 48), wasted generation drops to 22.0% and vLLM’s own queue depth never leaves zero for the whole run — Aquifer’s durable queue absorbs the backlog instead of vLLM’s internal one taking the hit.

Worth knowing when setting that ceiling: Aquifer’s ORCA fallback (section 9) paces down based on kv_cache_usage_perc — memory pressure specifically. A deployment bottlenecked on scheduler concurrency rather than memory can be fully saturated while that signal stays quiet, since it isn’t watching the thing that’s actually full. Concretely, --max-num-seqs 48 means kv-cache usage per concurrently-running sequence may never cross ORCA’s 0.70 pace-down threshold even at full 48-slot occupancy, depending on sequence length and cache budget. Set a static max_concurrent at or below the backend’s real capacity rather than relying on ORCA alone to catch a concurrency-driven ceiling — it’s built to catch memory pressure, not slot exhaustion. Auto-detecting that real ceiling instead of requiring it to be set by hand is filed as a follow-up: rjpruitt16/aquifer#11.


make perf runs two Go benchmarks in-process against both storage backends, with a local target API and webhook receiver that answer instantly and pacing set high enough never to wait. They measure Aquifer’s own cost, so a change that slows the hot path shows up here before it shows up in production. Run it before and after a change on the same machine and compare.

  • BenchmarkJobLatency sends one job at a time and splits its trip into accept_ms (validate and durably store, i.e. POST /jobs returns), queue_ms (accepted until the target API receives the request) and total_ms.
  • BenchmarkPipelineThroughput has 8 concurrent callers submit jobs and reports jobs/s that make it all the way through: accepted, dispatched, completed and their webhook delivered.

Baseline, 2026-10-06, Apple M3 Max laptop (macOS, 200 jobs per run):

Backend accept queue total pipeline
SQLite 0.55 ms 0.13 ms 0.67 ms 571 jobs/s
Pebble, AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0 (sync every write) 9.90 ms 0.23 ms 10.13 ms 100 jobs/s
Pebble, 100ms flush (default) 0.04 ms 0.07 ms 0.11 ms 11,107 jobs/s
Pebble, AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=10 0.05 ms 0.08 ms 0.13 ms 9,593 jobs/s

The flush-interval rows (2000 jobs each; the default is 100ms) acknowledge a write before it is on disk and sync the log on a timer, the same trade Spillway makes with its 100ms Mnesia flush: a crash can lose up to that window of accepted jobs. pebble_flush_test.go checks that a process killed after the window loses nothing. On a Mac most of the gain comes from skipping slow fsyncs; expect a smaller gap on Linux, where a sync took about 0.3ms on a Fly volume.

Read these as relative numbers, not capacity, and don’t compare the two rows as like for like. SQLite runs in WAL mode with synchronous=NORMAL: a commit survives a process crash but isn’t flushed to disk per write, so a power loss can drop the last few accepted jobs. Pebble with AQUIFER_PEBBLE_FLUSH_INTERVAL_MS=0 syncs every write (zero acknowledged-job loss) with a 5ms group-commit window, and macOS fsync is far slower than Linux, so on a Mac a lone job waits out that window plus a full flush. On a Fly performance-1x volume, fsync measured about 0.3ms p50 and Pebble outran SQLite (see section 8). Dispatch itself (queue_ms) is a fraction of a millisecond on both; the cost of a job is its storage writes.

On Fly (performance-1x with a volume, every job also delivering a webhook), syncing every write sustained about 150 jobs/s end to end and the 100ms flush about 700, which is why 100ms is the default.


12. Capacity per machine (Pebble, Fly.io, 2026-10-07)

Section titled “12. Capacity per machine (Pebble, Fly.io, 2026-10-07)”

How many jobs per second one Aquifer machine sustains end to end, measured on Fly in iad:

  • Target: one Aquifer machine with a volume and Pebble.
  • Load generator: a separate machine running benchmark/loadgen. It ramps POST /jobs open-loop across 50 users and also serves as the target API and the webhook receiver, both answering instantly.
  • Webhooks: every job also delivers a webhook, so each job is one request in and two requests out.
  • Config: a high rate ceiling (defaults: {rps: 100000, max_concurrent: 256}) and the 100ms flush (AQUIFER_PEBBLE_FLUSH_INTERVAL_MS, the default).

A step counts as sustained when Aquifer accepts what’s offered and completes jobs as fast as it accepts them, with the backlog staying flat.

Machine Sustained jobs/s p99, submit to webhook Notes
performance-1x, sync every write (FLUSH_INTERVAL_MS=0) ~150 31 ms at 100/s durable before acknowledging
performance-1x ~1,000 ~0.6 s one core is the limit
performance-2x ~1,500 ~0.4 s
performance-4x ~3,000, sometimes 4,000 56 ms at 3,000 4,000 held on one run, shed on two

Past the ceiling Aquifer sheds with 429 rather than falling over.

What moved the ceiling. Each stage was found by profiling Aquifer under load on Fly with pprof.

Stage Changes Result
1 100ms batched WAL sync instead of syncing every write 1x: ~150 → ~700
2 Cache admission readings (ReadMemStats and a walk of the Pebble directory ran on every submission); 64 MB Pebble block cache (AQUIFER_PEBBLE_CACHE_MB); no drain events unless drain mode is on; skip queue expiry scans until a deadline is due; one pooled HTTP client instead of a new one per request 1x: ~700 → ~1,000, but 2x no faster (one core busy)
3 Registry and worker locks cover only decisions, not store writes or queue handoffs; cleanup reads an expiry index instead of decoding every retained job 2x: uses both cores
4 Remember webhook receivers without L8 for 5 minutes instead of probing /.well-known/l8 before every webhook over a fresh connection (connect() was 65% of CPU on 2x) 2x: ~1,000 → ~1,500; 4x: ~3,000

Caveats.

  • These are best-case numbers. With an instant target API they measure Aquifer’s own cost per job. A slow target API doesn’t change that cost, but it does mean more requests in flight at once.
  • The 4x results vary. Finished jobs are kept for 30 minutes, so back-to-back runs leave a larger store behind. A fresh store held 3,000/s cleanly; 4,000/s held on one run out of three.
  • The load generator can be the limit. On a performance-2x generator, 6,000/s produced client-side errors, so the 4x ceiling was probed with a performance-4x generator.

Reproduce: build benchmark/loadgen for linux/amd64, run it on one machine, and point it at Aquifer over the private network:

Terminal window
loadgen -target http://<aquifer>.internal:8080 -self http://<loadgen>.internal:8080 -rates 700,1000,1500,2000,3000,4000 -step 20s

13. WebSocket sessions per machine (Fly.io, 2026-10-10)

Section titled “13. WebSocket sessions per machine (Fly.io, 2026-10-10)”

How many aqueduct.v1 WebSocket sessions one machine holds, and how many messages it moves through them. Measured on Fly in iad:

  • Target: one performance-1x machine (1 CPU, 2 GB).
  • Valkey: on its own performance-1x machine.
  • Backend: a no-limits backend that acks each command and sends one event (aquifer-websocket-bench -backend).
  • Client: a performance-2x machine running cmd/aquifer-websocket-bench.

Run with the Valkey reader-pool fix (#33). Lab configs are in benchmark/websocket. Spillway was measured on the same lab for comparison.

Sessions held (idle):

Sessions Aquifer connected Aquifer memory Aquifer connect p50 / p99 Spillway connected Spillway memory Spillway connect p50 / p99
1,000 1,000 110 MB 6 / 35 ms 1,000 244 MB 260 / 693 ms
5,000 5,000 493 MB 7 / 351 ms 5,000 498 MB 750 ms / 6.2 s
10,000 10,000 991 MB 13 ms / 1.0 s 10,000 818 MB 1.9 / 8.9 s
20,000 ran out of memory near 15,000 20,000 1.16 GB 5.2 / 44 s
  • Aquifer: about 90 KB per session, mostly six goroutines and socket buffers. Once sessions close, live heap returns to baseline: after a forced GC it was 30 MB live with no sessions.
  • Spillway: about 50 KB per session, but every new session goes through one queue process, which makes opening slow at scale.

Messages: each command is recorded durably, forwarded, acked, and answered with one event, which is also recorded before delivery.

Load Aquifer events/s Aquifer confirm p50 Spillway events/s Spillway confirm p50
100 sessions × 20/s 2,000 (round trip p50 1.8 s) 28 ms 1,720 3.4 s
500 sessions × 10/s ~1,300 9.9 s 920–1,660 7.9–11.8 s
5,000 sessions × 1/s ~1,000 11.4 s 840 13.5 s

One 1-CPU Aquifer moves about 1,500–2,000 WebSocket round trips/s. Past that, latency grows but the process stays up.

What limits it.

  • Aquifer is CPU-bound, at 88% of its core:

    • 51% of CPU goes to each session’s own blocking-read loop on Valkey;
    • Valkey sees about 17 operations per round trip, at about 50% of its core.

    A shared multi-stream reader and batched writes are the next step. The issue is filed separately.

  • Spillway had no backpressure at the time. Memory grew by about 5–10 MB/s under sustained overload, and in one run the process was killed out of memory. Fixed since in Spillway #27 and #28.

Configuration that matters.

  • File descriptors: each Aquifer session uses three (client, backend, Valkey reader). Container platforms often default to 10,240 open files, which caps an instance near 3,000 sessions; raise ulimit -n.
  • Valkey maxclients: needs one connection per live session, across every Aquifer sharing it.
  • Session ceiling: AQUIFER_WS_MAX_CLIENT_CONNECTIONS defaults to 1,000, which is safe. Raise it no higher than memory allows (about 90 KB per session). Since this benchmark, new sessions are also refused with 429 (limit_reason: memory) while the process is over AQUIFER_MEMORY_LIMIT_MB.

Bugs this benchmark found:

  • Aquifer #33: sessions starved the Valkey connection pool past ~10 per CPU. Command confirmations took 3.1 s at p50 on one CPU with 200 sessions; 0.34 ms after the fix.
  • Spillway #23: WebSocket target APIs on IPv6-only hosts, including Fly .internal, never connected.

Terminal window
cd benchmark
./throughput.sh <target-url> 50 30s
./burst.sh <target-url> 10 100
./admission_degradation.sh <target-url> 150 45s
./crash_recovery.sh <target-url> <fly-app-name> 30
./fairness.sh <target-url> 100
# Capacity/drain by machine size -- much slower (multiple full redeploys),
# run separately, not part of the GitHub Action's regular pass:
./capacity_by_size.sh <target-url> <fly-app-name> "256 512 1024" 500
# GPU retry tax -- needs a real vLLM instance (RunPod or otherwise),
# not part of the regular pass either:
./gpu_retry_tax.sh <vllm-url> <aquifer-url> 40 40 300
# GPU retry tax measured in tokens -- open-loop ramp with a retrying client,
# raw/useful/wasted tokens computed from vLLM's own /metrics. Needs
# `pip install -r requirements.txt` first.
python3 retry_tax_harness.py --mode direct --vllm-url <vllm-url> \
--start-rate 20 --end-rate 300 --ramp-seconds 60 --client-timeout 5 --max-retries 3 --max-tokens 300
python3 retry_tax_harness.py --mode aquifer --vllm-url <vllm-url> --aquifer-url <aquifer-url> \
--start-rate 20 --end-rate 300 --ramp-seconds 60 --max-tokens 300
python3 analyze_retry_tax.py --events <events.jsonl> --metrics <metrics.jsonl>

For Pebble instead of SQLite: deploy with -e AQUIFER_STORE_BACKEND=pebble -e DB_PATH=/data/pebble and run the same scripts unchanged.