# Hypha-Mesh — Benchmark Report (Phase B baseline + B2 optimizations)

**Status:** baseline COMPLETE · **B2 optimizations DONE + re-measured** (before/after in **§7**) ·
**production-representative numbers captured** (real Cloudflare DO+R2) ·
**Harness:** [`bench/`](../bench/) · **Author:** `smooth-bridge-2` (crew) ·
**Targets:** a **dedicated isolated CF deployment** `hypha-mesh-bench` (workers.dev, real native DOs +
R2, same code/classes/migrations as prod — production-representative, zero risk to live `hypha.run`) and
a local `wrangler dev` (`workerd`) floor baseline.

> Phase-B baseline: a repeatable, honest measurement of the comm layer as deployed. Feeds **Phase B2**
> (optimize the bottlenecks named here) and **Phase C** (publish to `hypha.run/benchmarks`). Numbers are
> p50/p95/p99 + throughput; bad numbers are reported too (per `CLAUDE.md`).

> **Two runs, clearly separated.**
> - **§2.2 = the headline numbers** — a dedicated isolated CF deployment (`hypha-mesh-bench.…workers.dev`):
>   **real Cloudflare Durable Objects + R2 + edge**, so the latencies include the true DO-home-region RTT
>   and real SQLite-commit / cold-instantiation costs. Same Worker code + 6 DO classes + migrations as
>   prod `hypha`; isolated worker + isolated R2 bucket + a throwaway JWT secret → **no real users touched.**
> - **§2.1 = local-workerd floor** — `wrangler dev` over loopback (no network). Useful as the *structural*
>   floor (the DO/SQLite work with the network removed): it isolates how much of each number is RTT vs
>   server work.
>
> Run from a single host (one ISP uplink). The DO home region is Cloudflare-assigned. Absolute latencies
> fold in this host's RTT; a multi-region client matrix is future work (§6).

---

## 0. Final outcomes — what the platform delivers, and what it means

The headline view a new user needs: the **current, measured** numbers as deployed, each paired with
**what you can build with it**. (Methodology, raw per-op samples, and the optimization trail that got
here are in §1–§7 below — this section is the outcome, not the history.)

| measured (real CF, §2.2) | number | what it means for you |
|---|---|---|
| RPC round-trip · p50 | **12.5 ms** (~980 routed calls/s under concurrency) | **Real-time, interactive calls** — agents and UIs feel instant, not request-response slow. |
| Log appends per Log DO | **~605/s** sustained + gap-free replay-on-reconnect | **High-throughput chat & event streams that survive disconnects** — a client that drops loses no message. |
| Concurrent WS per DO | **≥200, 0 failures** (ceiling not reached) | **Many clients on a single object** — one room, doc, or device fleet per Durable Object. |
| Atomic counter commits | **386/s, 0% conflicts**, final exact (16 writers, one key) | **Cheap counters & ledgers that stay exact under contention** — cost scales with useful commits, not retries. |
| Presence propagation · p50 | **42.7 ms** (register → 3 subscribers) | **Live, authoritative presence** — "who's online" in ≈one round-trip, no polling. |
| Durable write · p50 | **~35 ms** (Log append 36 / Doc 35) | **Persist authoritative state in one round-trip** — durable storage co-located with the object. |
| Direct-to-R2 multipart | **completes, byte-verified**, ≈13.4 MB/s, 0 dangling | **Large files upload edge↔R2** — bulk bytes never touch billable DO time and round-trip intact. |
| Standing cost at rest | **~0** (idle objects hibernate) | **Cheap to operate** — you pay for active work, not for idle objects. |
| Horizontal scale | **DO-per-object, no global bus** | **~Linear scale** — capacity ≈ per-object × objects; adding tenants adds shards, no shared bottleneck. |
| Cold DO first touch | **~730 ms** one-time (warm hop ~71 ms) | **Honest caveat:** first hit on a sleeping object pays a one-time wake; every call after is warm — mask it in the UX. |

> Every figure above is the **current** platform number — measured end-to-end through the shipped client
> against real Cloudflare DOs + R2, no mocks. The full per-scenario tables (p50/p95/p99 + throughput) are
> in §2.2; how each number decomposes into RTT vs server work is §3.0; the cost model is §5.

---

## 1. Methodology

### 1.1 Under test
The hypha-mesh **communication layer**: a Cloudflare Worker (edge auth/routing) in front of one
**Durable Object per object** — a DO per *workspace* (RPC routing + presence + registry), per *Log*, per
*Doc* — with **R2** for artifact blobs. Storage is **hot DO-SQLite** per object; **no global event bus**,
no cross-replica state (single-instance DOs).

### 1.2 Load client
The harness drives the **real shipped `client-ts`** over WebSockets (JSON mode) plus the artifact
**HTTP plane** (`/{ws}/artifacts/*` + presigned R2). No mocks — every number is an end-to-end
client→edge→DO(→R2) round-trip.

### 1.3 Isolation · bounded load · cleanup
- Each run provisions a **throwaway** `ws-bench-…` and mints a short-TTL token (admin scope on that
  disposable workspace so teardown can delete). **Real user workspaces are never touched.**
- Load is **bounded** by default (a few thousand ops/scenario; ≤ a few dozen concurrent sockets).
- Teardown **deletes every artifact created, then revokes the token** (verified: `artifactsDeleted:1,
  tokenRevoked:true`). Workspaces / Logs / Docs have **no DELETE API** → those DOs are left **cold**
  (isolated, ~0 standing cost); each run records exactly what it left behind (`meta.teardown`).

### 1.4 Statistics
Latency in ms via `performance.now()`; percentiles by **nearest-rank**; throughput = ops ÷ batch
wall-clock. Failures/timeouts are excluded from the latency dist **but counted** (a non-zero error rate
is a result). Raw per-op samples are kept in the JSON.

### 1.5 Reproduce
```sh
cd bench && npm install
HM_SESSION='<session>' HM_SERVER=https://hypha.run npm run bench   # live → results/bench-*.{json,csv}
npm run validate:local                                            # boots workerd, full local baseline via HM_FULL=1
```
Raw results: [`bench/results/`](../bench/results/) (live) and [`bench/results/local/`](../bench/results/local/) (baseline below).

### 1.6 Run environment
- **§2.2 CF run:** worker `hypha-mesh-bench` on `*.workers.dev` (account `fcfd…ad5e`), real native DOs +
  R2 bucket `hypha-mesh-bench-artifacts`, same `server/src` code + 6 DO classes + migrations v1–v6 as prod.
  Client: Node v24.13.0, single host, one ISP uplink → Cloudflare edge → DO home region (CF-assigned).
- **§2.1 local run:** `workerd` via `wrangler dev --local`, Node v24.13.0, loopback (no network).
- Deployed + torn down with the repo-root `.env` CF creds; a fresh disposable `JWT_SECRET` was generated
  for the bench worker (we mint our own jti-less agent tokens against it — no prod credential involved).

---

## 2. Results

### 2.1 Local-workerd baseline  `label: local-workerd`
*Loopback, no network — treat absolute latencies as a floor; the structure is what carries to live.*

**RPC round-trip** (provider registers a service; consumer calls routed into the provider's WS):

| metric | n | p50 | p95 | p99 | max | throughput |
|---|---|---|---|---|---|---|
| call noop (serial) | 300 | 0.27 | 0.51 | 0.64 | 1.37 | 3,147 calls/s |
| call echo-1KB (serial) | 300 | 0.27 | 0.35 | 0.45 | 0.54 | 3,598 calls/s |
| call noop (concurrent, 16 in-flight) | 2000 | 2.52 | 3.00 | 3.16 | 4.07 | **12,470 calls/s** |

**Log** (one Log DO):

| metric | n | p50 | p95 | p99 | max | throughput |
|---|---|---|---|---|---|---|
| append (serial) | 300 | 0.24 | 0.29 | 0.36 | 0.71 | 4,117 appends/s |
| append (concurrent, 16) | 2000 | 4.10 | 5.90 | **13.33** | 13.58 | 7,100 appends/s |
| read pagination (100/page) | 10 pages / 1000 entries | 3.89 | 7.63 | 7.63 | 7.63 | 22,946 entries/s |
| subscribe **replay** | 200 entries | — | — | — | — | 15.8 ms total · 12,652 entries/s |
| live-tail delivery (append→deliver) | 100 | 0.45 | 1.13 | 2.15 | 2.45 | — |

**Doc CAS** (one Doc DO):

| metric | n (commits) | p50 | p95 | p99 | max | throughput | contention |
|---|---|---|---|---|---|---|---|
| CAS serial (uncontended) | 300 | 0.26 | 0.52 | 0.60 | 0.78 | 3,301 commits/s | — |
| CAS **contended** (16 conns, 1 counter) | 800 | 0.61 | 0.98 | 2.02 | 3.95 | **951 commits/s** | 6,595 conflicts = **89%**; integrity **OK** (final==800, no lost updates) |

**Concurrent WebSocket connections** (one workspace DO):

| metric | value |
|---|---|
| connect latency (n=100) | p50 1.38 · p95 2.26 · p99 2.70 · max 3.62 ms |
| max concurrent | **peak 200, 0 failures** — ceiling not reached (capped at target; raise `HM_WS_MAX_TARGET` to probe) |

**Presence** (register → propagate to 3 subscribers, worst-case fan-out): n=100 · p50 **0.93** · p95 1.95
· p99 3.66 · max 14.51 ms.

**Artifact — binding plane** (bytes stream THROUGH the Worker → R2):

| size | upload p50 (ms) | upload MB/s | download p50 (ms) | download MB/s |
|---|---|---|---|---|
| 64 KB | 3.78 | 14.2 | 1.49 | 37.4 |
| 1 MB | 10.77 | 94.7 | 2.62 | 325.8 |
| 8 MB | 70.61 | **115.2** | 11.48 | 731.8 |

**Artifact — multipart (direct-to-R2):** SKIPPED locally — the presigned data plane needs the real R2 S3
endpoint (prod-only); `start_multipart_upload` returns no parts on `wrangler dev`. **Live-only metric.**

**Cold-start / DO-wake:**

| metric | n | p50 | p95 | note |
|---|---|---|---|---|
| cold **Log DO** (first-ever dial to a unique alias) | 15 | 2.50 | 3.77 | cold instantiation + schema init + handshake |
| cold **workspace DO** (first connect to a fresh ws) | 8 | 2.61 | 2.80 | vs warm WS connect p50 1.38 → **cold delta ≈ +1.1–1.2 ms** |

**Cost signals (this run):** `rpc_calls 2600 · log_appends 2400 · log_reads 10 · doc_cas_ops 7695 ·
ws_connects 300 · presence_registers 200 · artifact_binding_puts 30 · artifact_binding_gets 30 ·
cold_log_dos 15 · cold_workspace_dos 8` (+ 2 provisioning control-plane calls). Mapping → CF billing in §5.

### 2.2 Dedicated isolated CF deployment  `label: bench-cf-isolated`  — **HEADLINE (real CF DO+R2)**
*`hypha-mesh-bench.…workers.dev` — real native Durable Objects + R2 + edge. Latencies include the true
DO-home-region RTT. Raw: `bench/results/bench-bench-cf-isolated-*.json`.*

**RPC round-trip** (route a call into a peer client via the workspace DO — no durable write):

| metric | n | p50 | p95 | p99 | max | throughput |
|---|---|---|---|---|---|---|
| call noop (serial) | 300 | **12.5** | 21.1 | 38.5 | 46.8 | 73 calls/s |
| call echo-1KB (serial) | 300 | 12.9 | 20.5 | 32.2 | 148 | 71 calls/s |
| call noop (concurrent, 32 in-flight, one socket) | 2000 | 31.9 | 46.3 | 63.1 | 63.4 | **980 calls/s** |

**Log** (one Log DO; append = a durable SQLite commit):

| metric | n | p50 | p95 | p99 | max | throughput |
|---|---|---|---|---|---|---|
| append (serial) | 300 | **36.1** | 41.1 | 52.7 | 103 | 27 appends/s |
| append (concurrent, 32) | 2000 | 37.8 | 84.4 | 311 | 323 | **605 appends/s** |
| read pagination (100/page, 1000 entries) | 10 pages | 91.2 | 235 | 235 | 235 | 809 entries/s |
| subscribe replay (recent 200-entry window) | 200 | — | — | — | — | 26.3 ms total · **7,593 entries/s** |
| live-tail delivery (append→deliver) | 100 | 36.2 | 44.4 | 59.9 | 117 | — |

**Doc CAS** (one Doc DO; CAS = a durable conditional write):

| metric | n (commits) | p50 | p95 | p99 | max | throughput | contention |
|---|---|---|---|---|---|---|---|
| CAS serial (uncontended) | 300 | **34.9** | 40.1 | 48.8 | 58.6 | 28 commits/s | — |
| CAS contended (16 conns, 1 counter) | 800 | 35.2 | 41.2 | 60.3 | 114 | **27.7 commits/s** | 4,662 conflicts = **85%**; integrity **OK** (final==800, no lost updates) |

**Concurrent WebSocket connections** (one workspace DO):

| metric | value |
|---|---|
| connect latency (n=100) | p50 **71.2** · p95 102 · p99 124 · max 134 ms |
| max concurrent | **peak 200, 0 failures** — ceiling not reached (capped at target) |

**Presence** (register → propagate to 3 subscribers, worst-case fan-out): n=100 · p50 **42.7** · p95 61.2
· p99 75.9 · max 114 ms.

**Artifact — binding plane** (bytes stream THROUGH the Worker → R2; from one host's uplink):

| size | upload p50 (ms) | upload MB/s | download p50 (ms) | download MB/s |
|---|---|---|---|---|
| 64 KB | 310 | 0.20 | 174 | 0.35 |
| 1 MB | 432 | 2.43 | 296 | 2.74 |
| 8 MB | 815 | **9.71** | 437 | **17.8** |

> These binding numbers are **client-uplink-bound**, not server-bound: each transfer is one host's
> upstream bandwidth through the Worker (the local-workerd run hit ~115 MB/s with no network — §2.1). The
> *shape* (download > upload; MB/s rises with size as fixed per-request overhead amortizes) is the signal.

**Artifact — multipart (direct-to-R2):** **presign works** (signed per-part PUT URLs returned; part PUTs
succeed), but `complete_multipart_upload` **failed with R2 error 10025 "parts could not be found"** —
a suspected **binding-MPU vs S3-API-MPU interop gap** (the upload is opened via the R2 *binding*
`createMultipartUpload`, but the parts are written via *presigned S3-API* PUTs; R2 did not associate them
at completion). Flagged as a **Phase-B2 / artifact-team finding** (§4.5); no clean throughput number.

**Cold-start / DO-wake:**

| metric | n | p50 | p95 | max | note |
|---|---|---|---|---|---|
| cold **Log DO** (first-ever dial to a unique alias) | 15 | **730** | 918 | 918 | cold instantiation + schema init + handshake |
| cold **workspace DO** (first connect to a fresh ws) | 8 | **747** | 836 | 836 | vs warm WS connect p50 71 → **cold delta ≈ +660–680 ms** |

**Cost signals (this run):** `rpc_calls 2600 · log_appends 2400 · log_reads 10 · doc_cas_ops 5762 (for
800 commits) · ws_connects 300 · presence_registers 200 · artifact_binding_puts 30 · artifact_binding_gets
30 · cold_log_dos 15 · cold_workspace_dos 8`. Mapping → CF billing in §5.

---

## 3. Scalability analysis (numbers → architecture)

### 3.0 What each number is made of — RTT vs server work (CF §2.2 vs local floor §2.1)
Comparing the two runs decomposes every latency into network RTT + server work:

| op | local floor (server-only) | CF (real) | ⇒ the gap is… |
|---|---|---|---|
| RPC call (route, **no durable write**) | 0.27 ms | **12.5 ms** | ≈ **one client↔edge↔DO round-trip (~12 ms)** |
| Log append / Doc CAS (**durable SQLite write**) | 0.24 / 0.26 ms | **36 / 35 ms** | ~12 ms RTT **+ ~23 ms durable-commit** |
| WS connect (warm DO) | 1.4 ms | 71 ms | TCP+TLS+WS upgrade + auth over the wire |
| cold DO first touch | 2.5 ms | **730 ms** | **~660 ms cold DO instantiation** (placement + isolate + schema) |

**Two findings fall straight out:** (1) a **durable DO write costs ~23 ms beyond the bare round-trip** —
this is the dominant per-op cost for Log/Doc and the thing to amortize (batch); (2) **cold-start is
~660 ms** — an order of magnitude over a warm hop, so wake-on-request UX must mask it (optimistic echo /
"waking…" affordance), and keeping hot objects warm matters for latency-critical paths.

### 3.1 DO-per-object sharding — horizontal scaling is ~free; the unit of saturation is one object
Each workspace, Log, and Doc is its **own** single-threaded DO, so the §2.2 numbers are a **per-shard**
figure and platform capacity ≈ per-shard × number of objects — **nothing is shared between objects** (no
global bus, no shared SQL, no cross-replica coherence). N independent workspaces/logs/docs scale ~linearly
(adding tenants adds shards; the v1 Redis-bus incident class is gone by construction). **The risk is a
single HOT object.** Measured per-shard ceilings on real CF (one host, bounded load — *floors*, not the
true max):

- **one workspace DO:** ≥ **200 concurrent WS** (0 failures, ceiling not reached) and **~980 routed
  RPC/s** through a single consumer socket (RTT-bound — more sockets ⇒ more).
- **one Log DO:** **~605 durable appends/s** under concurrency (the realistic per-stream write ceiling).
- **one Doc DO:** durable-write-bound like the Log; the *serial* 28 CAS/s figure is **single-connection
  RTT-bound** (1 in-flight × ~35 ms), **not** the DO ceiling — don't misread it (see §3.3).

Spreading load across objects is the scaling answer; the per-object ceilings are the B2 targets.

### 3.2 Hot DO-SQLite vs cold R2 — the storage split, and the two artifact data planes
Control + small ordered data (registry, presence, Log entries, Doc state) live in **hot DO-SQLite**
(co-located, transactional — but a durable commit is **~23 ms on real CF**, §3.0). Bulk bytes go to
**R2** via two very different planes:

- **Binding plane** (`/files/{path}`) streams **through the Worker** → R2. Works everywhere; every byte
  traverses the Worker/DO tier (CPU + duration on the hot path). On CF it measured **0.2–9.7 MB/s from
  one host** — but that is **client-uplink-bound**, not server-bound (the local-floor run hit ~115 MB/s
  with the network removed). So the binding plane's server ceiling is high; the real-world limit is the
  client's pipe and the per-request overhead (hence MB/s rising with size).
- **Multipart/presigned plane** goes **edge↔R2 direct** — the architecture's intended bulk path ("bulk
  bytes never via the DO"), removing the Worker from the byte path. On this bench **presign worked but
  `complete_multipart_upload` failed (R2 10025)** — see §4.5; the direct-to-R2 throughput number is still
  open and is the highest-value remaining measurement.

**Takeaway:** binding is fine for small/medium blobs (config, manifests, sub-MB files); large files
*should* use the direct-to-R2 path — once §4.5 is fixed — to keep bytes off billable DO/Worker time.

### 3.3 Contention on one object — optimistic CAS stays correct, but wastes round-trips
The single-writer DO **never loses an update** (contended-CAS integrity check passed: 800 commits, final
counter exactly 800). But optimistic CAS-on-one-key under 16 contenders produced an **85% conflict rate**
— **4,662 wasted CAS attempts for 800 commits (5.8× amplification)**. Over the network the *throughput*
didn't collapse the way it does locally (27.7 vs 28 commits/s — because each commit is RTT-bound either
way), so on real CF **contention shows up as cost (wasted requests) more than as a latency cliff.** The
serialization is a correctness feature; the wasted round-trips are the bill (§4.1, §5).

### 3.4 No global bus — presence + streams are per-DO push
Presence and subscribe-streams are served from the **owning DO's** SQLite + connection set and pushed to
subscribers — no Redis, no fan-out tier. Register→propagate to 3 watchers is **p50 42.7 ms** on real CF
(≈ one RTT — i.e. propagation adds ~no server-side fan-out cost at this scale); the fan-out ceiling is
the subscriber count on that one DO. (PRESENT→STALE→ABSENT runs on a heartbeat **sweep alarm**, 30–60 s
cadence **by design** — a cost/correctness trade, not a latency regression.) Subscribe **replay**
(cheap re-subscribe) replays a bounded recent window — **200 entries in 26 ms (7,593/s)**. Replaying the
*entire* large log from seq 0 in one burst is a different story — see §4.4.

---

## 4. Bottlenecks (ranked, from the data)

1. **Optimistic CAS under contention on one Doc.** Evidence: 85% conflict rate, **5.8× request
   amplification** (5,762 CAS ops for 800 commits). Cause: read→CAS→retry storms on a single hot key.
   **B2 levers:** a server-side atomic op (fetch-add / conditional-debit that commits in one round-trip),
   client backoff/jitter, or counter sharding. *(The Doc spec already frames CAS as the lease/debit
   primitive — a native atomic increment would erase this.)* On CF this is primarily a **cost** bottleneck.
   **✅ RESOLVED in B2 (§7.1):** server-side atomic `incr` → 5,762→800 DO requests, 85%→0% conflicts,
   27.7→386 commits/s on the contended counter.
2. **Durable-write latency (~23 ms/commit) + single-stream append ceiling.** Evidence: Log/Doc writes are
   ~35 ms (vs 12.5 ms for a no-write RPC), and one Log DO sustains ~605 appends/s. Cause: each append/CAS
   is a durable SQLite commit. **B2 levers:** batched/pipelined append (amortize the commit), shard
   high-volume streams across Log DOs.
3. **Cold-start ~660 ms.** Evidence: cold Log/workspace DO p50 730/747 ms vs 71 ms warm connect. **B2
   levers:** keep latency-critical objects warm; mask wake-on-request in the UX (optimistic echo); treat
   first-touch as a one-time tax (it is — subsequent ops are warm).
4. **Full-log replay-from-zero can drop the Log WS.** Evidence: subscribing `sinceSeq:0` over a ~2,300-entry
   log in one burst stalled/closed the Log WS against real CF DOs (the harness now replays a bounded recent
   window — 200 entries / 26 ms — cleanly). Cause: a single large replay burst over one WS. **B2 lever:**
   server-side chunked/paced replay + backpressure for large catch-ups; clients should use read-pagination
   for full history and the resume cursor only for recent deltas.
5. **Multipart direct-to-R2 completion broken on this deployment.** Evidence: presign returns valid
   per-part PUT URLs and the PUTs succeed, but `complete_multipart_upload` fails with **R2 10025 "parts
   could not be found"** — the upload is opened via the R2 *binding* (`createMultipartUpload`) yet the
   parts are written via *presigned S3-API* PUTs; R2 appears not to associate the two at completion.
   **B2 / artifact-team:** open the MPU via the same plane the parts are written through (S3-API
   `CreateMultipartUpload` for presigned parts), or complete via the S3 API — then re-measure throughput.
   **✅ RESOLVED in B2 (§7.2):** open + complete via the S3 API; 64 MB completes + byte-match verified,
   ≈13.4 MB/s, 0 dangling MPUs.
6. **Connection amplification.** Each Log/Doc opens its **own** dedicated WS/DO. A client using many
   logs/docs opens many sockets (each ~71 ms to connect, ~730 ms if cold). Fine at small scale; watch
   clients that fan out across many primitives. **B2 lever (maybe):** a multiplexed transport, weighed
   against the isolation the per-object model buys.

## 5. Cost drivers (the cheap-to-operate goal)

Cloudflare bills DOs on **requests** + **active duration** (wall-time the isolate runs), and R2 on
**class-A/B ops** + storage. From the §2.2 cost signals:

- **DO requests scale 1:1 with ops and amplify under contention.** Each RPC call ≈ 1 inbound request to
  the workspace DO (the forward to the peer rides the peer's existing WS); each append/CAS = 1 request.
  Concurrency does not reduce request count. The contended-CAS case spent **5,762 DO requests for 800
  useful commits (≈7× incl. the gets)** — contention is a direct cost multiplier. *Atomic/batch ops cut
  both requests and latency.*
- **DO active duration:** hot-path streaming and **through-Worker artifact bytes** accrue billable
  wall-time; the **direct-R2 path keeps byte-time off the DO** (the cost case for fixing §4.5). **Idle
  DOs hibernate (~0 cost)** — the cheap-at-rest win, and why per-object sharding is also per-object cost
  isolation (no shared always-on infra). **Cold wake costs ~660 ms of latency but is a one-time tax**, not
  standing cost.
- **R2 ops:** one class-A per binding PUT + one class-B per GET; multipart would add `part_count` class-A
  PUTs + 1 complete (cheap relative to the DO duration it saves for large files).
- **Standing cost ≈ 0 at rest:** cold workspaces/logs/docs left behind (no DELETE API) hibernate to ~0 —
  which is why "no DELETE API" is tolerable for isolated tenants, and why a **dedicated teardownable
  deployment** (this run) is the clean choice for repeated benchmarking. (Bench teardown deletes the
  worker + R2 bucket entirely — §1.6.)

---

## 6. Honesty notes / limitations
- **§2.2 is real CF** but from **one host / one ISP uplink** to a single CF-assigned DO region — absolute
  latencies fold in this host's RTT (~12 ms base). A multi-region client matrix is future work.
- **§2.1 is loopback** — a *floor* used only to separate server work from RTT (§3.0); its absolute ms are
  not real-world.
- **Bounded load by default** — max-concurrent (200 WS) and throughput numbers are *floors* unless the run
  reports a failure at the ramp edge. The WS ceiling was **not** reached (no failures at 200).
- **Multipart direct-to-R2 not cleanly measured** — presign works, completion fails (§4.5); the headline
  bulk-throughput number remains open pending that fix.
- **Cold-start ≠ hibernation-wake** — idle eviction is runtime-timer-driven (not client-controllable), so
  cold-start uses first-ever instantiation as the actionable proxy (a fair upper bound on a wake).
- **Doc CAS serial = single-connection RTT-bound (28/s), not the DO ceiling** — see §3.1/§3.3; a concurrent
  uncontended CAS sweep (to find the true Doc write ceiling, cf. Log's 605/s) is a worthwhile B2 addition.

---

## 7. Phase B2 — optimizations: BEFORE / AFTER (real CF, fresh isolated bench)

**Status:** the three §4 levers earned by the data were implemented (additive, behavior-preserving) and
**re-measured on a fresh, isolated `hypha-mesh-bench` CF deployment** (same code/classes/migrations as
§2.2, own throwaway JWT secret + own R2 bucket, then **torn down** — prod `hypha` verified untouched).
Author: `gentle-island` (crew). Raw AFTER results: `bench/results/bench-bench-cf-after-*.json`.

> Method note: AFTER ran the SAME bench harness, SAME config (16 contenders × 50 rounds; 64 MB / 8 MB
> multipart; 15/8 cold DOs) against an equivalent fresh CF deployment, so AFTER↔§2.2-BEFORE is
> apples-to-apples. For the Doc lever the harness runs the OLD CAS path AND the NEW atomic path in the
> SAME run, so the comparison is on one deployment, same contention, same RTT.

### 7.1 Lever 1 — Doc CAS op-amplification → **server-side atomic `incr`** (the #1 bottleneck)

A new `incr` op on the Doc DO performs read-add-write in **one single-threaded turn** (atomic for free),
so a counter/ledger commits in **one round-trip with no read-modify-CAS-retry storm**. Optional `floor`
makes it the conditional-debit primitive (`incr(balance, -amount, {floor:0})` — never double-spends).
Client method `doc.incr(path, delta, {floor?})`; authorized exactly like `cas` (tier rw, no new permission).

**Contended counter — 16 connections racing on ONE key, 800 commits (same workload as §2.2):**

| metric | BEFORE — optimistic CAS | AFTER — atomic `incr` | change |
|---|---|---|---|
| conflicts | 4,662 = **85%** | **0 = 0%** | eliminated |
| DO requests for 800 commits | **5,762 (≈7.2×)** | **800 (1.0×)** | **7.2× fewer requests** |
| effective throughput | **27.7 commits/s** | **386 commits/s** | **≈14× higher** |
| integrity (final counter) | exact (800, no lost updates) | exact (800, no lost updates) | preserved |
| p50 latency / op | 35.2 ms | 37.3 ms | unchanged (RTT-bound) |

The per-op latency is the same (~36 ms, one client↔edge↔DO durable write — §3.0); the entire win is
**erasing wasted round-trips**. On CF, contention showed up as *cost* (DO requests) more than latency
(§3.3), so collapsing 5,762→800 requests for the same 800 commits is a direct ~7× cost cut on this
workload AND a ~14× throughput lift. Verified repeatable across runs (0 conflicts every time;
throughput 386–427/s). The OLD CAS path, re-measured in the same AFTER run, was unchanged (4,762
conflicts, 27.4/s) — confirming the comparison is honest. **REAL test** (`server/test/doc.test.ts`, full
suite green): N concurrent atomic increments → exact final value, EXACTLY one op per commit (no
amplification), strictly fewer ops than the CAS loop, and the floor-guarded debit never double-spends.

### 7.2 Lever 2 — R2 multipart completion (R2 error 10025) → **FIXED, byte-verified end-to-end**

Root cause (§4.5): the upload was OPENED via the R2 *binding* (`createMultipartUpload`) but parts were
written via *presigned S3-API* PUTs — two different MPU namespaces, so the binding's `complete` could
not find the parts. **Fix:** open AND complete the upload through the **same S3 API** the part PUTs use
(presigned `POST …?uploads` → presigned per-part PUTs → presigned `POST …?uploadId=…` with a parts XML
body), then `HEAD` the assembled object via the binding for its size/etag. Local dev has no S3 endpoint,
so that `fetch` throws and the code falls back to the binding plane — the local suite stays green.

| metric (64 MB in 8 × 8 MB parts, direct-to-R2) | BEFORE | AFTER |
|---|---|---|
| `complete_multipart_upload` | **FAILED — R2 10025 "parts could not be found"** | **succeeds** |
| download-back bytes match (size + tail marker) | n/a (never completed) | **✅ verified** |
| errors | — | 0 / 8 parts |
| throughput (aggregate) | — | **≈13.4 MB/s** (67 MB in 5.0 s; per-part p50 510 ms) |
| dangling MPUs left at teardown | — | **0** (clean completion) |

**REAL test:** the bench multipart scenario now uploads a large file → completes → **downloads it back and
verifies length + a stamped tail marker** (`bytes_match:1`), on real R2. Unit tests
(`server/test/artifact-r2.test.ts`) cover the new pure helpers (signed `?uploads` / `?uploadId` POST
presigns, `UploadId` parse, error-body detection, parts-XML build: sorted + quoted + escaped). The
binding plane (small/medium blobs) is unchanged: 8 MB binding upload p50 715 ms (≈11 MB/s) ≈ §2.2.

### 7.3 Lever 3 — cold DO wake (~660 ms): assessed → **platform-inherent (documented, no regression)**

| metric | BEFORE | AFTER |
|---|---|---|
| cold Log DO p50 | 730 ms | 795 ms |
| cold workspace DO p50 | 747 ms | 794 ms |

The cold delta vs a warm hop is **CF DO instantiation** (placement + isolate spin-up + first SQLite
open + handshake), not our code: the DO constructors do exactly one `CREATE TABLE IF NOT EXISTS`
(microseconds) and no I/O — already minimal, nothing cheap to shave. The small AFTER↔BEFORE difference
is run-to-run/region variance (AFTER p95 856 ms vs BEFORE 918 ms), and the additive Doc/artifact changes
don't touch the constructor path — **no regression**. Per the brief, keep-warm is deliberately NOT added
(over-engineering for a one-time tax); the actionable mitigation stays **UX masking** of wake-on-request
(optimistic echo / "waking…") per §3.0 / §4.3, and treating first-touch as the one-time cost it is.

### 7.4 Net effect

The #1 bottleneck (Doc CAS amplification) is **erased** for counter/ledger workloads — ~7× fewer DO
requests and ~14× throughput at equal correctness — and the highest-value open measurement (direct-to-R2
multipart) now **completes and round-trips bytes correctly**. Cold-start is confirmed platform-inherent.
All changes are additive and preserve every security invariant + the thin-layer discipline (no new core
primitive; `incr` reuses the Doc capability model, multipart reuses the existing S3 presign + R2 layer).
Levers #2 (durable-write batching), #4 (paced large replay), #6 (connection multiplexing) from §4 remain
open for a future pass — none were earned as urgently by the data as the three addressed here.