[idempotency]

This page

🔒 Idempotency & Durable Objects

How Hoox uses Cloudflare Durable Objects for at-most-once acceptance at the gateway (two-phase reserve/commit/release) so webhook retries cannot double-dispatch orders during network dropouts.

⚠️ The Danger: How Webhook Retries Lead to Double-Ordering

Gateway two-phase flow (v0.13+): reserve the key as pending before queue/service work, commit only when the trade truly succeeds (including soft-fail HTTP 2xx with success: false → release), so failed attempts can retry. Auto fingerprints include a per-minute time bucket so intentional same-size trades later are not blocked for the full TTL. Prefer a client idempotencyKey / Idempotency-Key header when you need cross-minute uniqueness.

Without idempotency, a typical signal failure sequence looks like this:

[TradingView® Webhook] ─── (Signal Post) ───> [Gateway Node] ─── (Submit Order) ───> [Exchange API]
                                                                                            │
                                                                                    (Order Filled!)
                                                                                            │
[TradingView® Webhook] <── (TLS/TCP Dropout) ── [Gateway Node] <── (Send Success) ─── (Connection drops)
         │
(No response: Retries!)
         │
[TradingView® Webhook] ─── (Signal Post) ───> [Gateway Node] ─── (Submit Order) ───> [Exchange API]
                                                                                            │
                                                                                   (DOUBLE-FILLED! ❌)

🛡️ The Hoox Solution: Durable Objects Mutex Locking

To enforce at-most-once acceptance within the TTL, Hoox implements an atomic two-phase dedup inside workers/hoox-worker utilizing Cloudflare Durable Objects.

A Durable Object is a unique, single-threaded compute isolate managed by Cloudflare that maintains its own highly optimized, in-memory state and persistent on-disk SQLite storage. Because access to a specific Durable Object instance is single-threaded, it acts as an absolute distributed lock (mutex).

The Idempotency Workflow (two-phase)

[Incoming Webhook Payload]
         │
         ▼
[Resolve key: Idempotency-Key | body.idempotencyKey | auto fingerprint]
         │  auto = trade:{ex}:{sym}:{action}:{qty}:{live|test}:{minuteBucket}
         ▼
[IdempotencyStore DO — fail-closed if binding missing]
         │
 ┌───────┴──────────────────────────────────────────┐
 │ Phase 1: reserve(key) under blockConcurrencyWhile│
 │  · committed / pending in TTL → reject (409)     │
 │  · new → store status=pending, schedule alarm    │
 └───────┬──────────────────────────────────────────┘
         │
         ├─► [Duplicate / in-flight] ──► 409 Conflict (no exchange call)
         │
         └─► [Reserved] ──► queue and/or trade-worker
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
            true success                    soft-fail / hard error
            (order path OK)                 (incl. HTTP 2xx success:false)
                    │                               │
                    ▼                               ▼
            Phase 2: commit(key)            Phase 2: release(key)
            (blocks retries for TTL)        (retry may reserve again)

🔍 The Dedup & Cleanup Algorithm

1. Key resolution

Preference order:

  1. Body idempotencyKey or HTTP Idempotency-Key header (max 256 chars) — use this for cross-minute uniqueness.
  2. Auto fingerprint: trade:{exchange}:{symbol}:{action}:{quantity}:{live|test}:{minuteBucket} so intentional same-size trades in a later minute are not blocked for the full TTL. Live and test modes never share a key.

2. Atomic two-phase evaluation

Before any queue or service dispatch:

  1. The gateway shards to an IdempotencyStore instance from the key. Missing IDEMPOTENCY_STORE binding → fail-closed (503 / IDEMPOTENCY_UNAVAILABLE).

  2. reserve: under single-threaded DO execution, pending or committed keys in TTL return duplicate; new keys become pending so concurrent retries cannot double-dispatch.

  3. On pipeline success: commit so further retries stay blocked for the TTL.

  4. On soft-fail (success: false even with HTTP 2xx) or hard error: release so a later retry can reserve again.

3. Automatic TTL & Storage Alarms

  • Default window is on the order of minutes (configured TTL); alarms reclaim expired keys so storage does not grow without bound.
  • Sharded object names keep hot keys from serializing the entire gateway behind one DO instance.

4. Cold-Start Resilience

When a DO instance has been idle (no requests for the TTL period), Cloudflare may evict it from memory. Upon the next request:

  1. The DO is reconstructed from its persisted SQLite state on disk.

  2. All previously recorded trace IDs are loaded back into memory.

  3. The dedup check continues seamlessly — no data loss, no duplicate risk.

This means idempotency protection survives worker restarts, cold starts, and infrastructure failovers without any manual intervention.

📊 Performance Impact

MetricValueNotes
DO ping overhead&lt; 2msPer-request latency added
Lock acquisition&lt; 1msSingle-threaded, no contention
Dedup check (cache hit)&lt; 0.5msIn-memory SQLite query
Storage alarm cleanup&lt; 5msRuns in background, non-blocking

Warning

Never disable idempotency check bindings in your wrangler.jsonc file in production. The performance cost of pinging the DO is less than 2 milliseconds, while the cost of a duplicate order could be catastrophic.

🔗 Next Steps