[Monitoring Operations]

Health probes, queue backlogs, live edge logs, and emergency kill-switch controls via the CLI.

Algorithmic trading demands high-integrity, real-time observability. Because Hoox microservices are distributed across Cloudflare's global edge network, tracking health, logs, and message queues requires a consolidated management plane.

This guide outlines how to use the Hoox CLI to audit system status, monitor queue backlogs, stream live edge logs, and manage the emergency kill switch.


1. Probing Worker Health Status

Every Hoox worker is configured with a secure /health endpoint that validates local CPU capacity, binding access, and D1 database connection integrity.

To probe and audit all active endpoints simultaneously:

hoox check health

hoox monitor status was removed in v0.8.0. The single source of truth for health checks is now hoox check health.

Expected Output

The CLI runs asynchronous concurrent health probes, displaying green indicators and public route locations:

┌─────────────────────────────────────────────────────────────┐
│               Hoox Microservice Health Probes                │
├─────────────────────────────────────────────────────────────┤
│  hoox (Gateway)     ...  OK  https://hoox.alpha.workers.dev│
│  trade-worker       ...  OK  (Internal Binding Verified)   │
│  d1-worker          ...  OK  (Internal Binding Verified)   │
│  telegram-worker    ...  OK  (Internal Binding Verified)   │
│  agent-worker       ...  OK  (Internal Binding Verified)   │
│  web3-wallet-worker ...  OK  (Internal Binding Verified)   │
│  email-worker       ...  OK  (Internal Binding Verified)   │
│  report-worker      ...  OK  (Internal Binding Verified)   │
│  analytics-worker   ...  OK  (Internal Binding Verified)   │
│                                                             │
│  System Status: HEALTHY (0 warnings, 0 errors)              │
└─────────────────────────────────────────────────────────────┘

Interpreting Health States

IndicatorMeaningAction Required
OKAll bindings active, D1 responsiveNone — operating normally
DEGRADEDSome bindings unavailableCheck hoox logs tail <worker> for errors
FAILEDWorker unreachable or D1 disconnectedRun hoox repair check immediately
STARTINGWorker is still booting after deployWait 30–60 seconds and retry

2. Governing the Global Kill Switch

The Global Kill Switch is your emergency brake. Stored inside the sub-millisecond CONFIG_KV namespace, flipping this parameter instantly blocks all incoming trade signals globally in under 10 seconds, without requiring code updates or worker redeployments.

hoox monitor kill-switch show

# B. Emergency HALT - disable all edge trade execution immediately
hoox monitor kill-switch on

# C. Resume normal operations
hoox monitor kill-switch off

Kill Switch Behavior

When activated, the following sequence occurs automatically:

  1. hoox gateway rejects all incoming /webhook POST requests with 503 Service Unavailable.

  2. A KILL_SWITCH_ACTIVE event is logged to D1 for audit trail.

  3. agent-worker halts its 5-minute cron execution loop.

  4. trade-worker queues any in-flight payloads for retry (not abandoned).

  5. Telegram notification is sent to alert you.

Warning

When the Kill Switch is turned ON, the hoox gateway router immediately rejects all incoming TradingView alerts and API webhook requests with a 503 Service Unavailable error and logs a KILL_SWITCH_ACTIVE warning. Active positions must be flattened manually or via hoox monitor trades close-all.


3. Auditing Recent Transactions

To review execution history directly from your terminal:

# Print the 10 most recent trades executed across all exchanges
hoox monitor trades

# Print the 50 most recent trades
hoox monitor trades 50

# Filter by specific exchange
hoox monitor trades --exchange bybit

# Filter by date range (ISO format)
hoox monitor trades --from 2026-05-15 --to 2026-05-20

This aggregates entries from your production D1 database and prints a formatted terminal table outlining: TIMESTAMP | REQUEST_ID | EXCHANGE | SYMBOL | ACTION | QUANTITY | PRICE | STATUS

Trade Status Codes

StatusMeaning
FilledOrder fully executed on the exchange
PartialOnly portion filled; remainder queued
RejectedExchange returned a business logic error
FailedMax queue retries exhausted; logged for review
EnqueuedSubmitted to Cloudflare Queues for async retry

4. Inspecting Queue Depth (Guaranteed Delivery Backlog)

If network volatility causes exchange API dropouts, Hoox offloads trade execution payloads to Cloudflare Queues. To check if there is an active execution backlog:

hoox monitor queue-depth

This displays the number of pending messages in the queue:

  • 0: System is running normally (all trades executing on the fast-path direct service binding).
  • > 0: Exchange API is experiencing rate-limits or downtime. Trade-worker is retrying transactions asynchronously in the background.

Queue Health Indicators

Queue DepthRisk LevelRecommended Action
0NormalNo action needed
1–5WatchfulMonitor exchange health pages
6–20ElevatedCheck exchange maintenance schedule; consider hoox config kv set webhooks:queue_mode direct_only
20+CriticalVerify exchange API keys are valid; check rate limit headers; scale manually if needed

5. Real-Time Log Streaming & Telemetry

When debugging custom strategies, you can stream verbose console logs directly from Cloudflare's global edge nodes to your local terminal using Wrangler bindings:

# A. Stream live console logs for a specific worker
hoox logs tail trade-worker

# B. Stream gateway traffic in real-time
hoox logs tail hoox

# C. Stream all workers simultaneously (diagnostic mode)
hoox logs tail --all

# D. Filter logs by severity level
hoox logs tail trade-worker --level error

Downloading Logs for Off-Line Analysis

To download historical logs offloaded to your R2 storage bucket for compliance audits or tax reports:

hoox logs download hoox --output logs/gateway-backup.log

# Download trade-worker logs for the last 24 hours
hoox logs download trade-worker --output logs/trade-execution.log --since 24h

6. System Metrics Overview

Hoox tracks key performance indicators via Cloudflare Analytics Engine. Access summary metrics via:

# View current system metrics snapshot
hoox monitor metrics

# Or use the Dashboard Command Center for visual charts
hoox dashboard

Key Metrics Tracked

MetricSourceDescription
Orders/Minanalytics-workerTrade execution throughput
Avg Latency (ms)analytics-workerSignal-to-fill round-trip time
Error Rate (%)analytics-workerFailed orders / total attempts
Queue DepthCloudflare QueuesPending execution backlog
Daily P&LD1 DatabaseRealized profit/loss calculation
Drawdown %agent-workerCurrent daily asset drawdown

Tip

If the system status check returns a red error indicator (e.g. d1-worker ... FAILED), immediately proceed to the Self-Healing & Repair Guide to run automated diagnostics and restore service bindings!

Next Steps