[monitor-trading]
This page
📈 Monitoring Operations
Algorithmic trading demands high-integrity, real-time observability. Because Hoox microservices are distributed across Cloudflare®'s global edge network, tracking health, logs, and message queues requires a consolidated management plane.
This guide outlines how to use the Hoox CLI to audit system status, monitor queue backlogs, stream live edge logs, and manage the emergency kill switch.
🟢 1. Probing Worker Health Status
Every Hoox worker is configured with a secure /health endpoint that validates local CPU capacity, binding access, and D1 database connection integrity.
To probe and audit all active endpoints simultaneously:
hoox check health
Use
hoox check healthas the single source of truth for worker health probes (not a monitor subcommand).
Expected Output
The CLI runs asynchronous concurrent health probes, displaying green indicators and public route locations:
┌─────────────────────────────────────────────────────────────┐
│ Hoox Microservice Health Probes │
├─────────────────────────────────────────────────────────────┤
│ hoox (Gateway) ... ✅ OK https://hoox.alpha.workers.dev│
│ trade-worker ... ✅ OK (Internal Binding Verified) │
│ d1-worker ... ✅ OK (Internal Binding Verified) │
│ telegram-worker ... ✅ OK (Internal Binding Verified) │
│ agent-worker ... ✅ OK (Internal Binding Verified) │
│ web3-wallet-worker ... ✅ OK (Internal Binding Verified) │
│ email-worker ... ✅ OK (Internal Binding Verified) │
│ report-worker ... ✅ OK (Internal Binding Verified) │
│ analytics-worker ... ✅ OK (Internal Binding Verified) │
│ │
│ System Status: HEALTHY (0 warnings, 0 errors) │
└─────────────────────────────────────────────────────────────┘
Interpreting Health States
| Indicator | Meaning | Action Required |
|---|---|---|
| ✅ OK | All bindings active, D1 responsive | None — operating normally |
| ⚠️ DEGRADED | Some bindings unavailable | Check hoox logs worker <worker> for errors |
| ❌ FAILED | Worker unreachable or D1 disconnected | Run hoox repair check immediately |
| 🔄 STARTING | Worker is still booting after deploy | Wait 30–60 seconds and retry |
🚨 2. Governing the Global Kill Switch
The Global Kill Switch is your emergency brake. Stored inside the sub-millisecond CONFIG_KV namespace, flipping this parameter instantly blocks all incoming trade signals globally in under 10 seconds, without requiring code updates or worker redeployments.
hoox monitor kill-switch show
# B. Emergency HALT - disable all edge trade execution immediately
hoox monitor kill-switch on
# C. Resume normal operations
hoox monitor kill-switch off
Kill Switch Behavior
When activated, the following sequence occurs automatically:
hooxgateway rejects all incoming/webhookPOST requests with503 Service Unavailable.A
KILL_SWITCH_ACTIVEevent is logged to D1 for audit trail.agent-workerhalts its cron execution loop.trade-workerqueues any in-flight payloads for retry (not abandoned).- Telegram notification is sent to alert you.
Warning
When the Kill Switch is turned ON, the hoox gateway router immediately
rejects all incoming TradingView® alerts and API webhook requests with a 503 Service Unavailable error and logs a KILL_SWITCH_ACTIVE warning. Active
positions must be flattened manually or via hoox monitor trades close-all.
🗃️ 3. Auditing Recent Transactions
To review execution history directly from your terminal:
# Print the 10 most recent trades executed across all exchanges
hoox monitor trades
# Print the 50 most recent trades
hoox monitor trades 50
# Filter by specific exchange
hoox monitor trades --exchange bybit
# Filter by date range (ISO format)
hoox monitor trades --from 2026-05-15 --to 2026-05-20
This aggregates entries from your production D1 database and prints a formatted terminal table outlining: TIMESTAMP | REQUEST_ID | EXCHANGE | SYMBOL | ACTION | QUANTITY | PRICE | STATUS
Trade Status Codes
| Status | Meaning |
|---|---|
EXECUTED | Live order filled; written by trade-worker to D1 |
TEST_EXECUTED | Testnet ("test": true) fill; excluded from dashboard headline stats |
Filled | Exchange-side full fill (report/CLI display) |
Partial | Only portion filled; remainder queued |
Rejected | Exchange returned a business logic error |
Failed | Max queue retries exhausted; logged for review |
Enqueued | Submitted to Cloudflare® Queues for async retry |
Sandbox fills use position ids like bybit-testnet-BTCUSDT-LONG. Filter them in the Dashboard Positions mode control (Live only / Testnet / Live + Test). See Test Trading.
📨 4. Inspecting Queue Depth (Guaranteed Delivery Backlog)
If network volatility causes exchange API dropouts, Hoox offloads trade execution payloads to Cloudflare® Queues. To check if there is an active execution backlog:
hoox monitor queue-depth
This displays the number of pending messages in the queue:
0: System is running normally (all trades executing on the fast-path direct service binding).> 0: Exchange API is experiencing rate-limits or downtime. Trade-worker is retrying transactions asynchronously in the background.
Queue Health Indicators
| Queue Depth | Risk Level | Recommended Action |
|---|---|---|
| 0 | 🟢 Normal | No action needed |
| 1–5 | 🟡 Watchful | Monitor exchange health pages |
| 6–20 | 🟠 Elevated | Check exchange maintenance schedule; consider hoox config kv set webhooks:queue_mode direct_only |
| 20+ | 🔴 Critical | Verify exchange API keys are valid; check rate limit headers; scale manually if needed |
📝 5. Real-Time Log Streaming & Telemetry
When debugging custom strategies, you can stream verbose console logs directly from Cloudflare®'s global edge nodes to your local terminal (hoox logs worker):
# A. Stream live console logs for a specific worker
hoox logs worker trade-worker
# B. Stream gateway traffic in real-time
hoox logs worker hoox
# C. Stream all workers simultaneously (diagnostic mode)
hoox logs all
# D. Filter logs by severity level
hoox logs worker trade-worker --level error
Downloading Logs for Off-Line Analysis
To download historical logs offloaded to your R2 storage bucket for compliance audits or tax reports:
hoox logs worker hoox --output logs/gateway-backup.log
# Download trade-worker logs for the last 24 hours
hoox logs worker trade-worker --output logs/trade-execution.log --since 24h
📊 6. System Metrics Overview
Hoox tracks key performance indicators via Cloudflare® Analytics Engine. Access summary metrics via:
# View current system metrics snapshot
hoox monitor metrics
# Or use the Dashboard Command Center for visual charts
hoox dashboard
Key Metrics Tracked
| Metric | Source | Description |
|---|---|---|
| Orders/Min | analytics-worker | Trade execution throughput |
| Avg Latency (ms) | analytics-worker | Signal-to-fill round-trip time |
| Error Rate (%) | analytics-worker | Failed orders / total attempts |
| Queue Depth | Cloudflare® Queues | Pending execution backlog |
| Daily P&L | D1 Database | Realized profit/loss calculation |
| Drawdown % | agent-worker | Current daily asset drawdown |
Tip
If the system status check returns a red error indicator (e.g. d1-worker ... ❌ FAILED), immediately proceed to the Self-Healing & Repair
Guide to run automated diagnostics and restore service bindings!
🔗 Next Steps
- Self-Healing & Repair — Diagnose connection drops, recreate broken bindings, and rebuild environments.
- Terminal UI Operations — Monitor health status, tail logs, and edit configurations interactively in a full-screen GUI cockpit.
- Infrastructure Management — Manage KV config namespaces, Queue parameters, and R2 storage buckets.