Alerting Drill Runbook

Staging Alerting Drill & Synthetic Fault Injection Runbook

1. Purpose & Scope

This operational runbook defines the standard operating procedures for executing synthetic fault injection drills and end-to-end alerting verification across SiteSwarm staging environments.

Synthetic alerting drills validate the complete operational pipeline:

Telemetry & Probe Emission: Workers Analytics Engine (WAE) domain metrics via @siteswarm/telemetry and synthetic HTTP /api/health probes.
Watchdog Worker Evaluation: Decoupled scheduled evaluation worker (services/watchdog) running every 2 minutes (*/2 * * * *) with a strict 25-second execution ceiling.
KV State Storage: Cloudflare KV (WATCHDOG_STATE) at alert:<rule_id>, storing lightweight metadata for $O(1)$ single-call listing alongside full body incident details.
Debouncing & Routing: KV-backed 15-minute suppression window to prevent alert storm paging, with 2-run flapping prevention for auto-resolution.
Multi-Channel Dispatch: P1 incident paging via BetterStack On-Call API (phone, SMS, push) and multi-channel notification webhooks (Discord & Slack).
Command Center Fleet Grid: Real-time fleet health aggregation in SiteSwarm Central Command Center (apps/command-center at /observability).

2. Production Safety Guardrail & Invariants

[!CAUTION] STRICT PRODUCTION GUARD: Synthetic fault injection and drill harnesses MUST NEVER execute against production workers, live tenant domains, or production KV namespaces.

The drill harness enforces the following non-negotiable safety guardrails:

Hard Abort Assertion: scripts/drill-alerting.ts asserts that env !== "production" and NODE_ENV !== "production". Any execution against production throws an uncatchable error and terminates with exit code 1 immediately.
Namespace Isolation: Synthetic drill rules strictly use the reserved prefix alert:drill:* (probe:drill-canary, wae:drill-dlq-spike), preventing interference with production rule IDs (probe:bakery, wae:dlq-error-spike).
Automated Teardown: The drill CLI automatically purges all temporary drill keys upon completion. Engineers can also trigger standalone cleanup via npx tsx scripts/drill-alerting.ts --cleanup.

3. Alerting Architecture & Operational Contracts

flowchart TD
    subgraph Ingress & Probes
        A["Client Worker /api/health"] -->|HTTP Probe 2500ms| W["services/watchdog (Scheduled Cron)"]
        T["@siteswarm/telemetry"] -->|dlq_routed| WAE["Workers Analytics Engine (WAE)"]
        WAE -->|SQL Query| W
    end

    subgraph Evaluation & State
        W -->|Evaluate Rules| EV["Watchdog Evaluator"]
        EV -->|Read Previous State| KV[("WATCHDOG_STATE (KV)")]
        EV -->|Write State & Metadata| KV
    end

    subgraph Router & Outbound
        EV -->|Breach / Recovery| R["Alert Router"]
        R -->|P1 Critical (Call/SMS/Push)| BS["BetterStack On-Call"]
        R -->|P1/P2 Webhooks| DC["Discord & Slack"]
        R -->|Debounce (15m)| S["Suppress Duplicate Outbound"]
    end

    subgraph Observability Cockpit
        KV -->|Single kv.list({ prefix: 'alert:' })| CC["Command Center (/observability)"]
    end

Key Operational Constants

Constant	Value	Description
Watchdog Cron Frequency	`/2 * * *` (120s)	Scheduled evaluation cycle triggering `evaluateScheduled`
Watchdog Timeout Ceiling	`25,000 ms`	Strict timeout ceiling enforcing bounded execution before CF 30s limit
Probe Timeout Ceiling	`2,500 ms`	Strict per-probe timeout via `AbortSignal.timeout(2500)`
Alert Detection SLA (TTD)	`<= 180 s` (3m)	Maximum acceptable time from fault injection to alert firing
Debounce Suppression Window	`15 minutes` (900,000 ms)	Active incident suppression window preventing repeated paging on consecutive cron runs
Auto-Resolution Requirement	`2 consecutive runs`	Required healthy evaluations before clearing firing state and resolving incident
BetterStack Severity	`P1`	P1 triggers phone call, SMS, email, and mobile push; P2/P3 dispatches webhooks only

4. Step-by-Step Staging Drill Execution

Step 1: Pre-Flight Verification

Verify that required dependencies and credentials are ready. For a dry-run drill, zero credentials are required.

# Verify tooling binaries
pnpm run check
pnpm run lint

For live staging verification against Cloudflare KV, set the following staging environment variables (or pull from Doppler / Vault):

export CLOUDFLARE_ACCOUNT_ID="<staging-account-id>"
export CLOUDFLARE_API_TOKEN="<staging-api-token>"
export BETTERSTACK_TOKEN="<staging-test-betterstack-token>"
export DISCORD_WEBHOOK_URL="<staging-test-discord-webhook>"
export SLACK_WEBHOOK_URL="<staging-test-slack-webhook>"

Step 2: Dry-Run Simulation (Recommended First Step)

Always run the drill with --dry-run first to test rule logic, synthetic timers, and dispatch paths without external network side-effects:

npx tsx scripts/drill-alerting.ts --dry-run

Expected output:

Baseline State Verification: Asserts initial KV state is clean.
Synthetic Fault Injection: Injects simulated 500 error & DLQ spike.
Watchdog Breach Detected: Verifies status: "firing" recorded and P1 incident paged.
Alert Debouncing Enforced: Verifies duplicate outbound alerts are suppressed within the 15-minute window.
Fault Cleared & Recovery Triggered: Simulates recovery (probe 200 OK, DLQ count 0).
Resolution Pending: Run 1 verifies status remains firing (consecutive_healthy: 1), confirming anti-flapping guard.
Auto-Resolution Completed: Run 2 verifies status transitions to healthy (consecutive_healthy: 2), BetterStack resolved, and recovery webhook posted.
Drill State Cleanup: Purges test drill keys from KV.
Exit code: 0.

Step 3: Targeted Fault Drills

Scenario A: Degraded Client Health Probe Drill (`--fault=probe`)

Simulates a client application outage (e.g. /api/health returning HTTP 500 or timing out after 2,500ms):

npx tsx scripts/drill-alerting.ts --env=staging --fault=probe --target=bakery

Scenario B: DLQ Error Spike Metric Drill (`--fault=dlq`)

Simulates an asynchronous event processing failure triggering dead-letter queue routing exceeding the threshold:

npx tsx scripts/drill-alerting.ts --env=staging --fault=dlq

Scenario C: Full Dual-Fault Drill (`--fault=all`)

Simulates concurrent edge storefront degradation and event processing pipeline failure:

npx tsx scripts/drill-alerting.ts --env=staging --fault=all

5. Verifying On-Call Paging & Communication Channels

During a live staging drill, on-call engineers should verify receipt of notifications across configured channels:

1. BetterStack Incident Verification

Incident Creation: Verify that BetterStack triggers a new incident with title [P1] Synthetic Drill Canary Health Probe Breach.
Escalation Policy: Verify phone call, SMS, and mobile push notifications are delivered within 60 seconds of alert_fired.
Auto-Resolution: Confirm that after 2 consecutive healthy probe runs, the incident automatically transitions to Resolved in BetterStack without manual engineer intervention.

2. Discord Webhook Notification

Verify rich embed formatting in the designated alerts channel:

Firing Alert: Red embed (#ED4245), 🚨 [P1] Alert Firing: Synthetic Drill Canary Health Probe, displaying App, Environment, Severity, Commit SHA, and Fleet Status deep link.
Resolved Alert: Green embed (#57F287), ✅ Alert Resolved: Synthetic Drill Canary Health Probe, reporting auto-resolution after 2 consecutive healthy runs.

3. Slack Webhook Notification

Verify block-kit message formatting in the designated Slack channel:

Header: 🚨 *[P1] Alert Firing: Synthetic Drill Canary Health Probe*
Fields: App Name, Environment, Status, Commit SHA.
Resolution: ✅ *Alert Resolved: Synthetic Drill Canary Health Probe*

4. Central Command Center UI Verification

Navigate to https://command-center.staging.siteswarm.dev/observability (or http://localhost:4321/observability):

Verify the Fleet Observability Grid displays the drill target card with INCIDENT status and pulsing red indicator.
Verify the deep-link launcher buttons:
- Launch Probe: Deep-links to /api/health.
- View Metrics: Deep-links to Cloudflare Workers Analytics Engine query dashboard.
- BetterStack Incident: Deep-links directly to the active BetterStack incident page.
Once resolved, verify the card immediately restores to HEALTHY status and green indicator.

6. Interpreting Watchdog KV State

The Watchdog Worker persists alert state directly in Cloudflare KV under the binding WATCHDOG_STATE.

KV Key Structure

alert:<rule_id>

Examples:

alert:probe:bakery
alert:probe:auto-repair
alert:wae:dlq-error-spike
alert:drill:probe:drill-canary

KV Metadata Schema (`AlertMetadataRecord`)

Stored in Cloudflare KV key metadata to enable single-call listing without body GETs:

{
  "rule_id": "probe:bakery",
  "status": "firing",
  "severity": "P1",
  "updated_at": "2026-11-10T00:34:00.000Z",
  "last_breach": "2026-11-10T00:34:00.000Z",
  "last_notified": "2026-11-10T00:34:00.000Z",
  "consecutive_healthy": 0
}

KV Value Schema (`AlertStateRecord`)

Stored as the body payload:

{
  "rule_id": "probe:bakery",
  "status": "firing",
  "severity": "P1",
  "updated_at": "2026-11-10T00:34:00.000Z",
  "last_breach": "2026-11-10T00:34:00.000Z",
  "last_notified": "2026-11-10T00:34:00.000Z",
  "consecutive_healthy": 0,
  "incident_id": "bs_inc_99812",
  "message": "Health check returned HTTP 500 Internal Server Error",
  "details": {
    "statusCode": 500,
    "url": "https://greenleafbakery.com/api/health"
  }
}

Inspecting KV State via Wrangler

# List all active alerts and metadata in staging
wrangler kv key list --namespace-id=watchdog-state-staging --prefix="alert:" --env=staging

# Get full body for a specific alert
wrangler kv key get --namespace-id=watchdog-state-staging "alert:probe:bakery" --env=staging

7. Troubleshooting & Escalation Procedures

Issue 1: Alert Fails to Fire within 3 Minutes

Check Watchdog Cron Schedule: Verify that the scheduled event fired in Cloudflare Worker logs:
Terminal window
```
wrangler tail siteswarm-service-watchdog-staging
```
Inspect HTTP Probe Response: Curl target /api/health manually:
Terminal window
```
curl -I https://<target-domain>/api/health
```
If response time exceeds 2,500ms, the probe times out as expected.
Verify KV Permissions: Ensure the worker has KVNamespace binding permissions to write WATCHDOG_STATE.

Issue 2: Alert Fails to Auto-Resolve after Fault Recovery

Flapping Guard Verification: Ensure at least 2 consecutive cron runs (4 minutes) have elapsed since recovery. Run 1 transitions state to healthy_pending (status: "firing" with consecutive_healthy: 1).
WAE Query Window: For DLQ spike alerts, WAE queries timestamp > NOW() - INTERVAL '10' MINUTE. If error events occurred within the last 10 minutes, the alert will remain active until the 10-minute window rolls over.

Issue 3: BetterStack Incident Not Created

Verify Token: Verify BETTERSTACK_TOKEN is configured in .dev.vars or worker secrets.
Check Fail-Closed Logs: BetterStack HTTP errors (429, 500, 503) log console.warn without throwing. Check worker tail logs for: [BetterStack] Failed to dispatch incident for <ruleId>: HTTP <status>

Emergency Cleanup of Drill Keys

If a drill process was terminated before cleanup ran, purge test keys immediately:

# Automated cleanup via drill harness
npx tsx scripts/drill-alerting.ts --cleanup

# Manual cleanup via Wrangler CLI
wrangler kv key delete --namespace-id=watchdog-state-staging "alert:drill:probe:drill-canary" --env=staging

8. Pre-Release Verification Checklist

Before releasing changes to services/watchdog or alerting rules to production, complete the following checklist: