Watchdog Platform Service

Watchdog Platform Service & Fleet Alerting Architecture (`WATCHDOG_PLATFORM_SERVICE.md`)

1. Executive Summary & Platform-Hosted Topology

The Watchdog Platform Service (@siteswarm/service-watchdog) delivers an autonomous, edge-native sentinel and health watchdog across the SiteSwarm multi-tenant platform. Operating as a decoupled, platform-hosted service worker (MULTI_WORKER_PLATFORM_SERVICES.md), it continuously audits fleet-wide operational health, evaluates synthetic probes and telemetry metric breaches, tracks incident lifecycles in Cloudflare KV, and orchestrates on-call paging through BetterStack and multi-channel webhooks.

While individual client applications (apps/*) and asynchronous workers (services/*) emit telemetry and serve traffic, the Watchdog operates on an external, out-of-band evaluation cycle. It guarantees continuous observability and rapid breach detection without adding bundle weight or cold-start latency to visitor-facing client storefronts.

flowchart TD
    subgraph ClientAndPlatformWorkers ["1. Edge Workloads & Telemetry Producers"]
        direction TB
        ClientApps["Client Applications (apps/*)\n(10 Edge Applications)\n• Expose /api/health probes\n• Emit WAE Domain Telemetry"]
        EventPlatform["Private Event Platform (services/event-platform)\n• DLQ Routing Events\n• Broker Ingestion Metrics"]
    end

    subgraph TelemetryIngress ["2. Telemetry Aggregation"]
        WAE[("Workers Analytics Engine (WAE)\nDataset: siteswarm_domain_metrics\n• blob1: appId (Indexed)\n• blob2: metricName (Indexed)\n• timestamp: Event Time")]
        ClientApps -.->|"emitDomainMetric()"| WAE
        EventPlatform -.->|"dlq_routed"| WAE
    end

    subgraph WatchdogIsolate ["3. Watchdog Service Worker (services/watchdog)"]
        direction TB
        CronTrigger["Cron Trigger: */2 * * * *\n(25s Strict Execution Ceiling)"]
        Evaluator["Alert Evaluator (evaluator.ts)\nConcurrent Promise.allSettled"]

        subgraph DualEngine ["Dual Evaluation Engine"]
            ProbeRunner["Synthetic Probe Runner\n• 10 Client /api/health\n• AbortSignal.timeout(2500ms)"]
            WaeRunner["WAE SQL Query Engine\n• Parameter Sanitization\n• Bounded Timeframes"]
        end

        Registry["Rule Registry (registry.ts)\n• PROBE_ALERT_RULES\n• WAE_ALERT_RULES"]
        Router["Alert Router (router.ts)\n• 15m Debounce Window\n• 2-Run Auto-Resolution"]

        CronTrigger ==> Evaluator
        Registry --> Evaluator
        Evaluator --> ProbeRunner
        Evaluator --> WaeRunner
        ProbeRunner --> Router
        WaeRunner --> Router
    end

    subgraph StorageLayer ["4. Incident State Persistence"]
        KV[("Cloudflare KV: WATCHDOG_STATE\n• Key: alert:<ruleId>\n• Metadata: { status, severity, last_breach }\n• Value: Full AlertStateRecord JSON")]
    end

    subgraph EgressDispatch ["5. Outbound Paging & Notification"]
        BetterStack["BetterStack Incident API\n(P1 Escalations: Phone, SMS, Push)"]
        Webhooks["Multi-Channel Webhooks\n(Discord & Slack Rich Embeds)"]
        CommandCenter["Command Center Cockpit (apps/command-center)\n(/observability single-call KV list)"]
    end

    %% Wiring
    ProbeRunner ==>|"Concurrent HTTP GET /api/health (2.5s)"| ClientApps
    WaeRunner ==>|"GraphQL / SQL API Query (blob1, blob2)"| WAE
    Router <===>|"Read State / Record State"| KV
    Router ==>|"P1 Firing / Resolve"| BetterStack
    Router ==>|"P1/P2 Firing / Resolve"| Webhooks
    KV ==>|"Single-Call kv.list({ prefix: 'alert:' })"| CommandCenter

2. Core Architectural Principles & Invariants

The Watchdog Platform Service operates under six non-negotiable architectural invariants:

1. Segregation of Duties & Circular Dependency Immunity

The Watchdog runs inside a completely isolated Cloudflare Worker environment (siteswarm-service-watchdog). It maintains zero runtime dependencies on Cloudflare D1 databases, queue brokers, caching layers, or client application internal code. If the platform database experiences an outage or queue workers stall, the Watchdog remains fully functional, evaluating probes, reading analytics, and firing emergency on-call pages.

2. Bounded Cron Execution Ceiling (`DEFAULT_OVERALL_TIMEOUT_MS = 25000`)

Cloudflare scheduled cron triggers impose a 30-second CPU/wall-clock limit. To guarantee predictable termination and eliminate zombie executions, evaluateScheduled() wraps entire evaluation cycles in a strict 25-second timeout via Promise.race(). Any probe or query that exceeds this ceiling is cancelled before worker eviction.

3. Sub-2.5s Synthetic Probe Abort (`timeoutMs = 2500`)

Every synthetic probe to an application’s /api/health endpoint is bound by AbortSignal.timeout(2500). A degraded or hanging client application cannot exhaust Watchdog thread pools or delay evaluations for subsequent applications. Socket hangs, slow TLS negotiations, or edge worker timeouts abort at 2,500ms and immediately transition the probe to firing.

4. WAE Indexable Column Pruning & Timeframe Bounds

Unbounded analytical scans against Workers Analytics Engine risk slow query latency and API rate-limiting. All queries generated by buildWaeAlertQuery():

Filter exclusively on indexed blob columns (blob1 = '<appId>' and blob2 = '<metricName>').
Enforce mandatory timeframe interval windows (timestamp > NOW() - INTERVAL '<N>' MINUTE).
Strictly forbid wildcard or table full-scan patterns.

5. 15-Minute Alert Debounce Window (`DEBOUNCE_WINDOW_MS = 900000`)

Because the evaluation cron runs every 2 minutes (*/2 * * * *), an active incident would generate 30 paging alerts per hour without rate suppression. The alert router enforces a 15-minute suppression window on firing rules:

Outbound pages to BetterStack and webhooks are dispatched immediately on initial transition to firing.
Subsequent breach evaluations within the 15-minute window update KV state (updated_at, details) but return action debounced, suppressing duplicate outbound notifications.

6. Anti-Flapping & 2-Consecutive-Healthy Auto-Resolution Gate (`REQUIRED_CONSECUTIVE_HEALTHY = 2`)

Transient network blips or edge cold-starts can cause rapid flapping between healthy and degraded states. Watchdog enforces an anti-flapping recovery gate:

When a rule transitions from firing to healthy, the incident is not immediately resolved.
Run 1 sets state to healthy_pending (consecutive_healthy = 1), preserving the active incident.
Only upon achieving 2 consecutive healthy evaluations (consecutive_healthy = 2) does Watchdog execute automated resolution: resolving the BetterStack incident, posting recovery webhooks, and clearing KV incident state.

3. Dual Evaluation Engine

Watchdog executes two complementary evaluation strategies in parallel during each scheduled run:

flowchart LR
    Schedule["Cron Schedule\n(*/2 * * * *)"] --> Concurrency["Promise.allSettled()"]

    subgraph SyntheticEngine ["Synthetic Health Probes (10 Target Apps)"]
        direction TB
        P1["GET /api/health (Apex Auto)"]
        P2["GET /api/health (Green Leaf Bakery)"]
        P3["GET /api/health (CLK Construction)"]
        Pn["GET /api/health (7 Other Apps...)"]
    end

    subgraph AnalyticsEngine ["WAE Metric Telemetry (SQL Queries)"]
        direction TB
        Q1["DLQ Error Spike (dlq_routed > 0 in 10m)"]
        Qn["Future Telemetry Queries..."]
    end

    Concurrency --> SyntheticEngine
    Concurrency --> AnalyticsEngine

3.1 Synthetic HTTP Health Probe Engine

Synthetic probes simulate real client traffic by dispatching HTTP GET requests to /api/health across all registered applications.

Probe Evaluation Contract (`evaluateProbeRule`)

Target Construction: Evaluates against https://${rule.domain}/api/health (or staging origin overrides supplied via env.PROBE_BASE_URLS).
Signal Timeout: Sets AbortSignal.timeout(rule.timeoutMs ?? 2500).
Response Inspection:
- HTTP status must be 200 OK (any 4xx/5xx status triggers immediate firing).
- JSON payload must parse successfully and report status: "ok" or status: "healthy" (case-insensitive).
- Any report of degraded, error, or unparseable JSON triggers firing.
Exception Mapping:
- Network errors, DNS resolution failures, and connection resets are caught and classified as Health check probe failed: <message>.
- TimeoutError or aborted signals are classified as Health check timed out after <timeoutMs>ms.

Registered Fleet Probes

The authoritative probe registry tracks all 10 deployable client applications:

Rule ID	Application (`appId`)	Canonical Domain	Target Endpoint	Severity	Timeout
`probe:auto-repair`	`auto-repair`	`apexautoservice.com`	`/api/health`	P1	2,500 ms
`probe:bakery`	`bakery`	`greenleafbakery.com`	`/api/health`	P1	2,500 ms
`probe:clk-construction`	`clk-construction`	`clkconstructioninc.com`	`/api/health`	P1	2,500 ms
`probe:me-portfolio`	`me-portfolio`	`jacobmiller22.com`	`/api/health`	P1	2,500 ms
`probe:rcc-stjohn`	`rcc-stjohn`	`stjohnscatholicchurch.org`	`/api/health`	P1	2,500 ms
`probe:rcc-stpatrick`	`rcc-stpatrick`	`saintpatrickchurchhill.org`	`/api/health`	P1	2,500 ms
`probe:rcc-stpeter`	`rcc-stpeter`	`stpeterchurch1834.org`	`/api/health`	P1	2,500 ms
`probe:software-agency`	`software-agency`	`apexlabs.dev`	`/api/health`	P1	2,500 ms
`probe:marketing`	`marketing`	`siteswarm.dev`	`/api/health`	P1	2,500 ms
`probe:docs`	`docs`	`docs.jacobmiller22.com`	`/api/health`	P2	2,500 ms

3.2 Workers Analytics Engine (WAE) SQL Metric Engine

While synthetic probes verify edge reachability and HTTP availability, WAE metric rules detect deep platform anomalies, background task failures, and Dead Letter Queue (DLQ) escalations.

SQL Query Builder & Sanitization Pipeline (`buildWaeAlertQuery`)

To guard against SQL injection while maintaining sub-100ms query performance, all analytical queries are constructed using a strict sanitization pipeline:

// Query construction with indexable columns and bounded timeframes
export function buildWaeAlertQuery(options: WaeQueryOptions = {}): string {
  const dataset = sanitizeSqlIdentifier(options.dataset ?? "siteswarm_domain_metrics", "dataset");
  const metricAlias = sanitizeSqlIdentifier(options.metricAlias ?? "dlq_count", "metricAlias");
  const timeframeMinutes = sanitizeTimeframeMinutes(options.timeframeMinutes ?? 10);

  const whereClauses: string[] = [];

  // Indexable column: blob1 (appId)
  if (options.appId !== undefined) {
    const safeAppId = sanitizeSqlLiteral(options.appId, "appId");
    whereClauses.push(`blob1 = '${safeAppId}'`);
  }

  // Indexable column: blob2 (metricName)
  if (options.metricName !== undefined) {
    const safeMetricName = sanitizeSqlLiteral(options.metricName, "metricName");
    whereClauses.push(`blob2 = '${safeMetricName}'`);
  }

  // Timeframe bound prevents table full-scans
  whereClauses.push(`timestamp > NOW() - INTERVAL '${timeframeMinutes}' MINUTE`);

  return `SELECT count() AS ${metricAlias} FROM ${dataset} WHERE ${whereClauses.join(" AND ")}`;
}

SQL Defense-in-Depth Invariants:

Identifier Whitelisting (sanitizeSqlIdentifier): Requires ^[a-zA-Z0-9_]+$. Any quotes, spaces, or semicolons throw immediately.
Literal Value Sanitization (sanitizeSqlLiteral): Rejects quotes, semicolons, comments (--, /*), and backslashes. Permits only safe alphanumeric and hyphen/colon characters (^[a-zA-Z0-9_\-\.:]+$).
Timeframe Range Bounds (sanitizeTimeframeMinutes): Requires a positive integer between 1 and 10,080 minutes (7 days). Prevents unbounded queries.
Structural Query Inspection (assertSafeSqlQuery): Rejects destructive keywords (drop, delete, truncate, union select) and tautology patterns (where 1=1, or 1=1).

Registered WAE Metric Rules

Rule ID	Metric Target	Service / Dataset	Query Profile	Threshold	Severity
`wae:dlq-error-spike`	`dlq_routed`	`siteswarm-service-event-platform` / `siteswarm_domain_metrics`	`blob1 = 'siteswarm-service-event-platform' AND blob2 = 'dlq_routed'` (10m window)	`> 0`	P1

4. Incident State Machine & KV Storage Schema

Watchdog uses Cloudflare KV (WATCHDOG_STATE) to maintain state between stateless cron invocations.

4.1 Storage Architecture: The Dual-Write Pattern

Cloudflare KV allows attaching a lightweight JSON metadata object (up to 1,024 bytes) to any key write. Watchdog leverages this for single-call listing performance:

flowchart TD
    subgraph KVEntry ["KV Key: alert:<ruleId>"]
        direction TB
        Meta["KV Metadata (Lightweight, Indexed)\n{\n  rule_id: 'probe:bakery',\n  status: 'firing',\n  severity: 'P1',\n  updated_at: '2026-11-11T06:00:00Z',\n  last_breach: '2026-11-11T06:00:00Z',\n  last_notified: '2026-11-11T06:00:00Z',\n  consecutive_healthy: 0\n}"]
        Body["KV Value (Full JSON String)\n{\n  rule_id: 'probe:bakery',\n  status: 'firing',\n  severity: 'P1',\n  message: 'Health check probe failed: HTTP 500',\n  details: { statusCode: 500, url: 'https://greenleafbakery.com/api/health' },\n  incident_id: 'bs_inc_982341',\n  ...\n}"]
    end

KV Value (Full State): Stores complete debug diagnostic payloads, stack traces, HTTP status codes, and upstream BetterStack incident IDs.
KV Metadata (Indexed Summary): Stores essential routing flags (status, severity, last_breach, last_notified, consecutive_healthy).
Single-Call Listing Contract: getFleetHealth(kv) calls kv.list({ prefix: 'alert:' }). It extracts all fleet statuses directly from the list metadata without performing individual kv.get() calls, avoiding N+1 fan-out latency.

4.2 Incident Lifecycle State Machine

stateDiagram-v2
    [*] --> HEALTHY: Initial State / First Run

    HEALTHY --> FIRING: Probe Fails or WAE Metric Exceeds Threshold
    note right of FIRING
        • Record last_breach & last_notified
        • Dispatch BetterStack P1 incident
        • Dispatch Discord/Slack webhooks
        • consecutive_healthy = 0
    end note

    FIRING --> FIRING: Subsequent Breach
    note right of FIRING
        • If < 15m since last_notified: Debounce (Suppress outbound page)
        • If >= 15m: Re-notify
        • Update updated_at in KV
    end note

    FIRING --> HEALTHY_PENDING: Single Healthy Evaluation (Run 1)
    note right of HEALTHY_PENDING
        • Anti-Flapping Gate: Incident remains active
        • consecutive_healthy = 1
        • Outbound alerts suppressed
    end note

    HEALTHY_PENDING --> FIRING: Probe Fails on Run 2 (Flap Detected)
    note left of FIRING
        • Reset consecutive_healthy = 0
        • Incident continues uninterrupted
    end note

    HEALTHY_PENDING --> RESOLVED: Second Healthy Evaluation (Run 2)
    note right of RESOLVED
        • consecutive_healthy >= 2 achieved
        • Resolve BetterStack Incident
        • Dispatch Recovery Webhook
        • Clear incident_id from KV
        • Transition status to "healthy"
    end note

    RESOLVED --> HEALTHY: State Re-harmonized
    HEALTHY --> HEALTHY: Consecutive Healthy Run (consecutive_healthy++)

4.3 Data Contract Schemas

export type AlertSeverity = "P1" | "P2" | "P3";
export type AlertStatus = "healthy" | "firing";

/**
 * Full persistent alert state stored in KV body at alert:<ruleId>
 */
export interface AlertStateRecord {
  rule_id: string;
  status: AlertStatus;
  severity: AlertSeverity;
  updated_at: string;
  last_breach?: string;
  last_notified?: string;
  incident_id?: string;
  consecutive_healthy?: number;
  message?: string;
  details?: Record<string, unknown>;
}

/**
 * Lightweight metadata attached to KV entry for single-call listing
 */
export interface AlertMetadataRecord {
  rule_id: string;
  status: AlertStatus;
  severity: AlertSeverity;
  updated_at: string;
  last_breach?: string;
  last_notified?: string;
  consecutive_healthy?: number;
}

/**
 * Ephemeral result of a single rule evaluation
 */
export interface AlertEvaluationResult {
  rule_id: string;
  status: AlertStatus;
  severity: AlertSeverity;
  message?: string;
  details?: Record<string, unknown>;
  evaluated_at: string;
}

5. Alert Dispatch & Multi-Channel Routing Pipeline

When a breach occurs, the Alert Router (router.ts) orchestrates outbound paging across two dedicated egress pipelines:

flowchart TD
    Result["AlertEvaluationResult\n(status: 'firing')"] --> Router["Alert Router (routeAlert)"]

    Router --> CheckDebounce{"Is Rule Firing &<br/>Within 15m Debounce Window?"}
    CheckDebounce -->|"Yes"| Debounce["Suppress Outbound Dispatch\n(Return action: 'debounced')"]

    CheckDebounce -->|"No (Initial Breach or Expired Window)"| Severity{"Rule Severity"}

    Severity -->|"P1 (Critical)"| BS["BetterStack Incident API\n• Triggers On-Call Escalation\n• Voice Call, SMS, Mobile Push\n• Returns incidentId"]
    Severity -->|"P1 or P2"| Webhooks["Multi-Channel Webhooks\n• Discord Webhook (Rich Embed)\n• Slack Webhook (Block Kit)\n• Non-blocking Promise.allSettled"]

    BS --> RecordKV["Record State in WATCHDOG_STATE\n(Save incident_id, last_notified)"]
    Webhooks --> RecordKV
    Debounce --> RecordKV

5.1 BetterStack Incident API Integration (`sendBetterStackIncident`)

For P1 critical incidents, Watchdog interfaces directly with the BetterStack Uptime / Incident API (https://uptime.betterstack.com/api/v2/incidents):

Authentication: Bearer token via env.BETTERSTACK_TOKEN.

Payload:

{
  "name": "[P1] Green Leaf Bakery Health Probe Breach",
  "summary": "Health check returned HTTP 500 Internal Server Error",
  "requester_email": "watchdog@siteswarm.dev",
  "call": true,
  "sms": true,
  "email": true,
  "push": true
}

Incident Linkage: BetterStack returns a unique incidentId (e.g. bs_inc_12345). Watchdog saves this identifier into KV state (state.incident_id).
Automated Resolution (resolveBetterStackIncident): Upon achieving 2 consecutive healthy runs, Watchdog issues a POST /api/v2/incidents/{incidentId}/resolve request to automatically resolve the active alert on engineer devices.

5.2 Multi-Channel Webhook Pipeline (`dispatchWebhooks`)

All P1 breaches and P2 warnings post rich embeds to team communications channels:

Discord Webhook (sendDiscordWebhook): Posts structured Discord embeds color-coded by severity:
- P1 Firing: Crimson Red (0xed4245)
- P2 Firing: Warning Yellow (0xfee75c)
- Resolved: Emerald Green (0x57f287)
- Fields: App Name, Environment, Severity, Status, Commit SHA, Rule ID, Deep Links to Fleet Status and GitHub repository.
Slack Webhook (sendSlackWebhook): Posts formatted Slack Block Kit messages with contextual header blocks and section fields.

5.3 Non-Blocking Outbound Resiliency

Outbound network requests to third-party APIs (BetterStack, Discord, Slack) are wrapped in defensive try/catch blocks. If BetterStack returns HTTP 503 or Discord rate-limits with HTTP 429:

Watchdog logs a structured warning via console.warn.
The evaluation loop never throws or terminates the scheduled cron cycle.
Local KV state updates proceed unimpeded.

6. Command Center Operator Cockpit Integration

The SiteSwarm Command Center (apps/command-center) provides an operator cockpit page (/observability) delivering real-time fleet health awareness.

flowchart LR
    Operator["Operator / Engineering Browser"] --> Cockpit["Astro SSR Page\n(/observability)"]
    Cockpit --> CacheHeader["Cache-Control:\nprivate, max-age=60, stale-while-revalidate=30"]
    Cockpit --> ResolveKV["Dynamic KV Resolution\n(cloudflare:workers cf.env.WATCHDOG_STATE)"]
    ResolveKV --> FetchFleet["fetchFleetObservability()\nSingle-Call kv.list({ prefix: 'alert:' })"]
    FetchFleet --> RenderGrid["FleetGrid.astro\n10 Client Cards + Platform Metrics"]

6.1 Architectural Implementation

Edge SSR Rendering: Rendered on Cloudflare Pages/Workers with prerender = false.
Sub-60s Caching Strategy: Sets Cache-Control: private, max-age=60, stale-while-revalidate=30, ensuring operators see fresh incident state within 60 seconds while absorbing browser refresh spikes.
Dynamic KV Resolution: Resolves WATCHDOG_STATE via standard Cloudflare Workers bindings (import("cloudflare:workers")).
Deep Linking: Each card links directly to:
- Live Application Health Endpoint (https://<domain>/api/health)
- BetterStack Incident Console (if an active incident_id exists)
- Cloudflare Analytics Engine SQL Dashboard

6.2 Test Mode & Playwright Simulation Hooks

To facilitate deterministic end-to-end testing without mutating production or staging KV:

The cockpit recognizes x-playwright-test: true headers or ?testAlerts= query parameters.
Passing ?testAlerts=bakery:degraded,software-agency:incident generates synthetic in-memory alert states for UI visual verification.

7. Operational Runbooks & Synthetic Fault Drills

SiteSwarm maintains a CLI drill harness (scripts/drill-alerting.ts) to verify end-to-end alerting flows in non-production environments.

7.1 Production Guardrail

export function assertNotProduction(env?: string): void {
  const normalizedEnv = (env || process.env.NODE_ENV || "").trim().toLowerCase();
  if (normalizedEnv === "production" || process.env.NODE_ENV === "production") {
    throw new Error(
      "CRITICAL GUARD: Synthetic fault injection drill is strictly prohibited in production! (Detected: env=production or NODE_ENV=production)"
    );
  }
}

Any attempt to invoke the drill script against production immediately aborts execution.

7.2 Drill Execution Workflows

Drill Command	Target Fault Injected	Verification Points
`npx tsx scripts/drill-alerting.ts --dry-run`	None (Read-Only)	Validates environment credentials, probe endpoint reachability, and WAE dataset connectivity.
`npx tsx scripts/drill-alerting.ts --env=staging --fault=probe`	Client probe degraded (HTTP 500)	1. Injects probe fault under `alert:drill:probe:<app>` 2. Verifies breach detection in $\le 2$ minutes 3. Verifies BetterStack P1 paging 4. Verifies 15m debounce suppression 5. Verifies 2-run recovery auto-resolution
`npx tsx scripts/drill-alerting.ts --env=staging --fault=dlq`	WAE DLQ error spike (`dlq_routed > 0`)	1. Emits synthetic WAE metric event 2. Verifies SQL query breach detection 3. Verifies alert dispatch to Discord/Slack
`npx tsx scripts/drill-alerting.ts --cleanup`	Teardown	Purges all keys matching `alert:drill:*` from KV storage.

7.3 Latency & Detection SLOs

Detection Latency SLO: $\le 3$ minutes (MAX_DETECTION_LATENCY_MS = 180000) from initial fault injection to active alert state in KV.
Probe Timeout SLO: Exactly 2,500 ms ceiling per target.
Cron Cycle Budget: $\le 25$ seconds for all 10 probes and WAE queries combined.

8. Failure Modes & Edge Mitigations

Failure Mode	Blast Radius	Watchdog Runtime Behavior	Mitigation / Recovery
Client Application Hang (Slow DB / Cold Start)	Single client probe affected	Probe triggers `AbortSignal.timeout(2500)`. Status transitions to `firing` (`Health check timed out after 2500ms`).	Other 9 client probes proceed without delay. Overall cron cycle stays well within 25s ceiling.
WAE API Rate Limiting (HTTP 429)	Analytical metric evaluation	`CloudflareAnalyticsClient` catches HTTP 429. Returns firing error result or logs warning.	Synthetic HTTP probes are evaluated independently and remain 100% operational.
Upstream BetterStack API Outage (5xx / Timeout)	On-call paging dispatch	`sendBetterStackIncident` catches HTTP 5xx or fetch exception. Logs structured `console.warn`.	Local incident state is still recorded in KV. Webhook notifications to Discord/Slack proceed unimpeded.
KV Write Contention / Rate Limit	Alert state persistence	KV returns transient error. Caught by `routeAlert` error handler.	Rule re-evaluates on next scheduled cron cycle (2 minutes later).
Global 25-Second Execution Ceiling Overrun	Scheduled cron cycle	`Promise.race()` triggers rejection: `Watchdog evaluation exceeded strict 25000ms ceiling`.	Cycle aborts cleanly before Cloudflare’s 30s CPU limit evicts the worker isolate.

9. Decisions Intentionally Deferred

To adhere strictly to Minimal Viable Abstraction (MVA) and avoid premature platform complexity:

Multi-Tier On-Call Escalation Ladders:
- Rationale: SiteSwarm delegates complex on-call rotation schedules, escalation tiers, and phone overrides directly to BetterStack rather than implementing bespoke scheduling logic inside the Worker.
In-Worker Historical Metric Graphing:
- Rationale: The Watchdog persists only active incident state in KV. Long-term metric storage, graphing, and trend analysis are delegated to Workers Analytics Engine (WAE) and Cloudflare dashboards.
Machine Learning Anomaly Detection:
- Rationale: Static thresholding (> 0 DLQ events, HTTP 500 status codes, sub-2.5s timeouts) provides 100% deterministic, explainable alerting with zero model drift or inference compute costs.
Push-Based Client Telemetry Handshakes:
- Rationale: Client storefronts emit fire-and-forget telemetry via WAE. The Watchdog never accepts incoming push traffic from client frontends, preserving complete architectural decoupling.