Docs Portal Architecture

Documentation Portal Architecture Specification

Standard: RFC-00679 / SiteSwarm Zero-Copy Documentation Portal
Component: @siteswarm/app-docs (apps/docs)
Status: Living Canonical Specification
Deployment Target: docs.jacobmiller22.com (Platform Edge)
Parent Epic: Epic #677

1. Executive Summary & Architectural Philosophy

SiteSwarm maintains an authoritative, living corpus of technical architecture specifications, operational runbooks, and business product documentation. To ensure high developer velocity, seamless operational onboarding, and absolute transparency across human engineers and autonomous AI agents, this documentation must be viewable, searchable, and navigable through a high-performance, edge-rendered developer portal (@siteswarm/app-docs).

Historically, documentation websites in multi-package monorepos have relied on brittle pre-build mirroring scripts (e.g. sync-docs.ts or cp -r docs/* src/content/docs/) or filesystem symlinks. These synchronization hacks introduce catastrophic failure modes:

Git Churn and Worktree Dirtying: Mirrored markdown files frequently pollute Git worktrees, trigger phantom diffs, or get accidentally committed as duplicate artifacts.
Local Dev Watcher Desynchronization: Editing a source specification under docs/ fails to trigger Hot Module Replacement (HMR) in the docs application unless bespoke file watchers (chokidar) are running in parallel.
Broken Relative Links: Copying markdown files into a different directory depth breaks relative cross-specification markdown links.
Duplicate Storage & CI Latency: Pre-build mirroring doubles disk space requirements and adds unnecessary build steps to CI pipelines.

To eliminate these failure modes permanently, SiteSwarm adopts Astro Starlight powered by the Astro 5 Content Collections Layer (loader). By leveraging Astro 5’s decoupled Content Layer, @siteswarm/app-docs ingests Markdown documentation zero-copy and directly in-place across all 4 pillars: docs/architecture/, docs/runbooks/, docs/business/, and docs/write-ups/.

flowchart TD
    subgraph MonorepoRoot["SiteSwarm Monorepo Root (docs/)"]
        Arch["docs/architecture/*.md\n(Canonical Architecture Specs)"]
        Runbooks["docs/runbooks/*.md\n(Operational Runbooks)"]
        Biz["docs/business/*.md\n(Business Requirements & PRD)"]
        WriteUps["docs/write-ups/*.md\n(Historical ADRs, Spikes & Evaluations)"]
    end

    subgraph AppDocs["@siteswarm/app-docs (apps/docs)"]
        Loader["Astro 5 Content Layer Loader\n(zero-copy glob loader)"]
        Store["Astro Local Data Store\n(context.store with Heading-1 Title Extraction)"]
        Starlight["Astro Starlight Integration\n(@astrojs/starlight)"]
        Pagefind["Build-Time Pagefind Search Indexer\n(Client-Side WASM, 0 External APIs)"]
        HealthRoute["/api/health Endpoint\n(@siteswarm/health Server Route)"]
    end

    subgraph EdgeDeployment["Cloudflare Edge Delivery"]
        EdgeWorkers["Cloudflare Workers\n(docs.jacobmiller22.com / docs-staging.jacobmiller22.com)"]
        Cache["Cloudflare Global Tiered Cache\n(Sub-50ms TTFB Worldwide)"]
    end

    Arch -->|"In-Place Ingestion"| Loader
    Runbooks -->|"In-Place Ingestion"| Loader
    Biz -->|"In-Place Ingestion"| Loader
    WriteUps -->|"In-Place Ingestion"| Loader

    Loader --> Store
    Store --> Starlight
    Starlight --> Pagefind
    Pagefind --> EdgeWorkers
    HealthRoute --> EdgeWorkers
    EdgeWorkers --> Cache

2. Zero-Copy Ingestion Architecture (Astro 5 Content Layer)

2.1 The Ingestion Seam

The foundational invariant of @siteswarm/app-docs is: zero file duplication, zero mirror scripts, and zero symlinks.

Astro 5 introduces the Content Layer API, which replaces legacy filesystem-bound collections (src/content/docs/) with configurable loader functions in src/content.config.ts. Rather than requiring Markdown files to exist inside the application directory, the Content Layer allows loaders to query and ingest files directly from arbitrary filesystem directories.

2.2 Content Layer Collection Schema

The documentation portal defines a single unified docs collection that ingests all 4 pillars while preserving relative route taxonomy:

import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';
import { docsSchema } from '@astrojs/starlight/schema';

/**
 * SiteSwarm Zero-Copy Documentation Portal Content Loader
 * Reads Markdown documents directly in-place from the monorepo root docs/
 * with zero pre-build copying or mirroring.
 */
export const collections = {
  docs: defineCollection({
    loader: glob({
      base: '../../docs',
      pattern: '{architecture,runbooks,business,write-ups}/**/*.{md,mdx}',
    }),
    schema: docsSchema({
      extend: z.object({
        pillar: z.enum(['architecture', 'runbooks', 'business', 'write-ups']).optional(),
        source_issue: z.union([z.number(), z.string()]).optional(),
        standard: z.string().optional(),
        status: z.string().optional(),
        date: z.union([z.string(), z.date()]).optional(),
        distilled_into: z.array(z.string()).optional(),
        candidate_targets: z.array(z.string()).optional(),
      }).passthrough(),
    }),
  }),
};

2.3 Astro Configuration

In apps/docs/astro.config.mjs, @astrojs/starlight is mounted with custom sidebar groupings reflecting the living documentation hierarchy, utilizing @astrojs/cloudflare for edge delivery:

import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
import cloudflare from '@astrojs/cloudflare';

export default defineConfig({
  adapter: cloudflare({
    imageService: 'passthrough',
  }),
  integrations: [
    starlight({
      title: 'SiteSwarm Platform Documentation',
      description: 'Canonical living architecture, operational runbooks, and business specifications for SiteSwarm.',
      defaultLocale: 'root',
      locales: {
        root: {
          label: 'English',
          lang: 'en',
        },
      },
      sidebar: [
        {
          label: 'Architecture & Invariants',
          autogenerate: { directory: 'architecture' },
        },
        {
          label: 'Operational Runbooks',
          autogenerate: { directory: 'runbooks' },
        },
        {
          label: 'Business Specifications',
          autogenerate: { directory: 'business' },
        },
        {
          label: 'Write-Ups & Research Spikes',
          autogenerate: { directory: 'write-ups' },
        },
      ],
      social: [
        {
          icon: 'github',
          label: 'GitHub Repository',
          link: 'https://github.com/jptechnologysolutions/siteswarm',
        },
      ],
    }),
  ],
});

3. Ingestion Scope & Write-Up Omission Policy

3.1 The 4-Pillar Repository Documentation Taxonomy

SiteSwarm enforces a strict 4-pillar taxonomy governed by RULES.md and packages/governance/src/cli/linters/docs-taxonomy.ts:

docs/architecture/: Canonical architectural specifications, RPC contracts, edge caching models, and invariants.
docs/runbooks/: Operational procedures, client intake/onboarding guides, offboarding runbooks, and release protocols.
docs/business/: Product requirements (PRD), business boundaries, target personas, and market positioning.
docs/write-ups/: Point-in-time research spikes, architectural decision records (ADRs), exploratory evaluations, and founder alignment notes.

3.2 Omission Policy for `docs/write-ups/`

Point-in-time engineering write-ups (docs/write-ups/) are strictly excluded from portal navigation and search indexes:

Temporal Ephemerality: Write-ups represent exploratory, point-in-time technical evaluations authored during active development tickets. Once distilled into living canonical specifications via /docs distill, write-ups serve solely as immutable historical Git logs.
Search Result Poisoning: Ingesting historical write-ups corrupts search relevance. Operators searching for architectural answers (e.g. cache purging or domain ingress) would encounter obsolete options rejected during spikes rather than the ratified canonical specification.
Governance Alignment: Governed by the One-Way Citation Guard in docs-taxonomy.ts. Canonical specifications are prohibited from linking downward to docs/write-ups/. Consequently, docs/write-ups/ forms a disconnected historical leaf that must remain in Git history only.

3.3 Enforcement Mechanism

The omission of docs/write-ups/ is enforced mechanically:

Loader Whitelist: The Astro Content Layer glob pattern exclusively matches {architecture,runbooks,business}/**/*.{md,mdx}. The write-ups/ directory is never read by the loader.
Pagefind Exemption: Because write-ups/ are never converted to HTML routes, Pagefind never indexes them.
Automated Architectural Linter: pnpm run lint:arch asserts that canonical documentation pillars remain pristine and free from point-in-time references.

4.1 Automated Heading-1 Title Extraction

In SiteSwarm, living documentation is authored as native Markdown where the first top-level heading (# Title) serves as the human-readable document title. Many canonical files do not declare YAML frontmatter title: "...".

Because Astro Starlight generates the page <title>, browser tab label, and top-level <h1> from the frontmatter title field, a content transformation pipeline resolves title extraction automatically:

Title Detection: If frontmatter title is missing, the AST parser extracts the plain text of the first # Heading encountered in the document.
Title Injection: The extracted text is injected into the entry metadata as data.title.
Leading H1 Demotion / Removal: To prevent duplicate <h1> tags on the rendered page (one generated by Starlight layout and one in the body), the leading Markdown H1 is consumed as page metadata rather than re-rendered in the main content container.
Fallback Slug Generation: If neither frontmatter title nor # Heading exists, the filename without extension (e.g. DOCS_PORTAL_ARCHITECTURE -> Docs Portal Architecture) is humanized and utilized as the title.

The documentation portal structure maps directly to the 3 canonical pillars:

docs.jacobmiller22.com/
├── architecture/
│   ├── app-serving-cache/
│   ├── capability-management/
│   ├── cloudflare-for-platforms-and-domain-ingress/
│   ├── content-invalidation-and-cache-freshness/
│   ├── docs-portal-architecture/
│   ├── rules/
│   └── linters/
│       └── ...
├── runbooks/
│   ├── client-app-review/
│   ├── client-intake-and-onboarding/
│   ├── client-intake-checklist/
│   ├── client-offboarding/
│   └── project-management-and-intent-governance/
└── business/
    └── prd/

Clean URL Slugs: Slugs are lowercase and kebab-cased matching standard web routing conventions.
Autogenerated Sidebars: Sections use Starlight’s autogenerate capability scoped to each pillar directory, ensuring that creating a new living specification in docs/architecture/ automatically appears in the sidebar on the next build without manually editing a navigation config file.

5. Strict Internal Linking Invariant across Canonical Specs

5.1 Dual-Resolution Markdown Linking Invariant

Living documentation must remain 100% functional in two distinct contexts:

Raw Git Checkout / GitHub Web UI: Clicking a relative link in GitHub (e.g. [RULES.md](./RULES.md)) must immediately navigate to the target file on disk.
Starlight Documentation Portal: Clicking the same link on docs.jacobmiller22.com must navigate to the edge-rendered HTML route (/architecture/rules/).

5.2 Link Normalization Pipeline

During Astro build, Starlight processes internal relative Markdown links using remark and rehype transforms:

A relative link pointing to ./CONTENT_INVALIDATION_AND_CACHE_FRESHNESS.md is resolved relative to the source document’s directory.
The link target is mapped to the corresponding Starlight collection route slug (/architecture/content-invalidation-and-cache-freshness/).
Fragment hashes (e.g. #2-service-binding-rpc-contract) are preserved and transformed into valid heading anchors.

5.3 Static Verification via Architectural Linter

Broken links are prevented mechanically in CI before any code reaches production. The architectural linter (packages/governance/src/cli/linters/docs-taxonomy.ts) runs lintRelativeMarkdownLinks:

It traverses every Markdown file in README.md, HIGH_LEVEL_DESIGN.md, AGENTS.md, and docs/**/*.{md,mdx}.
It parses all Markdown relative links and verifies that the destination file physically exists on disk.
If any relative link points to a non-existent file or attempts to link into docs/write-ups/, the build and CI gate immediately fail with exit code 1.

6. Build-Time Search Indexing & Cloudflare Edge Deployment

6.1 Pagefind Build-Time Search Indexing

Search in @siteswarm/app-docs is powered by Pagefind, Starlight’s integrated search library:

Build-Time Index Generation: During astro build, Pagefind analyzes static HTML output across the 3 living pillars, building chunked index shards under dist/pagefind/.
Zero Runtime Dependencies: The search index executes 100% client-side via WebAssembly (WASM). There are zero calls to external search SaaS providers (such as Algolia or Elasticsearch).
Hermetic & Offline: Complies with RULES.md invariant of zero API keys and zero network tokens required for documentation build, test, and execution.
Bandwidth Efficiency: Pagefind downloads only the index shards corresponding to the user’s active search query words, delivering sub-30ms search results with minuscule payload transfer.

6.2 Cloudflare Edge Deployment Strategy

@siteswarm/app-docs is deployed to Cloudflare’s edge network under the authoritative platform zone:

sequenceDiagram
    autonumber
    actor Dev as Developer / Operator
    participant Edge as Cloudflare Edge CDN (docs.jacobmiller22.com)
    participant Worker as @siteswarm/app-docs (Cloudflare Worker / Pages)
    participant WASM as Pagefind WebAssembly Index

    Dev->>Edge: GET /architecture/capability-management/
    alt Cached at Edge
        Edge-->>Dev: HTTP 200 (HTML from Global Tiered Cache, <20ms TTFB)
    else Cache Miss
        Edge->>Worker: Route request
        Worker-->>Edge: HTML stream with immutable asset links
        Edge-->>Dev: HTTP 200
    end

    Dev->>WASM: Type search query ("cache purge")
    WASM-->>Dev: Instant client-side search results (<15ms)

Hostname & DNS: Bound to docs.jacobmiller22.com, matching the platform domain architecture defined in platform.config.ts and CLOUDFLARE_FOR_PLATFORMS_AND_DOMAIN_INGRESS.md.
Edge Output: Configured with @astrojs/cloudflare static/prerendered edge distribution.
Cache Control:
- Hashed static assets (/_astro/*, /pagefind/*): Cache-Control: public, max-age=31536000, immutable.
- HTML documents: Cache-Control: public, max-age=0, must-revalidate coupled with Cloudflare CDN caching and deployment-based tag purges (via CONTENT_INVALIDATION_AND_CACHE_FRESHNESS.md).

7. Anti-Patterns & Failure-Mode Guardrails (Definition of NOT Done)

To preserve codebase hygiene, any implementation matching the following patterns will be rejected:

Anti-Pattern	Defect Signal	Remediation Requirement
Pre-Build File Copying	Scripts such as `scripts/sync-docs.ts` or `cp -r docs/* ...` in `package.json`.	Reject. Use Astro 5 Content Collections `glob({ base: '../../docs', ... })` loader exclusively.
Symlink Forests	Generating symlinks inside `apps/docs/src/content/docs/` pointing to root `docs/`.	Reject. Symlinks fail cross-platform checks (Windows) and break parallel Git worktrees.
Ingesting Write-Ups	Including `docs/write-ups/` in portal navigation, search index, or sidebar.	Reject. Enforce strict 3-pillar ingestion whitelist; write-ups stay in Git history only.
Duplicated Heading-1 Rendering	Pages displaying double `<h1>` titles (one from Starlight layout, one from Markdown).	Reject. Automatically demote or extract the top Markdown `# Heading` into Starlight entry metadata.
Broken Relative Links	Relative links between specs returning 404 or remaining unlinked `.md` strings.	Reject. Ensure remark link transformer maps relative markdown targets to valid Starlight web routes.
External Runtime Search SaaS	Integrating Algolia DocSearch or cloud search services requiring external API keys.	Reject. Use Starlight’s hermetic, build-time Pagefind WASM search.

8. Decisions Intentionally Deferred

In strict alignment with the Minimal Viable Abstraction (MVA) invariant (RULES.md), the following capabilities are intentionally deferred from initial portal implementation:

Client / Tenant Authentication & Private Portal Gating:
- Deferred: Protecting routes behind Cloudflare Access / Zero Trust JWT login or role-based access control (RBAC).
- Rationale: SiteSwarm canonical specifications and runbooks represent non-sensitive engineering methodology and operational standards. No proprietary customer credentials or private client data reside within docs/. An edge auth gate will only be evaluated if enterprise multi-tenant proprietary runbooks are introduced.
Multi-Version Documentation & Release Branch Snapshots:
- Deferred: Release version dropdowns (v1.0, v2.0) and branch snapshot archivers.
- Rationale: SiteSwarm operates strictly on trunk-based continuous delivery (main is always the authoritative production truth per DEVELOPMENT.md). Multi-version snapshotting introduces significant build overhead and maintenance without current business justification.
Interactive Sandboxes & Live Component Playgrounds:
- Deferred: Embedding WebContainers, live Astro component previews, or browser-executable sandbox workers into documentation pages.
- Rationale: Living specifications focus on architectural invariants, schemas, and contracts. Interactive playgrounds add runtime JavaScript bloat and edge bundle complexity.
Multi-Language (i18n) Localization:
- Deferred: Localized documentation translation dictionaries and multilingual routing.
- Rationale: Engineering specifications and runbooks are authored exclusively in English. Localization infrastructure will only be introduced if international operator teams require it.
Automated OpenAPI / TypeDoc AST Ingestion:
- Deferred: Parsing monorepo TypeScript declarations and auto-generating API reference documentation pages.
- Rationale: Handwritten canonical specifications with curated code samples and Mermaid state machines provide vastly superior clarity compared to raw generated AST dumps.

Docs Portal Architecture

Documentation Portal Architecture Specification

1. Executive Summary & Architectural Philosophy

2. Zero-Copy Ingestion Architecture (Astro 5 Content Layer)

2.1 The Ingestion Seam

2.2 Content Layer Collection Schema

2.3 Astro Configuration

3. Ingestion Scope & Write-Up Omission Policy

3.1 The 4-Pillar Repository Documentation Taxonomy

3.2 Omission Policy for `docs/write-ups/`

3.3 Enforcement Mechanism

4. Automated Heading-1 Title Extraction & Navigation Hierarchy

4.1 Automated Heading-1 Title Extraction

4.2 Navigation Hierarchy

5. Strict Internal Linking Invariant across Canonical Specs

5.1 Dual-Resolution Markdown Linking Invariant

5.2 Link Normalization Pipeline

5.3 Static Verification via Architectural Linter

6. Build-Time Search Indexing & Cloudflare Edge Deployment

6.1 Pagefind Build-Time Search Indexing

6.2 Cloudflare Edge Deployment Strategy

7. Anti-Patterns & Failure-Mode Guardrails (Definition of NOT Done)

8. Decisions Intentionally Deferred

Docs Portal Architecture

Documentation Portal Architecture Specification

1. Executive Summary & Architectural Philosophy

2. Zero-Copy Ingestion Architecture (Astro 5 Content Layer)

2.1 The Ingestion Seam

2.2 Content Layer Collection Schema

2.3 Astro Configuration

3. Ingestion Scope & Write-Up Omission Policy

3.1 The 4-Pillar Repository Documentation Taxonomy

3.2 Omission Policy for docs/write-ups/

3.3 Enforcement Mechanism

4. Automated Heading-1 Title Extraction & Navigation Hierarchy

4.1 Automated Heading-1 Title Extraction

4.2 Navigation Hierarchy

5. Strict Internal Linking Invariant across Canonical Specs

5.1 Dual-Resolution Markdown Linking Invariant

5.2 Link Normalization Pipeline

5.3 Static Verification via Architectural Linter

6. Build-Time Search Indexing & Cloudflare Edge Deployment

6.1 Pagefind Build-Time Search Indexing

6.2 Cloudflare Edge Deployment Strategy

7. Anti-Patterns & Failure-Mode Guardrails (Definition of NOT Done)

8. Decisions Intentionally Deferred

3.2 Omission Policy for `docs/write-ups/`