WhiteGlove AV — Operations and Reliability

This is the reliability picture for WhiteGlove AV: how the system stays up, how it recovers, and what keeps a public demo from ever going dark. It is the public companion to the internal operations runbook, focused on the design choices that make the room dependable rather than the commands that operate it.

Nothing depends on one machine

The room runs on small, single-purpose pieces, each deployed to infrastructure that is built to stay up on its own. The public site and the run feed run on Cloudflare's edge, the always-on event reaction runs as a managed container service, and the decision logic lives in versioned swamp workflows rather than in any one app. Because the decision is a workflow and not app code, the same policy runs whether it fires from a person at the door or a calendar event on the wire, and no single laptop or workstation is in the path of a production run.

The monitor always renders

The public monitor reads its run history directly from a persistent feed store at the edge, so the activity it shows survives restarts and reconnections without any operator intervention. If that feed is ever empty, the monitor falls back to a committed snapshot of real runs within a single poll, so the public URL renders a meaningful view every time it loads. The room's history is durable, the dashboard is stateless, and the failure mode is a slightly older view rather than a blank screen.

Recognition can degrade without stopping the room

Recognition is the one probabilistic input, and the system is built so that input can degrade gracefully. Whether an identity comes from a face, a badge, or a QR scan, it resolves to the same kind of identifier, so if one recognition path is unavailable the others carry the room without changing anything downstream. Recognition is deliberately confined to one seam that the rest of the system does not wait on, so the room keeps running end to end no matter which path supplied the identity.

Control failures never corrupt the record

When swamp enacts a scene on the room's Cisco endpoint, the control step is set so that a missing macro or an offline codec cannot invalidate the decision or its audit. The room records what it decided and what it attempted, separately, so an AV outage shows up honestly in the record as an attempted action rather than silently rewriting what the system chose to do. The decision and the audit are the load-bearing parts, and they hold even when a device does not.

Secrets stay in the vault

Every secret the system needs resolves from a swamp vault at deploy time and is pushed into a managed secret on the host that needs it. Nothing is hardcoded, nothing is committed, and nothing secret ever lands on a command line or a configuration file in the repository. Rotation starts in the vault and propagates to the consumer, so the source of truth for any credential is always the vault and never a hand-edited copy on a host.