WhiteGlove AV — Architecture
The thesis
A conference room should know who walked in and what they came to do, and it should set itself up accordingly without anyone touching a panel. WhiteGlove AV delivers that: the room recognizes who is present, reverse-looks-up that person's Microsoft Entra identity, reads their department and job title, and auto-loads the AV scene that fits the work in front of them. The shift we are making is from command-based control, where a human tells the room what to do one button at a time, to context-aware assistance, where the room reads its own context and acts.
The line that holds the whole design together is this: we let AI see, and we let the automation platform decide and act. Recognition is the only probabilistic input in the system, and we keep it confined to a single seam at the very front. Everything downstream of that seam is a reproducible, fully-audited automated run, so the room's behavior is decided by the platform rather than guessed by a model, and every decision the room makes carries a cryptographic audit trail you can replay after the fact. That is the part that turns a clever demo into something you would actually put in a boardroom, because the question an enterprise asks about an AI-driven room is never "is it impressive" but "can you prove what it did and why," and the platform is the answer to that question.
The platform underneath is swamp. swamp runs the room's logic as declarative, step-by-step workflow definitions, executes them deterministically, and records every step so each decision can be replayed and audited after the fact. The recognition layer sees, swamp correlates and decides, and the AV control layer acts, and because the decision lives in a declarative swamp workflow rather than in app code, the logic is the same whether it fires from a face at the door or a calendar event on the wire. swamp is the orchestration and audit spine.
The data flow
Two paths converge on the same deterministic core. The interactive path starts with a person; the always-on path starts with a calendar event. Both end in an audited swamp run that decides a scene and enacts it on the room's Cisco endpoint.
INTERACTIVE PATH (someone walks in / badges in / is recognized)
recognition correlation decision control audit
─────────── ─────────── ──────── ─────── ─────
face bridge ──objectId──▶ correlation ──dept/ decide ──scene▶ cisco-macros ──▶ swamp run
(room-display/face) workflow jobTitle scene .activate record
badge / QR ──objectId──▶ │ Entra reverse ─────▶ (deterministic (RoomOS macro (Graph read +
│ lookup via dept/role→ on the codec) decision +
│ Microsoft Graph scene table) rationale +
▼ users read control attempt)
Microsoft Entra
Stage-safe twin: a paired correlation workflow runs the identical contract
against a sandboxed synthetic persona directory (no Graph call, no production
tenant, no real identity data ever on screen).
ALWAYS-ON EVENT PATH (a meeting is booked / changed / cancelled on a room)
Microsoft Graph Cloudflare Worker swamp serve swamp workflow
─────────────── ───────────────── ────────────── ──────────────
calendar change ──POST──▶ change-notification ──▶ /hooks/room-invite ──▶ room-reaction workflow
notification on front door (always-on serve 1. get current meeting (Graph)
/users/{room}/events • echo ?validationToken on a managed 2. decide scene
(created/updated/ • validate clientState container service) 3. activate scene (cisco macro)
deleted) • return 202 immediately 4. audit = the swamp run record
• HMAC-sign, forward
X-Hub-Signature-256
The subscription renews itself on a schedule; Graph caps /events
subscriptions at a few days.
The recognition seam and the calendar seam are interchangeable triggers into the same decision policy. The interactive path is what you drive on stage; the always-on path is what runs unattended in production, where no one is standing at a laptop and the room reacts the instant Graph asserts a change.
Component map
Each component is a small, single-purpose piece. The front doors are stateless and run no decision logic; all of the decision, control, and audit logic lives in the swamp workflows, which are the enforceable source of truth. A change-notification front door receives Microsoft Graph calendar events, validates and acknowledges them, and forwards them to an always-on event service that runs the deterministic decision on every event. A room display drives the interactive recognition path. A persistent feed store keeps the run history at the edge so it survives restarts and reconnections. A conversational Teams intervention bot lets a human step in and confirm before any AV change is dispatched. And the public monitor surfaces every run, the attribute that drove each scene, the swamp call chain, and the audit lines on a dark ops console for the big screen. None of these pieces holds the policy; they feed it.
The deterministic core: workflows
The decision logic that makes the room behave lives in a set of catalog workflows, not in any of the apps above. This is deliberate: a swamp workflow is a declarative, versioned, fully-audited DAG, so the same policy that runs on stage is the policy that runs in production, and every run leaves a replayable record.
The interactive correlation workflow is the correlation and control-policy step. Given an asserted identity from the recognition sensor (an Entra object id from a badge, QR code, or face), it looks the user up in Microsoft Entra via the Microsoft Graph users read path, then deterministically maps their department (falling back to job title) to the AV scene that should load. The recognition sensor is the only probabilistic input; the Graph read and the scene decision both land in the run's data lineage.
A stage-safe twin of that workflow honors the identical contract, objectId to identity to scene decision to audit lineage, but it reads from a sandboxed synthetic persona directory embedded in the workflow, with no Graph call and no production tenant, so no real identity data ever appears on screen. Everything downstream, the app, the scene mapping, the audit record, is byte-for-byte the same shape as production, which is what lets us demo the real decision path without putting tenant data on a conference screen. An object id that is not in the directory resolves to an unrecognized visitor and a Welcome/Standby scene, which is itself a real demo beat.
The room-reaction workflow is the always-on event reaction. Fired by a Graph calendar change-notification routed through the Cloudflare Worker into the serve, it fetches the room's current meeting from Microsoft Graph, applies the same deterministic decision policy plus a scene-to-RoomOS-macro mapping, and enacts the scene on the room's Cisco endpoint through the macro control surface, set so a missing macro or an offline codec never invalidates the decision and its audit. The only non-deterministic input is the calendar event itself.
A room-context workflow ties a room resource to its context and decides the AV action with an explicit rationale, cross-referencing the asserted occupant identity against the room's calendar, Neat Pulse sensors, Utelogy room state, and an admin knowledge set, so every action is explainable and auditable. Identity resolves against the synthetic directory for stage safety, and the calendar and knowledge inputs drop in as model-method steps without changing the contract.
A subscription-lifecycle workflow manages the Graph change-notification subscriptions that feed the room-reaction path, with list, create, renew, and delete actions, authing with a Microsoft Entra app's app-only token resolved at runtime from a swamp vault. It renews on a schedule because Graph validates the notification URL synchronously at create time, so the front door must be deployed and reachable first.
The extension layer
The workflows wire together a set of swamp model extensions, each owning one integration domain so the workflows stay declarative and the vendor specifics stay inside a model. A broad Microsoft Graph surface (calendar, places, users, groups, mail, teams, presence) is the read path for the Entra correlation and the room calendar. A Neat extension reads Neat Pulse room and occupancy sensors. An Azure Cognitive Services Face capability handles face detection. A Cisco RoomOS macro control surface is what actually moves the room. A Utelogy extension supplies live AV device and room state. And an adopted community vault-filesystem extension backs the admin knowledge set the room-context workflow reasons over.
Recognition
The recognition seam is real and the architecture is built so it stays a single, swappable input. A browser face bridge in the room display maps a face to an objectId, and badge and QR input map to the same objectId, so the correlation path runs end to end regardless of the source. Azure Cognitive Services Face detection is provisioned and verified. Because recognition is confined to that single front seam, an objectId from any source, browser model, badge, or QR, flows into the exact same deterministic correlation, so a stronger recognition method is an upgrade to one input rather than a change the rest of the room has to absorb.
Control
Scene to action runs through the Cisco RoomOS macro control surface, which fires a macro on the room codec. Control holds to the wired Cisco macros plus the Neat, Utelogy, and calendar signals. The scene-to-macro mapping (presentation, collaboration, videocall, briefing, welcome) and the requirement to deploy those macros to the room codecs before the scene activates are part of the event-service design, and the system is operated through the operations runbook.