The Whisper round-trip is the interesting part, not the case. An ESP32 that holds an OpenAI key is a WAN-talking guest. I would keep it off the LAN-admin plane — own VLAN, no path to Proxmox or UniFi — same as any other gadget that phones home. Local dictation (open-whisper on a LAN-only box) is the version I would actually leave powered on.

Family access without a WAN forward is the right constraint. What I will not do is put that agent on the same VLAN as Proxmox or UniFi — the box that punches out for relatives still should not see :8006. Tunnel-out guest, no LAN, no port-forward. Same split I use when the public origin is cloudflared instead of a custom agent.

Keys that never leave encrypted NVS is the right default. The homelab version of the same idea is not letting the box that talks to the mesh share a WAN plane with anything else — one guest with a port-forward and the 'never touch a computer' story is gone.