Voice Operating Interface for the Workstation: an STT/TTS Loop with Access to Projects and Agents

Context

The workstation of an engineer or manager who runs a dozen parallel projects and commands their own team of LLM agents is fragmented by nature: context is scattered across terminals, folders, tabs, and dashboards. Every question to the system — "how is the build doing", "which agents are active", "what do we even have in inventory" — requires switching windows, finding the right tool, and typing by hand. Individually these are seconds; over a working day they accumulate into persistent friction that lowers the operator's throughput.

Mass-market voice assistants do not solve this. They are built around the cloud and consumer scenarios: weather, timers, music. They do not know which projects live on a particular machine, have no access to it, and ship audio to external servers — which, for a working environment holding private codebases and client data, is a risk in its own right. A workstation voice interface has to be local in its data and aware of its context, and products like that barely exist on the market.

There is also a third, engineering layer of difficulty. A voice loop is a chain of recognition, dialogue core, and synthesis, where every link adds latency — and cumulative latency determines whether a real person will keep using the tool. Separately, there is the boundary between conversation and action: an assistant that can only talk is useless the moment machine access is needed, while an assistant that executes commands demands an explicit and reliable request-routing mechanism. Finally, packaging into a native desktop application exposes low-level platform constraints — such as unreliable microphone behavior in WKWebView. What makes this case interesting is that all three layers had to be closed within one system.

The Task

The original brief: remove the keyboard from part of the operator's daily cycle. One window on the desktop, a living assistant face you can address by voice, and an answer — also by voice. Not a chat box in a browser but a presence: a counterpart that knows the project list and the agent roster and talks about them out loud, with no tool-switching and no manual typing.

Voice interfaces carry a well-known trap. A conversational assistant that only chats is useless precisely when it is needed most: "check what's wrong with the disk", "show the processes", "build the project". So the second half of the task was to make the assistant not just answer but act on the real machine: read files, run commands, check system state.

The third requirement: the assistant must understand on its own when it is time to move from words to action. Forcing the user to switch modes by hand would reintroduce the very friction the system is built to eliminate. Routing between the conversational and agent loops had to be automatic and predictable.

Approach

The system is assembled as a local desktop web application. The start.sh script brings up the server; the application opens at http://127.0.0.1:8800. Everything runs on localhost: the microphone is available through the standard browser permission prompt, and neither audio nor project context ever leaves the machine. This is an architectural decision, not a convenience — the privacy of the working environment is guaranteed by topology, not policy.

The backend is deliberately built on the Python standard library — http.server in a single server.py, with no external web framework. For a local single-user tool this is the right boundary of responsibility: fewer dependencies, instant startup, full control over every endpoint. The engineering weight goes not into a framework but where it creates value — into the voice processing chain and the agent layer.

The voice pipeline has three links. Speech to text — via Groq Whisper. The answer is formed by an Anthropic model whose system prompt carries the user's project list and the agent roster from agents.json — the assistant talks about the real estate of the machine, not in the abstract. Speech synthesis — via edge-tts. The client receives an mp3 packed in base64 and plays it in a single request-response cycle, with no intermediate states.

The frontend is HTML embedded in the server: a dark cinematic console with the assistant's portrait. The portrait reacts to sound — a Web Audio analyser reads the level and pulses a CSS glow variable in time with the voice. The assistant visibly breathes with its own speech; this is not decoration but feedback that tells the user the system hears and answers, without a glance at the logs.

Architecture

The server surface is a small set of endpoints with clear roles. The / page is the console itself. /face.jpg is the portrait. /state returns agents and projects for the interface. POST /converse accepts audio and runs it through the full pipeline — recognition, dialogue core, synthesis — returning mp3 in base64. POST /ask is text input for moments when speaking is inconvenient. Each pipeline link is independently replaceable: STT, core, and TTS are coupled only by data format, so swapping the provider of any one of them does not touch the others.

The key architectural fork is the two execution loops. Chat is fast conversation directly through the API, for questions and explanations. Do is a full agent via claude -p that actually performs tasks on the machine: inspects the system, reads files, runs commands. The difference is not cosmetic: chat can only talk, do can act. The split is also a security boundary: the conversational mode physically has no tools, so an accidental or misheard request cannot change anything on the machine.

To spare the user from choosing a mode by hand, automatic routing sits between the loops. The needs_tools() function in server.py uses a regular expression to recognize questions about the system, files, and actions and directs them to agent mode, leaving everything else in fast chat. This resolves the voice assistant's central pain: previously, in conversational mode, it could not see the computer and could do nothing; now a request finds its own loop. The deterministic classifier is a conscious choice — it is transparent, debuggable by reading a single line, and adds neither latency nor the cost of an LLM call to every utterance.

A separate engineering track is the 3D face. It is implemented in WebGL via Three.js r128: a 220×220 PlaneGeometry with a displacement map derived from the depth.jpg depth map, textured through emissiveMap to preserve the noir look without overexposure. A perspective camera provides cursor-driven parallax and automatic sway, and the audio level is mixed into the displacement — the face comes alive in sync with speech. It is important to name the tier honestly: this is 2.5D depth parallax from a single portrait and a depth map — genuine WebGL depth, but not a full 3D rig with lip sync and facial animation. That terminological discipline is part of the project's engineering culture: the system's claimed level matches its actual one.

Native Application

A browser tab is not the presence a desktop assistant requires: it gets lost among other tabs and lives by the browser's rules. So the system is packaged as a native Mac application on pywebview: app.py brings up the same server and opens a native WKWebView window instead of a browser. Double-click the icon — and the assistant is on screen like any ordinary program.

The native wrapper brought its own engineering problem. In WKWebView, acquiring the microphone via getUserMedia is unreliable — and for a voice assistant this is critical: without stable input the whole loop is dead. The constraint was bypassed by moving recording to the server: the /listen/start and /listen/stop endpoints capture audio via ffmpeg with avfoundation directly from input :0. Recording stopped depending on the whims of the web layer and works identically in the native window and in the browser — one codebase, two ways of delivering the interface.

So that the system requests microphone access correctly, NSMicrophoneUsageDescription is declared in Info.plist — on first use the user sees the standard system prompt, just like any native application.

Outcome

The voice operating interface runs locally in production: one window, a living face, spoken dialogue, and access to all projects and the agent team. The user speaks, the assistant answers by voice, and — when the question concerns the machine — switches itself into agent mode and performs the task rather than deflecting. The keyboard has been removed from a significant portion of the daily operating cycle.

Three things that projects like this usually defer "for later" are built and working here: a native Mac application instead of a tab, server-side microphone recording that bypasses WKWebView limitations, and automatic routing between conversation and action. It is exactly this trio that moves the project from the voice-chat-demo category into the category of a working desktop tool used every day.

Some capabilities are deliberately left for the next tier, and the system does not claim more than it delivers. The face is high-quality depth parallax, not facial animation; there are two modes, and they are honestly separated; the routing classifier is deterministic, not probabilistic. This framing allows the system to grow one node at a time — wake-word activation, lip sync, multi-agent orchestration — without rewriting the core.

For the studio, this case is a proven pattern for a voice operating loop over an arbitrary working environment: local in its data, aware of its context, with an explicit boundary between conversation and action. The same skeleton — STT, a dialogue core with working-context injection, TTS, an agent executor, and a deterministic router — transfers to other environments where the operator's hands are busy and the system must answer and act on voice.

What we built

• Local server on stdlib

  A single server.py on http.server with no external web framework: page, portrait, state, voice and text endpoints. Launched via start.sh, port 8800.

• Voice pipeline

  Speech to text via Groq Whisper, answers via an Anthropic model, synthesis via edge-tts. POST /converse returns a ready mp3 in base64.

• Working-context injection

  The dialogue core's system prompt carries the user's project list and the agent roster from agents.json — the assistant answers about the real inventory, not in the abstract.

• Two loops and auto-routing

  Chat — fast conversation through the API; do — a full agent via claude -p. The needs_tools() function uses a regular expression to route system requests into the agent loop on its own.

• Audio-reactive console

  Dark embedded HTML with a portrait that pulses its glow with the voice via a Web Audio analyser and a CSS variable.

• 3D face on WebGL

  Three.js r128, a 220×220 PlaneGeometry with a displacement map from depth.jpg and an emissiveMap texture, cursor parallax and auto-sway — 2.5D depth from a single portrait.

• Native Mac application

  A pywebview (WKWebView) wrapper: app.py brings up the server and opens a native window. Info.plist with NSMicrophoneUsageDescription for the system microphone prompt.

• Server-side microphone recording

  The /listen/start and /listen/stop endpoints capture audio via ffmpeg avfoundation from input :0 — independent of getUserMedia in WKWebView.

Engineering challenges