Remote Control of an Engineering Agent over a Secured Telegram Channel

Context

Autonomous development agents — Claude Code and comparable systems — have become a working tool for engineering teams over the past two years. An agent running on the working machine has what no cloud chat has: full access to the file system, the terminal, a set of specialized skills, and the accumulated context of the projects. That locality is what makes it productive — and it is also what chains it to the desk. To submit a task, the engineer has to be physically at the computer, and that constraint discounts a large part of the agent's potential: ideas, urgent fixes, and checks do not arrive on the workstation's schedule.

Remote access to the agent looks trivial right up until you design it seriously. This is remote control of a privileged execution environment: the agent has access to the shell, files, keys, and working databases. Classic enterprise practice closes such perimeters with bastion hosts, VPNs, and audit systems — infrastructure whose cost is out of all proportion to a personal tool. Opening a port to the outside or standing up a public webhook server, meanwhile, would create a permanent attack surface on the machine that holds the entire working environment.

The third dimension of difficulty is autonomy itself. An agent that executes tasks with no operator at the screen can run away into uncontrolled execution: hang, reach for a production database, burn through the budget. A remote channel amplifies the risk — the operator cannot see the screen and cannot intervene immediately. So the engineering brief read: give the agent maximum autonomy, but build independent control mechanisms around it that fire even when the agent itself goes silent.

The Task

The goal was to unbind the agent from the workstation: state a task from a phone, from anywhere, and receive the result through the same channel the engineer already uses daily — Telegram. The channel had to accept both text and voice, and return not only answers but work artifacts — finished files.

The key requirement: the executor stays the local agent itself — not a separate stripped-down bot with its own logic, and not a cloud copy without access to the working environment. In this architecture, Telegram is transport only: convenient mobile input and a delivery channel for results. All the work happens on the machine where the files, skills, and tools live.

The second requirement was zero infrastructure. No public servers, domains, tunnels, or outward-facing ports: the tool runs on a single machine, and standing up hosting around it would be disproportionate to the task and unjustifiable in attack-surface terms. The third was controlled autonomy: the agent works without per-step confirmations, but the system must stop runaway processes on its own and prevent context leaking between tasks.

Approach

We deliberately chose a minimalist architecture over frameworks. The bridge core, bridge.py, is written on the Python standard library alone — not a single third-party dependency at the transport level. The decision is driven not by asceticism but by operational requirements: the fewer the dependencies, the smaller the surface of failures and updates in a tool that has to run unattended for years.

The infrastructure question is closed by the choice of polling model. The bot reaches out to Telegram itself via the getUpdates method (long-polling) from inside the machine: no inbound connection is required, no public server is needed, and an open port is absent as a class. The traffic direction is inverted — the bot goes to Telegram, not Telegram to the bot — which eliminates a whole layer of security questions at the schema level.

Channel access is restricted to an allowlist of Telegram IDs. If the list is empty, the bot enters introduction mode: it tells whoever contacts it their own ID and executes no tasks. The configuration file holding the bot token and API key is stored with chmod 600 permissions. Agent authorization runs through the ANTHROPIC_API_KEY environment variable; the alternative path is subscription sign-in via /login, which needs no key in the environment.

Development proceeded in iterations validated in real operation. The first version delivered the basic text channel; the second, a non-blocking task queue and live progress; the third, voice input and output and file delivery. Every operational incident was converted into a concrete architectural decision, not a line in a policy document.

Architecture

The transport loop works like this: an incoming message is picked up by the long-polling cycle and passed into a claude -p invocation with JSON response format, a resume session, and a set permission mode. The CLI returns structured JSON with type, is_error, result, session_id, and total_cost_usd fields — from these the bridge deterministically reads the task's outcome, maintains session continuity, and accounts the cost of every call. The agent's answer goes back to the chat in chunks, fitting within Telegram's message-length limits.

The execution loop is built on a queue and a dedicated worker thread. In the first version, receiving a message blocked the channel for the entire duration of the task; after the rework, intake does not block — a new message gets an immediate queued-confirmation, and execution proceeds asynchronously. The agent runs in stream-json mode, and the bridge relays tool calls to the chat as work progresses: terminal access, file reads, searches, subagent launches. At the end comes a summary with execution time and call cost. The operator sees not silent waiting but a telemetry stream of what the agent is doing right now. The execution mode is bypassPermissions: the agent is fully autonomous, including shell access, with control moved into a separate layer described below.

The voice loop is bidirectional. An incoming voice message goes to Groq Whisper API for recognition, model whisper-large-v3-turbo, via a multipart request; there was a non-trivial detail here — without an explicit User-Agent header, Cloudflare responds with error 1010, and the header had to be pinned in the client. Answers are synthesized via edge-tts (Microsoft neural voices) with opus conversion through ffmpeg; the default voice is the multilingual Vivienne with natural Russian pronunciation, with a fallback to the macOS system synthesizer say in case of unavailability.

The artifact-delivery loop closes the cycle: the bridge parses the agent's answer, finds absolute paths to created files in it, and sends them to the chat as documents via sendDocument (multipart, up to 50 MB). The scenario is verified on an xlsx export — the agent assembles a spreadsheet on the machine, and it arrives on the phone as a finished file. The operations layer is closed with operating-system facilities: a macOS LaunchAgent with RunAtLoad and KeepAlive flags brings the bridge up at boot and restarts it automatically after crashes, logs are written unbuffered (python3 -u), the process is stopped by targeting the bridge.py path specifically, and the condition for continuous operation is that the machine must not sleep (caffeinate).

The Incident and the Safety Loop

An autonomous agent with shell access is a source of productivity and a source of risk at once — and the risk materialized. One task in full-autonomy mode ran away into uncontrolled execution: the process hung for about seven hours (25491 seconds by the log), accessed a production database, and held resources open. The postmortem revealed an architectural defect in time control: the deadline check sat inside the output-reading loop, so a silent hung process formally never violated the timeout — there was simply nothing to kill it.

The control mechanism was redesigned at the root. Time is now watched by an independent watchdog on threading.Timer that terminates the process unconditionally — regardless of whether output is flowing. The task timeout was tightened from 1800 to 600 seconds. Automatic session reset on timeout or error was added: the context of a failed task does not leak into subsequent, unrelated messages — at once data hygiene and protection against cascading failures.

On top of process control, a system SAFETY_PROMPT is attached via append-system-prompt: a ban on operations against production databases, a ban on exposing secrets, a stop after 3–4 steps, and a confirmation requirement before any irreversible actions. The protection is deliberately designed as a soft loop — a prompt plus an independent process watchdog, not a hard permissions lock: the system's owner chose full autonomy with a safety net, and the architecture honestly reflects that choice instead of cutting back the agent's capabilities.

Outcome

The channel is in production, running in autonomous agent mode under the soft safety loop. A task is submitted from a phone by text or voice, executed by the local agent with all the files, skills, and tools of the working machine, and the result — including finished documents — returns to the same chat together with execution telemetry, run time, and call cost. The workstation is no longer a tether point: the engineering environment is reachable from anywhere with a messenger.

The operating economics are transparent and controlled. The approximate cost of a single invocation is around $0.04 including the load of the working directory context; every task carries a hard $5 budget cap that cuts off anomalously expensive runs, and the cost of each call is accounted from the CLI's own total_cost_usd field. Beyond API usage the solution carries no infrastructure costs at all: no hosting, no domain, no server maintenance.

Operations also yielded a systemic lesson, now codified: an agent controlled through the channel must never restart itself via launchctl bootout — that command kills the bridge along with the running task. Restarts are performed only from outside. It is a precise illustration of the principle the whole project is built on: a system that gives an agent access to the machine must protect itself from its own agent as well. The patterns earned here — long-polling with no inbound connections, an independent process watchdog, execution telemetry, per-call budget caps — are reused by the studio as the base skeleton for client deployments of autonomous agents.

What we built

• Transport core on stdlib

  bridge.py on the Python standard library alone: long-polling getUpdates → claude -p with JSON output, a resume session, and permission mode → answers in chunks to the chat. No public server, domain, or open port.

• Queue and worker thread

  Message intake never blocks: the task goes into a queue, a dedicated worker executes it, and a queued-confirmation is sent to the chat immediately.

• Execution telemetry

  stream-json mode forwards tool calls as work progresses — terminal, file reads, searches, subagents; the final summary shows run time and call cost.

• Voice input

  Voice-message recognition via Groq Whisper API (whisper-large-v3-turbo, multipart, mandatory User-Agent header against Cloudflare 1010).

• Voice replies

  Synthesis via edge-tts (Microsoft neural voices) with opus conversion through ffmpeg; the default voice is the multilingual Vivienne, with macOS system say as fallback.

• Artifact delivery

  The bridge finds absolute paths to created files in the agent's answer and sends documents to the chat via sendDocument (multipart, up to 50 MB); verified on an xlsx export.

• Safety loop

  Telegram ID allowlist, configuration with chmod 600 permissions, an independent watchdog on threading.Timer, SAFETY_PROMPT via append-system-prompt, automatic session reset on failure, a $5 per-task budget cap.

• Autostart and resilience

  A macOS LaunchAgent with RunAtLoad and KeepAlive brings the bridge up at boot and survives crashes; logs are unbuffered (python3 -u), shutdown targets the process by path.

Engineering challenges