Mods
Standalone MCP servers ("Mods") that expose host capabilities to the remote backend agent. Each Mod is an independent process, decoupled from the YUI app — the agent attaches them as tool sources alongside the Expression Broker (deprecated, removed in v0.6.0; the client declares its expression tool on the request). They live under Mods/ in the repository root. Mods are optional: YUI runs without any of them.
Convention
Each mod is a self-contained uv project in Mods/<mod>/ with its own pyproject.toml, uv.lock, and dependency set — no shared lock, so one mod's deps never leak into another (e.g. the shell-sandbox image carries no pyobjc). Containerized mods carry a Dockerfile whose build context is that same folder; Mods/docker-compose.yml builds and deploys them together (docker compose up -d --build). Run a mod's tests with cd Mods/<mod> && uv run pytest. Python lint is ruff (format + check, line-length = 110), enforced in CI per mod — run it locally with cd Mods/<mod> && uv run ruff format . && uv run ruff check ..
Catalog
| Mod | Port | Runs | What |
|---|---|---|---|
| router | 8080 | host-native | One HTTP front door — path-routes /<mod>/mcp to every mod over a single SSH tunnel |
| desktop-control | 9000 | host-native | See the screen, read the day's activity log, and open/close apps on the macOS host |
| shell-sandbox | 9001 | container | Unrestricted shell over a bind-mounted host directory |
| avatar | 9002 | host-native | Query the avatar's own body state and move it with semantic verbs |
| factorio | 9003 | host-native | Poll the Factorio game mod over RCON, cache game events, and post urgent wake signals |
Not a mod, but lives here: browser-cdp exposes no MCP tools — it bridges the remote agent's own Playwright MCP to your local Mac browser over CDP.
Exposure
Every mod binds 127.0.0.1 only — there is no transport auth, so any local process that can reach the port gets the mod's full capability. Reach a mod from the remote agent over an SSH reverse tunnel, never by binding 0.0.0.0. Over HTTP, every mod and the router refuse a request whose Host is not a loopback name (127.0.0.1, localhost, ::1) with 421 and a request with a non-loopback Origin with 403, while a local process that sends no browser headers can still use them. The router collapses all mods onto one tunnel (8080), so you forward one port instead of one per mod:
ssh -R 8080:localhost:8080 <remote-host> # all mods, via the router
ssh -R 9000:localhost:9000 <remote-host> # one mod directlyThe agent then adds the MCP tool source — http://localhost:8080/<prefix>/mcp through the router (the prefixes are listed under router), or http://localhost:<port>/mcp directly (from the remote's view).
router
One HTTP front door that path-routes to every mod, so a single SSH reverse tunnel exposes them all:
http://host:8080/<prefix>/mcp -> 127.0.0.1:<mod port>/mcpIt runs host-native as a thin proxy — MCP's Streamable HTTP transport is SSE, so responses stream through unbuffered. The routing table is the UPSTREAMS dict in router/router/server.py; its keys are the route prefixes: desktop (desktop-control), shell (shell-sandbox), avatar, and factorio. An unregistered prefix returns 404; a registered-but-unreachable mod returns 502. A request whose Host is not a loopback name returns 421 and one with a non-loopback Origin returns 403, both before any mod is contacted. GET /_mods lists the registered mods so a client can discover them without reading the code.
desktop-control
Lets the agent see the screen, read the day's activity log, and open/close apps on the macOS host. Runs host-native — it needs the host GUI, which a container on macOS cannot reach.
| Tool | Description |
|---|---|
screenshot | Capture every display as PNG (one image per monitor), long edge downscaled to 1280px |
list_running_apps | Names of visible (non-background) apps |
get_frontmost_window | { app, title } for the frontmost app; title is null when it has no front window |
get_activity_timeline(date) | One day of the witness log as ordered app / idle segments; an absent day file is an empty timeline |
open_app(name) | Launch + focus an allowlisted app |
close_app(name) | Gracefully quit an allowlisted app |
The DESKTOP_CONTROL_ALLOWED_APPS allowlist is the safety boundary — only listed apps can be opened or closed, and an empty allowlist rejects everything. The read tools are ungated. macOS TCC grants gate the tools, attributed to the launching process:
| Permission | Tools |
|---|---|
| Screen Recording | screenshot; the title half of get_frontmost_window |
| Automation → System Events | list_running_apps |
| Automation → each target app | close_app — its Apple Event goes to the named app, so every allowlisted app is its own grant |
| none | open_app; get_activity_timeline; the app half of get_frontmost_window |
A startup preflight logs an explicit [setup] … NOT granted warning for Screen Recording and for System Events. It cannot cover close_app: probing an app's Automation grant prompts the user, so the per-app grants surface on first quit instead.
get_activity_timeline reads the witness log from WITNESS_LOG_DIR, defaulting to the app's own witness/ directory, and merges its transitions into { date, segments } — one segment per app stretch ({start, end, type: "app", app, window_title, duration_min}) and one per idle stretch ({start, end, type: "idle", duration_min}). An app change recorded while the machine is idle is background churn and does not end the idle stretch; a missing, unreadable, or retention-expired day file reads as an empty timeline rather than an error. Merging is the only processing; the summarizing belongs to the agent reading it. Those segments carry a day of window titles, so exposing this mod exposes the titles too.
shell-sandbox
An unrestricted shell exposed to the agent, running inside a container against a bind-mounted host directory. There is no command allowlist — rm, pnpm, pip, and build steps all run. The container is the boundary, and the operator picks what to mount.
| Tool | Description |
|---|---|
run(command) | Run a shell command in the workspace; returns exit_code, stdout, stderr, truncated, timed_out |
read_file(path, offset?, limit?) | Read a UTF-8 text file as a line window |
write_file(path, content) | Create or overwrite a text file (parent dirs created) |
edit_file(path, old, new, replace_all?) | Exact-replace old→new; unique match required unless replace_all |
read_image(path) | Read an image (png/jpg/jpeg/gif/webp) and return it as a viewable image |
The mounted directory is the only reachable host state, and it is writable — mount a copy or a scratch directory if you don't want the agent to mutate originals. Network egress is open (needed for pnpm/pip installs). The container runs as root with --cap-drop ALL and no-new-privileges; for genuinely untrusted input, run under gVisor or a microVM.
avatar
Gives the agent its own body: where the avatar is, and where to move it. Runs host-native — it calls the YUI client's loopback ingress, which only exists on the Mac running the app. The ingress binds only while the agent-notify feature is enabled, so the app must be running with that on.
| Tool | Description |
|---|---|
get_body_state() | Window position + monitor, posture (standing / sitting / peeking / dragging / walking / climbing / lying and what it is perched on), loaded VRM, whether a move is running |
list_perch_targets() | The client's tracked perch candidates plus the peek edges; each window carries app, title and rect, with app and title null for an unnamed window |
sit_on_window(app) | Sit on the top edge of that app's window — the frontmost one whose top edge is reachable |
peek(side) | Peek around the left or right edge of the frontmost window |
move_to(spot, monitor=None) | Move to center or a named corner, optionally on a given monitor |
stand() | Return to the standing position: leaves a perch or a peek, and gets her out of bed. In bed it starts the wake clip (she sits up, stretches and stands, about 22 s) and answers ok as that clip starts; posture keeps reading lying, and sit_on_window, peek and move_to keep answering busy, until she is up. While her lie-down clip still plays it plays no wake clip: the bed fades out and she is standing again within a second |
go_to_bed() | Lay her on a bed at the spot she stands on: she leaves any perch, comes down to the floor, the bed appears there and she lies down. Answers ok when the lie-down clip starts, and lying is reported from that moment; at once when she already lies. She sleeps until the client's wake timeout (the Character-tab setting, the backend passes no duration), a click or a message from the user, or stand |
Movement only — expression stays on the generate_express stream and screen capture belongs to desktop-control. A gesture that did not happen raises a tool error carrying the client's reason: not_found, blocked (every candidate window is covered, so nothing moved), interrupted (the user is holding the avatar, or grabbed it mid-move), busy (another gesture is running, or she is on the bed in any phase from the bed appearing to her standing, where sit_on_window, peek and move_to answer busy; stand gets her up), or unsupported (go_to_bed where the bed scene cannot start, such as with reduced motion on). interrupted also covers a click, a message or a drag by the user while the bed is being set up for go_to_bed. The ingress address defaults to http://127.0.0.1:8770; set AVATAR_INGRESS_URL when the client's ingress port was changed.
sit_on_window walks the app's windows front-to-back and takes the first whose top edge is not covered, so a Stage Manager thumbnail sitting in front of the real window does not shadow it. blocked means every one of them was covered.
The user always outranks the agent: a drag aborts a running gesture and blocks new ones until it ends.
factorio
Bridges the Factorio game to the agent through the coop-agent game mod, which answers /coop-rpc <json> over RCON. Runs host-native — RCON is a loopback console on the machine running the game. The RCON host, port, and password are read from Factorio's own config.ini (local-rcon-socket, local-rcon-password); the Mod never writes that file. The config path is config_ini in Mods/factorio/config.toml, and FACTORIO_CONFIG_TOML overrides the config file path. The game mod is the Lua mod in Mods/factorio/game-mod/; its tests under game-mod/tests track a pyproject.toml, so the mods CI job runs them.
| Tool | Description |
|---|---|
factorio_status() | Game mod version, map id, newest event sequence, tick, and host state |
factorio_events(since_seq=None, limit=200) | Cached events after since_seq (exclusive). None starts at the unread cursor; a value above it is clamped down, a value below it re-reads. Returns {events, next_seq, more} with a contiguous page of at most 200 |
factorio_snapshot(sections=None) | State snapshot of the factory; omit sections for every section |
factorio_find(area=None, around=None, name=None, type=None, limit=50) | Find entities by name, type, area, or near the host player |
factorio_inspect(unit_number) | Read one entity by its unit number |
factorio_recipe(name) | Read one recipe by name |
factorio_say(text) | Say one line in game chat under the body's name |
factorio_ping(pos, text) | Ping a map position with a text label |
factorio_jobs() | Every job record the game mod holds, oldest first |
factorio_jobs_enqueue(jobs) | Queue 1 to 8 jobs in the order given; each job is {kind, args, timeout_ticks?}. Returns the assigned ids. The game mod validates the kind and the args and answers bad_request with the field at fault |
factorio_jobs_cancel(id: int | "all") | Cancel one queued or running job by integer id, or every job with all. A cancel fires no event |
factorio_idle(mode, pos=None) | Set the idle mode follow or stay; stay takes an optional pos and otherwise stands at the body's position. Lifts the stop latch |
factorio_stop() | Stop the body's walking, mining, and crafting and latch it stopped. A job enqueue or factorio_idle lifts the latch |
The Mod polls events_since{seq, limit: 200} every second, looping until a page comes back short. Every event enters a 5000-event cache and the JSONL outcome log at outcome_log. An event whose kind is in wake.kinds rings the doorbell: after wake.coalesce_ms, the Mod posts one group to signals_url with delivery: "urgent", source: "factorio", and event_type: "game.wake". The group carries pending_from_seq (the oldest event the agent has not read), pending_to_seq (the newest cached event), and reset; its note is the newest unread chat line verbatim, otherwise the unread wake kinds. One post is in flight at a time. A non-2xx response backs off from 5 s to 300 s and retries the same range; unread wake events post once more 300 s after the last 2xx, once per unread range. When the agent has read every wake event by the time a post is due, nothing is posted.
A history_gap reply rewinds the poll cursor to oldest_seq - 1. A rollback on the same map (head_seq below the poll cursor) rewinds both cursors and drops cached events above head_seq. A different map id, or stale_session from any op, makes the Mod call status, adopt the new map, move both cursors to status.seq, and clear the cache. The last two set reset: true on the next doorbell. Otherwise both cursors only move forward. When the poll and a tool meet the same map change, the first recovery adopts the map and the later one leaves the cursors and the cache as they are. A rollback and a map change each write queue_length from status.queue.length in their session_reset outcome line; queue_length is null when the status carries no queue.
The default wake kinds are chat, queue_idle, job_failed, stopped, body_died, player_died, and rocket_launched. These opt-in kinds stay cached and do not wake unless config.toml lists them: job_done, research_finished, attack_started, attack_ended, body_respawned, idle_changed, player_joined, player_left, snapshot_summary, body_threat (an enemy came within the body's safe radius), body_damaged (an enemy damaged the body).
A call whose reply is lost is re-sent with the same id once status confirms the map; the game mod replays a stored write reply. An id_conflict reply means the id already ran a write with different arguments: the call ends with a tool error and is never re-sent under a new id.
The safety boundary is the game's own RCON configuration: the Mod reads it and touches nothing else. It holds no game logic and cannot edit the game's files or settings. The MCP server binds 127.0.0.1:9003 and has no transport auth, so reach it over the SSH reverse tunnel or through the router. The tools can speak in game chat and place map pings.
browser-cdp
Not an MCP mod — it exposes no tools. The agent already ships Playwright MCP, so the browser tools exist on its side; what is missing is the browser to drive. browser-cdp provides it by exposing a Chrome DevTools Protocol (CDP) endpoint that the agent's Playwright connects to, driving your local Mac browser with its logged-in sessions.
Launch the local Chrome with a CDP endpoint (launch-chrome-cdp.sh, a dedicated persistent profile), tunnel the CDP port, and point the agent's Playwright MCP at it:
./launch-chrome-cdp.sh # local Chrome on 127.0.0.1:9222
ssh -R 9222:localhost:9222 <remote-host> # CDP is not MCP — separate from the router's 8080
npx @playwright/mcp@latest --cdp-endpoint http://localhost:9222The attachment is configuration, not a chat message: set --cdp-endpoint (or cdpEndpoint in the agent's mcp.json) and restart the agent's Playwright MCP. CDP is full, unauthenticated control of that browser — keep it on 127.0.0.1 and reach it only through the SSH tunnel.