YUI — Install and Wiring Guide
YUI is the frontend (body): VRM character rendering, desktop-pet behavior, and I/O surfaces. The brain and voice services — backend agent, TTS, STT — run as separate, config-swappable processes that YUI points at via configs/endpoints.json or the in-app panel (right-click the character → Settings → Connection tab).
What you need
One VRM model — and one ships in the repo (resources/vrms/Sendagaya_Shino.vrm). Everything else is optional: an unset service simply keeps its feature off, and you can wire it any time later from the in-app panel.
| Component | Status | Without it |
|---|---|---|
| VRM model | Bundled — bring your own optional (§2) | — |
Chat backend (chat_base_url) | Optional | Character appears and idles; a chat turn answers with an inline "Backend not configured" pointer to Connection |
| Chat API key | Optional | Only needed when the endpoint enforces one; set in the panel or .env.local |
| Expression MCP Broker | Deprecated, removed in v0.6.0 | Both request modes bake the vocabulary into the client-declared tool, no broker involved |
| TTS | Optional | Speech bubble works; no audio output |
| STT | Optional | Text input works; no voice input |
| Screenshot context | Optional | Agent receives no screen context |
Mods (Mods/) | Optional | No desktop-control or other Mod capabilities |
Fastest path: one OpenAI key
With YUI running (§1), one OpenAI key covers chat and voice: Chat Completions for chat and OpenAI for TTS. Neither service runs on your machine.
- Right-click the character → Settings → Connection tab.
- Chat section: keep Chat Completions as the chat API, pick the OpenAI provider, paste the key into Chat API key, set Chat model (
gpt-5-mini). - TTS section: pick the OpenAI provider and paste the same key into TTS API key.
- Close Settings and press
/to type a message.
Her personality is the Card section of the Talk tab, sent on Chat Completions and Responses. Each preset starts with the default card in the UI language; edit any field to make the card your own.
1. Run YUI itself
A coding agent can run sections 1–2 and the wiring in 4–7 for you from Install with a coding agent. The rest of this page is the manual path and the reference for the external services.
Prerequisites
- Node.js + pnpm
- Rust + the Tauri v2 toolchain
Install and launch
pnpm install
pnpm tauri dev # transparent desktop-pet window on port 1420 (recommended)
pnpm tauri:dev # same window on an auto-picked free port, so worktrees run side by side
pnpm dev # browser-only, no Tauri shell
pnpm build # production buildChat auth key
If your chat endpoint requires a key, either paste it into the panel's Chat key field or copy .env.example to .env.local and set VITE_YUI_CHAT_KEY. The key set in the panel wins; .env.local (gitignored) is the fallback used when the panel field is empty. It is read at build time, so restart the dev server after editing it. This fallback applies to dev runs (pnpm dev, pnpm tauri:dev) only — a release build carries no key from the environment, so release users enter their keys in the panel's Connection tab, which stores them on the device.
cp .env.example .env.local
# then edit .env.local and set VITE_YUI_CHAT_KEY=<your-key>2. VRM model (bring your own, optional)
The bundled Sendagaya_Shino.vrm is what configs/avatar.json → vrm_url loads by default. To use another VRM 1.0 model:
- In the Tauri app — open the panel's VRM section and import the file with the OS picker. The file is copied into the app data directory and added to the model list; nothing in the repo changes.
- From the repo (
pnpm devorpnpm tauri dev) — drop the file intoresources/vrms/(gitignored except the bundled default; Vite serves/vrms/*from there) and pointconfigs/avatar.jsonat it: setvrm_urlto/vrms/<file>.vrmand add a matching entry toavailable({ "id", "label", "url", "source": "bundled" };idis limited to[A-Za-z0-9._-]).
Per-model framing (framing.margin, framing.fov, and framing.upper_body, the vertical band of the model the phone frames by height as fractions of the model height from the feet) and the hit-test alpha threshold (hit_test.alpha_threshold) live in configs/avatar.json. Every tunable section in that file is required — the client reads each value from it and fails the load naming any key the file leaves out.
3. Chat backend
The shortest path to a first chat needs no config file: right-click the character to open Settings, switch to the Connection tab (plug icon), and pick a Provider preset in the Chat section, fill in Chat model (and Chat API key for OpenAI or Groq), close the panel, press / (or Cmd/Ctrl+Shift+Y) to open the text input, and send a message. The preset — OpenAI, Ollama, LM Studio, or Groq — autofills the endpoint URL; the prerequisite is a running Ollama or LM Studio, or an OpenAI or Groq key. Chat Completions, the shipped default, needs a tool-calling model because the client always declares its generate_express tool (src/io/chat/tools/client-tools.ts): gpt-5-mini on OpenAI, qwen3 on Ollama (pull it first with ollama pull qwen3), llama-3.3-70b-versatile on Groq.
YUI supports three chat protocols, selected by chat_api in configs/endpoints.json. Options A and B work with any server that speaks the corresponding OpenAI API; push is a WebSocket contract for backends that deliver without a request, described in push-transport.md. The shipped file sets chat_completions; if the key is removed the client behaves as responses.
Which one to pick
Pick by who keeps the conversation history. With Chat Completions the client sends it every turn. With Responses the server keeps it.
| Server | Mode to pick |
|---|---|
| A server that only serves a model (Ollama, LM Studio, vLLM, Groq) | Chat Completions (default) |
| A server that keeps responses (OpenAI, agent servers) | Responses |
| A backend that speaks first — currently only Hermes Agent supports this | Push |
On Responses against a server that keeps no responses, every turn after the first answers 404 for the stored previous_response_id. The client retries once without the id, the conversation context starts over, and the speech bubble shows a notice that it was reset; choose Chat Completions for such a server.
What the connection remembers
With Chat Completions the client sends the session transcript every turn, trimmed to chat_model_context_window. With Responses the server keeps the history. Conversation history is no long-term memory. Long-term memory comes from a backend agent with its own memory, or from an external memory setup added to the backend.
Option A — Chat Completions mode ("chat_api": "chat_completions", shipped default)
Any tool-calling OpenAI-compatible /v1/chat/completions endpoint drives expression on its own: the client declares generate_express with the emotion/motion/voice vocabulary baked into the tool schema, runs the call locally, returns the result, and keeps the conversation transcript client-side (no previous_response_id), trimmed to chat_model_context_window.
- Stand up (or pick) a Chat Completions endpoint.
- In
configs/endpoints.json, set:json"chat_api": "chat_completions", "chat_base_url": "<API root including /v1, e.g. http://localhost:8000/v1>", "chat_model": "<model id served by that endpoint>", "chat_model_context_window": 200000 - Provide the endpoint's API key if it needs one (§1 Chat auth key).
generate_express tool-call deltas are accumulated from the chat.completion.chunk stream, and each call's cue plays when a chunk carries a finish_reason or the stream ends. A model that stops on its cue calls without speaking gets the tool results back and re-requests, so the turn still reaches speech — up to three round trips. A model that speaks alongside its cues finishes in one request. See CC mode transport for the request shape and Tool-call round trip for the round trip.
Backend capability still varies: a plain OpenAI-compatible server (e.g. vLLM) speaks standard tool-call streaming, while the Hermes api-server's /v1/chat/completions never surfaces tool calls. With Hermes, use Responses mode.
Option B — Responses mode ("chat_api": "responses")
Any backend served over the OpenAI Responses API (/v1/responses). While broker_base_url is unset, the client declares generate_express on every request with the vocabulary baked into the tool schema, runs the call locally, and sends a follow-up request that re-sends the input items plus the call and its result, so the in-turn round trip does not depend on the server keeping responses (a plain OpenAI-compatible server such as vLLM works). Across turns the client chains on previous_response_id; a server that answers 404 for the stored id gets one retry without it, and the conversation context resets. The Hermes Agent API server does not pass a request's tools to its model; on Hermes use push mode, or the Expression Broker (§4) on Responses until v0.6.0.
- Stand up the backend agent with the Responses API served.
- Hand the agent the cue contract so it understands how to drive the character:
- With Hermes: create a profile, add
docs/reference/client-context.mdto that profile's context, and instruct it to remember the contract. - With other agents: include the contents of
docs/reference/client-context.mdin the system prompt or context.
- With Hermes: create a profile, add
- In
configs/endpoints.json, set:jsonThe client appends"chat_api": "responses", "chat_base_url": "http://localhost:8643/v1", "chat_model": "<model id>"/responsestochat_base_urlitself.
What each chat mode carries
| Feature | chat_completions | responses | push |
|---|---|---|---|
| Who keeps the conversation history | client | server | backend |
Speech text and generate_express cues | yes | yes | yes |
| Status pill's tool segment (which tool the backend is running) | yes | yes | yes |
| Reasoning at the top of the message window's bubble | yes, when the server streams reasoning_content or reasoning deltas | yes, when the backend streams reasoning events | yes |
| A reply the backend starts on its own | — | — | yes |
| Delegation list and reports | — | — | yes |
The first two modes work with any server that speaks the corresponding OpenAI API. push needs a backend that implements the contract in push-transport.md; the Hermes Agent integration under integrations/hermes/ is the implementation that exists today.
Reasoning effort
The in-app agent settings expose reasoning effort (none · minimal · low · medium). Responses mode sends it as reasoning.effort; Chat Completions mode as the top-level reasoning_effort.
The reasoning-effort control and the Card section are shown in Chat Completions and Responses modes only. Push mode sends neither, so the agent settings hide both there.
4. Expression MCP Broker (deprecated)
broker_base_url is deprecated and is removed in v0.6.0; the client declares its tools on the request, with the vocabulary in the tool schema. While the key is set the broker publishes YUI's renderable emotion/motion/emotion_text vocabulary so a backend agent that cannot take request tools (the Hermes API server) learns what the body can express, and the client logs one deprecated warning per launch. YUI publishes in every chat mode whenever broker_base_url is set, and silently skips it otherwise.
- Install and serve the broker from https://github.com/yw0nam/tts_express_broker.
- The broker listens by default at
http://localhost:3201/mcp(streamable-http MCP). - In
configs/endpoints.json, set:json"broker_base_url": "http://localhost:3201/mcp"
5. TTS — Voice Output (optional)
Without TTS, YUI displays text in the speech bubble but produces no audio. tts_provider names the engine at tts_base_url: irodori (the shipped default), openai, or fish. Each provider has its own endpoint shape and decides how the cue's emotion_text and caption ride in the request and where the voice list comes from. The Provider select in the Connection tab's TTS section sets the provider and fills in its server URL and default model.
If the server requires auth, set VITE_YUI_TTS_KEY in .env.local. YUI sends it as Authorization: Bearer. Like all key fallbacks this applies to dev runs only; a release build reads the key entered in the Connection tab.
Irodori
Irodori TTS Server is the reference deployment. Follow that repo's README to run it (default port 8088). YUI prepends the emoji emotion_text tag to the spoken text and sends caption as irodori.caption.
Caveat: Irodori serves Japanese only. When using it, instruct your backend agent to respond in Japanese.
In configs/endpoints.json:
"tts_provider": "irodori",
"tts_base_url": "http://localhost:8088",
"tts_model": "irodori-tts",
"tts_speaker": "<voice-id>"tts_model must match the name the server is configured under, or the server answers 400.
The TTS server is the source of truth for the available voice IDs (GET /v1/audio/voices), and YUI ships no bundled catalog. The panel's voice section lists them; tts_speaker picks the one used until you choose another there. Voices live in the server's voices/ directory, and the panel uploads imported reference clips with POST/PUT /v1/audio/voices and removes them with DELETE /v1/audio/voices/{voice_id}. When the server lists no voice a local clip was imported under, the voice list refresh uploads that clip again.
OpenAI
- Create a key at https://platform.openai.com/api-keys.
- In the Connection tab's TTS section, pick OpenAI as the provider. The server URL becomes
https://api.openai.comand the modelgpt-4o-mini-tts. - Paste the key into the TTS API key field.
The voice section lists OpenAI's 13 built-in voices: alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, cedar. Importing, deleting, and re-uploading voices is off for this provider. YUI sends the spoken text as input and joins emotion_text and caption into instructions. The tts-1 and tts-1-hd models ignore instructions and speak only alloy, ash, coral, echo, fable, nova, onyx, sage, and shimmer.
Fish
Fish Audio is a hosted service. Sign in at https://fish.audio/, create an API key, pick Fish Audio as the provider in the Connection tab's TTS section (the server URL becomes https://api.fish.audio and the model s2.1-pro-free), and paste the key into the TTS API key field.
The model field takes any S2-family model id (s2-pro, s2.1-pro, s2.1-pro-free, drama-3-preview); an unknown id falls back to s2.1-pro on Fish's side.
The voice section lists the models in your Fish account (GET /model?self=true) by their titles. Importing an audio file uploads it as a new voice model (POST /model, fast training) and selects the model Fish creates; deleting a voice removes the model from your account. A voice from Fish's public library that is not in your list can still be used: paste its voice id, or its voice page URL such as https://fish.audio/m/<id>, into the field under the voice list and it becomes the active voice. Deleting a pasted voice only takes it off the list. Imported and pasted voices belong to the provider they were added under, so they are listed only while that provider is selected.
YUI sends the spoken text as text to POST <tts_base_url>/v1/tts and the model as a model HTTP header. emotion_text and caption each ride as their own [...] bracket ahead of the sentence (S2 inline direction).
6. STT — Voice Input (optional)
Without STT, text input still works. VAD (Silero + ONNX) runs client-side; STT sends segmented audio to a transcription server.
Serve any OpenAI-compatible transcription server at the configured URL, then set:
"stt_base_url": "http://localhost:5517/v1"YUI sends audio to <stt_base_url>/audio/transcriptions. If the server requires auth, set VITE_YUI_STT_KEY in .env.local (dev runs; a release build reads the key entered in the Connection tab) — YUI sends it as Authorization: Bearer. A hosted API such as Groq also needs a model: set stt_model in configs/endpoints.json (for example "stt_model": "whisper-large-v3-turbo") or in the Connection tab's STT model field, which overrides the file. To fix the spoken language, set stt_language the same way; the Connection tab's STT language field suggests Auto, English, and Japanese and takes any other ISO-639-1 code.
7. Wire It All Together — configs/endpoints.json
YUI ships with no service addresses: every URL in the bundled configs/endpoints.json is unset, and an unset URL means that feature is off — STT, TTS, and the expression broker stay quiet, and a chat turn with no chat_base_url answers with an inline "Backend not configured" error pointing at Connection.
Point YUI at your services by editing configs/endpoints.json or using the in-app Endpoint settings panel, which persists overrides to local storage and leaves the bundled file untouched.
Key reference:
| Key | Shipped default | Purpose |
|---|---|---|
chat_api | chat_completions | Chat protocol: "chat_completions" (client-declared generate_express, any tool-calling endpoint), "responses" (backend agent honoring the expression contract), or "push" (one WebSocket, see push transport) |
chat_base_url | unset | In chat_completions and responses modes, the API root including /v1, to which the client appends /chat/completions or /responses. In push mode, the WebSocket base without /v1, where the client opens <chat_base_url>/ws |
chat_model | unset | Model ID sent to the backend |
chat_model_context_window | 200000 | Token window — display in Responses mode; also trims the client-side transcript in Chat Completions mode |
chat_instructions | expression prompt | System-level nudge on how to use generate_express; appended to the card, sent as instructions (Responses) or the first system message (Chat Completions) |
stt_base_url | unset | STT server base URL |
stt_model | unset | model sent to the STT server; omitted when unset |
stt_language | unset | language sent to the STT server, an ISO-639-1 code such as ja; omitted when unset or auto, and the server detects the language |
tts_provider | irodori | TTS engine at tts_base_url: irodori, openai, or fish |
tts_base_url | unset | TTS server root without /v1 |
tts_model | irodori-tts | Model sent to the TTS server — Irodori/OpenAI as the request's model field (must match the server's configured name), Fish as a model HTTP header (an S2-family id) |
tts_speaker | unset | Default voice id, until another is picked in the panel |
tts_max_inflight | 1 | Concurrent TTS synthesis requests |
broker_base_url | unset | Expression broker MCP URL. Deprecated, removed in v0.6.0: the client declares its tools on the request |
All service addresses come from this file or the in-app overrides.
8. Platform Notes
- Idle watching works on both.
os_idle_mspolling is implemented on both macOS and Windows, so idle-triggered and co-working proactive cues fire on either platform. - macOS TCC grants. The optional screenshot context feature and the
desktop-controlMod (if used) require Screen Recording permission. App control via thedesktop-controlMod additionally requires Automation / Apple Events permission. Grant these in System Settings → Privacy & Security.