Live input and the Live API

The Live panel (command palette: “Show Live”) drives the open model from live sources:

• Webcam face tracking in the browser (MediaPipe Face Landmarker, no server);
• VTube Studio (receive its tracking, or send the editor’s parameter values to it);
• the Live API: the editor connects to a WebSocket server of your own (protocol below).

Live input moves the preview only: it never writes the document, has no undo entries and is not seen by collaborators. Stopping a source puts the parameters it drove back to their values from before it started. To keep a performance, record it into an animation track (Record in the Live panel, when the project has a scene and a model track).

Mapping

Each tracking source has a mapping table: one row per input → parameter, with

Rows writing the same parameter are added, and the sum is clamped to the parameter’s range. The defaults map the standard parameters (ParamAngleX/Y/Z, ParamBodyAngleX/Y/Z, ParamEyeL/ROpen, ParamEyeBallX/Y, ParamBrowL/RY, ParamMouthOpenY, ParamMouthForm) that the model has; “Defaults” restores them.

Calibrate neutral pose stores the current input values as the neutral pose: angles, positions, gaze, brows and mouth form read 0 there; eye openness reads fully open; mouth open and blendshape scores read closed / 0.

Where settings are saved. Mappings and calibration are saved per project (per model name for runtime models opened without a project) in the browser’s local storage, not in the document: they describe one person’s face, camera and taste, so every collaborator keeps their own; live input never changes the rig or its export, so it does not belong in the document’s history, undo or the room’s storage; and tuning changes them continuously, which would otherwise flood every collaborator with operations. The VTube Studio and Live API addresses are per browser.

Webcam inputs

Computed per frame from MediaPipe’s facial transformation matrix and its 52 blendshape scores. With Mirror on (default) the preview acts as a mirror: X-type values are negated and Left/Right swap, so the model copies you like a reflection (your right eye drives the model’s left eye, which is on the right of the screen).

Tracking runs in a worker: camera frames are handed over as ImageBitmaps, one in flight at a time (frames arriving meanwhile are skipped). The GPU delegate (WebGL2 on an OffscreenCanvas) is used when available, else the CPU delegate (WebAssembly SIMD). The panel shows the cost per frame: worker time (avg / p50 / p95 over the last 120 frames), main-thread time and the tracked frame rate. Every file is served by the editor itself: the tasks-vision WebAssembly runtime (@mediapipe/tasks-vision, Apache-2.0) and the Face Landmarker model bundle (face_landmarker.task, float16 v1, Apache-2.0, from Google’s MediaPipe model repository).

VTube Studio

The editor is a VTube Studio plugin (“Ikimotion Studio Live” by “Ikimotion Studio”) using the VTube Studio Public API at ws://localhost:8001 (change the address if you changed the port in VTube Studio). In VTube Studio, turn on Start API (allow plugins) in the settings.

• First connection: VTube Studio asks whether to allow the plugin; the status reads “Waiting for you to allow the plugin in VTube Studio…”. The token it grants is stored in this browser (per address); later connections authenticate with it without asking. “Forget access token” makes the next connection ask again. If you deny access, or revoke it in VTube Studio later, the link stops until you connect again.
• Reconnects: when VTube Studio closes or restarts, the link retries (1 s, doubling to 10 s) and re-authenticates with the stored token.
• Receive tracking: polls InputParameterListRequest (one request in flight, up to 30 per second) and feeds VTube Studio’s tracking parameters (FaceAngleX, EyeOpenLeft, MouthSmile, custom plugin parameters, …) through the VTube Studio mapping table.
• Send preview: sends the model’s evaluated parameter values (physics included) with InjectParameterDataRequest (mode set, faceFound: true) 30 times per second (VTube Studio releases a parameter after 1 s without data). Targets:
  • VTS tracking parameters: the inverse of the VTube Studio mapping table (first row per VTS parameter): e.g. ParamAngleX → FaceAngleX. Works with the parameter mapping VTube Studio sets up for a model with standard parameters.
  • One plugin parameter per model parameter: creates custom parameters Rig<ParameterId> (alphanumeric, ≤ 32 characters; VTube Studio allows 100 per plugin, so the first 100 parameters are sent) with the parameter’s range; map each to its Live2D parameter once in VTube Studio for an exact 1:1 test of the exported model. Receiving and sending are exclusive (sending your own values back would feed back).

Live API (version 1)

The editor connects as a client to the WebSocket URL entered in the Live panel and reconnects (1 s doubling to 10 s) while connected. Messages are UTF-8 JSON text frames, one object per frame, each with "v": 1 and a "type". Server messages may carry an "id" (string); replies to them carry it back as "ref". Unknown fields are ignored; unknown types are answered with an error. Inbound messages larger than 1 MiB are rejected.

Parameters are addressed by their ID (the runtime ID, e.g. ParamAngleX). Values set through the API move the preview only and are clamped to the parameter’s range.

Editor → server

// On connect, in this order: hello, parameters, selection.
{ "v": 1, "type": "hello", "app": "rig-platform", "editor": "rig-platform-web", "api": 1,
  "capabilities": ["parameters", "values", "set", "selection"] }

// The open model's parameters (also after getParameters, and whenever the model changes).
{ "v": 1, "type": "parameters", "ref": "p1",
  "model": { "name": "Hiyori", "project": "a1b2c3" },          // null: no model open
  "parameters": [ { "id": "ParamAngleX", "name": "Angle X", "min": -30, "max": 30, "default": 0, "repeat": false } ] }

// Subscribed values: the first message has every subscribed parameter, later ones only
// those that changed; nothing is sent while nothing changes. At most `rate` per second.
{ "v": 1, "type": "values", "seq": 42, "time": 1760450000000, "values": { "ParamAngleX": 12.5 } }

// The editor's selection (on connect, on every change, and after getSelection).
{ "v": 1, "type": "selection", "ref": "q",
  "items": [ { "kind": "parameter", "id": "ParamAngleZ", "name": "Angle Z" } ] }   // kind: drawable | part | deformer | parameter

{ "v": 1, "type": "subscribed", "ref": "s", "rate": 30, "parameters": ["ParamAngleX"] }
{ "v": 1, "type": "result", "ref": "set1", "applied": 1, "unknown": ["NoSuchParam"] }
{ "v": 1, "type": "pong", "ref": "r" }
{ "v": 1, "type": "error", "ref": "x", "code": "bad_version", "message": "…" }

values.time is the editor’s wall clock (ms since the epoch); seq increases by one per values message. Values are the evaluated ones (physics outputs included).

Error codes: bad_json, too_large, bad_version, unknown_type, bad_message (a field has the wrong type), no_model (set while no model is open).

Server → editor

{ "v": 1, "type": "hello", "name": "My tracker" }          // optional; shown in the panel
{ "v": 1, "type": "getParameters", "id": "p1" }
{ "v": 1, "type": "subscribe", "id": "s", "rate": 30, "parameters": ["ParamAngleX"] }
// rate: Hz, default 30, capped at 60; parameters: default all; unknown IDs are left out
// (see the "subscribed" reply). A new subscribe replaces the previous one.
{ "v": 1, "type": "unsubscribe", "id": "u" }               // replied with subscribed, rate 0
{ "v": 1, "type": "set", "id": "set1", "values": { "ParamAngleX": 12.5 } }
// Holds the values on the preview until release or disconnect; result is sent when `id` is given.
{ "v": 1, "type": "release", "id": "r1", "parameters": ["ParamAngleX"] }
// Ends holds (default: all) and restores the values from before the first set.
{ "v": 1, "type": "getSelection", "id": "q" }
{ "v": 1, "type": "ping", "id": "r" }

When the connection closes, every hold ends (values restored) and the subscription is dropped. Tracking sources and API holds write the same preview values: whichever writes last wins.

Connecting to local apps: browser policies

The deployed editor is served over https; VTube Studio and most local tools listen on plain ws://localhost. Verified with apps/web/e2e/live-network.spec.ts (Chromium 156, Playwright WebKit 27.2):

Consequences:

• Chrome / Edge: click Allow on the local-network prompt the first time. Managed installations can pre-allow the editor’s origin with the LocalNetworkAccessAllowedForUrls policy.
• Safari: the VTube Studio link and Live API servers on ws:// cannot be reached from the https editor. Use Chrome/Edge/Firefox, run the editor locally (pnpm -C apps/web dev or preview, served on http://localhost), or serve your Live API server over wss:// with a certificate Safari trusts (VTube Studio itself offers only ws://).
• The Live API server may run on another machine; then it needs wss:// with a valid certificate (any browser blocks ws:// to a remote host from an https page).