Skip to content

innestemme

innestemme is a local voice assistant that replaces Home Assistant's Assist pipeline on the living room Voice PE. It connects to the device over the ESPHome native API, hears "Hey Jarvis" with openWakeWord, transcribes with Parakeet, answers with its own skills and Home Assistant as "Jarvis" in a butler's style, and speaks with Kokoro (voice am_onyx) inside the engine. How the engine works is in the innestemme repo's docs/. Its image, ghcr.io/mortennordbye/innestemme, is built in the separate mortennordbye/innestemme repo. The manifests live in k8s/talos/apps/innestemme/.
Namespace innestemme
VIP 10.3.10.99:9090: spoken answers, /metrics, and the skills page at /, which is also at https://jarvis.local.bigd.no through the private gateway
Device Voice PE "Home Assistant Voice 0a1f4d", 10.3.20.67 on IoT, area Living Room
Answers announced on the living room Sonos, media_player.living_room, at volume 0.50; "Hey Jarvis" over an answer interrupts it
Voice Kokoro inside the engine (sherpa-onnx), voice am_onyx; rendered speech kept in /models/speech-cache
Name and style wake-name: Jarvis, honorific: sir ("As you wish, sir."), honorific-female: ma'am for female voices, no honorific when the voice is unclear
Delivery Kargo project innestemme-cd: each build is promoted straight to main and deployed, then smoke-tested (kargo.md); roll back by promoting older Freight
Backup none: the volume is a model cache that downloads again
Wake words Hey Jarvis, detected in the engine (openWakeWord, wake-model, wake-threshold: 0.65) on the device's continuous stream
Firmware innestemme's firmware/ build (26.9.0 plus a patch), "Wake word in innestemme" switch on
Unhandled requests /models/unhandled.jsonl (unhandled-log), listed on the web page

How a request flows

The Voice PE on the IoT network streams its microphone on port 6053 to the innestemme pod, where openWakeWord, Parakeet speech to text, the rule-based skills and Kokoro text to speech run in turn; the skills read and write Home Assistant, which tells the living room Sonos to announce the answer, and the Sonos downloads the .wav from the LoadBalancer VIP 10.3.10.99:9090

  1. The pod dials the Voice PE on 10.3.20.67:6053 (Noise-encrypted) and takes its microphone. Trusted to IoT is open, so no firewall rule is needed for this direction.
  2. The device streams its microphone all the time; the engine scores it with openWakeWord (wake-model: hey_jarvis, wake-threshold: 0.65) and takes what follows "Hey Jarvis".
  3. Parakeet TDT 0.6B (int8) transcribes the request, and the engine answers from its rules and Home Assistant (home-assistant Service). A complete command is acted on after 0.24 s of quiet, anything else after 0.64 s. Home Assistant writes (lights, scenes, the phone's ring) are queued, so the answer does not wait for the Hue lights. No language model is configured, so a request the rules do not understand gets a short "didn't catch that" (only when the name was in the wake word; a false wake by the TV stays silent).
  4. Speech: about 4,600 fixed sentences (openers, remarks, lead-ins, jokes, confirmations, every volume level and minute of the day, this home's lights and room temperatures) are rendered once by Kokoro in idle time and kept on the models volume (/models/speech-cache, about 560 MB); only other sentences with live data are synthesized per request. The home's forecast and the electricity prices are fetched every 4 minutes, so those answers skip their lead-in. A slow answer (weather, departures, prices, what's on, who's home) opens with a kept lead-in ("Checking the forecast, sir.") while its data sentence is synthesized. A sentence heard before it was rendered is logged as speech not rendered ahead.
  5. Home Assistant announces the answer on the Sonos (answer-player), which downloads it from http://10.3.10.99:9090/speech/<id>.wav while it is still being synthesized. Without answer-player the Voice PE plays it itself; IoT may only reach that one address and port, from the Voice PE's MAC: voice_pe_to_innestemme in terraform/unifi/network/firewall.tf.

Configuration

Settings are the innestemme-config ConfigMap, one voice-engine --help option per key. Reloader restarts the pod when it changes. innestemme-secret comes from Bitwarden:

Key Bitwarden secret Holds
HA_TOKEN innestemme-ha-token Home Assistant long-lived token ai-voice
VOICE_SATELLITE_KEY innestemme-satellite-key the Voice PE's API encryption key (base64)
VOICE_ADDRESS innestemme-address street address for weather and departures
VOICE_HUE_APP_KEY id 32ceeacb-… innestemme's key on the Hue bridge, for party mode
VOICE_HUE_CLIENT_KEY id 2e946f69-… the bridge's Entertainment stream key
VOICE_PLEX_TOKEN innestemme-plex-token the Plex server's own token, for "play on Plex"</td> </tr> </tbody> </table> <p>The satellite key is the <code>noise_psk</code> Home Assistant set when the device was added, in the ESPHome entry of <code>/config/.storage/core.config_entries</code> on the HA box. Re-adding the device in Home Assistant can generate a new key; update the Bitwarden secret when that happens.</p> <h2 id="home-assistant-side">Home Assistant side<a class="headerlink" href="#home-assistant-side" title="Permanent link">¶</a></h2> <ul> <li>The device's Assist satellite entity is disabled, so Home Assistant keeps the LED ring, volume and mute but leaves the voice to innestemme. Re-enable it to fall back to Assist.</li> <li>The engine finds the room from the device's area in Home Assistant, looked up by the device name. Renaming the device in Home Assistant breaks that lookup; set <code>room</code> in the ConfigMap if it is renamed.</li> <li>"Play <title> on Plex" plays through the TV's Cast player as <code>plex://<key></code>, which needs Home Assistant's Plex integration (manual setup, <code>10.3.10.103:32400</code>, the same token). innestemme itself searches <code>plex-url</code>, the Plex service in the cluster, which the Plex CiliumNetworkPolicy lets the innestemme namespace reach.</li> <li>The engine sets the device's wake words (<code>satellite-wake-words</code>) each time it connects. The wake word selects in Home Assistant go through the disabled satellite entity and do nothing; the sensitivity select is the device's own and still works.</li> <li>The device runs innestemme's firmware: with its "Wake word in innestemme" switch on it streams the microphone continuously and the engine detects the wake word. Turning the switch off in Home Assistant goes back to the device's own wake word without a flash; the engine still works then. Rolling back fully: flash the official 26.9.0 binary (<code>esphome upload ... --file</code>).</li> </ul> <h2 id="operating">Operating<a class="headerlink" href="#operating" title="Permanent link">¶</a></h2> <ul> <li>Only one engine can hold the device. Stop any laptop instance before the pod starts, or the two take turns connecting.</li> <li>Logs: <code>kubectl -n innestemme logs deploy/innestemme</code>. A working start logs <code>satellite connected</code> and <code>satellite room from home assistant</code>.</li> <li>Work on the engine from a Mac against the real device: scale the pod to zero (<code>kubectl -n innestemme scale deploy/innestemme --replicas=0</code>; Argo CD ignores replicas), run <code>make kokoro</code> and <code>make satellite</code> in the innestemme repo, and scale back to one afterwards.</li> <li>The skills page (<code>https://jarvis.local.bigd.no</code>) lists every skill with its data source, the calls it makes and example phrases, shows how a typed phrase is parsed, and "Hear response" speaks the answer in the browser for requests that change nothing.</li> </ul> <h2 id="kokoro">Kokoro<a class="headerlink" href="#kokoro" title="Permanent link">¶</a></h2> <p>Kokoro v1.0 runs inside the engine through sherpa-onnx (<code>english-tts: kokoro-onnx</code>): the f32 model, its voices and English pronunciation data (about 380 MB) download into <code>/models/hub</code> on first start. It was a Kokoro-FastAPI sidecar (Python, 1.7 GB resident) until 2026-10-18; in the engine it is 16-31% faster on the same CPUs (innestemme <code>docs/benchmarks.md</code>), so the pod has the 6 CPUs the two shared. Its speech cache is <code>/models/speech-cache/kokoro-onnx-am_onyx</code>; after the switch the about 4,600 fixed sentences render again in idle time, a few hours, and until then some answers are synthesized when asked.</p> <h2 id="troubleshooting">Troubleshooting<a class="headerlink" href="#troubleshooting" title="Permanent link">¶</a></h2> <p>Measured on 2026-10-10, when the assistant "worked once, then stopped":</p> <ul> <li><strong>Loud answers deafened the device's wake word detector</strong> (official firmware): after an answer at volume 65 it missed normal voices for several seconds. With the wake word detected in the engine this no longer applies; answers are at 0.50.</li> <li><strong>Saying "Hey Jarvis" again during a run stops it</strong> (official firmware); the engine logs <code>device ended the run</code>.</li> <li><strong>Reading the device's own log:</strong> a small aioesphomeapi script calling <code>subscribe_logs</code> with the <code>noise_psk</code>; it shows <code>Detected 'Hey Jarvis' with sliding average probability is X</code> and the voice assistant's state changes. A detection only logs when it passes the cutoff (0.83 at Very sensitive); a voice from 1.5 m scores about 0.85.</li> <li><strong>A stuck device</strong> can be restarted with its Restart button entity (disabled by default in Home Assistant) or over the API.</li> <li><strong>Testing without talking:</strong> play a Piper "Hey Jarvis." clip on the Sonos with <code>media_player.play_media</code> (<code>announce: true</code>, <code>extra.volume</code>) and read the device log. It is audible in the living room.</li> </ul> </article> </div> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type="button" class="md-top md-icon" data-md-component="top" hidden> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class="md-footer"> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class="md-copyright"> Made with <a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener"> Material for MkDocs </a> </div> </div> </div> </footer> </div> <div class="md-dialog" data-md-component="dialog"> <div class="md-dialog__inner md-typeset"></div> </div> <script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": ["navigation.indexes", "navigation.sections", "navigation.top", "search.suggest", "content.code.copy"], "search": "../../assets/javascripts/workers/search.a7633da7.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script src="../../assets/javascripts/bundle.c21346df.min.js"></script> <script> // Material renders the site name as plain text; link it home like the logo. (function () { var logo = document.querySelector(".md-header__button.md-logo"); var name = document.querySelector(".md-header__topic:first-child .md-ellipsis"); if (!logo || !name) return; var link = document.createElement("a"); link.href = logo.getAttribute("href"); link.className = "md-header__home"; link.textContent = name.textContent.trim(); name.replaceChildren(link); })(); </script> </body> </html>