Voice assistant

Say “turn off everything on the terrace”, “aprinde lumina din birou”, “run movie mode” or “is the garage door open?”. An LLM agent maps the phrase to concrete actions against your real catalogue, executes them through the same engine the web UI uses, and speaks the answer back.

How it works

Browser / PWA / Android                  Backend
─────────────────────────                ───────────────────────────────────
Web Speech API (STT)                     voiceCommand GraphQL mutation
   transcript ───────────────────────▶     1. build a catalogue from the live DB
                                           2. agentic tool-use loop with the LLM
   plays reply  ◀──────────────────────    3. execute the tool calls it returns
   (neural MP3, else browser voice)        4. optionally synthesise the reply
   keeps sessionId for multi-turn

Speech-to-text happens in the browser; the backend receives text, never audio. That keeps the server contract simple and means any client — the web app, the Android app, a script — can use the same mutation. You can also just type the command.

The agentic loop

This is not one-shot classification. Each request runs a bounded tool-use loop (at most six iterations):

1. Build the catalogue of controllable entities.
2. Load prior messages for this session, or start fresh.
3. Append the transcript and call the configured LLM provider.
4. The model returns either tool calls — execute them, feed the results back, loop — or a final reply.
5. Persist the conversation, synthesise speech if enabled, return.

Clarification is emergent rather than engineered: when a request is ambiguous the model simply ends its turn with a question instead of guessing. The client speaks it, keeps the session id, and the next utterance continues the conversation. There is no dedicated “ask” tool.

Grounding: the catalogue

On every request the backend builds a fresh catalogue from the database and passes it to the model as JSON. The model may only choose ids that appear in it, and the server re-validates every id before acting.

{
  "peripherals": [ { "id", "name", "category", "zones": [...], "aliases": [...] } ],
  "zones":       [ { "id", "name", "peripherals": [names], "aliases": [...] } ],
  "scenarios":   [ { "jobId", "name", "description" } ],
  "mowers":      [ { "deviceId", "name" } ]
}

• Only peripherals with at least one connected port are listed.
• Zones carry their peripheral names recursively, so the model can target a whole area.
• Voice aliases — alternate names you set on a peripheral or zone — are included, which is how Romanian nicknames get matched reliably.

Tools

If a tool throws — the mower cloud rejects a command, say — the loop feeds the error back to the model so it can explain what happened, rather than aborting the turn.

Providers

The orchestration lives in myHAB; a provider is a pure translator between the neutral conversation shape and one vendor's API. Two ship today:

The system prompt, tool schemas and catalogue format are shared, so both vendors behave identically. Adding a third means implementing one interface and registering it.

Conversation state

Multi-turn state is held server-side in Hazelcast, keyed by a client session id: a five-minute TTL, capped at forty messages, trimmed only at a clean user-message boundary so tool-call pairs stay intact. A failed turn is not persisted, so a broken sequence can never poison the next one.

Text-to-speech

Replies can be rendered by Google Cloud Text-to-Speech instead of the browser's built-in voice. Google TTS does not accept API keys — it needs a service account, whose JSON key myHAB exchanges for a short-lived OAuth token, cached for about an hour.

If TTS fails for any reason the result simply omits audio and the browser voice is used. It is never a hard failure.

Setting it up

1. Create an API key at your chosen LLM provider.
2. In Settings → App configuration (or the configuration repository) set:

   feature:
     voice:
       enabled: true
       llm:
         provider: anthropic        # or openai
         apikey: "<your key>"
         # model: claude-haiku-4-5  # optional override

3. Optional — neural speech. Create a Google Cloud service account, enable the Cloud Text-to-Speech API and billing, download the JSON key, then:

   feature:
     voice:
       tts:
         enabled: true
         provider: google
         # either minified inline JSON, or a path on the server
         apikey: "/opt/myhab/gcp-tts.json"
         voice:
           ro: ro-RO-Wavenet-A
           en: en-US-Neural2-C

4. Add voice aliases to peripherals and zones whose spoken names differ from their display names. This is the single highest-value tuning step.

Using it

The Android client

client/android/ is a native Kotlin + Jetpack Compose companion that adds the one thing the browser cannot do: hands-free activation.

(wake word | tap-to-talk) → on-device speech-to-text → voiceCommand mutation
   → play the server's MP3 reply (fallback: Android TextToSpeech)
   → if the assistant asked something, listen again in the same conversation

cd client/android
./gradlew :app:assembleDebug     # build
./gradlew :app:installDebug      # install on a connected device

# Windows helper: versioned APK into dist/
.\build-apk.ps1 -Install

It is a thin client over the existing server contract: no backend changes, no parallel execution path. Everything the app can do, the web client can do too.

Security and cost

• The voiceCommand mutation requires authentication; the executing user is recorded on the event for audit.
• The model can only act on ids present in the catalogue, and the server re-validates each one — a hallucinated device id is rejected, not executed.
• API credentials come from the configuration repository or environment variables. Because a git history keeps old values, rotation means issuing a new key and revoking the old one.
• The catalogue and system prompt are prompt-cached, so the marginal cost per command is small and roughly constant.

Current limitations

• Only ON / OFF / TOGGLE. Dimming, colour and set-value commands are not exposed to voice.
• Scenarios can be run by voice, not authored.
• Peripherals and zones support aliases; mower devices do not yet.
• Speech recognition is single-language per session — no automatic language detection.
• Browser coverage is whatever the Web Speech API offers, which is uneven outside Chrome.