Big push driven by user feedback ("делай имбу") and web research on what
voice assistants need to be the ideal:
TTS backend abstraction (P0.1)
- new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs
- TtsBackend trait with SapiBackend (current PowerShell), PiperBackend
(rhasspy/piper, neural quality), SileroBackend (python subprocess)
- JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if
binary + voice present in tools/piper/. Falls back to SAPI on missing.
- SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils
sanitiser applied unless raw=true.
- llm_fallback + lua/api/tts both routed through tts::backend().
- tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx
from rhasspy releases + huggingface. Smoke-test included.
- tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess.
IMBA-1 Agentic LLM router
- crates/jarvis-app/src/llm_router.rs
- When fuzzy/intent matcher fails, LLM picks the closest command from the
full registry. Returns JSON {command_id, confidence, reason}.
- Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1
enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default.
- Inserted in app.rs::execute_command between "no match" and existing
llm_fallback chat fallback.
IMBA-2 Long-term memory
- crates/jarvis-core/src/long_term_memory.rs — JSON store at
APP_CONFIG_DIR/long_term_memory.json. Atomic write-through.
- remember/recall/search/forget/all/build_context API.
- Lua bindings: jarvis.memory.* (5 functions).
- llm_fallback auto-injects relevant facts (substring search of prompt)
into system message before LLM call.
- Pack resources/commands/memory_pack/ with 4 commands: remember, recall,
forget, list.
IMBA-3 Profile switching (work/game/sleep/driving/default)
- crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/
Auto-seeds 5 defaults on first run with personality + allow/deny lists +
greetings + emoji icons.
- active_profile.txt persists choice across restart.
- Lua bindings: jarvis.profile.{active,set,list,allows,active_name}.
- llm_fallback prepends profile personality to system prompt.
- Pack resources/commands/profile_switch/ with 6 voice triggers.
IMBA-4 Multimodal screenshot + vision LLM
- crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox.
- jarvis.vision.screenshot() captures via PowerShell System.Drawing.
- jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model
(default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL).
- Pack resources/commands/vision/ with 2 commands: describe + read_error.
P0.2 Continuous conversation grace window
- config::CONVERSATION_GRACE_MS = 30_000.
- app.rs: after command result, if grace_ms > 0 keep listening WITHOUT
re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so
the existing timeout fires at start + grace_ms.
Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils).
Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed
on Windows MSVC (VS 2026 Enterprise vcvars64).
Notes for setup:
- Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB).
- GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision).
- All features are opt-in via env vars or auto-detect; existing SAPI +
fuzzy match path remains the default.
User complaint: SAPI reads every dotted acronym character-by-character with a
"точка" between letters. Unbearable on every speak. No real TTS analog
shipping yet (Silero/Coqui/Inworld in roadmap), so the immediate fix is text
preprocessing.
New module `jarvis-core::text_utils` with `sanitize_for_speech()`:
- Specific brand mapping: J.A.R.V.I.S. → Джарвис, U.S.A. → США, U.K. →
Британия, U.S. → США, S.O.S. → сос.
- Generic dotted-acronym collapse: any `[letter].[letter].[letter].(...)`
run of 3+ letters gets its dots stripped. "T.O.N." → "TON".
- URL stripping: anything starting with http/https gets replaced with the
word "ссылка". SAPI reading URLs char-by-char is unlistenable.
- Em/en-dash normalisation: — → -, « » " stripped.
- Whitespace collapse.
Wired in two places:
- `jarvis.speak(text, opts?)` Lua API runs sanitiser unless opts.raw=true.
- `jarvis-app/llm_fallback::speak_via_sapi` runs sanitiser before the SAPI
PowerShell shell-out for LLM auto-fallback replies.
5 unit tests in `text_utils::tests` covering jarvis-name collapse, generic
acronym pattern, URL stripping, dash normalisation, and "leave clean
russian sentences alone" — all green.
Real TTS upgrade (Silero v4 first cut) tracked as P0.1 in roadmap.
Architecture — close the trigger-strip duplication:
lua/api/text.rs (new):
+ jarvis.text.strip_trigger(phrase, triggers) — case-insensitive
"longest trigger wins" stripping, returns trimmed remainder. The
11 packs that previously inlined a `for _, t in ipairs(triggers) do
string.find ... s == 1 then sub` loop can drop to one call.
Demonstrated in spelling/, window_switch/, summarize/.
+ jarvis.text.contains_any(phrase, needles) — boolean check for any
of N substrings, case-insensitive. Useful for keyword routing.
5 new portable Lua packs (sandbox=full). 36 → 41 total. cargo test
commands::tests still 3/3.
brightness/ — brightness_up / down / max / min. WMI WmiMonitorBrightness
(read CurrentBrightness) + WmiMonitorBrightnessMethods.WmiSetBrightness
(write). ±20% step for up/down; max=100, min=10. Desktop monitors that
don'\''t expose the WMI namespace get a friendly "не поддерживается"
toast instead of a silent failure.
window_switch/ — switch_to_window. Strip trigger ("переключись на" /
"switch to" / etc.), apply the same alias map as process_kill
(хром→chrome, телега→telegram, ...), then PowerShell finds processes
by name OR window-title substring sorted by StartTime desc and uses
(New-Object -ComObject WScript.Shell).AppActivate($pid) on the top
match. Speaks the window title that got focus.
spelling/ — spell_out. Strips "произнеси по буквам" / "spell out" /
etc., iterates `word:gmatch(".")`, joins letters with ". " so SAPI
gives each one a natural micro-pause. Uses jarvis.text.strip_trigger
+ jarvis.speak — 24 lines total.
network/ — open_wifi_settings (ms-settings:network-wifi) +
open_bluetooth_settings (ms-settings:bluetooth) + my_ip. my_ip pulls
the first non-loopback, non-APIPA IPv4 with Get-NetIPAddress and
fetches WAN IP from api.ipify.org (5s timeout); speaks "Локальный X.
Внешний Y".
summarize/ — summarize_selection. Synthesises Ctrl+C through user32!
keybd_event (so the focused window copies its selection), waits
220 ms for the clipboard to populate, reads via
jarvis.system.clipboard.get(), bails if < 20 chars, truncates at
8000, sends to Groq @ temp 0.3 with a "3-5 sentences, keep names"
prompt, drops the summary back into the clipboard and speaks it.
Lets you select ANY text in ANY app (browser, PDF, IDE) and say
"суммируй" to get a spoken TL;DR.
The new packs explicitly use the new jarvis.text + jarvis.llm +
jarvis.speak API surface — no inline PS-SAPI boilerplate, no inline
Groq plumbing. Code is now intent-first.
Architecture — DRY at the Lua API surface:
lua/api/tts.rs (new): jarvis.speak(text, opts?). opts.lang = ISO
two-letter code (default "ru"); auto-picks first installed voice
matching the culture. opts.async = true|false (default true). The
9 packs that previously inlined a 60-character PS-SAPI string for
every spoken reply can now reduce to one line. Refactored fun/,
dice/, clipboard_read/ as proof — fun/ask.lua went from 75 lines
of HTTP+SAPI boilerplate to ~28 lines of intent code.
lua/api/llm.rs (new): jarvis.llm(messages, opts?). messages is a
list of {role, content} tables, opts.max_tokens / .temperature /
.top_p. Returns the assistant string or nil. Gated by sandbox
allows_http() (so Standard+ packs only). Internally uses the
existing jarvis_core::llm::LlmClient::complete_with, no duplicated
HTTP plumbing.
llm/client.rs: split complete() into complete_with(messages,
max_tokens, temperature, top_p) for the new caller; old complete()
is now a thin wrapper at temp=0.7 top_p=1.0 for backward compat.
GUI — /commands page rewrite:
Replaces the "Раздел в разработке" placeholder. Calls Tauri command
get_commands_list (already existed in tauri_commands/commands.rs,
was wired but unused). Renders a search-filterable card grid:
- cmd id (monospace)
- type badge with colour by kind (lua=blue, ahk=red, cli=grey,
voice=green, terminate=red, stop_chaining=violet)
- sandbox badge
- description line (if non-empty)
- all phrases for currentLanguage, fallback to en, fallback to
first available; each phrase in a small inline pill
Toolbar: text input with magnifier icon + reload button (↻).
Counter line shows visible/total + "Скажи Джарвис + любую фразу"
hint. Counts down as you type the filter.
New packs (3, total 33 → 36):
games/ — launch_game / list_games. Reads or creates
%USERPROFILE%\Documents\jarvis-games.json (the sample preloads dota,
cs2, elden ring, witcher 3, factorio + minecraft launcher path +
fortnite epic URI). Trigger-strip + fuzzy name/alias match, then
launches via steam://rungameid/N (Steam), epic:// (Epic Games), or
start "" "path" (anything else). list_games speaks the configured
library.
mouse/ — left/right/middle/double click + scroll up/down. Single
dispatch.lua + _mouse_helper.ps1. PS P/Invokes user32!mouse_event
with the right flag pair (LEFTDOWN+LEFTUP / etc.), 30ms gap, doubles
do two LEFT cycles with a 50ms gap, wheel uses ±360 ticks.
random_choice/ — "выбери из X или Y или Z". Strips the trigger,
splits on " или " / " or " / ", " / " либо " / " чи ", math.random
picks one, speaks "Я выбираю N".
cargo test commands::tests still 3/3. 36 packs verified via
jarvis-gui startup log.
LLM auto-fallback (jarvis-app/src/app.rs + jarvis-core/src/config.rs):
- When neither intent classifier nor levenshtein finds a command match,
route the utterance straight to the LLM instead of just playing the
"not found" sound. Triggers ("скажи X", "answer Y") still work and
take precedence — they short-circuit before command lookup.
- Two new config knobs:
LLM_AUTO_FALLBACK = true — master toggle
LLM_AUTO_FALLBACK_MIN_CHARS = 4 — suppress for very short utterances
so background noise doesn't burn
Groq quota
- Requires GROQ_TOKEN; if absent, behaviour is unchanged (play_not_found).
codegen/ command pack:
- Phrases: "напиши код X", "сгенерируй скрипт Y", "write code Z", etc.
- Strips the trigger from the recognized phrase, sends what remains to
Groq with a strict system prompt ("return ONLY code, no fences, no
commentary") at temperature 0.2.
- Parses the JSON content, unescapes \n / \" / \\ / \t, strips any
remaining ```lang ... ``` fences, drops the result into the clipboard
via jarvis.system.clipboard.set.
- Notifies a 120-char preview + plays ok-sound. Works for any language
the model handles (Python by default if unspecified).
- GROQ_TOKEN / GROQ_MODEL / GROQ_BASE_URL read from env at call time —
same envvars the voice-loop fallback already uses.
ocr/ command pack:
- Phrases: "прочитай экран", "что на экране", "read screen", etc.
- Captures the primary screen via System.Windows.Forms + System.Drawing
to a temp PNG, then shells to tesseract.exe (-l rus+eng for ru/ua,
-l eng for en).
- Resolves Tesseract by PATH first, then C:\Program Files\Tesseract-OCR
and the x86 install dir; if none works the user gets a friendly
"winget install UB-Mannheim.TesseractOCR" hint in a notification.
- Recognized text goes to clipboard + a 200-char preview notification.
All three features are portable (env-var resolution, no hardcoded user
paths). Command-pack total is now 11. cargo test -p jarvis-core --lib
commands::tests passes 3/3.
cargo test on jarvis-core was failing with `LNK1181 cannot open input
file "libvosk.lib"`. The link-search path is declared in jarvis-app's
build.rs (and jarvis-cli's), but the test binary for jarvis-core has
no such config and the workspace .cargo/config.toml didn't cover it.
Adds a minimal build.rs that emits `cargo:rustc-link-search=native=...`
pointing at lib/windows/amd64 (resolved relative to CARGO_MANIFEST_DIR,
no hardcoded absolute paths so the project still builds for anyone
cloning under a different drive/path).
Verified: cargo test -p jarvis-core --lib commands::tests now passes
3/3 (every_command_toml_parses, lua_command_scripts_exist,
every_command_has_phrases).
Three unit tests catch the kind of bug that broke weather/set_city (a
phrases array instead of lang→array map silently dropped the whole pack):
- every_command_toml_parses: every resources/commands/*/command.toml round-
trips through toml::from_str::<JCommandsList>. Reports all failures at
once instead of failing on the first.
- lua_command_scripts_exist: for every type="lua" command, the named (or
default script.lua) script file exists in the pack dir.
- every_command_has_phrases: structural commands (terminate/stop_chaining)
excluded, every other command has ≥1 phrase across all languages.
Tests use env!("CARGO_MANIFEST_DIR") -> ../.. -> resources/commands so they
work from any cwd.
NB: cargo test does not run on this machine right now — aws-lc-sys needs
cl.exe (MSVC Build Tools missing). The release binary was built earlier
with MSVC available; toolchain install is a follow-up. The tests are still
valid for CI / a re-armed dev box.
new module crates/jarvis-app/src/llm_fallback.rs holds an optional
LlmClient + ConversationHistory built once at startup. if GROQ_TOKEN is
unset the module logs a warning and stays disabled — voice commands keep
working as before.
both the wake-word voice path (recognize_command in app.rs) and the
gui-side text command path (process_text_command) now check whether the
recognized phrase starts with one of the configured trigger phrases
(ru: 'скажи' / 'ответь' / 'произнеси'). when it does, the remainder of
the phrase is sent to Groq and the reply is published as a new
IpcEvent::LlmReply { text } so the gui can speak it.
on api error the trailing user turn is popped from history, the russian
fallback line is sent over ipc and a 'error' voice cue plays. the loop
itself never panics.
add LLM_DEFAULT_ENABLED / LLM_DEFAULT_MAX_HISTORY / LLM_DEFAULT_MAX_TOKENS
defaults, the russian J.A.R.V.I.S. system prompt, a fallback error line,
and per-language helpers get_llm_trigger_phrases() and get_llm_system_prompt()
so the voice loop can opt into Groq with a 'скажи …' style prefix.
split llm.rs into a module with separate client and history submodules.
ConversationHistory holds an optional system prompt plus a FIFO of user/
assistant turns capped at max_turns; oldest turns evict on overflow,
system prompt is preserved across truncation and clear().
New jarvis-core::llm module providing a blocking client for
OpenAI-compatible chat completions endpoints (Groq by default).
- LlmClient::new / from_env (GROQ_TOKEN, GROQ_BASE_URL, GROQ_MODEL)
- complete(messages, max_tokens) -> String
- thiserror-based LlmError / ConfigError
- gated behind a new llm feature, included in default jarvis_app
Not yet wired into the wake-word/intent loop; that lands in v0.3.
