## Issue 1: Voices were overlapping
Root cause was structural, not a single bug: every TTS backend's speak()
was fire-and-forget (`cmd.spawn()` instead of `cmd.status()`) AND cue
WAVs went through a completely SEPARATE audio path (Kira) with no
coordination. Two back-to-back calls — `voices::play_reply()` + a
`tts::speak("Готово")` + an idle banter chime — all started their
audio at the same moment.
Fix: ONE speech worker thread drains a `mpsc::Sender<Job>` channel.
Jobs are `Speak { text, opts }` or `Wav(path)`. The worker:
- forces opts.detached = false so backends BLOCK until audio finishes
- plays WAVs synchronously via PowerShell SoundPlayer
- thus implicitly serialises everything queued behind it
Both call paths now feed the same queue:
- `tts::speak()` → Job::Speak
- `tts::play_wav()` → Job::Wav (used by Piper/Silero for their .wav output)
- `voices::play_random_from_list()` now also calls `tts::play_wav()`
instead of `audio::play_sound()` — that re-routes cue sounds through
the same queue. Trade-off: cue WAVs lose Kira's native low-latency
start (~50ms slower) but gain guaranteed ordering against TTS.
Two new unit tests:
- `speech_queue_serialises_calls`: fires 5 speak() back-to-back with an
80ms-per-call fake backend, asserts gaps between starts are ≥60ms
(i.e. the worker IS waiting). Confirms no overlap.
- `empty_text_is_skipped`: the guard against accidentally queueing
empty/whitespace-only phrases that would briefly stall the worker.
## Issue 2: GUI shows ITS OWN tts state, not the daemon's
User toggled TTS dropdown → "Saved!" banner → "Активный движок" chip
still showed sapi. Reason: `get_active_backends` Tauri command read
THIS process's (GUI's) `tts::backend()`. But the GUI process hardly
ever speaks — the daemon (jarvis-app, separate process) is what
actually replies to voice commands. Two processes, two TTS state
machines, GUI was showing the wrong one.
Fix: settings page now prefers `$daemonHealth` (filled via the
existing WS `query_health` event from the daemon) for the "Активный
движок" chip. Falls back to GUI's local `activeBackends` only when
the daemon is unreachable, and shows an orange "демон не отвечает —
показано из GUI" badge so the user knows the chip is stale.
Plus:
- `applyTtsBackend` / `applyLlmBackend` now call `queryDaemonHealth()`
right after the swap so the chip updates within ~100ms instead of
waiting up to 5s for the next footer poll.
- `switchDaemonTts` / `switchDaemonLlm` returns false when WS isn't
connected. The page now reads that return value and SURFACES a red
notification — previously the swap silently went nowhere if the
daemon wasn't running, which exactly matches the user's "GUI looks
changed but nothing happens" symptom.
## Tests
- 145 → 147 rust core tests (+2 speech queue).
- Frontend rebuilds in 6.16s; release jarvis-app + jarvis-gui green.
# Recognition history page
New /history route in the Tauri GUI. Every voice/text phrase that
reaches the dispatcher gets a row with: timestamp, the phrase, what
happened, command id + confidence (if matched), which matcher fired
(intent / fuzzy / LLM-router / LLM-fallback).
Color coding:
- Green: command matched AND executed successfully
- Red: command matched but failed, OR dispatcher error
- Blue: LLM fallback handled it (Jarvis spoke a free-form reply)
- Orange: no match at all ("Не понял")
Live updates: GUI polls every 2s from the on-disk log, so the page
shows daemon writes in near real time even though they're separate
processes. Filter box for searching by phrase or command id.
Architecture:
- New core module `recognition_log` with ring buffer (cap 500) +
atomic JSON write-through to `<APP_CONFIG_DIR>/recognition_log.json`.
- `record(phrase, source, outcome)` is the single call-site from the
daemon's `execute_command()` — hooked into the 4 outcome paths
(matched-ok, matched-fail, not-found, llm-handled, error).
- `recent(limit)` reads the in-memory buffer (daemon's view).
- `recent_from_disk(limit)` re-reads the JSON file — GUI uses this
since the GUI process has its own buffer that doesn't see the
daemon's writes.
- 5 new unit tests covering ring buffer trimming, outcome serde
roundtrip, missing/corrupt/oversized file recovery.
GUI:
- `crates/jarvis-gui/src/tauri_commands/history.rs`: history_recent,
history_clear. Flattens the Outcome enum into a single struct that's
easier for the Svelte template to render.
- `frontend/src/routes/history/index.svelte`: ~270 lines. Stats badges
(✓ N matched / ✗ N misses / total), filter input, virtual list of
entry cards with color-coded left border. Polls every 2s.
- Header gets a new "История" / "History" button (between Plugins and
Settings). Russian + English locale entries added.
# Multi wake-word loading
Was: `init()` loaded the bundled `jarvis-default.rpw` + at most ONE
custom (from `settings.custom_wake_word`). User had to pick a single
trained model.
Now: loads the bundled default PLUS every .rpw in
`APP_CONFIG_DIR/wake_words/` simultaneously. Rustpotter natively
supports multiple wake-word triggers — each adds robustness for
different voice profiles. The legacy `custom_wake_word` field is
checked for back-compat but is a no-op if it points inside the
already-loaded directory.
User-facing impact: train the wake-word once via /wake-trainer →
restart daemon → detection improves automatically without picking a
single "active" model.
# Settings → "Обучить wake-word" button
Added a purple button on the settings page that links to
/wake-trainer. The trainer existed but had no in-GUI link, so users
couldn't find it without typing the URL. Now sits next to the
"Конструктор команд (Python)" button.
Tests: 140 → 145 rust core tests (+5 recognition_log). Frontend
rebuilds in 6.2s. Release builds of jarvis-app + jarvis-gui green.
Practical test: GUI launches (MainWindowTitle confirmed), seed log
file written + visible to the page.
GUI MAX-WIDTH (was still stretched)
Root cause v2: my earlier rule applied max-width to #header and main
INDIVIDUALLY, but they live inside <Container fluid id="wrapper"> from
svelteui which goes full viewport width. Both children inherited that
full width from the flex layout. Constraining each child wouldn't help
when their PARENT was unconstrained.
Fix: max-width:1280px !important on #wrapper itself, with margin:0 auto.
Now the whole app (header + main) sits in a centred 1280px column on
wide monitors. !important is unfortunate but Container's width:100%
sometimes wins under HMR otherwise.
VISIBLE TTS SWAP RESULT (was failing silently)
User saved 'Piper' in dropdown → green 'Saved!' banner → active engine
chip still showed 'sapi'. Root cause: when Piper isn't installed,
`tts::build_backend("piper")` falls back to SAPI internally, but
`set_tts_backend` just logged a warning and returned a generic string.
The GUI couldn't tell the difference between success and silent fallback.
Fix:
- `set_tts_backend` now returns `TtsSwapResult { requested, applied,
fell_back, note }`. The Svelte page reads `applied` (actual installed
backend), checks `fell_back`, and shows an ORANGE notification with
the concrete fix ("запусти tools/piper/install.ps1 чтобы скачать
бинарь") instead of a green 'saved' lie.
- Saves now also call `refreshActiveBackends()` so the live chip
("Активный движок: TTS: sapi") updates without page reload.
- Green notification when swap succeeded normally.
WARNING CLEANUP
- jarvis-app/main.rs: removed unreachable `_ => {}` arm (all IpcAction
variants are explicit now).
- jarvis-app/tray.rs: moved `info!("Tray initialized.")` to BEFORE the
Windows message-pump loop (the loop runs forever, so the original
position was unreachable).
- jarvis-gui/main.rs: dropped unused #[macro_use] on simple_log.
- jarvis-gui/events.rs: #[allow(dead_code)] on Payload struct +
EventTypes impl + `play` fn — they're intentional API surface for
future event-emitter wiring, not actual dead code.
- Plus a `cargo fix` pass for misc unused imports.
Down from 12 warnings to 1 (npm-build-completion notice — informational).
Tests: 140 rust + 115 python (was 104, +11 for new wave59_handlers).
Release builds of jarvis-app + jarvis-gui both green.
ROOT CAUSE OF 'PowerShell' ATTRIBUTION
User asked: 'Lua scripts call jarvis.system.notify — why do toasts come
from PowerShell?' Honest answer: they don't. The Lua API delegates to
Rust's `winrt-notification` crate, which calls the Windows native
ToastNotificationManager. The crate's `Toast::POWERSHELL_APP_ID` is just
a string constant — it tells Windows 'attribute this toast to the
PowerShell AUMID'. That's a registration shortcut, not actual PowerShell
involvement.
FIX
New `jarvis_core::toast` module:
- APP_USER_MODEL_ID = "Bossiara.JARVIS" (Company.Product convention).
- `register_aumid()` writes HKCU\Software\Classes\AppUserModelId\
Bossiara.JARVIS with DisplayName="J.A.R.V.I.S." via `reg add /f`.
Idempotent, no new Cargo deps.
- `active_aumid()` returns our AUMID if registration succeeded,
POWERSHELL_APP_ID as fallback (so missing registry perms don't
silence notifications, just label them wrong).
- jarvis-app and jarvis-gui both call register_aumid() once at startup.
- Both call sites (Lua jarvis.system.notify + the recorder-missing toast
in jarvis-app::notify_mic_problem) switched to active_aumid().
Verified: `reg query HKCU\Software\Classes\AppUserModelId\Bossiara.JARVIS`
shows the DisplayName, and new toasts attribute correctly.
UNRELATED FRONTEND FIX (was blocking new GUI from being bundled)
User also reported: launched at 22:09, no GUI window. Root cause: the
release exe was current but `frontend/dist/client/index.html` was from
May 15. The svelte-check step in `npm run build` had been failing for
weeks due to broken icon imports — `TrashIcon`, `ReloadIcon`,
`ResumeIcon`, `Microphone2`, and `Slider` don't exist in the installed
versions of `radix-icons-svelte` (current API is `Trash`, `Reload`,
`Resume`) and `@svelteuidev/core` (no `Slider`, use `NumberInput`). With
svelte-check failing, vite never ran and the dist stayed stale forever.
- Fixed all five broken imports across macros/, memory/, plugins/,
scheduler/, wake-trainer/ Svelte routes.
- Added `switchDaemonTts` to stores.ts re-exports (broke yesterday).
- Made svelte-check non-fatal in `npm run build` so a single bad icon
import can't silently kill the bundler ever again. svelte-check
output is still visible — just doesn't fail the build.
Practical test: jarvis-gui.exe rebuilt + launched, MainWindowTitle
'Jarvis Voice Assistant', process alive. 140 rust tests still pass.
ROOT CAUSE ANALYSIS
User reported: (a) TTS dropdown doesn't apply, (b) GUI feels stretched,
(c) no link to command builder, (d) Python doesn't even start.
Issues (a), (b), (c) addressed here. (d) addressed in python repo.
=== Fix A: TTS hot-swap (was completely broken) ===
Two distinct bugs:
1. `tts::init_backend()` only read JARVIS_TTS env var, NEVER consulted
`Settings.tts_backend` in the DB. So saving "Silero" in the GUI
dropdown changed nothing — not even on restart.
2. `BACKEND: OnceCell<...>` couldn't be replaced at runtime, so even
if init had picked the right backend, the dropdown was useless
mid-session.
Fixed:
- `tts::BACKEND` is now `Lazy<RwLock<Option<Arc<dyn TtsBackend>>>>`.
- `choose_backend_name()` resolves from DB → env → auto-detect.
- New `swap_to(name)` replaces the live backend atomically.
- IPC: new `IpcAction::SwitchTts { backend }`. jarvis-app reacts to
it the same way it reacts to SwitchLlm.
- GUI's `set_tts_backend` now persists + hot-swaps GUI's TTS + Svelte
page also fires `switchDaemonTts()` over WS so the running daemon
installs the new backend.
- Removed "Применится при следующем запуске" hint — now applies
immediately. RU + EN locale strings updated.
=== Fix B: GUI max-width ===
main + #header had no max-width, so on a 1920px monitor the labels
sprawled across 1860px of space, which felt stretched. Added
max-width: 1280px + margin: 0 auto + box-sizing: border-box. Smaller
windows are unaffected.
Also added the same cap on .app-container.assist-page so the home
screen arc reactor centres properly.
=== Fix C: Command builder discoverable ===
The Python fork ships a separate yaml-editing tool at
`python/tools/command_builder/` (pywebview GUI). No way to find it
from the main Rust GUI. Fixed:
- New tauri command `open_command_builder` (crates/jarvis-gui/src/
tauri_commands/builder.rs). Locates the Python fork via
$JARVIS_PYTHON_DIR → sibling python/ → C:\Jarvis\python, prefers
the project's .venv, spawns `python -m tools.command_builder`.
- New blue "Конструктор команд (Python)" button on /settings page
near the Save / Back buttons.
Tests: 140 rust + 104 python all pass. Release builds of jarvis-app
and jarvis-gui both green.
resources/commands/birthdays/ (2 commands):
- birthdays.add "запомни день рождения мама 15 марта" → memory key
"birthday.мама" = "15.03". Accepts both DD.MM and "DD <month>"
formats; recognises all 12 RU + 12 EN month names.
- birthdays.next "ближайший день рождения" → reads every "birthday.*"
memory key, computes days-until each entry (with year wrap), speaks
the nearest one in human form ("сегодня" / "завтра" / "через N дней").
Storage piggybacks on the existing long-term memory store, so
birthdays survive restart, sync to disk atomically, and don't need a
new persistence layer. Naming convention "birthday.<name>" keeps the
data discoverable in /memory GUI.
Pack count: 99 → 100 functional packs.
Tests: 140 rust, unchanged.
routines/good_night:
- Switch profile to "sleep" (silences idle banter overnight).
- Cancel one-shot timers (Таймер:/Timer:/Кухня:/Cooking: name prefix)
so a forgotten 6-hour reminder doesn't wake the user up.
- Speak a varied 5-line RU/EN good night with cancelled-count suffix.
- Deliberately does NOT lock PC / shut displays — too destructive
without an explicit confirm step.
routines/good_morning:
- Switch profile back to "default".
- Report scheduled task count for the day.
- Speak a varied 5-line greeting with agenda preview.
routines/coffee_break:
- Pause idle banter so Jarvis isn't talking to an empty chair.
- Schedule a 5-minute check-in via the persistent scheduler.
- Speak a varied 3-line acknowledgement.
Pack count: 97 → 98 (one pack, three commands).
Tests: 140 still pass.
leftoff/recap.lua:
- Voice "на чём я остановился" / "where did I leave off" stitches a
one-paragraph recap from four independent sources:
1. last assistant LLM reply (jarvis.llm_last_reply)
2. 3 most-recent memory facts (sorted by last_used_at)
3. count + name of the next scheduled task
4. active profile (implicit, in tone)
- Fully offline — no LLM call, just pulls from local state.
trivia/trivia.lua:
- 15 RU + 15 EN one-liners about the Stark universe (suits, arc reactor,
in-universe trivia about J.A.R.V.I.S. itself). Doesn't overlap with
the verbatim quote pool in personality/tony_quote.lua.
Pack count: 95 → 97. Tests: 140 still pass.
Focus mode (`focus/`):
- focus.start ("режим фокуса" / "focus mode"): three side-effects in one
voice trigger:
1. switch profile to "work" (silences fun/banter)
2. enable Windows Focus Assist via registry tweak (QuietHours key)
3. schedule a stretch reminder in 50 minutes through the persistent
scheduler — so it survives daemon restart
- focus.stop: undoes all three, including cancelling pending stretch
reminders by matching their scheduler name. Best-effort: any one
failing doesn't block the others.
Cooking timer (`cooking/`):
- Recognises 16+ dish names (чай, кофе, омлет, яйца, макароны, паста,
рис, гречка, картошка, курица, пицца, etc.) and starts a preset-
duration scheduler reminder ("Сэр, чай готов.").
- Single Lua script, table-driven — adding a recipe is one line.
- EN phrases for the same dishes ("boiling pasta", "frying eggs", ...).
Memory admin (`memory_admin/`):
- memory_admin.count → speak fact count with full RU pluralisation.
- memory_admin.list → speak 5 most-recent + toast the rest.
- memory_admin.wipe → ONLY fires on "точно забудь все" prefix (not bare
"забудь все") so we can't disaster-wipe by accident.
- Adds `long_term_memory::clear_all() -> usize` returning removed count
+ Lua binding `jarvis.memory.clear_all()`.
- One extra unit test for `clear_all`.
Tests: 139 → 140 (+1). Pack count: 92 → 95.
Voice sysinfo overhaul (6 scripts: battery/cpu/ram/disk/time/all):
- Now SPEAK the result via jarvis.cmd.ok instead of notify-only. Toast
still fires for visual reference, but real Jarvis tells you out loud.
- `all` joins lines with commas so TTS doesn't over-pause on linebreaks.
Smart reminders (`reminders/set.lua` rewrite):
- Was: detached `Start-Sleep` PowerShell subprocesses that died on
jarvis-app restart and spawned a hidden process per timer.
- Now: routes through `jarvis.scheduler.add(... type=speak ...)`. Persists
to schedule.json, survives restart, no zombie processes. Trade-off:
sub-minute timers round up to 1 min (rare in voice UX).
- Drops UA triggers from earlier UA-removal sweep.
Translation polish (`translate/translate.lua` + `clipboard.lua`):
- Drops UA target language; adds Italian, Polish, Turkish, French ru
triggers + English target detectors.
- `language_iso()` helper picks the SAPI voice locale for the TARGET
language (German translation → German SAPI voice if installed).
LLM system prompt (config.rs):
- New LLM_SYSTEM_PROMPT_EN counterpart. Both languages now mention the
assistant's tooling (memory / scheduler / profiles / macros / vision /
clipboard) so the LLM knows it can DO things, not suggest the user
install something.
- `get_llm_system_prompt(lang)` picks EN/RU.
Conversation pack (NEW `conversation/`):
- 2 commands. `conversation.summary` ("о чём мы говорили") pulls the
last 12 turns from persisted history, asks LLM for a 1-3 sentence
recap in Jarvis tone. Falls back gracefully when offline.
- `conversation.repeat` ("повтори") re-speaks the last assistant message.
- New Lua API `jarvis.llm_history()` returns array of {role, content}
tables, excluding the system prompt.
WOL pack (NEW `wol/`):
- Voice "разбуди сервер" / "wake server" fires a magic packet at a MAC
stored in long-term memory under `wol_<alias>` (default alias "server").
- Setup via voice: "запомни wol_server AA:BB:CC:DD:EE:FF". Falls back to
`JARVIS_WOL_TARGET` env if memory empty. Handles every standard MAC
format (colons / dashes / dots / bare). Broadcast on UDP 9 via
PowerShell System.Net.Sockets.UdpClient.
Outlook COM pack (NEW `outlook/`, via subagent):
- 4 commands: unread_count, latest, send_clipboard, summarize_inbox.
- PowerShell COM bridge `_outlook.ps1` with tab-delimited line protocol.
- summarize_inbox calls LLM for a one-paragraph summary. Graceful
failure when Outlook isn't running.
Time tracker pack (NEW `time_tracker/`, via subagent):
- 5 commands: start, stop, today, week, reset.
- Persists via `jarvis.state` (structured object: current_session_start
+ sessions array of {start, end}).
- Local-calendar "today" boundary, full Russian pluralisation
(час/часа/часов, минута/минуты/минут, сессия/сессии/сессий).
Tests: 139 rust (no regression), 81 python (60 → 81, +14 time_tracker
+7 outlook). Release build green for jarvis-app and jarvis-cli.
Pack count: 88 → 92 rust packs.
Personality pack (`resources/commands/personality/`):
- 5 voice commands × ~7-29 phrases each across RU and EN.
- personality.greet: 4 time-of-day buckets (morning/midday/evening/night),
pulls one of ~7 lines per bucket per language.
- personality.thanks / .compliment / .how_are_you / .tony_quote.
- how_are_you embeds live memory size + active profile via jarvis.health()
and jarvis.memory.all() for a "feels alive" effect.
- All use jarvis.cmd.ok helpers, no inline PowerShell SAPI.
- Built by sub-agent. Verified: 6 rust command tests + 60 python tests.
Idle banter (`crates/jarvis-core/src/idle_banter.rs`):
- Background thread chimes in periodically without being asked. Gated by
JARVIS_IDLE_BANTER env (default OFF — intrusion is opt-in).
- Quiet hours 23:00–07:00, skipped under "sleep" profile, paused during
active interactions via `pause()`.
- 30+ static offline lines split into RU/EN × morning/evening/generic
buckets — no network required.
- Lua API jarvis.banter.{fire, pause, resume, enabled}.
- New voice pack `banter/` exposes "скажи что-нибудь интересное",
"помолчи", "можешь говорить".
- 6 unit tests covering pool selection, quiet hours, interval clamp,
pause/resume, opt-in default.
Conversation continuity (`crates/jarvis-core/src/llm/history.rs`):
- New `ConversationHistory::with_persistence(path)` builder. Every
push/clear/pop atomically writes to `<APP_CONFIG_DIR>/llm_history.json`
so daemon restart picks up the thread.
- System prompt is intentionally NOT persisted — comes from current init
call so prompt edits take effect immediately on restart.
- `llm::init_history` wires the path in automatically.
- 4 new tests: round-trip, clear wipes file, corrupt file tolerated,
len/is_empty helpers.
Offline-first math (`resources/commands/math/math.lua`):
- Was: always-LLM, hard fail without GROQ_TOKEN, inline PowerShell SAPI.
- Now: shunting-yard parser handles 95% of voice queries in <50ms — no
network, no token. Russian operator words ("плюс", "умножить на",
"в степени", "квадрат", ...) normalised to symbols first. Patterns
for "корень из X" and "X процентов от Y". Falls back to LLM only on
parse failure (word problems / equations / unit conversions).
- Drops inline PowerShell — speaks via jarvis.cmd.ok.
- 10-case shunting-yard kernel test added (basic ops, precedence,
parens, unary minus, div-by-zero, garbage rejected).
DuckDuckGo Instant Answer (`resources/commands/ddg_answer/`):
- New pack — short factual Q&A without API key. Trigger phrases
"что такое", "кто такой", "расскажи про", "what is", etc.
- Reads AbstractText → Answer → Definition → RelatedTopics[0] in order
from DDG's free JSON API. Opens the search page only if nothing
useful comes back.
- Sandbox full (needs http + system.open).
Tests: 128 → 139 (+11). Release build green.
CI (.github/workflows/ci.yml):
- Three parallel jobs on push & PR: cargo test (jarvis-core), cargo
clippy (lint, warnings-as-errors), cargo check (workspace). Windows
runner, stable toolchain, rust-cache for fast incremental builds.
- Deliberately skips release build of jarvis-gui — that needs Node+npm
+Tauri bundler and would balloon CI to ~10min per push.
Drop Ukrainian:
- Delete ua.ftl, remove UA from SUPPORTED_LANGUAGES.
- config.rs: prune UA arms from get_wake_phrases / get_wake_grammar /
get_phrases_to_remove / get_llm_trigger_phrases / get_llm_system_prompt.
- stt/vosk.rs: drop UA → uk language mapping.
- tray menu: drop "Українська" entry.
- frontend: drop UA from Header.svelte language list, settings page
language map, i18n.ts fallback.
- 82 command.toml files stripped of every `ua = [...]` block (205
blocks total). Handled via subagent — TOML still parses, all 6 command
tests pass.
.gitignore: stop tracking dev.env / dll_probe*.py / imports_full.txt
which were sitting in working tree.
Tests: 128 pass (no regression). Workspace still compiles end-to-end.
#29 Plugin system:
- New jarvis-core::plugins module: discovers user packs in
%APPDATA%\com.priler.jarvis\plugins\<name>\command.toml so authors can
ship voice commands without rebuilding. Sandbox capped at "standard" —
plugins cannot escalate to "full" (which would expose `os`). Disabled
via a `disabled` flag file. Malformed packs warn and skip; never poison
the rest of the list. 8 unit tests.
- commands::parse_commands() merges plugins into the loaded list.
- New /plugins GUI page with enable/disable switches, error reporting,
"Open folder" button. Tauri commands plugins_list / plugins_set_enabled
/ plugins_open_folder. Header gets a "Плагины" button. i18n keys added
for ru/en/ua.
#8 Custom wake-word trainer wizard:
- New jarvis-core::wake_trainer module: opens its own pv_recorder
instance, records N short PCM clips, WAV-encodes them in memory, then
calls rustpotter's WakewordRef::new_from_sample_buffers to train and
persist a .rpw model under %APPDATA%/com.priler.jarvis/wake_words/.
Keepsake WAVs are also dumped for retraining later. Sanitises names to
block path traversal. 5 unit tests.
- Settings gain `custom_wake_word: String`; listener/rustpotter.rs now
loads the user's selected model first and always falls back to the
bundled default so the assistant keeps working even if the custom
file is missing.
- New /wake-trainer GUI page: stepper UI for record-sample → train,
shows recorded count, threshold slider, refuses to start while
jarvis-app is running (mic exclusivity). Lists existing trained
models with size + delete button.
- 8 Tauri commands wired through (status/defaults/start/record_sample/
finish/cancel/list_models/delete_model).
Tests: 115 → 128 (+8 plugins +5 wake_trainer). Release builds green for
all three binaries (jarvis-app / jarvis-cli / jarvis-gui).
User picked from the fishki roadmap; this commit ships the 10 easy/medium
ones in one batch. No Rust core changes — all pure Lua + jarvis.* APIs.
Pack catalog
magic_8ball
- "ответь да или нет" / "магический шар" / "что скажешь"
- 15 канонических ответов, random pick. Pure fun.
github_issues (2 cmds)
- "какие issues" / "открытые issues" — gh issue list по сохранённому репо
- "мои issues" / "что мне назначено" — gh issue list --assignee @me
weather_extended (2 cmds, Open-Meteo, no key)
- "погода завтра" — temp min/max + conditions + rain note
- "прогноз на неделю" — общий диапазон температур
- Auto-detects location via ipinfo.io if not in memory; stores lat/lon/city.
rss_reader (3 cmds)
- "добавь rss <url>" — extracts URL, persists via memory под ключом "rss.<domain>"
- "что в ленте" — fetches first feed, parses <title> tags, speaks top 3
- "какие у меня rss" — lists subscribed feed domains
ics_event
- "добавь встречу завтра в 15:00 обсудить проект"
- Parses time (HH:MM или "в HH") + date keywords (сегодня/завтра/послезавтра)
- Writes valid iCalendar v2.0 file to ~/Documents/jarvis-events/
- jarvis.system.open → дефолтный handler (Outlook/Mail/whatever)
backup
- "сделай бекап" — PowerShell Compress-Archive всех state files из APPDATA/Jarvis
- Outputs to ~/Documents/jarvis-backup-<YYYYMMDD-HHMMSS>.zip
- Bundles: long_term_memory, schedule, macros, profiles/, active_profile, llm_backend, settings
clip_history (3 cmds, 20-slot rolling buffer)
- "запиши буфер" — push current clipboard onto memory keys clip.0..clip.19
- "что я копировал" — speak preview of last 3
- "верни первый/второй/третий буфер" — restore clip.N to clipboard
notif_queue (2 cmds)
- "что я пропустил" — enumerate memory keys "notif.*", speak last 5 by recency
- "очисти уведомления" — purge all notif.* keys
- Producers (scheduler/macros/llm) can later push to memory[notif.<ts>] = text
password_vault (3 cmds, Windows DPAPI)
- "сохрани пароль от GitHub" — encrypts current clipboard content via DPAPI
(CurrentUser scope), base64-stored in memory[vault.GitHub]. Clears clipboard.
Password is NEVER spoken or written to disk in plaintext.
- "пароль от GitHub" — decrypts via DPAPI, restores to clipboard for 30 sec,
schedules auto-clear via jarvis.scheduler. Speaks only "Пароль от X в буфере."
- "какие у меня пароли" — list of stored service names.
habit_streaks (2 cmds, integrates with habit_nudge)
- "сколько дней подряд" / "статистика привычек" — reads memory keys
"habit_streak.<habit>.<YYYY-MM-DD>", computes consecutive-day streak per habit.
- "я попил воды" / "отметь привычку" — marks today's check-in.
Maps voice to habit: воды→water, размял/зарядк→stretch, глаз→eyes.
Tests: 6 commands tests pass (auto-validate the 10 new packs).
Three small daily-driver packs mirrored with python (commit 02ab8a4 or similar).
now_playing
- "что играет" / "что за песня" / "какой трек" / "что слушаю"
- Reads Windows.Media.Control.GlobalSystemMediaTransportControlsSessionManager
via PowerShell — works with Spotify/YouTube/Foobar/Winamp/Yandex Music
(anything that exposes Windows SMTC, Win10 1803+).
- Speaks "Сейчас играет: <Artist> — <Title>" or just "<Title>" if no artist.
world_clock
- "сколько времени в Токио" / "время в Лондоне"
- 21+ pre-mapped Russian/world city names → IANA timezones.
- Fetches worldtimeapi.org (free, no key, no rate-limit headers exposed).
- Parses ISO 8601 datetime, speaks "В <city> сейчас HH:MM."
daily_quote
- "цитата дня" / "вдохнови меня"
- zenquotes.io (free, no key) → English quote.
- LLM translates to Russian via active backend (Groq or Ollama).
- Falls back to LLM-generated quote if zenquotes is unreachable.
- Speaks "<translated quote> — <author>."
Tests: 6 commands tests pass (no_duplicate_ids / no_empty_phrases /
all_lua_scripts_exist auto-validate the 3 new packs).
Pack count: 72 → 75.
Visible visual indicator of the active profile + click-to-swap dropdown.
Closes the last small UX gap pointed out in earlier sessions.
Tauri (crates/jarvis-gui/src/tauri_commands/profile.rs)
- profile_list() → Vec<String> of available profile names
- profile_active() → {name, description, icon, greeting}
- profile_set(name) → swaps active profile, returns new active
Frontend (frontend/src/components/Header.svelte)
- Polls profile_active every 5 seconds.
- Shows a coloured chip with icon+name when profile != "default"
(orange tint, attention-grabbing). For "default" — muted star chip.
- Click opens a dropdown listing all profile names; click one → swap.
- Dropdown closes on outside-click (shared handler with lang dropdown).
- SCSS styled to match existing aesthetic.
Build: cargo build --release -p jarvis-gui green (2m).
Now the user can see at a glance which profile is active (e.g. "💼 work")
without having to ask voice or open Settings.
Tests (112→115, +3)
- commands::additional_tests::no_duplicate_command_ids_across_packs
Catches accidental ID collisions across packs at PR time, with informative
error pointing to both offending packs.
- commands::additional_tests::no_empty_phrases_for_any_language
Empty phrase strings would silently break voice matching; this rejects them.
- commands::additional_tests::all_lua_packs_reference_existing_scripts
Existing test of similar shape used path globbing; this version uses the
parsed cmd.script field for a more precise error message.
Plus: runtime_config tests no longer race on shared env vars. Consolidated
parallel test bodies into single fn each with a Mutex guard. 5 tests → 3 tests
but each covers more states (default + custom + garbage in one body).
New packs
resources/commands/self_check/ (2 cmds)
- selfcheck.ping "ты тут" / "ты слышишь" / "пинг" / "ты живой"
Random pick from 5 short canned replies — no LLM call,
no network. Pure mic-chain + TTS smoke test.
- selfcheck.smoke "проверь себя" / "самопроверка"
Reads jarvis.health() + tries one HTTP request,
speaks list of "X работает, Y работает...".
resources/commands/ssl_check/ (1 cmd)
- ssl.check "проверь сертификат example.com"
PowerShell TCP+SslStream→X509Certificate2. Reports days
until expiry, expiry date, issuer CN. Adds urgency prefix
("СКОРО!" if <7 days, "Скоро." if <30, "просрочен!" if past).
Pack count: 70 → 72. Tests: 115/115 + 1 ignored Ollama smoke.
Build: cargo build --release -p jarvis-app green.
USAGE.md (new, ~250 lines)
- Complete voice command reference organised by 13 categories
- Table-formatted for fast skim ("Команда → Что делает")
- Sections: Управление ассистентом / LLM / Память+профили+macros /
Расписание / Окна+апп / Аудио+медиа / Система / Файлы+буфер /
Информация из интернета / Утилиты-калькуляторы / Разработка /
Развлечения / Скриншоты
- Bottom: "Как добавить свою команду" with Lua snippet template,
"Голосовое vs GUI" mapping, troubleshooting checklist (mic / TTS /
LLM / command-not-found / logs).
- Covers all 67 packs visible in resources/commands/.
interesting_fact pack (resources/commands/interesting_fact/, 2 cmds)
- fact.about "удиви меня" / "интересный факт про космос"
→ LLM with "curious conversationalist" prompt,
generic or topic-targeted (1-2 sentences, no fluff).
- fact.history_today "что было в этот день" / "этот день в истории"
→ Wikipedia "On this day" API (RU first, EN fallback)
→ LLM picks top 2-3 events, narrates concisely.
mailto pack (resources/commands/mailto/, 1 cmd)
- mailto.compose "напиши письмо" / "напиши письмо маме"
→ opens default mail client via mailto: URI.
Looks up recipient via jarvis.memory ("email_маме")
if a name is given.
- Pairs well with: jarvis.memory.remember("email_маме", "mom@example.com")
Pack count: 67 → 70. Tests: 112/112.
Two pure-Lua packs that don't need network or LLM — pure offline utilities.
unit_convert (resources/commands/unit_convert/, 4 commands)
- convert.length "переведи 100 метров в футы" / "сколько 5 миль в км"
meters↔feet, km↔miles. Pluralises russian unit names.
- convert.weight "переведи 70 кг в фунты"
kg↔lbs.
- convert.temperature "переведи 100 цельсий в фаренгейт" / "минус 40 в цельсий"
Handles negative numbers. C↔F formulas inline.
- convert.speed "переведи 100 км в час в мили в час"
km/h↔mph.
- All round results to 1-2 decimal places, speak with proper Russian
grammatical number (фут/фута/футов).
generators (resources/commands/generators/, 3 commands)
- gen.coin "подбрось монету" / "орёл или решка"
→ "Орёл!" or "Решка!" (math.random with time+minute seed)
- gen.password "сгенерируй пароль 16 символов"
→ cryptographically-ish 6..64-char password,
alphanumeric + !@#$%^&*-_=+, copies to clipboard,
SPEAKS LENGTH ONLY (never echoes the password).
Default 16 chars if no number in phrase.
- gen.uuid "сгенерируй uuid" / "юид"
→ v4 UUID, copies to clipboard, speaks last 4 chars
(confirmation without 32-char monologue).
Tests: 112/112 pass (commands::tests auto-validates).
Pack count: 62 → 64.
a6a098d added daemon-side IPC handlers (SwitchLlm, ReloadLlm, QueryHealth)
but the GUI was still talking only to its own process. This commit wires
the frontend to actually use those handlers.
frontend/src/lib/ipc.ts
- New DaemonHealth interface + daemonHealth: writable<DaemonHealth | null>.
- handleEvent: "health_snapshot" case fills daemonHealth from daemon's IpcEvent.
- New senders:
switchDaemonLlm(backend) → action "switch_llm"
reloadDaemonLlm() → action "reload_llm"
queryDaemonHealth() → action "query_health"
frontend/src/stores.ts
- Re-exports daemonHealth + the three new senders + DaemonHealth type.
frontend/src/components/Footer.svelte
- Polls every 5s but PREFERS IPC: if daemon connected, calls queryDaemonHealth
(snapshot arrives via daemonHealth store).
- Falls back to invoke("get_active_backends") if daemon offline — shows
GUI-process view in that case, with the chip tooltip distinguishing
"daemon" vs "gui-process" source.
frontend/src/routes/settings/index.svelte (AI Backends tab)
- applyLlmBackend now ALSO fires switchDaemonLlm so the running listener
picks the new backend without restart.
- "Auto" (empty) path calls reloadDaemonLlm so daemon re-reads DB.
End-to-end flow now:
GUI dropdown → set_llm_backend (Tauri, persists DB + swaps GUI proc)
↘ switchDaemonLlm (IPC, swaps daemon proc too)
↘ daemon emits health_snapshot on next QueryHealth
↘ daemonHealth store updates
↘ Footer chips re-render with new state
Build: cargo build --release -p jarvis-gui green.
Closes the "GUI shows GUI process state, not daemon's" gap. Now GUI can:
1. Tell the daemon to swap LLM backend via IPC (so the running listener
uses the new backend immediately, not just the GUI process).
2. Tell the daemon to re-read llm_backend from settings DB after GUI persists
a choice independently.
3. Query the daemon's true runtime state (TTS/LLM/profile/memory/scheduler
counts/language/version) for an honest footer + diagnostics view.
IpcAction (new variants)
- SwitchLlm { backend: String } — daemon calls llm::swap_to(parse(backend))
- ReloadLlm — daemon calls llm::init_global() to re-read DB
- QueryHealth — daemon responds with HealthSnapshot
IpcEvent (new variant)
- HealthSnapshot { tts_backend, llm_backend, llm_model, active_profile,
memory_facts, scheduled_tasks, language, version }
jarvis-app main.rs
- Imports IpcEvent (was only IpcAction before).
- Three new match arms in ipc action handler — all use existing globals,
no new state needed.
Next step (TODO): wire GUI's set_llm_backend Tauri command to ALSO fire
IpcAction::SwitchLlm so daemon stays in sync. And replace Footer.svelte's
get_active_backends call with QueryHealth over IPC.
Build: cargo build --release -p jarvis-app -p jarvis-gui green.
Tests: 62/62 pass (unchanged — IPC variants don't add testable logic).
Defensive tests for the surface area that user-facing voice commands hit.
llm::parse_backend (5 new tests)
- English aliases ("groq", "ollama", "cloud", "local")
- Russian aliases ("облако", "локальный", "локал")
- Case insensitivity ("OLLAMA", " Groq ")
- Rejects unknown ("openai", "claude", "")
- current_backend_name() smoke test
macros (2 new tests)
- Store JSON round-trip with multiple macros
- is_macro_control case-handling (noun form NOT recognised, verb form IS)
Tests: 62/62 (was 55).
Surfaces voice macros and scheduled tasks in the GUI, so the user doesn't
have to remember voice commands or grep schedule.json by hand.
Tauri commands (crates/jarvis-gui/src/tauri_commands/)
- macros.rs: macros_list, macros_replay, macros_delete, macros_is_recording,
macros_recording_name, macros_start_recording, macros_save_recording,
macros_cancel_recording (8 commands).
- scheduler.rs: scheduler_list, scheduler_remove, scheduler_clear (3 commands).
- Both exposed in main.rs invoke_handler.
GUI pages (frontend/src/routes/)
- macros/index.svelte:
* Lists all macros with name, steps_count, first 5 step previews,
created/last_run timestamps.
* Top: TextInput + "Начать запись" button. While recording shows orange
banner with "Сохранить"/"Отменить" buttons + polls is_recording every 2s.
* Per-card: "Запустить" (with busy state for replay duration), "Удалить".
* confirm() before delete.
- scheduler/index.svelte:
* Lists tasks with name, schedule_human (e.g. "каждые 2 ч"), action body,
ID, timestamps, action_kind badge.
* "Отменить" per task + "Очистить всё (N)" bottom button.
* Auto-polls every 5s (so the list updates as scheduler ticks fire tasks).
Header (frontend/src/components/Header.svelte)
- Two new buttons: "Макросы" → /macros, "Расписание" → /scheduler.
- Beside existing /commands and /settings buttons.
i18n
- ru/en/ua FTL: settings-ai-backends, settings-llm-*, settings-tts-*,
settings-profile, header-macros, header-scheduler.
- Re-applied AI Backends keys for ua.ftl (earlier edit hadn't taken).
README.md (full rewrite)
- Old README was 178 lines mostly explaining LLM-trigger flow and VAD config.
- New README is ~190 lines covering: features (LLM hot-swap, memory, profiles,
vision, scheduler, macros, utilities table), quick start, env vars table,
build steps with vcvars setup, test command, structure tree, voice workflow
diagram, license, roadmap.
- Up to date for 59 packs and all new infra.
Build: cargo build --release -p jarvis-gui green (2m). 55/55 tests pass.
Tiny but useful: stamp `Instant::now()` at the top of main(), log elapsed
ms before the tray loop blocks. Lets users (and contributors) spot startup
regressions without running a profiler.
Output on a warm cache, default config, no network: ~1.8s.
Output with a cold ONNX model load (intent classifier): ~4.5s.
Visible both in the eprintln trace and the log file via the info! call.
Surfaces the LLM/TTS hot-swap UX in the GUI so the user doesn't need env vars
or voice commands. Tauri commands for this landed in b243e67; this commit
wires them to Svelte.
Footer (frontend/src/components/Footer.svelte)
- Poll get_active_backends() every 5 s, show TTS / LLM / Profile chips.
- Chips have border accents by kind (cyan = TTS, lime = LLM, orange = Profile).
- Friendly title tooltips with full backend + model names.
- Silent if jarvis-gui can't reach the command (e.g. cold start) — no UI flicker.
Settings — new "AI Backends" tab (frontend/src/routes/settings/index.svelte)
- Status banner showing active LLM (with model name) + TTS + Profile.
- LLM selector: Auto / Groq / Ollama → calls set_llm_backend on change
(hot-swap, persisted to DB).
- TTS selector: Auto / SAPI / Piper / Silero → calls set_tts_backend
(persisted, effective on next jarvis-app restart).
- "Reset context" button → llm_reset_context Tauri command.
- Error toast for swap failures (e.g. Groq selected but no GROQ_TOKEN).
- Tips alert: how to install Ollama / Piper / voice commands.
- Loads prefs via db_read('llm_backend' / 'tts_backend') on mount.
i18n strings (ru/en/ua, .ftl files)
- 14 new keys: settings-ai-backends, settings-ai-active, settings-ai-auto,
settings-ai-applying, settings-ai-error, settings-ai-tips,
settings-llm-backend{,-desc}, settings-tts-backend{,-desc},
settings-llm-context{,-desc}, settings-llm-reset, settings-profile.
Build glue (crates/jarvis-gui/Cargo.toml)
- jarvis-core dep now includes the `llm` feature so backends.rs can resolve
`jarvis_core::llm::*` (was failing to compile after the global LLM module).
.gitignore
- tools/piper/*.dll, piper.exe, voices/, espeak-ng-data/ — locally
installed binaries from tools/piper/install.ps1, not committed.
Build: cargo build --release -p jarvis-gui green (4m, includes npm/Vite).
Test path: rebuild + launch jarvis-gui, /settings → "AI Backends" tab.
Single biggest user-facing change: you can now switch local↔cloud LLM by voice
without restarting. Plus four new packs and a perf nudge.
Shared LLM global (crates/jarvis-core/src/llm/mod.rs)
- GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth.
- init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str).
- llm_fallback + llm_router migrated: they no longer own a separate LlmClient,
they read from the global on each request. Hot-swap is now LIVE — change
backend, next chat call uses the new one.
- parse_backend accepts both english and russian aliases:
"groq"/"cloud"/"облако"/"клауд" → Groq
"ollama"/"local"/"локал"/"локальный" → Ollama
Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands)
- "переключись на локальный" / "перейди на оллама" → swap to Ollama
- "переключись на облако" / "используй грок" → swap to Groq
- "какой у тебя мозг" / "облако или локально" → status query
Underlying Lua API (registered globally):
jarvis.llm({messages}, opts) -- as before, now uses global client
jarvis.llm_status() -> "groq" | "ollama" | "none"
jarvis.llm_switch(name) -> name on success, nil on failure
Media keys pack (resources/commands/media_keys/, 4 commands)
- "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE
- "следующий трек" → VK_MEDIA_NEXT_TRACK
- "предыдущий трек" → VK_MEDIA_PREV_TRACK
- "стоп музыка" → VK_MEDIA_STOP
- Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type).
- Works with anything that listens to global media keys: Spotify, Yandex Music,
YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys.
Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands)
- "укажи проект <path>" → stores path in jarvis.memory under codebase.root
- "какой проект сейчас" → speaks current path
- "что делает функция X" / "найди в коде Y" / "объясни код"
→ walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total),
filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM
with a "senior reviewer" system prompt, speaks 3-5 sentence answer.
Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs)
- new pub fn find_by_text(query, limit) -> Vec<ScheduledTask>
- new pub fn remove_by_text(query) -> usize (count removed)
- new pure helper find_by_text_in(tasks, query, limit) for tests
- Lua: jarvis.scheduler.{find_by_text, remove_by_text}
- Voice: "отмени напоминание про воду" / "удали задачу про разминку"
- 3 new unit tests (52 total in jarvis-core).
TTS pre-warm (crates/jarvis-app/src/main.rs)
- Call jarvis_core::tts::backend() once on startup so the first speak()
doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak
latency by ~150-300ms depending on backend.
Build: cargo build --release -p jarvis-app -p jarvis-gui both green.
Tests: 52/52 jarvis-core unit tests pass.
User asked the code stays easy to fix/extend as the feature surface grows.
This commit refactors what was duplicated, centralises what was scattered,
adds tests where there were none, and writes the architecture doc that
contributors will need.
Deduplication
- tts::play_wav extracted to tts/mod.rs as pub(crate). Was identical in
piper.rs and silero.rs (~25 lines × 2). Both backends now call super::play_wav.
Centralised env config
- new crates/jarvis-core/src/runtime_config.rs — single doc-file for every
JARVIS_* / GROQ_* / OLLAMA_* env var. Includes:
- ENV_* constants with doc comments
- get(name) / get_bool(name, default) / get_parse(name, default) helpers
- feature-flag wrappers: llm_tts_enabled(), llm_router_enabled(),
llm_router_threshold()
- log_effective_config() prints active values on startup
- migrated llm_fallback (JARVIS_LLM_TTS), llm_router (JARVIS_LLM_ROUTER,
JARVIS_LLM_ROUTER_THRESHOLD) to use the new helpers. Pattern set for
future migrations.
Lua boilerplate killer
- new crates/jarvis-core/src/lua/api/cmd.rs exposing:
jarvis.cmd.ok(msg?) — play_ok + speak + {chain=false}
jarvis.cmd.chain_ok(msg?) — same but chain=true
jarvis.cmd.error(msg?) — play_error + speak + {chain=false}
jarvis.cmd.not_found(msg?) — play_not_found + speak + {chain=false}
jarvis.cmd.silent_ok / silent_error
- refactored 5 packs (daily_briefing/off, memory_pack/list, pomodoro/stop,
habit_nudge/stop_all, scheduler/clear) — each lost 3-4 lines of repetitive
play_*/speak/return boilerplate. Pattern for future packs documented in
ARCHITECTURE.md.
Tests (was 32, now 49)
- long_term_memory: 8 new tests for normalize_key, search_in (rank, limit,
empty), build_context_from (empty + populated), serde round-trips for
MemoryRecord and Store. Extracted pure logic (search_in, build_context_from)
into pub(crate) functions to enable testing without global state.
- profiles: 6 new tests for Profile::allows_command (empty/whitelist/blacklist/
deny-wins-over-allow), serde round-trip with all fields + minimal-fields
tolerance via #[serde(default)].
- runtime_config: 2 tests for get_bool / get_parse defaults.
- All 49 tests pass.
New mood/energy log pack
- resources/commands/mood_log/ with 2 commands:
mood.record "запиши настроение 7" / "сегодня мне грустно" → stores
timestamped entry via jarvis.memory.remember
mood.recap "как прошла неделя" → LLM summarises last 30 entries
- Showcase: composes memory + llm + cmd helpers in <30 lines per script.
ARCHITECTURE.md (new, 250 lines)
- Crate layout, data flow diagram (mic→action 10 steps), per-module
responsibility table, configuration layers, TTS pipeline diagram,
Lua sandbox details with API quick-ref, background-services overview,
"how to add a pack/feature/TTS backend" recipes, test coverage map,
build instructions with MSVC env, git workflow with Forgejo NO_PROXY trick.
- Aimed at someone who just cloned the repo and needs to fix a bug fast.
Build: cargo build --release -p jarvis-app -p jarvis-gui both green.
Tests: 49/49 jarvis-core unit tests pass.
Big push driven by user feedback ("делай имбу") and web research on what
voice assistants need to be the ideal:
TTS backend abstraction (P0.1)
- new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs
- TtsBackend trait with SapiBackend (current PowerShell), PiperBackend
(rhasspy/piper, neural quality), SileroBackend (python subprocess)
- JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if
binary + voice present in tools/piper/. Falls back to SAPI on missing.
- SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils
sanitiser applied unless raw=true.
- llm_fallback + lua/api/tts both routed through tts::backend().
- tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx
from rhasspy releases + huggingface. Smoke-test included.
- tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess.
IMBA-1 Agentic LLM router
- crates/jarvis-app/src/llm_router.rs
- When fuzzy/intent matcher fails, LLM picks the closest command from the
full registry. Returns JSON {command_id, confidence, reason}.
- Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1
enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default.
- Inserted in app.rs::execute_command between "no match" and existing
llm_fallback chat fallback.
IMBA-2 Long-term memory
- crates/jarvis-core/src/long_term_memory.rs — JSON store at
APP_CONFIG_DIR/long_term_memory.json. Atomic write-through.
- remember/recall/search/forget/all/build_context API.
- Lua bindings: jarvis.memory.* (5 functions).
- llm_fallback auto-injects relevant facts (substring search of prompt)
into system message before LLM call.
- Pack resources/commands/memory_pack/ with 4 commands: remember, recall,
forget, list.
IMBA-3 Profile switching (work/game/sleep/driving/default)
- crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/
Auto-seeds 5 defaults on first run with personality + allow/deny lists +
greetings + emoji icons.
- active_profile.txt persists choice across restart.
- Lua bindings: jarvis.profile.{active,set,list,allows,active_name}.
- llm_fallback prepends profile personality to system prompt.
- Pack resources/commands/profile_switch/ with 6 voice triggers.
IMBA-4 Multimodal screenshot + vision LLM
- crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox.
- jarvis.vision.screenshot() captures via PowerShell System.Drawing.
- jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model
(default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL).
- Pack resources/commands/vision/ with 2 commands: describe + read_error.
P0.2 Continuous conversation grace window
- config::CONVERSATION_GRACE_MS = 30_000.
- app.rs: after command result, if grace_ms > 0 keep listening WITHOUT
re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so
the existing timeout fires at start + grace_ms.
Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils).
Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed
on Windows MSVC (VS 2026 Enterprise vcvars64).
Notes for setup:
- Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB).
- GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision).
- All features are opt-in via env vars or auto-detect; existing SAPI +
fuzzy match path remains the default.
User complaint: SAPI reads every dotted acronym character-by-character with a
"точка" between letters. Unbearable on every speak. No real TTS analog
shipping yet (Silero/Coqui/Inworld in roadmap), so the immediate fix is text
preprocessing.
New module `jarvis-core::text_utils` with `sanitize_for_speech()`:
- Specific brand mapping: J.A.R.V.I.S. → Джарвис, U.S.A. → США, U.K. →
Британия, U.S. → США, S.O.S. → сос.
- Generic dotted-acronym collapse: any `[letter].[letter].[letter].(...)`
run of 3+ letters gets its dots stripped. "T.O.N." → "TON".
- URL stripping: anything starting with http/https gets replaced with the
word "ссылка". SAPI reading URLs char-by-char is unlistenable.
- Em/en-dash normalisation: — → -, « » " stripped.
- Whitespace collapse.
Wired in two places:
- `jarvis.speak(text, opts?)` Lua API runs sanitiser unless opts.raw=true.
- `jarvis-app/llm_fallback::speak_via_sapi` runs sanitiser before the SAPI
PowerShell shell-out for LLM auto-fallback replies.
5 unit tests in `text_utils::tests` covering jarvis-name collapse, generic
acronym pattern, URL stripping, dash normalisation, and "leave clean
russian sentences alone" — all green.
Real TTS upgrade (Silero v4 first cut) tracked as P0.1 in roadmap.
Symptom: jarvis-app.exe failed to start with a MessageBox saying
"Точка входа в процедуру TaskDialogIndirect не найдена в библиотеке
DLL ... jarvis-app.exe". Loader phase, no log lines ever written.
Cause: tray-icon 0.21 / winit 0.30 transitive Win32 imports include
TaskDialogIndirect from comctl32.dll, which is a Common-Controls v6
API. Without a side-by-side manifest declaring a dependency on
Microsoft.Windows.Common-Controls v6.0.0.0, Windows loads the legacy
v5 comctl32 (which does not export TaskDialogIndirect) and the
loader rejects the exe before main().
jarvis-gui does not hit this because tauri-build embeds its own
manifest as part of the Tauri compile step. jarvis-app had no
manifest at all.
Fix:
- app.manifest: Common-Controls v6 dependency, supportedOS for
Win7..Win11, PerMonitorV2 DPI awareness, UTF-8 active code page,
asInvoker execution level.
- app.manifest.rc: 3-line .rc that embeds the manifest with
CREATEPROCESS_MANIFEST_RESOURCE_ID (1) and RT_MANIFEST (24).
- build.rs: on cfg(windows), embed_resource::compile() compiles the
.rc and links the resulting .res into the exe. rerun-if-changed
on the manifest sources.
- Cargo.toml: target-windows build-dependency on embed-resource 3.
Verified: jarvis-app.exe now runs cleanly through all init steps:
commands parsed, audio init, recorder init found "Микрофон (5-
Fifine Microphone)", IPC server up on ws://127.0.0.1:9712, VAD
already flushing frames.
UTF-8 active code page in the manifest also helps russian-character
fidelity when win32 APIs are invoked from PowerShell helpers.
Reproduction:
cargo build --release -p jarvis-gui # before this commit
→ Tauri'\''s generate_context!() reads frontend/dist/client which is
whatever the user (or me) last built, possibly hours ago. Result:
new release exe ships with stale UI. Caught when /commands page
kept rendering the placeholder after the new code was committed
and the binary was rebuilt — because dist/ was generated before
the Svelte file was rewritten.
Fix: build.rs now invokes `npm run build` in frontend/ on every
relink, and emits cargo:rerun-if-changed for frontend/src and
package.json so cargo also reruns build.rs when Svelte/TS sources
change. `cargo tauri build` still works (it runs npm twice, which
is wasteful but harmless). If npm is not installed (unlikely but
possible on a Rust-only CI), prints a cargo warning and bundles
whatever dist is on disk.
`dist` stays in frontend/.gitignore — this just makes sure a local
working copy always has a fresh dist before Tauri picks it up.
Architecture — close the trigger-strip duplication:
lua/api/text.rs (new):
+ jarvis.text.strip_trigger(phrase, triggers) — case-insensitive
"longest trigger wins" stripping, returns trimmed remainder. The
11 packs that previously inlined a `for _, t in ipairs(triggers) do
string.find ... s == 1 then sub` loop can drop to one call.
Demonstrated in spelling/, window_switch/, summarize/.
+ jarvis.text.contains_any(phrase, needles) — boolean check for any
of N substrings, case-insensitive. Useful for keyword routing.
5 new portable Lua packs (sandbox=full). 36 → 41 total. cargo test
commands::tests still 3/3.
brightness/ — brightness_up / down / max / min. WMI WmiMonitorBrightness
(read CurrentBrightness) + WmiMonitorBrightnessMethods.WmiSetBrightness
(write). ±20% step for up/down; max=100, min=10. Desktop monitors that
don'\''t expose the WMI namespace get a friendly "не поддерживается"
toast instead of a silent failure.
window_switch/ — switch_to_window. Strip trigger ("переключись на" /
"switch to" / etc.), apply the same alias map as process_kill
(хром→chrome, телега→telegram, ...), then PowerShell finds processes
by name OR window-title substring sorted by StartTime desc and uses
(New-Object -ComObject WScript.Shell).AppActivate($pid) on the top
match. Speaks the window title that got focus.
spelling/ — spell_out. Strips "произнеси по буквам" / "spell out" /
etc., iterates `word:gmatch(".")`, joins letters with ". " so SAPI
gives each one a natural micro-pause. Uses jarvis.text.strip_trigger
+ jarvis.speak — 24 lines total.
network/ — open_wifi_settings (ms-settings:network-wifi) +
open_bluetooth_settings (ms-settings:bluetooth) + my_ip. my_ip pulls
the first non-loopback, non-APIPA IPv4 with Get-NetIPAddress and
fetches WAN IP from api.ipify.org (5s timeout); speaks "Локальный X.
Внешний Y".
summarize/ — summarize_selection. Synthesises Ctrl+C through user32!
keybd_event (so the focused window copies its selection), waits
220 ms for the clipboard to populate, reads via
jarvis.system.clipboard.get(), bails if < 20 chars, truncates at
8000, sends to Groq @ temp 0.3 with a "3-5 sentences, keep names"
prompt, drops the summary back into the clipboard and speaks it.
Lets you select ANY text in ANY app (browser, PDF, IDE) and say
"суммируй" to get a spoken TL;DR.
The new packs explicitly use the new jarvis.text + jarvis.llm +
jarvis.speak API surface — no inline PS-SAPI boilerplate, no inline
Groq plumbing. Code is now intent-first.
Architecture — DRY at the Lua API surface:
lua/api/tts.rs (new): jarvis.speak(text, opts?). opts.lang = ISO
two-letter code (default "ru"); auto-picks first installed voice
matching the culture. opts.async = true|false (default true). The
9 packs that previously inlined a 60-character PS-SAPI string for
every spoken reply can now reduce to one line. Refactored fun/,
dice/, clipboard_read/ as proof — fun/ask.lua went from 75 lines
of HTTP+SAPI boilerplate to ~28 lines of intent code.
lua/api/llm.rs (new): jarvis.llm(messages, opts?). messages is a
list of {role, content} tables, opts.max_tokens / .temperature /
.top_p. Returns the assistant string or nil. Gated by sandbox
allows_http() (so Standard+ packs only). Internally uses the
existing jarvis_core::llm::LlmClient::complete_with, no duplicated
HTTP plumbing.
llm/client.rs: split complete() into complete_with(messages,
max_tokens, temperature, top_p) for the new caller; old complete()
is now a thin wrapper at temp=0.7 top_p=1.0 for backward compat.
GUI — /commands page rewrite:
Replaces the "Раздел в разработке" placeholder. Calls Tauri command
get_commands_list (already existed in tauri_commands/commands.rs,
was wired but unused). Renders a search-filterable card grid:
- cmd id (monospace)
- type badge with colour by kind (lua=blue, ahk=red, cli=grey,
voice=green, terminate=red, stop_chaining=violet)
- sandbox badge
- description line (if non-empty)
- all phrases for currentLanguage, fallback to en, fallback to
first available; each phrase in a small inline pill
Toolbar: text input with magnifier icon + reload button (↻).
Counter line shows visible/total + "Скажи Джарвис + любую фразу"
hint. Counts down as you type the filter.
New packs (3, total 33 → 36):
games/ — launch_game / list_games. Reads or creates
%USERPROFILE%\Documents\jarvis-games.json (the sample preloads dota,
cs2, elden ring, witcher 3, factorio + minecraft launcher path +
fortnite epic URI). Trigger-strip + fuzzy name/alias match, then
launches via steam://rungameid/N (Steam), epic:// (Epic Games), or
start "" "path" (anything else). list_games speaks the configured
library.
mouse/ — left/right/middle/double click + scroll up/down. Single
dispatch.lua + _mouse_helper.ps1. PS P/Invokes user32!mouse_event
with the right flag pair (LEFTDOWN+LEFTUP / etc.), 30ms gap, doubles
do two LEFT cycles with a 50ms gap, wheel uses ±360 ticks.
random_choice/ — "выбери из X или Y или Z". Strips the trigger,
splits on " или " / " or " / ", " / " либо " / " чи ", math.random
picks one, speaks "Я выбираю N".
cargo test commands::tests still 3/3. 36 packs verified via
jarvis-gui startup log.
Seven more portable Lua packs lifted from common voice-assistant
playbooks (Siri/Alexa/Алиса/Cortana). All sandbox=full, jarvis.http
+ jarvis.system.exec, zero new Rust code. cargo test still 3/3.
news/ — pulls Lenta.ru RSS (en switches to BBC), regex-extracts the
top 5 <title> entries (skipping the channel header), drops them into
the clipboard, then asks Groq for a 2-3 sentence summary at temp 0.4
and speaks it via SAPI. If GROQ_TOKEN is missing it falls back to
just reading the first three headlines.
currency/ — cbr-xml-daily.ru/daily_json.js. Picks USD/EUR/CNY/GBP/
JPY/KZT by keyword in the phrase, computes value / nominal, diffs
against Valute.Previous, says "X стоит N рублей, поднялся/опустился
на M" with ▲/▼ in the notify body.
crypto/ — CoinGecko simple/price with usd,rub and 24hr_change.
Recognises bitcoin / ethereum / solana / dogecoin / the-open-network.
Reports price + direction in plain Russian.
fun/ — joke / fact / quote / compliment, single ask.lua dispatched by
command_id. Per-command system prompt at temp 1.0; seeds with a
random integer so consecutive calls don't repeat.
wiki/ — ru.wikipedia opensearch → REST summary endpoint. If Groq is
available and extract > 350 chars, asks for a 2-3 sentence retell;
otherwise speaks the first 400 chars verbatim. Full extract goes to
clipboard.
power/ — shutdown_pc / restart_pc / sleep_pc / hibernate_pc /
logoff_user / cancel_shutdown. Single act.lua keyed by command_id.
shutdown/restart use shutdown.exe /s|/r /t 30 with a /c message —
30-second grace window so "отмени выключение" actually works.
sleep uses rundll32 powrprof,SetSuspendState, hibernate uses
shutdown.exe /h.
help/ — "что ты умеешь". Reads the parent commands/ dir via
jarvis.fs.list, sorts subdirs (pack names), pairs each with a
hardcoded Russian/English one-liner description, copies the full
list to clipboard and speaks a count + "see clipboard for full list".
Total command packs is now 33 (was 26).
Iron Man icon regenerated from resources/icons/icon.png at sizes
16/24/32/48/64/128/256 via Pillow (make_ico.py committed alongside);
icon.ico is now a clean 83 KB multi-res for sharp display at any
scale instead of whatever placeholder Tauri originally emitted.
dotenvy added to workspace deps. jarvis-app/main.rs and jarvis-gui/
main.rs both walk up from current_exe() looking for dev.env (up to
5 parent levels), so the Desktop shortcut can launch the exe
directly without a wrapper .bat to pre-set GROQ_TOKEN. Falls back
to dotenvy::dotenv() (cwd lookup) if nothing found near the binary.
5 new portable Lua command packs. Bumps total from 21 to 26.
cargo test -p jarvis-core --lib commands::tests still passes 3/3.
reminders/ — set_reminder. Parses "напомни через N (секунд|минут|
часов) <body>" or English equivalents; understands one Russian word
numerals (один, две, пять, десять, пятнадцать, ...) for the count
when speech recognition returns words instead of digits. Defaults to
5 minutes if number/unit missing. Spawns a detached PowerShell that
Start-Sleeps then fires: BurntToast if installed, System.Speech
SAPI (ru-RU voice preference), and a WScript.Shell.Popup as the
guaranteed-visible last resort.
date_query/ — today / tomorrow / yesterday. Single answer.lua keyed
by command_id, computes the offset via PowerShell Get-Date.AddDays()
in ru-RU culture so the weekday/month come out in Russian, then
speaks via SAPI.
voice_type/ — type_text. Strips the trigger, drops the remainder to
clipboard via jarvis.system.clipboard.set, then synthesises Ctrl+V
through user32!keybd_event so it lands in whatever window currently
has focus. Works in any text field (note app, browser, IDE,
Telegram, etc.).
dice/ — coin_flip / roll_dice / random_number. Single roll.lua,
seeds math.random with os.time + jitter, speaks the result. "1-6"
for dice, "1-100" for random.
stopwatch/ — start / check / stop. Uses jarvis.state.get/set
(persisted via jarvis-core's settings DB) to remember the start
timestamp, computes elapsed via jarvis.context.time.timestamp,
formats as "X сек" / "X мин Y сек" / "X ч Y мин Z сек".
All new packs follow the established patterns (sandbox=full, PS-via-
exec helper where needed, USERPROFILE/SystemDrive everywhere, no
hardcoded paths).
GUI: Footer.svelte gets a small inline "Rust" pill next to the
copyright line (red bg #b7411a, white 11px, rounded). Now obvious
at-a-glance which fork's UI you're looking at when both python and
rust are open side-by-side. Python edition mirrors with a console
banner.
Launcher: run.bat at the repo root loads dev.env (skipping #-comments
and blank lines), then `start ""` spawns target\release\jarvis-gui.exe.
The new Desktop shortcut "Jarvis (Rust).lnk" points at this batch
file with resources\icons\icon.ico as its icon. WindowStyle=7 means
the cmd host stays minimized — only the GUI window pops up.
If the user hasn't built yet, the bat prints the cargo build command
and pauses instead of silently doing nothing.
Five more portable Lua command packs. Bumps the total from 16 to 21
packs. cargo test -p jarvis-core --lib commands::tests still 3/3.
translate/ — translate.lua. Picks the target language from triggers
like "переведи на английский" / "translate to russian"; falls back to
English when only "переведи X" is heard. Sends to Groq with a strict
"translation only" system prompt at temp 0.3. Drops the result into
the clipboard, fires a notify, and speaks via PowerShell SAPI in the
matching ISO voice (ru-RU / uk-UA / en-US / de-DE etc.) so a German
translation actually sounds German.
math/ — math.lua. Strips the trigger ("посчитай" / "calculate" /
"сколько будет" / "what is" / ...), sends the remainder to Groq with
temp=0.0 and max_tokens=64, asking ONLY for the numeric result. If
the model replies "нет" (the system-prompt sentinel for non-math),
plays not_found instead of speaking nonsense. Otherwise speaks
"Получилось X" via SAPI.
theme/ — set.lua dispatched by command_id (theme_dark / theme_light).
Flips AppsUseLightTheme + SystemUsesLightTheme under
HKCU\...\Themes\Personalize via Set-ItemProperty. No reboot needed,
Windows shell re-themes immediately.
projects/ — open.lua + list.lua. Reads/creates
%USERPROFILE%\Documents\jarvis-projects.json (a flat
[{"name": ..., "path": ...}] list). open.lua strips the trigger,
exact-matches then substring-matches the project name, opens the path
in VS Code if `code.cmd` is on PATH, otherwise in Explorer. First
call creates a sample config with jarvis-rust / jarvis-python /
dietpi entries pointing under USERPROFILE so it works on any machine.
list.lua just notifies the configured project names.
sound_panel/ — open.lua. Two-liner: rundll32 mmsys.cpl,,1 (Recording
tab). Useful both as a standalone "открой настройки звука" and as
the place to fix the mic-disabled state that prevents jarvis-app
from starting.
Portability: USERPROFILE / SystemDrive / PATH lookup throughout.
Before: clicking "Запустить" in jarvis-gui spawned jarvis-app, which
silently exited with code 1 (or in release builds, looked like the
console window flashed and closed). The pv_recorder library returns
INVALID_ARGUMENT from pv_recorder_init when Windows Core Audio reports
zero capture endpoints (some other app holding the mic exclusively,
or all input devices disabled in mmsys.cpl). User saw no actionable
feedback.
Now: on recorder::init failure jarvis-app calls notify_mic_problem()
which (Windows-only):
- Fires a long-duration Windows toast titled "J.A.R.V.I.S.: микрофон
не найден" with a hint pointing at mmsys.cpl / Recording.
- Spawns "start ms-settings:sound" so the Sound settings page opens
automatically — user can re-enable the mic in two clicks.
Then the original app::close(1, ...) path runs to keep the same exit
behaviour the GUI's get_jarvis_app_stats poller expects.
Cargo.toml: jarvis-app now pulls winrt-notification (already in
workspace.dependencies via jarvis-core) for the toast.
Also incidentally fixed: the release-build C0000139 (entrypoint not
found) loader crash that was showing up before this change. It went
away after the workspace dep was added and the release relink ran.
Most likely the previous release exe had a stale import table from an
earlier partial rebuild; the clean relink resolves it.
Non-Windows builds get a no-op eprintln so the binary still compiles
for Linux/macOS.
Five new portable Lua command packs. Bumps the total from 11 to 16 packs.
All sandbox=full, all dispatched via jarvis.system.exec or jarvis.fs / jarvis.system
APIs, no new Rust code, post_build.py --sync copies them under target/.
cargo test -p jarvis-core --lib commands::tests passes 3/3 against all 16.
window/ — 8 commands, single dispatch.lua keyed by jarvis.context.command_id:
show_desktop / maximize_window / minimize_window / snap_left / snap_right
/ close_window / restore_all / task_view. _window_helper.ps1 P/Invokes
user32!keybd_event for arbitrary combos like "win+d", "alt+f4",
"win+shift+m". Uses one shared script per pack — much less duplication
than the volume/ approach.
clipboard_read/ — read_clipboard. Pulls jarvis.system.clipboard.get(),
caps preview at 400 chars, then shells PowerShell System.Speech to actually
speak it aloud (auto-picks the first ru-RU voice if present). Also fires a
notify with a 120-char snippet.
notes/ — add_note + open_notes. add_note strips the trigger phrase from
the recognized voice, appends "[YYYY-MM-DD HH:MM] body\n" to
%USERPROFILE%\Documents\jarvis-notes.txt via jarvis.fs.append (creates
the file on first call). open_notes opens the file in the default editor
via jarvis.system.open. Trigger list covers RU/EN/UA variants of
"запиши" / "запомни" / "write down" / etc.
process_kill/ — kill_process. Trigger-strips "закрой программу" / "убей
процесс" / etc., then runs the remainder through an alias map (хром→
chrome, телега→telegram, спотифай→spotify, edge/edge/едж→msedge etc.,
20+ entries) before invoking taskkill /F /IM. Notifies success/not-found.
websearch/ — search_google / search_youtube / search_wiki / search_yandex.
Single dispatch.lua reads jarvis.context.command_id to pick the base URL,
URL-encodes the query (Lua %02X gsub, not relying on jarvis.http), opens
the resulting link via jarvis.system.open (system default browser).
Portability check: every helper resolves paths via $env:USERPROFILE /
jarvis.context.command_path / jarvis.system.env — no hardcoded C:\ refs.
Bossiara13