# J.A.R.V.I.S. — Architecture This is the code map for the Rust fork. If you're here to fix a bug or add a feature, start by reading the relevant section, then jump to the file. The Python fork at `C:\Jarvis\python` follows the same conceptual flow but is a single-process Python daemon — much smaller surface area. This doc covers the Rust side. ## Crate layout ``` crates/ jarvis-core library: STT, wake-word, TTS, intent, commands, IPC, LLM, Lua sandbox, scheduler, memory, profiles. ZERO runtime, only building blocks. Other crates wire it together. jarvis-app daemon binary. Holds the microphone, runs the listening loop, dispatches commands, starts the scheduler thread. Tray icon. jarvis-gui Tauri 2 + Svelte frontend. Shows command list, settings, mic state. Communicates with jarvis-app via IPC websocket. jarvis-cli debug CLI (classify / execute / list). Currently has a known runtime panic on startup; use jarvis-app instead. ``` ``` resources/commands// command.toml — id, phrases per language, script path, sandbox level, timeout *.lua — script(s); one per command id when multiple ids share a pack ``` The `tools/` directory hosts optional native helpers (Piper TTS binary, Silero helper script). Both are downloaded on demand via `tools/piper/install.ps1` etc. ## Data flow (mic → action) ``` [1] pv_recorder ──► raw 16 kHz mono PCM [2] webrtc-vad ───► voice activity windows [3] Vosk (wake-word + STT) ──► utf-8 transcript [4] intent classifier (MiniLM) ──► (intent_id, confidence)? [5] levenshtein fallback ──► (pack, command)? [6] llm_router (IMBA-1) ──► canonical phrase substitution? [7] llm_fallback chat ──► spoken reply [8] command execution ──► Lua sandbox / Python subprocess / WinAPI call [9] TTS (sapi | piper | silero) ──► speech out [10] voices::play_* ──► reaction sounds (ok, error, not_found) ``` Each step lives in a separate module — replace any one and the rest still works. | Step | Module | |------|---------------------------------------------------| | 1 | `jarvis-core::recorder` | | 2 | `jarvis-core::audio_processing::vad::webrtc` | | 3 | `jarvis-core::stt` + `jarvis-core::listener` | | 4 | `jarvis-core::intent` | | 5 | `jarvis-core::commands::fetch_command` | | 6 | `jarvis-app::llm_router` | | 7 | `jarvis-app::llm_fallback` | | 8 | `jarvis-core::commands::execute_command` | | 9 | `jarvis-core::tts` | | 10 | `jarvis-core::voices` | The listening loop driving all of this is `jarvis-app/src/app.rs::start()`. ## Configuration Two layers: 1. **`db::structs::Settings`** — persistent user preferences (voice, language, wake-word engine, noise-suppression backend). Lives in a sqlite file under `/`. Edited via the GUI Settings page. 2. **`runtime_config`** — environment-variable knobs read at startup. See `crates/jarvis-core/src/runtime_config.rs` for the full list with doc comments. Examples: - `JARVIS_TTS=sapi|piper|silero` - `JARVIS_LLM=groq|ollama` - `JARVIS_LLM_ROUTER=1` (default on), `JARVIS_LLM_ROUTER_THRESHOLD=0.55` - `GROQ_TOKEN`, `GROQ_MODEL`, `OLLAMA_BASE_URL`, `OLLAMA_MODEL` `runtime_config::log_effective_config()` is called once on `main()` so the active values appear in the startup log. `dev.env` next to the exe is auto-loaded by `dotenvy`. Walks up 5 parents — so running from `crates/jarvis-app/` during development still works. ## TTS pipeline ``` text ─► text_utils::sanitize_for_speech ─► TtsBackend::speak(text, opts) │ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ SapiBackend PiperBackend SileroBackend (PowerShell) (subprocess) (python helper) │ │ │ ▼ ▼ ▼ SAPI direct WAV → play_wav WAV → play_wav (shared) (shared) ``` `tts::backend()` is initialised lazily on first call; subsequent calls reuse the same `Arc`. Backend is picked by `JARVIS_TTS` env var with auto-detect fallback: if `tools/piper/piper.exe` exists, use Piper. `play_wav` is the shared synchronous WAV player (PowerShell SoundPlayer on Windows, no-op stub elsewhere). Used by both Piper and Silero — kept in `tts/mod.rs` as `pub(crate)` to deduplicate. ## Lua sandbox `mlua` 0.11 with three sandbox levels: | Level | Allowed | |-----------|--------------------------------------------| | Minimal | log, sleep, speak, audio, context, cmd | | Standard | + http, llm, state, fs (within pack dir) | | Full | + system.exec, clipboard.set, abs paths | Default is `standard`. Set per command in `command.toml` via `sandbox = "full"`. Sandbox is enforced by: - `LuaEngine::new(level)` controls which `StdLib` flags are loaded - `lua/engine.rs::register_api` gates HTTP/state/fs registration on the level - `globals.set("io", Nil)` removes the `io` stdlib unless `Full` - `os.execute / os.exit / os.remove / os.rename / os.setlocale` removed even in `Full` mode (security defence) ### Available APIs for command scripts See `crates/jarvis-core/src/lua/api/*.rs`. Quick reference: ```lua jarvis.speak(text, opts?) -- TTS; opts.lang/.async/.raw jarvis.cmd.ok(msg?) -- boilerplate-killer: play_ok + speak + {chain=false} jarvis.cmd.error(msg?) -- play_error + speak + {chain=false} jarvis.cmd.not_found(msg?) -- play_not_found + speak + {chain=false} jarvis.cmd.chain_ok(msg?) -- like ok() but chain=true jarvis.audio.play_ok / play_error / play_not_found / play_reply / play_goodbye / play_thanks jarvis.context.{phrase, command_id, command_path, language, time, slots} jarvis.text.strip_trigger(phrase, {triggers...}) jarvis.text.contains_any(phrase, {needles...}) jarvis.llm({messages}, {opts}) -- Groq or Ollama, selected by JARVIS_LLM jarvis.http.{get, post, post_json, json} jarvis.memory.{remember, recall, search, forget, all} jarvis.profile.{active, active_name, set, list, allows} jarvis.scheduler.{add, list, count, remove, clear} jarvis.vision.{screenshot, describe} -- HTTP sandbox required jarvis.fs.{read, write, append, exists, is_file, is_dir, list, mkdir, remove} jarvis.state.{get, set} -- pack-scoped k/v jarvis.system.{open, exec, notify, env, platform, clipboard.{get, set}} jarvis.settings.{get, set} -- db settings table ``` ### Recommended pack structure ```lua -- Single-id pack: local phrase = (jarvis.context.phrase or ""):lower() local body = jarvis.text.strip_trigger(phrase, { "trigger1", "trigger2" }) body = body:gsub("^[%s,:%.]+", ""):gsub("%s+$", "") if body == "" then return jarvis.cmd.error("Что именно?") end -- ... do work ... return jarvis.cmd.ok("Готово.") ``` For multi-id packs (one Lua script shared between several `[[commands]]`), dispatch on `jarvis.context.command_id`. The `profile_switch/switch.lua` is a good example — maps `profile.work` → `work`, `profile.game` → `game`, etc. ## Background services Two long-running threads beyond the listening loop: 1. **`scheduler` tick thread** (`crates/jarvis-core/src/scheduler.rs`). Wakes every 30 seconds, fires due tasks (Speak via TTS or Lua scripts). Persists changes to `/schedule.json`. 2. **`ipc::start_server`** (`crates/jarvis-core/src/ipc/`). Tokio websocket server on a configurable port. The GUI connects to this to send commands and receive events (`SpeechRecognized`, `LlmReply`, `CommandExecuted`, etc.). ## How to add a new command pack 1. Create `resources/commands//`. 2. Add `command.toml`: ```toml [[commands]] id = "your_pack.thing" type = "lua" script = "thing.lua" sandbox = "standard" timeout = 5000 [commands.phrases] ru = ["сделай штуку", ...] en = ["do the thing", ...] ``` 3. Write `thing.lua`. Start from the snippet under "Recommended pack structure". 4. Add a `commands::tests` assertion in `crates/jarvis-core/src/commands/tests.rs` so the parser regression test covers your TOML (the existing `every_command_toml_parses` already loads any new dir automatically). 5. Run `cargo test -p jarvis-core --lib commands::tests`. 6. Rebuild jarvis-app: it'll pick up the new pack on next start. For Python parity, mirror the YAML entry + handler in `C:\Jarvis\python\`. ## How to add a new Rust core feature 1. Add module under `crates/jarvis-core/src/.rs`. Pure data layer + thin global wrapper if needed (see `long_term_memory.rs` as template). 2. Register in `crates/jarvis-core/src/lib.rs` (`pub mod ;`). 3. Expose Lua API at `crates/jarvis-core/src/lua/api/.rs` if scripts need it. Register in `lua/api.rs` and call from `lua/engine.rs`. 4. Wire init from `crates/jarvis-app/src/main.rs`. 5. Add a `runtime_config::ENV_*` constant if there's an env-driven knob. 6. Add unit tests in a `#[cfg(test)] mod tests {}` block in the new file. 7. Document the public surface in this file (data flow / API section). ## How to add a new TTS backend Implement `TtsBackend` (see `crates/jarvis-core/src/tts/mod.rs`). Use the `super::play_wav` helper if your engine emits WAV files. Wire selection in `init_backend()` near the top of the same file. Document the env var in `runtime_config.rs`. ## Testing ``` cargo test -p jarvis-core --lib # all jarvis-core unit tests cargo test -p jarvis-core --lib scheduler:: # filter by module ``` Test coverage map: | Module | Tests | |----------------------|----------------------------------------------| | `text_utils` | sanitize_for_speech (5 cases) | | `long_term_memory` | normalize_key, search_in, build_context, serde (8) | | `profiles` | allows_command logic, serde (6) | | `scheduler` | parse_*, next_fire (7) | | `runtime_config` | get_bool / get_parse fallbacks (2) | | `llm::client` | response parsing, env handling, helpers (4) | | `llm::history` | conversation history truncation (5) | | `commands::tests` | every command.toml parses + has phrases (3) | | `lua::tests` | sandbox levels, fs escape, timeout (4) | | `audio_processing::vad::listen_window` | listening window logic (4) | Run-time integration tests are manual: - mic available? → `jarvis-app` logs `recorder::init` - wake word fires? → log shows `WakeWordDetected` - TTS speaks? → check Piper/SAPI logs - Scheduler fires? → wait 30s after `scheduler::add` with `in 30 seconds` ## Build ```powershell # One-time: set up MSVC env (PowerShell 7 / pwsh): $vc = "C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Auxiliary\Build\vcvars64.bat" cmd /c "`"$vc`" >NUL && set" | % { if ($_ -match '^([^=]+)=(.*)$') { [Environment]::SetEnvironmentVariable($matches[1], $matches[2], 'Process') } } # Then: cargo build --release -p jarvis-app -p jarvis-gui ``` The `jarvis-gui` `build.rs` auto-invokes `npm run build` and emits `cargo:rerun-if-changed=frontend/src` so changes to the Svelte frontend are picked up by a plain `cargo build`. No need to run `cargo tauri build` manually unless you want a packaged installer. ## Git workflow - `master` is protected. - `feature/pc-tools` is the integration branch where everything is currently shipping. - Self-hosted git on Forgejo at `http://192.168.0.10:3000/bossiara13/J.A.R.V.I.S-rust`. Push command: `NO_PROXY=192.168.0.10,localhost,127.0.0.1` + `git -c http.proxy= push forgejo ` (the local V2Ray proxy at 10809 will otherwise block LAN traffic). - GitHub origin at `https://github.com/DmitryBykov-ISPO/J.A.R.V.I.S-rust` — manually pushed for public visibility. Commit messages: descriptive, no `Co-Authored-By: Claude` tag (project rule).