J.A.R.V.I.S-rust/ARCHITECTURE.md

J.A.R.V.I.S. — Architecture

This is the code map for the Rust fork. If you're here to fix a bug or add a feature, start by reading the relevant section, then jump to the file.

The Python fork at C:\Jarvis\python follows the same conceptual flow but is a single-process Python daemon — much smaller surface area. This doc covers the Rust side.

Crate layout

crates/
  jarvis-core   library: STT, wake-word, TTS, intent, commands, IPC, LLM,
                Lua sandbox, scheduler, memory, profiles. ZERO runtime, only
                building blocks. Other crates wire it together.
  jarvis-app    daemon binary. Holds the microphone, runs the listening loop,
                dispatches commands, starts the scheduler thread. Tray icon.
  jarvis-gui    Tauri 2 + Svelte frontend. Shows command list, settings, mic
                state. Communicates with jarvis-app via IPC websocket.
  jarvis-cli    debug CLI (classify / execute / list). Currently has a known
                runtime panic on startup; use jarvis-app instead.

resources/commands/<pack_name>/
  command.toml   — id, phrases per language, script path, sandbox level, timeout
  *.lua          — script(s); one per command id when multiple ids share a pack

The tools/ directory hosts optional native helpers (Piper TTS binary, Silero helper script). Both are downloaded on demand via tools/piper/install.ps1 etc.

Data flow (mic → action)

[1] pv_recorder ──► raw 16 kHz mono PCM
[2] webrtc-vad ───► voice activity windows
[3] Vosk (wake-word + STT) ──► utf-8 transcript
[4] intent classifier (MiniLM) ──► (intent_id, confidence)?
[5] levenshtein fallback ──► (pack, command)?
[6] llm_router (IMBA-1) ──► canonical phrase substitution?
[7] llm_fallback chat ──► spoken reply
[8] command execution ──► Lua sandbox / Python subprocess / WinAPI call
[9] TTS (sapi | piper | silero) ──► speech out
[10] voices::play_*  ──► reaction sounds (ok, error, not_found)

Each step lives in a separate module — replace any one and the rest still works.

Step	Module
1	jarvis-core::recorder
2	jarvis-core::audio_processing::vad::webrtc
3	jarvis-core::stt + jarvis-core::listener
4	jarvis-core::intent
5	jarvis-core::commands::fetch_command
6	jarvis-app::llm_router
7	jarvis-app::llm_fallback
8	jarvis-core::commands::execute_command
9	jarvis-core::tts
10	jarvis-core::voices

The listening loop driving all of this is jarvis-app/src/app.rs::start().

Configuration

Two layers:

1. db::structs::Settings — persistent user preferences (voice, language, wake-word engine, noise-suppression backend). Lives in a sqlite file under <APP_CONFIG_DIR>/. Edited via the GUI Settings page.

2. runtime_config — environment-variable knobs read at startup. See crates/jarvis-core/src/runtime_config.rs for the full list with doc comments. Examples:

   • JARVIS_TTS=sapi|piper|silero
   • JARVIS_LLM=groq|ollama
   • JARVIS_LLM_ROUTER=1 (default on), JARVIS_LLM_ROUTER_THRESHOLD=0.55
   • GROQ_TOKEN, GROQ_MODEL, OLLAMA_BASE_URL, OLLAMA_MODEL

   runtime_config::log_effective_config() is called once on main() so the active values appear in the startup log.

dev.env next to the exe is auto-loaded by dotenvy. Walks up 5 parents — so running from crates/jarvis-app/ during development still works.

TTS pipeline

text ─► text_utils::sanitize_for_speech ─► TtsBackend::speak(text, opts)
                                                │
                              ┌─────────────────┼─────────────────┐
                              ▼                 ▼                 ▼
                         SapiBackend      PiperBackend      SileroBackend
                         (PowerShell)     (subprocess)      (python helper)
                              │                 │                 │
                              ▼                 ▼                 ▼
                          SAPI direct      WAV → play_wav    WAV → play_wav
                                              (shared)         (shared)

tts::backend() is initialised lazily on first call; subsequent calls reuse the same Arc<dyn TtsBackend>. Backend is picked by JARVIS_TTS env var with auto-detect fallback: if tools/piper/piper.exe exists, use Piper.

play_wav is the shared synchronous WAV player (PowerShell SoundPlayer on Windows, no-op stub elsewhere). Used by both Piper and Silero — kept in tts/mod.rs as pub(crate) to deduplicate.

Lua sandbox

mlua 0.11 with three sandbox levels:

Level	Allowed
Minimal	log, sleep, speak, audio, context, cmd
Standard	+ http, llm, state, fs (within pack dir)
Full	+ system.exec, clipboard.set, abs paths

Default is standard. Set per command in command.toml via sandbox = "full".

Sandbox is enforced by:

• LuaEngine::new(level) controls which StdLib flags are loaded
• lua/engine.rs::register_api gates HTTP/state/fs registration on the level
• globals.set("io", Nil) removes the io stdlib unless Full
• os.execute / os.exit / os.remove / os.rename / os.setlocale removed even in Full mode (security defence)

Available APIs for command scripts

See crates/jarvis-core/src/lua/api/*.rs. Quick reference:

jarvis.speak(text, opts?)        -- TTS; opts.lang/.async/.raw
jarvis.cmd.ok(msg?)              -- boilerplate-killer: play_ok + speak + {chain=false}
jarvis.cmd.error(msg?)           -- play_error + speak + {chain=false}
jarvis.cmd.not_found(msg?)       -- play_not_found + speak + {chain=false}
jarvis.cmd.chain_ok(msg?)        -- like ok() but chain=true

jarvis.audio.play_ok / play_error / play_not_found / play_reply / play_goodbye / play_thanks
jarvis.context.{phrase, command_id, command_path, language, time, slots}

jarvis.text.strip_trigger(phrase, {triggers...})
jarvis.text.contains_any(phrase, {needles...})

jarvis.llm({messages}, {opts})   -- Groq or Ollama, selected by JARVIS_LLM
jarvis.http.{get, post, post_json, json}

jarvis.memory.{remember, recall, search, forget, all}
jarvis.profile.{active, active_name, set, list, allows}
jarvis.scheduler.{add, list, count, remove, clear}
jarvis.vision.{screenshot, describe}   -- HTTP sandbox required

jarvis.fs.{read, write, append, exists, is_file, is_dir, list, mkdir, remove}
jarvis.state.{get, set}                                -- pack-scoped k/v
jarvis.system.{open, exec, notify, env, platform, clipboard.{get, set}}
jarvis.settings.{get, set}                             -- db settings table

Recommended pack structure

-- Single-id pack:
local phrase = (jarvis.context.phrase or ""):lower()
local body = jarvis.text.strip_trigger(phrase, { "trigger1", "trigger2" })
body = body:gsub("^[%s,:%.]+", ""):gsub("%s+$", "")

if body == "" then
    return jarvis.cmd.error("Что именно?")
end

-- ... do work ...

return jarvis.cmd.ok("Готово.")

For multi-id packs (one Lua script shared between several [[commands]]), dispatch on jarvis.context.command_id. The profile_switch/switch.lua is a good example — maps profile.work → work, profile.game → game, etc.

Background services

Two long-running threads beyond the listening loop:

1. scheduler tick thread (crates/jarvis-core/src/scheduler.rs). Wakes every 30 seconds, fires due tasks (Speak via TTS or Lua scripts). Persists changes to <APP_CONFIG_DIR>/schedule.json.

2. ipc::start_server (crates/jarvis-core/src/ipc/). Tokio websocket server on a configurable port. The GUI connects to this to send commands and receive events (SpeechRecognized, LlmReply, CommandExecuted, etc.).

How to add a new command pack

1. Create resources/commands/<your_pack>/.
2. Add command.toml:

   [[commands]]
   id = "your_pack.thing"
   type = "lua"
   script = "thing.lua"
   sandbox = "standard"
   timeout = 5000
   
   [commands.phrases]
   ru = ["сделай штуку", ...]
   en = ["do the thing", ...]
   

3. Write thing.lua. Start from the snippet under "Recommended pack structure".
4. Add a commands::tests assertion in crates/jarvis-core/src/commands/tests.rs so the parser regression test covers your TOML (the existing every_command_toml_parses already loads any new dir automatically).
5. Run cargo test -p jarvis-core --lib commands::tests.
6. Rebuild jarvis-app: it'll pick up the new pack on next start.

For Python parity, mirror the YAML entry + handler in C:\Jarvis\python\.

How to add a new Rust core feature

1. Add module under crates/jarvis-core/src/<feature>.rs. Pure data layer + thin global wrapper if needed (see long_term_memory.rs as template).
2. Register in crates/jarvis-core/src/lib.rs (pub mod <feature>;).
3. Expose Lua API at crates/jarvis-core/src/lua/api/<feature>.rs if scripts need it. Register in lua/api.rs and call from lua/engine.rs.
4. Wire init from crates/jarvis-app/src/main.rs.
5. Add a runtime_config::ENV_* constant if there's an env-driven knob.
6. Add unit tests in a #[cfg(test)] mod tests {} block in the new file.
7. Document the public surface in this file (data flow / API section).

How to add a new TTS backend

Implement TtsBackend (see crates/jarvis-core/src/tts/mod.rs). Use the super::play_wav helper if your engine emits WAV files. Wire selection in init_backend() near the top of the same file. Document the env var in runtime_config.rs.

Testing

cargo test -p jarvis-core --lib          # all jarvis-core unit tests
cargo test -p jarvis-core --lib scheduler:: # filter by module

Test coverage map:

Module	Tests
text_utils	sanitize_for_speech (5 cases)
long_term_memory	normalize_key, search_in, build_context, serde (8)
profiles	allows_command logic, serde (6)
scheduler	parse_*, next_fire (7)
runtime_config	get_bool / get_parse fallbacks (2)
llm::client	response parsing, env handling, helpers (4)
llm::history	conversation history truncation (5)
commands::tests	every command.toml parses + has phrases (3)
lua::tests	sandbox levels, fs escape, timeout (4)
audio_processing::vad::listen_window	listening window logic (4)

Run-time integration tests are manual:

• mic available? → jarvis-app logs recorder::init
• wake word fires? → log shows WakeWordDetected
• TTS speaks? → check Piper/SAPI logs
• Scheduler fires? → wait 30s after scheduler::add with in 30 seconds

Build

# One-time: set up MSVC env (PowerShell 7 / pwsh):
$vc = "C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
cmd /c "`"$vc`" >NUL && set" | % { if ($_ -match '^([^=]+)=(.*)$') { [Environment]::SetEnvironmentVariable($matches[1], $matches[2], 'Process') } }

# Then:
cargo build --release -p jarvis-app -p jarvis-gui

The jarvis-gui build.rs auto-invokes npm run build and emits cargo:rerun-if-changed=frontend/src so changes to the Svelte frontend are picked up by a plain cargo build. No need to run cargo tauri build manually unless you want a packaged installer.

Git workflow

• master is protected.
• feature/pc-tools is the integration branch where everything is currently shipping.
• Self-hosted git on Forgejo at http://192.168.0.10:3000/bossiara13/J.A.R.V.I.S-rust. Push command: NO_PROXY=192.168.0.10,localhost,127.0.0.1 + git -c http.proxy= push forgejo <branch> (the local V2Ray proxy at 10809 will otherwise block LAN traffic).
• GitHub origin at https://github.com/DmitryBykov-ISPO/J.A.R.V.I.S-rust — manually pushed for public visibility.

Commit messages: descriptive, no Co-Authored-By: Claude tag (project rule).