J.A.R.V.I.S-rust/crates/jarvis-app/src/main.rs

236 lines
7.4 KiB
Rust
Raw Normal View History

use jarvis_core::slots;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc;
// include core
use jarvis_core::{
audio, audio_processing, commands, config, db, listener, recorder, stt, intent,
ipc::{self, IpcAction},
i18n, voices, models,
APP_CONFIG_DIR, APP_LOG_DIR, COMMANDS_LIST, DB,
};
// include log
#[macro_use]
extern crate simple_log;
mod log;
// include app
mod app;
mod llm_fallback;
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
mod llm_router;
// include tray
// @TODO. macOS currently not supported for tray functionality.
#[cfg(not(target_os = "macos"))]
mod tray;
static SHOULD_STOP: AtomicBool = AtomicBool::new(false);
fn main() -> Result<(), String> {
feat: 5 new packs + dotenvy + regen icon.ico + shortcuts redo Iron Man icon regenerated from resources/icons/icon.png at sizes 16/24/32/48/64/128/256 via Pillow (make_ico.py committed alongside); icon.ico is now a clean 83 KB multi-res for sharp display at any scale instead of whatever placeholder Tauri originally emitted. dotenvy added to workspace deps. jarvis-app/main.rs and jarvis-gui/ main.rs both walk up from current_exe() looking for dev.env (up to 5 parent levels), so the Desktop shortcut can launch the exe directly without a wrapper .bat to pre-set GROQ_TOKEN. Falls back to dotenvy::dotenv() (cwd lookup) if nothing found near the binary. 5 new portable Lua command packs. Bumps total from 21 to 26. cargo test -p jarvis-core --lib commands::tests still passes 3/3. reminders/ — set_reminder. Parses "напомни через N (секунд|минут| часов) <body>" or English equivalents; understands one Russian word numerals (один, две, пять, десять, пятнадцать, ...) for the count when speech recognition returns words instead of digits. Defaults to 5 minutes if number/unit missing. Spawns a detached PowerShell that Start-Sleeps then fires: BurntToast if installed, System.Speech SAPI (ru-RU voice preference), and a WScript.Shell.Popup as the guaranteed-visible last resort. date_query/ — today / tomorrow / yesterday. Single answer.lua keyed by command_id, computes the offset via PowerShell Get-Date.AddDays() in ru-RU culture so the weekday/month come out in Russian, then speaks via SAPI. voice_type/ — type_text. Strips the trigger, drops the remainder to clipboard via jarvis.system.clipboard.set, then synthesises Ctrl+V through user32!keybd_event so it lands in whatever window currently has focus. Works in any text field (note app, browser, IDE, Telegram, etc.). dice/ — coin_flip / roll_dice / random_number. Single roll.lua, seeds math.random with os.time + jitter, speaks the result. "1-6" for dice, "1-100" for random. stopwatch/ — start / check / stop. Uses jarvis.state.get/set (persisted via jarvis-core's settings DB) to remember the start timestamp, computes elapsed via jarvis.context.time.timestamp, formats as "X сек" / "X мин Y сек" / "X ч Y мин Z сек". All new packs follow the established patterns (sandbox=full, PS-via- exec helper where needed, USERPROFILE/SystemDrive everywhere, no hardcoded paths).
2026-05-15 12:28:44 +03:00
load_dotenv_near_exe();
eprintln!("[jarvis-app] step: init_dirs");
config::init_dirs()?;
eprintln!("[jarvis-app] step: init_logging");
log::init_logging()?;
info!("Starting Jarvis v{} ...", config::APP_VERSION.unwrap());
info!("Config directory is: {}", APP_CONFIG_DIR.get().unwrap().display());
info!("Log directory is: {}", APP_LOG_DIR.get().unwrap().display());
eprintln!("[jarvis-app] step: db::init");
let settings = db::init();
DB.set(settings.arc().clone())
.expect("DB already initialized");
eprintln!("[jarvis-app] step: voices::init");
let voice_id = settings.lock().voice.clone();
let language = settings.lock().language.clone();
if let Err(e) = voices::init(&voice_id, &language) {
warn!("Failed to init voices: {}", e);
}
eprintln!("[jarvis-app] step: i18n::init");
i18n::init(&settings.lock().language);
eprintln!("[jarvis-app] step: llm_fallback::init");
llm_fallback::init();
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
eprintln!("[jarvis-app] step: llm_router::init");
llm_router::init();
eprintln!("[jarvis-app] step: long_term_memory::init");
if let Err(e) = jarvis_core::long_term_memory::init() {
warn!("Long-term memory init failed: {}", e);
}
eprintln!("[jarvis-app] step: profiles::init");
if let Err(e) = jarvis_core::profiles::init() {
warn!("Profiles init failed: {}", e);
}
eprintln!("[jarvis-app] step: recorder::init");
if recorder::init().is_err() {
fix(app): show toast + open Sound settings when mic enumeration returns nothing Before: clicking "Запустить" in jarvis-gui spawned jarvis-app, which silently exited with code 1 (or in release builds, looked like the console window flashed and closed). The pv_recorder library returns INVALID_ARGUMENT from pv_recorder_init when Windows Core Audio reports zero capture endpoints (some other app holding the mic exclusively, or all input devices disabled in mmsys.cpl). User saw no actionable feedback. Now: on recorder::init failure jarvis-app calls notify_mic_problem() which (Windows-only): - Fires a long-duration Windows toast titled "J.A.R.V.I.S.: микрофон не найден" with a hint pointing at mmsys.cpl / Recording. - Spawns "start ms-settings:sound" so the Sound settings page opens automatically — user can re-enable the mic in two clicks. Then the original app::close(1, ...) path runs to keep the same exit behaviour the GUI's get_jarvis_app_stats poller expects. Cargo.toml: jarvis-app now pulls winrt-notification (already in workspace.dependencies via jarvis-core) for the toast. Also incidentally fixed: the release-build C0000139 (entrypoint not found) loader crash that was showing up before this change. It went away after the workspace dep was added and the release relink ran. Most likely the previous release exe had a stale import table from an earlier partial rebuild; the clean relink resolves it. Non-Windows builds get a no-op eprintln so the binary still compiles for Linux/macOS.
2026-05-15 11:58:47 +03:00
notify_mic_problem();
app::close(1, "recorder::init failed");
}
eprintln!("[jarvis-app] step: models::init");
if let Err(e) = models::init() {
warn!("Models registry init failed: {}", e);
}
eprintln!("[jarvis-app] step: stt::init");
if stt::init().is_err() {
app::close(1, "stt::init failed");
}
eprintln!("[jarvis-app] step: commands::parse_commands");
info!("Initializing commands.");
let cmds = match commands::parse_commands() {
Ok(c) => c,
Err(e) => {
warn!("Failed to parse commands: {}. Starting with empty command list.", e);
Vec::new()
}
};
info!("Commands initialized. Count: {}, List: {:?}", cmds.len(), commands::list_paths(&cmds));
COMMANDS_LIST.set(cmds).unwrap();
eprintln!("[jarvis-app] step: audio::init");
if audio::init().is_err() {
app::close(1, "audio::init failed");
}
eprintln!("[jarvis-app] step: listener::init");
if let Err(e) = listener::init() {
error!("Wake-word engine init failed: {}", e);
app::close(1, "listener::init failed");
}
eprintln!("[jarvis-app] step: tokio runtime");
let rt = Arc::new(
tokio::runtime::Runtime::new().expect("Failed to create tokio runtime")
);
eprintln!("[jarvis-app] step: intent::init");
rt.block_on(async {
2026-02-08 07:30:46 +05:00
if let Err(e) = intent::init(COMMANDS_LIST.get().unwrap()).await {
error!("Failed to initialize intent classifier: {}", e);
app::close(1, "intent::init failed");
}
});
eprintln!("[jarvis-app] step: slots::init");
slots::init().map_err(|e| error!("Slot extraction init failed: {}", e)).ok();
eprintln!("[jarvis-app] step: audio_processing::init");
info!("Initializing audio processing...");
if let Err(e) = audio_processing::init() {
warn!("Audio processing init failed: {}", e);
}
eprintln!("[jarvis-app] step: ipc::init");
info!("Initializing IPC...");
ipc::init();
// channel for text commands (manually written in the GUI)
let (text_cmd_tx, text_cmd_rx) = mpsc::channel::<String>();
ipc::set_action_handler(move |action| {
match action {
IpcAction::Stop => {
info!("Received stop command from GUI");
SHOULD_STOP.store(true, Ordering::SeqCst);
}
IpcAction::ReloadCommands => {
info!("Received reload commands request");
// TODO: implement reload
}
IpcAction::SetMuted { muted } => {
info!("Received mute request: {}", muted);
// TODO: implement mute
}
IpcAction::TextCommand { text } => {
info!("Received text command: {}", text);
if let Err(e) = text_cmd_tx.send(text) {
error!("Failed to send text command to app: {}", e);
}
}
IpcAction::Ping => {
// handled internally by server
}
_ => {}
}
});
// start WebSocket server on the shared runtime
let ipc_rt = Arc::clone(&rt);
std::thread::spawn(move || {
ipc_rt.block_on(ipc::start_server());
});
eprintln!("[jarvis-app] step: spawn app thread");
let app_rt = Arc::clone(&rt);
std::thread::spawn(move || {
let _ = app::start(text_cmd_rx, &app_rt);
});
eprintln!("[jarvis-app] step: tray::init_blocking");
tray::init_blocking(settings);
eprintln!("[jarvis-app] step: main returning Ok");
Ok(())
}
pub fn should_stop() -> bool {
SHOULD_STOP.load(Ordering::SeqCst)
}
fix(app): show toast + open Sound settings when mic enumeration returns nothing Before: clicking "Запустить" in jarvis-gui spawned jarvis-app, which silently exited with code 1 (or in release builds, looked like the console window flashed and closed). The pv_recorder library returns INVALID_ARGUMENT from pv_recorder_init when Windows Core Audio reports zero capture endpoints (some other app holding the mic exclusively, or all input devices disabled in mmsys.cpl). User saw no actionable feedback. Now: on recorder::init failure jarvis-app calls notify_mic_problem() which (Windows-only): - Fires a long-duration Windows toast titled "J.A.R.V.I.S.: микрофон не найден" with a hint pointing at mmsys.cpl / Recording. - Spawns "start ms-settings:sound" so the Sound settings page opens automatically — user can re-enable the mic in two clicks. Then the original app::close(1, ...) path runs to keep the same exit behaviour the GUI's get_jarvis_app_stats poller expects. Cargo.toml: jarvis-app now pulls winrt-notification (already in workspace.dependencies via jarvis-core) for the toast. Also incidentally fixed: the release-build C0000139 (entrypoint not found) loader crash that was showing up before this change. It went away after the workspace dep was added and the release relink ran. Most likely the previous release exe had a stale import table from an earlier partial rebuild; the clean relink resolves it. Non-Windows builds get a no-op eprintln so the binary still compiles for Linux/macOS.
2026-05-15 11:58:47 +03:00
feat: 5 new packs + dotenvy + regen icon.ico + shortcuts redo Iron Man icon regenerated from resources/icons/icon.png at sizes 16/24/32/48/64/128/256 via Pillow (make_ico.py committed alongside); icon.ico is now a clean 83 KB multi-res for sharp display at any scale instead of whatever placeholder Tauri originally emitted. dotenvy added to workspace deps. jarvis-app/main.rs and jarvis-gui/ main.rs both walk up from current_exe() looking for dev.env (up to 5 parent levels), so the Desktop shortcut can launch the exe directly without a wrapper .bat to pre-set GROQ_TOKEN. Falls back to dotenvy::dotenv() (cwd lookup) if nothing found near the binary. 5 new portable Lua command packs. Bumps total from 21 to 26. cargo test -p jarvis-core --lib commands::tests still passes 3/3. reminders/ — set_reminder. Parses "напомни через N (секунд|минут| часов) <body>" or English equivalents; understands one Russian word numerals (один, две, пять, десять, пятнадцать, ...) for the count when speech recognition returns words instead of digits. Defaults to 5 minutes if number/unit missing. Spawns a detached PowerShell that Start-Sleeps then fires: BurntToast if installed, System.Speech SAPI (ru-RU voice preference), and a WScript.Shell.Popup as the guaranteed-visible last resort. date_query/ — today / tomorrow / yesterday. Single answer.lua keyed by command_id, computes the offset via PowerShell Get-Date.AddDays() in ru-RU culture so the weekday/month come out in Russian, then speaks via SAPI. voice_type/ — type_text. Strips the trigger, drops the remainder to clipboard via jarvis.system.clipboard.set, then synthesises Ctrl+V through user32!keybd_event so it lands in whatever window currently has focus. Works in any text field (note app, browser, IDE, Telegram, etc.). dice/ — coin_flip / roll_dice / random_number. Single roll.lua, seeds math.random with os.time + jitter, speaks the result. "1-6" for dice, "1-100" for random. stopwatch/ — start / check / stop. Uses jarvis.state.get/set (persisted via jarvis-core's settings DB) to remember the start timestamp, computes elapsed via jarvis.context.time.timestamp, formats as "X сек" / "X мин Y сек" / "X ч Y мин Z сек". All new packs follow the established patterns (sandbox=full, PS-via- exec helper where needed, USERPROFILE/SystemDrive everywhere, no hardcoded paths).
2026-05-15 12:28:44 +03:00
// Look for dev.env next to the exe, then walk parents until we find one.
// Lets the user double-click jarvis-gui.exe / jarvis-app.exe without needing
// a wrapper .bat to pre-load GROQ_TOKEN and similar secrets.
fn load_dotenv_near_exe() {
if let Ok(exe) = std::env::current_exe() {
let mut dir = exe.parent().map(|p| p.to_path_buf());
for _ in 0..5 {
if let Some(d) = &dir {
let candidate = d.join("dev.env");
if candidate.is_file() {
let _ = dotenvy::from_path(&candidate);
eprintln!("[jarvis-app] loaded env: {}", candidate.display());
return;
}
dir = d.parent().map(|p| p.to_path_buf());
}
}
}
let _ = dotenvy::dotenv();
}
fix(app): show toast + open Sound settings when mic enumeration returns nothing Before: clicking "Запустить" in jarvis-gui spawned jarvis-app, which silently exited with code 1 (or in release builds, looked like the console window flashed and closed). The pv_recorder library returns INVALID_ARGUMENT from pv_recorder_init when Windows Core Audio reports zero capture endpoints (some other app holding the mic exclusively, or all input devices disabled in mmsys.cpl). User saw no actionable feedback. Now: on recorder::init failure jarvis-app calls notify_mic_problem() which (Windows-only): - Fires a long-duration Windows toast titled "J.A.R.V.I.S.: микрофон не найден" with a hint pointing at mmsys.cpl / Recording. - Spawns "start ms-settings:sound" so the Sound settings page opens automatically — user can re-enable the mic in two clicks. Then the original app::close(1, ...) path runs to keep the same exit behaviour the GUI's get_jarvis_app_stats poller expects. Cargo.toml: jarvis-app now pulls winrt-notification (already in workspace.dependencies via jarvis-core) for the toast. Also incidentally fixed: the release-build C0000139 (entrypoint not found) loader crash that was showing up before this change. It went away after the workspace dep was added and the release relink ran. Most likely the previous release exe had a stale import table from an earlier partial rebuild; the clean relink resolves it. Non-Windows builds get a no-op eprintln so the binary still compiles for Linux/macOS.
2026-05-15 11:58:47 +03:00
#[cfg(target_os = "windows")]
fn notify_mic_problem() {
use winrt_notification::{Toast, Duration as ToastDuration};
let _ = Toast::new(Toast::POWERSHELL_APP_ID)
.title("J.A.R.V.I.S.: микрофон не найден")
.text1("Windows не видит ни одного устройства записи.")
.text2("Откройте mmsys.cpl → Recording → включите микрофон и перезапустите Jarvis.")
.duration(ToastDuration::Long)
.show();
// Open Sound recording settings so the user can fix it in two clicks.
let _ = std::process::Command::new("cmd")
.args(["/C", "start", "", "ms-settings:sound"])
.spawn();
}
#[cfg(not(target_os = "windows"))]
fn notify_mic_problem() {
eprintln!("[jarvis-app] no recording device detected; check OS audio settings");
}