J.A.R.V.I.S-rust/crates/jarvis-app/src/llm_fallback.rs

152 lines
4.7 KiB
Rust
Raw Normal View History

use once_cell::sync::OnceCell;
use parking_lot::Mutex;
use jarvis_core::config;
use jarvis_core::i18n;
use jarvis_core::ipc::{self, IpcEvent};
feat: hot-swap LLM backend + media keys + codebase Q&A + scheduler cancel-by-text + TTS pre-warm Single biggest user-facing change: you can now switch local↔cloud LLM by voice without restarting. Plus four new packs and a perf nudge. Shared LLM global (crates/jarvis-core/src/llm/mod.rs) - GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth. - init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str). - llm_fallback + llm_router migrated: they no longer own a separate LlmClient, they read from the global on each request. Hot-swap is now LIVE — change backend, next chat call uses the new one. - parse_backend accepts both english and russian aliases: "groq"/"cloud"/"облако"/"клауд" → Groq "ollama"/"local"/"локал"/"локальный" → Ollama Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands) - "переключись на локальный" / "перейди на оллама" → swap to Ollama - "переключись на облако" / "используй грок" → swap to Groq - "какой у тебя мозг" / "облако или локально" → status query Underlying Lua API (registered globally): jarvis.llm({messages}, opts) -- as before, now uses global client jarvis.llm_status() -> "groq" | "ollama" | "none" jarvis.llm_switch(name) -> name on success, nil on failure Media keys pack (resources/commands/media_keys/, 4 commands) - "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE - "следующий трек" → VK_MEDIA_NEXT_TRACK - "предыдущий трек" → VK_MEDIA_PREV_TRACK - "стоп музыка" → VK_MEDIA_STOP - Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type). - Works with anything that listens to global media keys: Spotify, Yandex Music, YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys. Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands) - "укажи проект <path>" → stores path in jarvis.memory under codebase.root - "какой проект сейчас" → speaks current path - "что делает функция X" / "найди в коде Y" / "объясни код" → walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total), filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM with a "senior reviewer" system prompt, speaks 3-5 sentence answer. Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs) - new pub fn find_by_text(query, limit) -> Vec<ScheduledTask> - new pub fn remove_by_text(query) -> usize (count removed) - new pure helper find_by_text_in(tasks, query, limit) for tests - Lua: jarvis.scheduler.{find_by_text, remove_by_text} - Voice: "отмени напоминание про воду" / "удали задачу про разминку" - 3 new unit tests (52 total in jarvis-core). TTS pre-warm (crates/jarvis-app/src/main.rs) - Call jarvis_core::tts::backend() once on startup so the first speak() doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak latency by ~150-300ms depending on backend. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 52/52 jarvis-core unit tests pass.
2026-05-15 16:15:59 +03:00
use jarvis_core::llm::{self, ChatMessage, ConversationHistory};
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
use jarvis_core::long_term_memory;
use jarvis_core::profiles;
use jarvis_core::tts::{self, SpeakOpts};
use jarvis_core::voices;
feat: hot-swap LLM backend + media keys + codebase Q&A + scheduler cancel-by-text + TTS pre-warm Single biggest user-facing change: you can now switch local↔cloud LLM by voice without restarting. Plus four new packs and a perf nudge. Shared LLM global (crates/jarvis-core/src/llm/mod.rs) - GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth. - init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str). - llm_fallback + llm_router migrated: they no longer own a separate LlmClient, they read from the global on each request. Hot-swap is now LIVE — change backend, next chat call uses the new one. - parse_backend accepts both english and russian aliases: "groq"/"cloud"/"облако"/"клауд" → Groq "ollama"/"local"/"локал"/"локальный" → Ollama Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands) - "переключись на локальный" / "перейди на оллама" → swap to Ollama - "переключись на облако" / "используй грок" → swap to Groq - "какой у тебя мозг" / "облако или локально" → status query Underlying Lua API (registered globally): jarvis.llm({messages}, opts) -- as before, now uses global client jarvis.llm_status() -> "groq" | "ollama" | "none" jarvis.llm_switch(name) -> name on success, nil on failure Media keys pack (resources/commands/media_keys/, 4 commands) - "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE - "следующий трек" → VK_MEDIA_NEXT_TRACK - "предыдущий трек" → VK_MEDIA_PREV_TRACK - "стоп музыка" → VK_MEDIA_STOP - Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type). - Works with anything that listens to global media keys: Spotify, Yandex Music, YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys. Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands) - "укажи проект <path>" → stores path in jarvis.memory under codebase.root - "какой проект сейчас" → speaks current path - "что делает функция X" / "найди в коде Y" / "объясни код" → walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total), filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM with a "senior reviewer" system prompt, speaks 3-5 sentence answer. Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs) - new pub fn find_by_text(query, limit) -> Vec<ScheduledTask> - new pub fn remove_by_text(query) -> usize (count removed) - new pure helper find_by_text_in(tasks, query, limit) for tests - Lua: jarvis.scheduler.{find_by_text, remove_by_text} - Voice: "отмени напоминание про воду" / "удали задачу про разминку" - 3 new unit tests (52 total in jarvis-core). TTS pre-warm (crates/jarvis-app/src/main.rs) - Call jarvis_core::tts::backend() once on startup so the first speak() doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak latency by ~150-300ms depending on backend. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 52/52 jarvis-core unit tests pass.
2026-05-15 16:15:59 +03:00
// State now only holds conversation history — the actual LLM client lives in
// `jarvis_core::llm::GLOBAL` so that voice/Lua commands can hot-swap backends
// (Ollama ↔ Groq) without rebuilding this module's state.
struct State {
history: Mutex<ConversationHistory>,
max_tokens: u32,
}
static STATE: OnceCell<Option<State>> = OnceCell::new();
pub fn init() {
let _ = STATE.set(build_state());
}
fn build_state() -> Option<State> {
if !config::LLM_DEFAULT_ENABLED {
info!("LLM fallback disabled by config.");
return None;
}
feat: hot-swap LLM backend + media keys + codebase Q&A + scheduler cancel-by-text + TTS pre-warm Single biggest user-facing change: you can now switch local↔cloud LLM by voice without restarting. Plus four new packs and a perf nudge. Shared LLM global (crates/jarvis-core/src/llm/mod.rs) - GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth. - init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str). - llm_fallback + llm_router migrated: they no longer own a separate LlmClient, they read from the global on each request. Hot-swap is now LIVE — change backend, next chat call uses the new one. - parse_backend accepts both english and russian aliases: "groq"/"cloud"/"облако"/"клауд" → Groq "ollama"/"local"/"локал"/"локальный" → Ollama Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands) - "переключись на локальный" / "перейди на оллама" → swap to Ollama - "переключись на облако" / "используй грок" → swap to Groq - "какой у тебя мозг" / "облако или локально" → status query Underlying Lua API (registered globally): jarvis.llm({messages}, opts) -- as before, now uses global client jarvis.llm_status() -> "groq" | "ollama" | "none" jarvis.llm_switch(name) -> name on success, nil on failure Media keys pack (resources/commands/media_keys/, 4 commands) - "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE - "следующий трек" → VK_MEDIA_NEXT_TRACK - "предыдущий трек" → VK_MEDIA_PREV_TRACK - "стоп музыка" → VK_MEDIA_STOP - Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type). - Works with anything that listens to global media keys: Spotify, Yandex Music, YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys. Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands) - "укажи проект <path>" → stores path in jarvis.memory under codebase.root - "какой проект сейчас" → speaks current path - "что делает функция X" / "найди в коде Y" / "объясни код" → walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total), filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM with a "senior reviewer" system prompt, speaks 3-5 sentence answer. Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs) - new pub fn find_by_text(query, limit) -> Vec<ScheduledTask> - new pub fn remove_by_text(query) -> usize (count removed) - new pure helper find_by_text_in(tasks, query, limit) for tests - Lua: jarvis.scheduler.{find_by_text, remove_by_text} - Voice: "отмени напоминание про воду" / "удали задачу про разминку" - 3 new unit tests (52 total in jarvis-core). TTS pre-warm (crates/jarvis-app/src/main.rs) - Call jarvis_core::tts::backend() once on startup so the first speak() doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak latency by ~150-300ms depending on backend. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 52/52 jarvis-core unit tests pass.
2026-05-15 16:15:59 +03:00
// Initialise the shared global client (idempotent — fine if init was already called).
if let Err(e) = llm::init_global() {
warn!("LLM fallback disabled: {}. Set GROQ_TOKEN or run Ollama to enable.", e);
return None;
}
let lang = i18n::get_language();
let prompt = config::get_llm_system_prompt(&lang);
let history = Mutex::new(ConversationHistory::new(prompt, config::LLM_DEFAULT_MAX_HISTORY));
feat: hot-swap LLM backend + media keys + codebase Q&A + scheduler cancel-by-text + TTS pre-warm Single biggest user-facing change: you can now switch local↔cloud LLM by voice without restarting. Plus four new packs and a perf nudge. Shared LLM global (crates/jarvis-core/src/llm/mod.rs) - GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth. - init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str). - llm_fallback + llm_router migrated: they no longer own a separate LlmClient, they read from the global on each request. Hot-swap is now LIVE — change backend, next chat call uses the new one. - parse_backend accepts both english and russian aliases: "groq"/"cloud"/"облако"/"клауд" → Groq "ollama"/"local"/"локал"/"локальный" → Ollama Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands) - "переключись на локальный" / "перейди на оллама" → swap to Ollama - "переключись на облако" / "используй грок" → swap to Groq - "какой у тебя мозг" / "облако или локально" → status query Underlying Lua API (registered globally): jarvis.llm({messages}, opts) -- as before, now uses global client jarvis.llm_status() -> "groq" | "ollama" | "none" jarvis.llm_switch(name) -> name on success, nil on failure Media keys pack (resources/commands/media_keys/, 4 commands) - "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE - "следующий трек" → VK_MEDIA_NEXT_TRACK - "предыдущий трек" → VK_MEDIA_PREV_TRACK - "стоп музыка" → VK_MEDIA_STOP - Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type). - Works with anything that listens to global media keys: Spotify, Yandex Music, YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys. Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands) - "укажи проект <path>" → stores path in jarvis.memory under codebase.root - "какой проект сейчас" → speaks current path - "что делает функция X" / "найди в коде Y" / "объясни код" → walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total), filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM with a "senior reviewer" system prompt, speaks 3-5 sentence answer. Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs) - new pub fn find_by_text(query, limit) -> Vec<ScheduledTask> - new pub fn remove_by_text(query) -> usize (count removed) - new pure helper find_by_text_in(tasks, query, limit) for tests - Lua: jarvis.scheduler.{find_by_text, remove_by_text} - Voice: "отмени напоминание про воду" / "удали задачу про разминку" - 3 new unit tests (52 total in jarvis-core). TTS pre-warm (crates/jarvis-app/src/main.rs) - Call jarvis_core::tts::backend() once on startup so the first speak() doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak latency by ~150-300ms depending on backend. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 52/52 jarvis-core unit tests pass.
2026-05-15 16:15:59 +03:00
info!("LLM fallback enabled (backend: {}).", llm::current_backend_name());
Some(State {
history,
max_tokens: config::LLM_DEFAULT_MAX_TOKENS,
})
}
pub fn is_enabled() -> bool {
STATE.get().and_then(|s| s.as_ref()).is_some()
}
pub fn extract_prompt(text: &str) -> Option<String> {
let lang = i18n::get_language();
let triggers = config::get_llm_trigger_phrases(&lang);
let lowered = text.to_lowercase();
for trig in triggers {
if let Some(rest) = strip_trigger(&lowered, trig) {
let trimmed = rest.trim();
if !trimmed.is_empty() {
return Some(trimmed.to_string());
}
}
}
None
}
fn strip_trigger(text: &str, trigger: &str) -> Option<String> {
let t = text.trim_start();
if let Some(after) = t.strip_prefix(trigger) {
let next = after.chars().next();
if next.map_or(true, |c| !c.is_alphanumeric()) {
return Some(after.to_string());
}
}
None
}
pub fn handle(prompt: &str) {
let state = match STATE.get().and_then(|s| s.as_ref()) {
Some(s) => s,
None => {
warn!("LLM fallback called while disabled — ignoring.");
return;
}
};
info!("LLM prompt: {}", prompt);
let snapshot: Vec<ChatMessage> = {
let mut h = state.history.lock();
h.push_user(prompt);
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
let mut snap = h.snapshot();
// Inject (a) profile personality (b) relevant long-term memory as a fresh
// system message right after the base prompt. Both are optional.
let profile = profiles::active();
let mut overlay = String::new();
if !profile.llm_personality.is_empty() {
overlay.push_str(&format!("Активный профиль: {} {}\nХарактер для ответа: {}\n",
profile.icon, profile.name, profile.llm_personality));
}
let mem_ctx = long_term_memory::build_context(prompt, 5);
if !mem_ctx.is_empty() {
overlay.push_str(&mem_ctx);
}
if !overlay.is_empty() {
// Insert after the base system prompt (index 0 typically), before user msgs.
let insert_at = if !snap.is_empty() && snap[0].role == "system" { 1 } else { 0 };
snap.insert(insert_at, ChatMessage::system(overlay));
}
snap
};
feat: hot-swap LLM backend + media keys + codebase Q&A + scheduler cancel-by-text + TTS pre-warm Single biggest user-facing change: you can now switch local↔cloud LLM by voice without restarting. Plus four new packs and a perf nudge. Shared LLM global (crates/jarvis-core/src/llm/mod.rs) - GLOBAL: Lazy<RwLock<Option<Arc<LlmClient>>>>. Single source of truth. - init_global() / current() / swap_to(LlmBackend) / current_backend_name() / parse_backend(str). - llm_fallback + llm_router migrated: they no longer own a separate LlmClient, they read from the global on each request. Hot-swap is now LIVE — change backend, next chat call uses the new one. - parse_backend accepts both english and russian aliases: "groq"/"cloud"/"облако"/"клауд" → Groq "ollama"/"local"/"локал"/"локальный" → Ollama Voice + Lua LLM switcher (resources/commands/llm_switch/, 3 commands) - "переключись на локальный" / "перейди на оллама" → swap to Ollama - "переключись на облако" / "используй грок" → swap to Groq - "какой у тебя мозг" / "облако или локально" → status query Underlying Lua API (registered globally): jarvis.llm({messages}, opts) -- as before, now uses global client jarvis.llm_status() -> "groq" | "ollama" | "none" jarvis.llm_switch(name) -> name on success, nil on failure Media keys pack (resources/commands/media_keys/, 4 commands) - "пауза" / "плей" → VK_MEDIA_PLAY_PAUSE - "следующий трек" → VK_MEDIA_NEXT_TRACK - "предыдущий трек" → VK_MEDIA_PREV_TRACK - "стоп музыка" → VK_MEDIA_STOP - Helper _media.ps1 uses user32.keybd_event (P/Invoke through Add-Type). - Works with anything that listens to global media keys: Spotify, Yandex Music, YouTube (focused tab), Foobar2000, Winamp, etc. No OAuth, no API keys. Codebase Q&A pack (resources/commands/codebase_qa/, 3 commands) - "укажи проект <path>" → stores path in jarvis.memory under codebase.root - "какой проект сейчас" → speaks current path - "что делает функция X" / "найди в коде Y" / "объясни код" → walks the folder (depth 3, 30 files cap, 4KB per file, 50KB total), filters by source extensions (rs/py/ts/lua/go/...), feeds digest to LLM with a "senior reviewer" system prompt, speaks 3-5 sentence answer. Scheduler cancel-by-text (crates/jarvis-core/src/scheduler.rs) - new pub fn find_by_text(query, limit) -> Vec<ScheduledTask> - new pub fn remove_by_text(query) -> usize (count removed) - new pure helper find_by_text_in(tasks, query, limit) for tests - Lua: jarvis.scheduler.{find_by_text, remove_by_text} - Voice: "отмени напоминание про воду" / "удали задачу про разминку" - 3 new unit tests (52 total in jarvis-core). TTS pre-warm (crates/jarvis-app/src/main.rs) - Call jarvis_core::tts::backend() once on startup so the first speak() doesn't pay Piper binary discovery + voice loading cost. Cuts first-speak latency by ~150-300ms depending on backend. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 52/52 jarvis-core unit tests pass.
2026-05-15 16:15:59 +03:00
let client = match llm::current() {
Some(c) => c,
None => {
warn!("LLM call but global client missing — ignoring.");
return;
}
};
match client.complete(&snapshot, state.max_tokens) {
Ok(reply) => {
let reply = reply.trim().to_string();
info!("LLM reply: {}", reply);
state.history.lock().push_assistant(reply.clone());
ipc::send(IpcEvent::LlmReply { text: reply.clone() });
voices::play_ok();
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
speak_reply(&reply);
}
Err(e) => {
error!("LLM request failed: {}", e);
state.history.lock().pop_last_user();
let err_text = config::LLM_FALLBACK_ERROR_RU.to_string();
ipc::send(IpcEvent::LlmReply { text: err_text.clone() });
voices::play_error();
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
speak_reply(&err_text);
}
}
}
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
fn speak_reply(text: &str) {
refactor: maintainability pass — dedup, central env config, cmd helpers, tests, ARCHITECTURE.md User asked the code stays easy to fix/extend as the feature surface grows. This commit refactors what was duplicated, centralises what was scattered, adds tests where there were none, and writes the architecture doc that contributors will need. Deduplication - tts::play_wav extracted to tts/mod.rs as pub(crate). Was identical in piper.rs and silero.rs (~25 lines × 2). Both backends now call super::play_wav. Centralised env config - new crates/jarvis-core/src/runtime_config.rs — single doc-file for every JARVIS_* / GROQ_* / OLLAMA_* env var. Includes: - ENV_* constants with doc comments - get(name) / get_bool(name, default) / get_parse(name, default) helpers - feature-flag wrappers: llm_tts_enabled(), llm_router_enabled(), llm_router_threshold() - log_effective_config() prints active values on startup - migrated llm_fallback (JARVIS_LLM_TTS), llm_router (JARVIS_LLM_ROUTER, JARVIS_LLM_ROUTER_THRESHOLD) to use the new helpers. Pattern set for future migrations. Lua boilerplate killer - new crates/jarvis-core/src/lua/api/cmd.rs exposing: jarvis.cmd.ok(msg?) — play_ok + speak + {chain=false} jarvis.cmd.chain_ok(msg?) — same but chain=true jarvis.cmd.error(msg?) — play_error + speak + {chain=false} jarvis.cmd.not_found(msg?) — play_not_found + speak + {chain=false} jarvis.cmd.silent_ok / silent_error - refactored 5 packs (daily_briefing/off, memory_pack/list, pomodoro/stop, habit_nudge/stop_all, scheduler/clear) — each lost 3-4 lines of repetitive play_*/speak/return boilerplate. Pattern for future packs documented in ARCHITECTURE.md. Tests (was 32, now 49) - long_term_memory: 8 new tests for normalize_key, search_in (rank, limit, empty), build_context_from (empty + populated), serde round-trips for MemoryRecord and Store. Extracted pure logic (search_in, build_context_from) into pub(crate) functions to enable testing without global state. - profiles: 6 new tests for Profile::allows_command (empty/whitelist/blacklist/ deny-wins-over-allow), serde round-trip with all fields + minimal-fields tolerance via #[serde(default)]. - runtime_config: 2 tests for get_bool / get_parse defaults. - All 49 tests pass. New mood/energy log pack - resources/commands/mood_log/ with 2 commands: mood.record "запиши настроение 7" / "сегодня мне грустно" → stores timestamped entry via jarvis.memory.remember mood.recap "как прошла неделя" → LLM summarises last 30 entries - Showcase: composes memory + llm + cmd helpers in <30 lines per script. ARCHITECTURE.md (new, 250 lines) - Crate layout, data flow diagram (mic→action 10 steps), per-module responsibility table, configuration layers, TTS pipeline diagram, Lua sandbox details with API quick-ref, background-services overview, "how to add a pack/feature/TTS backend" recipes, test coverage map, build instructions with MSVC env, git workflow with Forgejo NO_PROXY trick. - Aimed at someone who just cloned the repo and needs to fix a bug fast. Build: cargo build --release -p jarvis-app -p jarvis-gui both green. Tests: 49/49 jarvis-core unit tests pass.
2026-05-15 16:06:18 +03:00
if !jarvis_core::runtime_config::llm_tts_enabled() {
return;
}
feat: TTS backend abstraction + 4 'imba' features + Lua packs Big push driven by user feedback ("делай имбу") and web research on what voice assistants need to be the ideal: TTS backend abstraction (P0.1) - new module crates/jarvis-core/src/tts/{mod,sapi,piper,silero}.rs - TtsBackend trait with SapiBackend (current PowerShell), PiperBackend (rhasspy/piper, neural quality), SileroBackend (python subprocess) - JARVIS_TTS env var picks (sapi|piper|silero). Auto-detect Piper if binary + voice present in tools/piper/. Falls back to SAPI on missing. - SpeakOpts {lang, detached, raw} replaces ad-hoc args. text_utils sanitiser applied unless raw=true. - llm_fallback + lua/api/tts both routed through tts::backend(). - tools/piper/install.ps1 downloads piper.exe + ru_RU-irina-medium.onnx from rhasspy releases + huggingface. Smoke-test included. - tools/silero/silero_tts.py helper (PyTorch); rust spawns it as subprocess. IMBA-1 Agentic LLM router - crates/jarvis-app/src/llm_router.rs - When fuzzy/intent matcher fails, LLM picks the closest command from the full registry. Returns JSON {command_id, confidence, reason}. - Threshold-gated re-dispatch via substitute phrase. JARVIS_LLM_ROUTER=1 enables; JARVIS_LLM_ROUTER_THRESHOLD overrides 0.55 default. - Inserted in app.rs::execute_command between "no match" and existing llm_fallback chat fallback. IMBA-2 Long-term memory - crates/jarvis-core/src/long_term_memory.rs — JSON store at APP_CONFIG_DIR/long_term_memory.json. Atomic write-through. - remember/recall/search/forget/all/build_context API. - Lua bindings: jarvis.memory.* (5 functions). - llm_fallback auto-injects relevant facts (substring search of prompt) into system message before LLM call. - Pack resources/commands/memory_pack/ with 4 commands: remember, recall, forget, list. IMBA-3 Profile switching (work/game/sleep/driving/default) - crates/jarvis-core/src/profiles.rs — JSON profiles at APP_CONFIG_DIR/profiles/ Auto-seeds 5 defaults on first run with personality + allow/deny lists + greetings + emoji icons. - active_profile.txt persists choice across restart. - Lua bindings: jarvis.profile.{active,set,list,allows,active_name}. - llm_fallback prepends profile personality to system prompt. - Pack resources/commands/profile_switch/ with 6 voice triggers. IMBA-4 Multimodal screenshot + vision LLM - crates/jarvis-core/src/lua/api/vision.rs — gated on HTTP sandbox. - jarvis.vision.screenshot() captures via PowerShell System.Drawing. - jarvis.vision.describe(prompt?) sends base64 PNG to Groq vision model (default llama-3.2-11b-vision-preview, override via GROQ_VISION_MODEL). - Pack resources/commands/vision/ with 2 commands: describe + read_error. P0.2 Continuous conversation grace window - config::CONVERSATION_GRACE_MS = 30_000. - app.rs: after command result, if grace_ms > 0 keep listening WITHOUT re-wake for the grace duration. Existing CMS_WAIT_DELAY back-dated so the existing timeout fires at start + grace_ms. Tests: 24/24 jarvis-core unit tests pass (including 5 text_utils). Build: cargo build --release -p jarvis-app and -p jarvis-gui both succeed on Windows MSVC (VS 2026 Enterprise vcvars64). Notes for setup: - Piper voice install: pwsh tools/piper/install.ps1 (downloads ~90 MB). - GROQ_TOKEN needed for IMBA-1 (router) and IMBA-4 (vision). - All features are opt-in via env vars or auto-detect; existing SAPI + fuzzy match path remains the default.
2026-05-15 15:32:44 +03:00
tts::speak(text, &SpeakOpts::lang("ru"));
}