7655720e11
An Assistant drawer in Sirius Studio with two providers and no key, no server and no spend. "This browser" runs an open model on the user's GPU through WebGPU with WebLLM (vendored, Apache-2.0); weights download once from the MLC mirror and stay in the browser cache. "Local endpoint" talks to an OpenAI-compatible runtime on the user's machine (Ollama, LM Studio), which unlocks larger models on a real GPU. Nothing the user writes or builds leaves their device in either mode. Three verbs: make a section, rewrite the selected text, restyle the selection. The model returns HTML and CSS as data; Studio sanitises it (no scripts, frames, handlers or imports) and inserts it through the editor, with Undo. Model output is never executed. The quantisation is chosen per GPU: q4f16 when the adapter exposes shader-f16, q4f32 otherwise (Pascal-era cards lack it). Small models often answer with bare HTML instead of JSON, so the parser accepts both, prompts avoid literal placeholders one model echoed back, and an out-of-memory or disposed runtime is reported as "pick a smaller model" with the engine reset. Switching models starts a fresh worker. Verified on an NVIDIA Pascal card: SmolLM2 360M rewrites text, Qwen2.5 Coder 0.5B builds a section; the 1.5B f32 build exceeded that card's memory and now fails gracefully.
7 lines
385 B
JavaScript
7 lines
385 B
JavaScript
// Web Worker for the in-browser assistant: runs the WebLLM engine off the UI
|
|
// thread. Weights and the model library are fetched by WebLLM itself and
|
|
// cached by the browser; nothing is sent anywhere.
|
|
import { WebWorkerMLCEngineHandler } from "../vendor/web-llm/index.js?v=0.2.85";
|
|
|
|
const handler = new WebWorkerMLCEngineHandler();
|
|
self.onmessage = (msg) => handler.onmessage(msg);
|