feat(studio): AI assistant that runs on the user's own device
An Assistant drawer in Sirius Studio with two providers and no key, no
server and no spend. "This browser" runs an open model on the user's
GPU through WebGPU with WebLLM (vendored, Apache-2.0); weights download
once from the MLC mirror and stay in the browser cache. "Local
endpoint" talks to an OpenAI-compatible runtime on the user's machine
(Ollama, LM Studio), which unlocks larger models on a real GPU. Nothing
the user writes or builds leaves their device in either mode.
Three verbs: make a section, rewrite the selected text, restyle the
selection. The model returns HTML and CSS as data; Studio sanitises it
(no scripts, frames, handlers or imports) and inserts it through the
editor, with Undo. Model output is never executed.
The quantisation is chosen per GPU: q4f16 when the adapter exposes
shader-f16, q4f32 otherwise (Pascal-era cards lack it). Small models
often answer with bare HTML instead of JSON, so the parser accepts
both, prompts avoid literal placeholders one model echoed back, and an
out-of-memory or disposed runtime is reported as "pick a smaller
model" with the engine reset. Switching models starts a fresh worker.
Verified on an NVIDIA Pascal card: SmolLM2 360M rewrites text, Qwen2.5
Coder 0.5B builds a section; the 1.5B f32 build exceeded that card's
memory and now fails gracefully.
2026-09-20 15:22:34 +02:00
|
|
|
// Web Worker for the in-browser assistant: runs the WebLLM engine off the UI
|
|
|
|
|
// thread. Weights and the model library are fetched by WebLLM itself and
|
|
|
|
|
// cached by the browser; nothing is sent anywhere.
|
|
|
|
|
import { WebWorkerMLCEngineHandler } from "../vendor/web-llm/index.js?v=0.2.85";
|
|
|
|
|
|
|
|
|
|
const handler = new WebWorkerMLCEngineHandler();
|
|
|
|
|
self.onmessage = (msg) => handler.onmessage(msg);
|