diff --git a/DESIGN-bns-indexer-service.md b/DESIGN-bns-indexer-service.md index 28e363ba..68e624a6 100644 --- a/DESIGN-bns-indexer-service.md +++ b/DESIGN-bns-indexer-service.md @@ -11,6 +11,8 @@ Already implemented: modules, the indexer/resolver split, the installer. See "Implementation notes" at the end for where the code deviates from this text, and what step 3 (the Theseus client) gets from the pipe. +- Migration step 3 (Theseus, not released yet): `bns-indexer.js` is now the + index host described under "Theseus" (see "Step 3 in Theseus" at the end). ## The model @@ -468,13 +470,43 @@ settings: - **Settings:** edit `policy.json` directly (`mode`, `power`, `economyWhen`, `startAtBoot`), and run the task where `ariadne-ctl` does (mode -> all-browsers, start-at-boot -> on). Task names: - - All users: `BNS Indexer`, `BNS Resolver Daemon` (unchanged, so 0.3.70's + - All users: `Ariadne BNS Indexer`, `BNS Resolver Daemon` (unchanged, so 0.3.70's panel still finds it); - - Just me: `BNS Indexer ()`, `BNS Resolver ()`. + - Just me: `Ariadne BNS Indexer ()`, `Ariadne BNS Resolver ()`. - **Until step 3 ships:** Theseus 0.3.70's Ariadne panel only knows `BNS Resolver Daemon`. In Theseus-only mode that task is Ready, not Running, so the panel shows Ariadne as off. +### Step 3 in Theseus +- **Where it runs:** `bns-indexer.js` stays the utilityProcess with the same + messages to main.js, so main.js changed only in small ways: the shared + module paths, a `status` message, and the Ariadne panel. Inside, it runs + the source chain, and runs the shared core as Theseus's own indexer only + on standby. +- **Packaging:** the shared modules ship as `bns-index-core.mjs`, + `bns-source-chain.mjs` and `bns-pipe.mjs` in the resources, like + `resolver-web.mjs`. That is why they import nothing from each other. +- **Finding Ariadne:** `InstallLocation` from its uninstall key (HKLM, then + HKCU), which gives the helper and the expected server image. Its files + are read from both scopes' `index\` folders even without the helper. The + copy with the most evidence wins, compared against Theseus's own raw copy + and the bundled one. +- **Health:** checked every 10 s. + - Healthy means: the pipe is connected, and Ariadne is either paused + (Economy) or confirmed its index within 10 min. + - Otherwise, 4 s after launch Theseus starts its own indexer, warm from + the richest copy (normally Ariadne's file). + - It hands back after 90 s of health. +- **Own copies:** + - `bns-name-index.json` (index format, from the pipe); + - `bns-name-snapshot.json` / `.prev.json` (raw, only while the own + indexer runs). +- **What does not change:** the quick single-name lookup stays in main.js + (it goes through Tor and the session proxy); the source chain in the host + has its public sources switched off. +- **Not done:** controls in Settings for mode, Power and Launch at start. + The panel's on/off switch still acts on the resolver task, as before. + ### Bundling Ariadne into the Theseus installer (designed, not wired) - **Payload:** the Ariadne setup exe, as an `extraResources` file of Theseus's NSIS build, so the two keep separate version lifecycles. diff --git a/bns-indexer.js b/bns-indexer.js index b6d40aa8..1bad3556 100644 --- a/bns-indexer.js +++ b/bns-indexer.js @@ -1,31 +1,47 @@ -// BNS name indexer — runs in its own process (Electron utilityProcess), -// started by main.js, so the snapshot parsing, index builds and electrum -// traffic never compete with the browser's main thread. +// BNS index host. Runs in its own process (Electron utilityProcess), started +// by main.js, so snapshot parsing, index builds, the Ariadne pipe and any +// electrum traffic never compete with the browser's main thread. +// Spec: DESIGN-bns-indexer-service.md ("Theseus"). // -// Boot is two-phase: -// 1. fast lane: load the local snapshot (user cache > bundled), build the -// index from it and publish it. No network. This is what the first tab -// needs, and it is all that runs while the browser is still starting. -// 2. full work, once main says "go" (its first page has loaded): electrum -// server discovery, the Sia snapshot refresh, and the 30 s delta poll. -// A lookup that misses the snapshot before then still gets a live -// delta poll on demand — only that, nothing else. +// Ariadne's Thread owns BNS indexing on a machine; Theseus reads from it +// through the shared source chain (Argus/src/lib/bns-source-chain.js): +// 1. Ariadne's indexer over its pipe, server-checked (ariadne-helper.exe): +// a snapshot, then pushes +// 2. local copies: Ariadne's two files (All users / Just me), Theseus's own +// raw copy, the bundled snapshot (the one with the most evidence wins), +// and Theseus's own index copy written from the pipe +// 3. (main.js) the quick single-name lookup on the gateway, provisional +// Theseus's own electrum indexer (the shared core, bns-index-core.js) is a +// standby, the last resort: +// - started when Ariadne is unhealthy: no pipe a few seconds after launch, +// a pipe that fails the server check, a pipe that went silent, or an +// index that is not being confirmed while Ariadne says it is not paused. +// Warm-started from the richest local copy (normally Ariadne's), so no +// download and no cold sync. +// - stopped again once Ariadne has been healthy for 90 s, so a flapping +// service cannot start and stop it over and over. +// Under Economy Ariadne is paused, not unhealthy: no takeover. +// With no Ariadne at all (portable, not installed, a pre-0.2 install) it is +// the 0.3.70 behaviour: the own indexer runs throughout. // -// Protocol (process.parentPort messages): -// main -> indexer { type: "init", resolverPath, userData, bundledSnapshot, torPort } -// { type: "go" } start phase 2 -// { type: "tor", port } electrum via Tor (null = direct) -// { type: "stop-polling" } -// { id, type: "poll" | "rebuild" | "ready" } replies { type: "reply", id, ok } -// indexer -> main { type: "index", builtAt, names: [[key, entry]...], network } -// { type: "fresh", builtAt } poll succeeded, nothing changed -// { type: "reply", id, ok, error? } +// The only thing Theseus ever does to Ariadne: when "Launch at start" is off, +// it runs the indexer's task once at launch. It never writes Ariadne's files. // -// The index entries are the resolver's own (records, owner, txid, height, …) -// built from the raw beacon transactions — the trust model is unchanged. +// Protocol (process.parentPort messages), unchanged for main.js: +// main -> host { type: "init", resolverPath, coreLib, chainLib, pipeLib, +// userData, bundledSnapshot, torPort } +// { type: "go" } network work may start +// { type: "tor", port } electrum via Tor (null = direct) +// { type: "stop-polling" } +// { id, type: "poll" | "rebuild" | "ready" } replies { type: "reply", id, ok } +// host -> main { type: "index", builtAt, names: [[key, entry]...], network, source } +// { type: "fresh", builtAt } confirmed, nothing changed +// { type: "status", ... } where the names come from (Settings) +// { type: "reply", id, ok, error? } "use strict"; const fs = require("fs"); const path = require("path"); +const { execFile } = require("child_process"); const { pathToFileURL } = require("url"); const WebSocket = require("ws"); @@ -41,17 +57,16 @@ const log = (...a) => console.log("[bns]", ...a); return close.apply(this, args); }; } -// A flaky network must not take the indexer down; main restarts it if it -// exits, but the index would go cold for a moment. process.on("uncaughtException", (err) => { try { console.error("[bns] uncaught:", (err && err.stack) || err); } catch {} }); process.on("unhandledRejection", (err) => { try { console.error("[bns] unhandled rejection:", (err && err.message) || err); } catch {} }); -let cfg = null; -let R = null; -async function getResolver() { - if (!R) R = await import(pathToFileURL(cfg.resolverPath).href); - return R; -} +const LAUNCH_GRACE_MS = 4_000; // the pipe gets this long before the own indexer starts +const STALE_MS = 10 * 60_000; // unpaused Ariadne whose index is not confirmed this long: unhealthy +const HANDBACK_MS = 90_000; // healthy this long before the own indexer is stopped +const CHECK_EVERY_MS = 10_000; + +let cfg = null, R = null, core = null, chainLib = null, pipeLib = null; +const importEsm = (p) => import(pathToFileURL(p).href); // ---- Tor ------------------------------------------------------------------ let torAgent = null; @@ -63,248 +78,232 @@ async function setTor(torPort) { class TorWebSocket extends WebSocket { constructor(url, opts) { super(url, { agent: torAgent, ...opts }); } } const currentWS = () => (torAgent ? TorWebSocket : WebSocket); -// ---- electrum server pool: hardcoded seed + on-chain discovery, persisted ---- -// Bootstrap from the baked-in seed (with pinned IPs), then refresh from the -// on-chain ELECTRUM_LIST_NAME record so the pool can be rotated without a new -// build. The last discovered list is cached to disk and tried first next launch. -let electrumPool = null; -let lastElectrumRefresh = 0; -// Per-network cache files. Chipnet keeps the historical names so existing -// profiles are not invalidated; any other network gets its own files. +// ---- Ariadne's Thread on this machine --------------------------------------- +// Each scope it may be installed in: where its files are, its pipe, and how +// to check the pipe's server (Ariadne's helper; the server's image is +// Ariadne's own node.exe). The install folder comes from Ariadne's uninstall +// key. A pre-0.2 Ariadne has no helper and no pipe: only files (none either). +const ARIADNE_KEY = "Software\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{7E7A5F1C-3B4E-4C8A-9E1D-ARIADNERSLVR}_is1"; +function execOut(file, args) { + return new Promise((resolve) => execFile(file, args, { windowsHide: true, timeout: 10_000 }, (err, out) => resolve(err ? "" : String(out)))); +} +async function installLocation(hive) { + const out = await execOut("reg.exe", ["query", `${hive}\\${ARIADNE_KEY}`, "/v", "InstallLocation", "/reg:64"]); + const m = /InstallLocation\s+REG_\w+\s+(.+)/.exec(out); + return m ? m[1].trim() : null; +} +async function findAriadne() { + // Tests point Theseus at a scratch Ariadne: {scope, pipe, stateDir, helper, imagePrefix}. + if (process.env.THESEUS_ARIADNE_TEST) { + try { return [JSON.parse(process.env.THESEUS_ARIADNE_TEST)]; } catch { return []; } + } + if (process.platform !== "win32") return []; + const scopes = []; + const programData = process.env.ProgramData || process.env.PROGRAMDATA || "C:\\ProgramData"; + const localAppData = process.env.LOCALAPPDATA || ""; + const machineLoc = await installLocation("HKLM"); + scopes.push({ + scope: "machine", pipe: "\\\\.\\pipe\\ariadne-bns", task: "Ariadne BNS Indexer", + stateDir: path.join(programData, "Ariadne"), loc: machineLoc, + }); + const userLoc = await installLocation("HKCU"); + if (userLoc || (localAppData && fs.existsSync(path.join(localAppData, "Ariadne", "index")))) { + const sid = (/"(S-1-[\d-]+)"/.exec(await execOut("whoami", ["/user", "/fo", "csv", "/nh"])) || [])[1]; + scopes.push({ + scope: "user", pipe: sid ? `\\\\.\\pipe\\ariadne-bns-${sid}` : null, task: `Ariadne BNS Indexer (${process.env.USERNAME || ""})`, + stateDir: path.join(localAppData, "Ariadne"), loc: userLoc, + }); + } + for (const s of scopes) { + if (!s.loc) continue; + const exe = path.join(s.loc, "tools", "ariadne-helper.exe"); + if (fs.existsSync(exe)) { s.helper = { exe }; s.imagePrefix = path.join(s.loc, "runtime") + path.sep; } + } + return scopes; +} +const ariadneCopies = (scopes) => scopes.flatMap((s) => [ + path.join(s.stateDir, "index", "bns-name-snapshot.json"), + path.join(s.stateDir, "index", "bns-name-snapshot.prev.json"), +]); + +// "Launch at start" off: the indexer runs on demand, i.e. now. The one thing +// Theseus starts on Ariadne's side; its task lets local users run it. +function runIndexerOnDemand(scopes) { + for (const s of scopes) { + if (!s.task || !s.loc) continue; + let pol = null; + try { pol = JSON.parse(fs.readFileSync(path.join(s.stateDir, "policy.json"), "utf8").replace(/^\uFEFF/, "")); } catch {} + if (pol && pol.startAtBoot === false) { + execFile("schtasks.exe", ["/Run", "/TN", s.task], { windowsHide: true }, (err) => { + log(err ? `Ariadne's indexer (${s.scope}) could not be started on demand` : `Ariadne's indexer (${s.scope}) started on demand (Launch at start is off)`); + }); + } + } +} + +// ---- state ------------------------------------------------------------------ let networkId = "chipnet"; const netSuffix = () => (networkId === "chipnet" ? "" : `.${networkId}`); -const electrumFile = () => path.join(cfg.userData, `electrum-servers${netSuffix()}.json`); -const snapshotUserPath = () => path.join(cfg.userData, `bns-name-snapshot${netSuffix()}.json`); -const serverKey = (s) => (typeof s === "string" ? s : s && s.url); -function mergeServers(preferred, rest) { - const seen = new Set(), out = []; - for (const s of [...(preferred || []), ...(rest || [])]) { - const k = serverKey(s); - if (k && !seen.has(k)) { seen.add(k); out.push(s); } - } - return out; +let scopes = []; +let chain = null; // the source chain (Ariadne + local copies) +let own = null; // Theseus's own indexer, while it runs +let ownReason = null; +let goReceived = false; +let launchedAt = Date.now(); +let healthySince = 0; +let lastPublished = null; // which index main has: "chain" | "own" + +function active() { return own && own.index ? "own" : "chain"; } +function publish(reason) { + const which = active(); + const idx = which === "own" ? own.index : chain && chain.index; + if (!idx || !idx.size) return; + lastPublished = which; + const builtAt = which === "own" ? own.builtAt : chain.builtAt; + port.postMessage({ type: "index", builtAt, names: [...idx], network: networkId, reason, source: which === "own" ? "theseus-indexer" : chain.source }); + sendStatus(); } -async function initElectrumPool() { - const r = await getResolver(); - const seed = r.ELECTRUM || r.CHIPNET_ELECTRUM; - let saved = []; - try { if (fs.existsSync(electrumFile())) saved = JSON.parse(fs.readFileSync(electrumFile(), "utf8")); } catch {} - electrumPool = mergeServers(saved, seed); // discovered first, seed always kept +function sendStatus() { + const cs = chain ? chain.status() : null; + port.postMessage({ + type: "status", + source: active() === "own" ? "theseus-indexer" : cs && cs.source, + ariadne: { + installed: scopes.filter((s) => s.loc).map((s) => s.scope), + pipe: cs ? cs.pipe : null, + paused: cs ? cs.paused : false, + builtAt: cs ? cs.builtAt : 0, + healthy: ariadneHealthy(), + }, + ownIndexer: own ? { running: true, reason: ownReason, ...own.status() } : { running: false }, + }); } -async function refreshElectrumPool() { - try { - const { fetchElectrumServers } = await getResolver(); - const found = await fetchElectrumServers({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }); - if (found && found.length) { - electrumPool = mergeServers(found, electrumPool); - try { fs.writeFileSync(electrumFile(), JSON.stringify(found, null, 2)); } catch {} + +// ---- health ----------------------------------------------------------------- +function ariadneHealthy() { + if (!chain || !chain.pipeConnected) return false; + // Economy: Ariadne pauses its network work and stops confirming. That is a + // setting, not a failure; taking over would undo it. + if (chain.paused) return true; + return Date.now() - chain.builtAt < STALE_MS; +} +function checkHealth() { + const now = Date.now(); + if (ariadneHealthy()) { + healthySince = healthySince || now; + if (own && now - healthySince >= HANDBACK_MS) stopOwn(); + } else { + healthySince = 0; + if (!own && now - launchedAt >= LAUNCH_GRACE_MS) { + const cs = chain && chain.status(); + const why = !scopes.some((s) => s.helper) ? "no Ariadne indexer installed" + : cs && cs.pipe && cs.pipe.error ? `Ariadne pipe: ${cs.pipe.error}` + : chain && chain.pipeConnected ? "Ariadne's index is not being confirmed" + : "Ariadne's indexer is not answering"; + startOwn(why); } - } catch { /* list unpublished or unreachable — keep the current pool */ } -} -function maybeRefreshElectrum() { - if (!started) return; // phase 2 only - if (Date.now() - lastElectrumRefresh < 30 * 60 * 1000) return; - lastElectrumRefresh = Date.now(); - refreshElectrumPool(); -} - -// ---- index state + publishing --------------------------------------------- -let index = null; // Map name -> entry -let currentSnapshotState = null; // raw { beacon, history, txs, … } behind `index` -let lastSig = ""; -// Cheap fingerprint of the snapshot: an index rebuilt from the same history -// at the same heights is the same index, so main isn't sent a copy. -const sigOf = (snap) => `${snap.history.length}:${snap.history.reduce((a, h) => a + (Number(h.height) || 0), 0)}`; -function publish(idx, snap, reason) { - index = idx; - const sig = snap ? sigOf(snap) : `full:${Date.now()}`; - const builtAt = reason === "snapshot" ? 0 : Date.now(); // a snapshot is a floor, not a ceiling - if (sig === lastSig) { port.postMessage({ type: "fresh", builtAt }); return; } - lastSig = sig; - port.postMessage({ type: "index", builtAt, names: [...idx], network: networkId, reason }); -} - -function readSnapshotFrom(p) { - try { - if (!fs.existsSync(p)) return null; - const parsed = JSON.parse(fs.readFileSync(p, "utf8")); - if (!parsed || !Array.isArray(parsed.history)) return null; - return parsed; - } catch { return null; } -} - -// Phase 1: the local snapshot, nothing else. -async function warmFromSnapshot() { - const r = await getResolver(); - networkId = r.NETWORK?.id || "chipnet"; - if (!r.buildIndexFromSnapshot) return false; // an older resolver-web.js - const snap = readSnapshotFrom(snapshotUserPath()) || readSnapshotFrom(cfg.bundledSnapshot); - if (!snap) return false; - try { - const t0 = Date.now(); - const idx = r.buildIndexFromSnapshot({ snapshot: snap }); - currentSnapshotState = snap; - publish(idx, snap, "snapshot"); - log(`warm-started from snapshot: ${idx.size} names in ${Date.now() - t0} ms @ height=${snap.asOfHeight ?? "?"} root=${snap.root ?? "?"}`); - return true; - } catch (e) { - log("snapshot warm-start failed:", e.message); - return false; } } -// ---- continuous background delta refresh ---------------------------------- -// One electrum connection per poll: fetch the beacon's history (one call), -// fetch only the tx bodies not already held, rebuild locally, persist. -const POLL_INTERVAL_MS = 30_000; -// Neither the electrum connect nor its WebSocket handshake has a timeout of -// its own; a server that accepts TCP and then stalls would wedge the loop. -const POLL_DEADLINE_MS = 45_000; -let pollInFlight = null, pollTimer = null; -async function pollAndMerge() { - if (pollInFlight) return pollInFlight; - let conn = null, deadline; - const work = (async () => { - const r = await getResolver(); - if (!r.connectElectrum || !r.BEACON_SCRIPTHASH || !r.buildIndexFromSnapshot) return rebuild(); - if (!electrumPool) await initElectrumPool(); - // Base state: memory > user cache > bundled > empty. The "empty" branch - // turns a first start without any snapshot into a full fetch. - const snap = currentSnapshotState - || readSnapshotFrom(snapshotUserPath()) - || readSnapshotFrom(cfg.bundledSnapshot) - || { beacon: r.BEACON_SCRIPTHASH, history: [], txs: {} }; - const el = conn = await r.connectElectrum({ electrum: electrumPool, WebSocket: currentWS(), directIP: true }); - try { - const freshHistory = await el.call("blockchain.scripthash.get_history", [r.BEACON_SCRIPTHASH]); - const known = new Set(snap.history.map((h) => h.tx_hash)); - // Merge fresh into snapshot history (dedup by tx_hash, keep fresh height — - // an event that was mempool at snapshot time now has a real height). - const merged = new Map(snap.history.map((h) => [h.tx_hash, h])); - const txs = { ...(snap.txs || {}) }; - let added = 0; - for (const h of freshHistory) { - if (!known.has(h.tx_hash)) { - try { txs[h.tx_hash] = await el.call("blockchain.transaction.get", [h.tx_hash, true]); added++; } - catch { /* unreadable — the reduction rules ignore missing txs */ } - } - merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height }); +// ---- Theseus's own indexer (standby) ------------------------------------------ +function startOwn(reason) { + if (own) return own; + ownReason = reason; + log(`own indexer starting: ${reason}`); + own = core.createIndexer({ + resolver: R, + WebSocket: currentWS(), + snapshotDir: cfg.userData, + snapshotBase: `bns-name-snapshot${netSuffix()}`, + // Ariadne's last snapshot first in line: no download, no cold sync. + warmPaths: [...ariadneCopies(scopes), cfg.bundledSnapshot], + electrumCacheFile: path.join(cfg.userData, `electrum-servers${netSuffix()}.json`), + log: (...a) => log(...a), + }); + own.on("change", () => { if (own) publish("own"); }); + own.on("fresh", ({ builtAt }) => { if (own && active() === "own") port.postMessage({ type: "fresh", builtAt }); }); + own.warm(); + if (own.index && lastPublished !== "own") publish("own-warm"); // its "change" normally did + if (goReceived) own.start(); + sendStatus(); + return own; +} +function stopOwn() { + if (!own) return; + log("Ariadne healthy again: own indexer stopped"); + own.stop(); + own.removeAllListeners(); + own = null; ownReason = null; + publish("handback"); +} + +// ---- boot ------------------------------------------------------------------- +async function init(m) { + cfg = m; + await setTor(m.torPort); + [R, core, chainLib, pipeLib] = await Promise.all([importEsm(m.resolverPath), importEsm(m.coreLib), importEsm(m.chainLib), importEsm(m.pipeLib)]); + networkId = (R.NETWORK && R.NETWORK.id) || "chipnet"; + scopes = await findAriadne(); + launchedAt = Date.now(); + const checked = scopes.filter((s) => s.helper && s.pipe); + chain = chainLib.createSourceChain({ + resolver: R, + // Machine scope first: if both are installed, readers use the machine one. + connectPipe: checked.length ? async () => { + let last = null; + for (const s of checked) { + try { return await pipeLib.connectIndexerPipe({ pipeName: s.pipe, helper: s.helper, expect: { scope: s.scope, imagePrefix: s.imagePrefix } }); } + catch (e) { last = e; } } - const history = [...merged.values()]; - currentSnapshotState = { ...snap, beacon: r.BEACON_SCRIPTHASH, history, txs }; - const idx = r.buildIndexFromSnapshot({ snapshot: currentSnapshotState }); - publish(idx, currentSnapshotState, "poll"); - try { - fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); - fs.writeFileSync(snapshotUserPath(), JSON.stringify(currentSnapshotState)); - } catch { /* readonly userdata / disk full — skip */ } - if (added > 0) log(`delta-refresh: +${added} new tx${added === 1 ? "" : "s"} (total ${history.length}, index has ${idx.size} names)`); - return true; - } finally { try { el.close(); } catch {} } - })().catch(() => false); - const timeout = new Promise((resolve) => { - deadline = setTimeout(() => { try { conn?.close(); } catch {} resolve(false); }, POLL_DEADLINE_MS); + throw last || new Error("no pipe"); + } : null, + localPaths: [ + ...ariadneCopies(scopes), + path.join(cfg.userData, `bns-name-snapshot${netSuffix()}.json`), + path.join(cfg.userData, `bns-name-snapshot${netSuffix()}.prev.json`), + cfg.bundledSnapshot, + ], + ownCopyPath: path.join(cfg.userData, `bns-name-index${netSuffix()}.json`), + // The public sources are not the chain's job here: the own indexer + // fetches the published snapshot when it runs, and main.js has the + // quick single-name lookup (through Tor or the session proxy). + publicSnapshotUrls: [], + publicIndexerUrl: null, + log: (...a) => log(...a), }); - const p = Promise.race([work, timeout]).finally(() => { - clearTimeout(deadline); - if (pollInFlight === p) pollInFlight = null; - maybeRefreshElectrum(); - }); - pollInFlight = p; - return p; + chain.on("change", () => { if (active() === "chain") publish("chain"); }); + chain.on("fresh", ({ builtAt }) => { if (active() === "chain") port.postMessage({ type: "fresh", builtAt }); }); + chain.on("pipe", ({ connected, error }) => { log(connected ? "Ariadne's indexer connected" : `Ariadne's indexer unavailable (${error})`); checkHealth(); sendStatus(); }); + chain.on("paused", (p) => { log(p ? "Ariadne is in Economy: sync paused" : "Ariadne resumed syncing"); sendStatus(); }); + const t0 = Date.now(); + chain.start(); // local copies now (synchronous), the pipe in the background + if (chain.index.size) log(`local copy: ${chain.index.size} names from ${chain.source} in ${Date.now() - t0} ms`); + if (!lastPublished) publish("local"); // the chain's "change" has normally sent it already + runIndexerOnDemand(scopes); + // Nothing to wait for (portable Theseus, no Ariadne, or one older than the + // pipe): the own indexer at once, as in 0.3.70. + if (!checked.length) startOwn("no Ariadne indexer installed"); + setTimeout(checkHealth, LAUNCH_GRACE_MS); + setInterval(checkHealth, CHECK_EVERY_MS); } +let initP = null; -// Full walk over electrum — only for a resolver without the delta primitives, -// or an explicit rebuild request. -let rebuilding = null; -function rebuild() { - if (rebuilding) return rebuilding; - rebuilding = (async () => { - const r = await getResolver(); - if (!electrumPool) await initElectrumPool(); - const idx = await r.buildIndex({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }); - publish(idx, null, "rebuild"); - return true; - })().catch(() => false).finally(() => { rebuilding = null; }); - return rebuilding; -} - -// The published raw snapshot (public, refreshed every 10 min on the VPS): -// one download that can carry a long-closed browser across days of beacon -// events, so the electrum poll after it only fetches what is newer still. -// Merged in now — not saved for the next start — and only ever adds: the -// index is rebuilt here from the transactions, as with any other source. -async function refreshFromPublishedSnapshot() { - try { - const r = await getResolver(); - // Two public copies (dl vhost, Sia via the gateway): first one that answers. - const urls = [r.NETWORK?.nameListUrl, r.NETWORK?.nameListMirrorUrl].filter(Boolean); - let pub = null; - for (const url of urls) { - try { - const res = await fetch(url, { redirect: "follow", signal: AbortSignal.timeout(15000) }); - if (!res.ok) { log(`published snapshot ${new URL(url).host}: HTTP ${res.status}`); continue; } - pub = await res.json(); - break; - } catch (e) { log(`published snapshot ${new URL(url).host}: ${e?.cause?.code || e?.message || e}`); } - } - if (!pub) return; - if (!pub || !Array.isArray(pub.history) || !pub.txs || (r.BEACON_SCRIPTHASH && pub.beacon !== r.BEACON_SCRIPTHASH)) return; - const base = currentSnapshotState || { beacon: r.BEACON_SCRIPTHASH, history: [], txs: {} }; - const merged = new Map(base.history.map((h) => [h.tx_hash, h])); - const txs = { ...(base.txs || {}) }; - let added = 0; - for (const h of pub.history) { - const mine = merged.get(h.tx_hash); - if (!pub.txs[h.tx_hash] && !txs[h.tx_hash]) continue; // no evidence, skip - if (!txs[h.tx_hash]) { txs[h.tx_hash] = pub.txs[h.tx_hash]; added++; } - // Keep a confirmed height over a mempool (0 / negative) one. - if (!mine || (!(Number(mine.height) > 0) && Number(h.height) > 0)) merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height }); - } - const next = { ...base, beacon: base.beacon || pub.beacon, history: [...merged.values()], txs }; - if (sigOf(next) === sigOf(base)) return; - currentSnapshotState = next; - publish(r.buildIndexFromSnapshot({ snapshot: next }), next, "published"); - try { - fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); - fs.writeFileSync(snapshotUserPath(), JSON.stringify(next)); - } catch {} - log(`published snapshot merged: +${added} tx${added === 1 ? "" : "s"} (total ${next.history.length}, ${index.size} names)`); - } catch (e) { log("published snapshot unavailable:", e?.message || e); } -} - -// ---- phases ----------------------------------------------------------------- -let warmed = null; // promise: phase 1 finished (index may still be null) -let started = false; // phase 2 running -function startFullWork() { - if (started) return; - started = true; - (async () => { - await warmed; - if (!electrumPool) await initElectrumPool(); - // Fastest catch-up first: one download of the published snapshot, then - // the electrum delta for whatever is newer than that. - await refreshFromPublishedSnapshot(); - lastElectrumRefresh = Date.now(); - refreshElectrumPool(); - // No snapshot at all (first start of a build without one): the poll's - // "empty" base state does the full fetch. - pollAndMerge(); - pollTimer = setInterval(() => pollAndMerge(), POLL_INTERVAL_MS); - })().catch((e) => log("start failed:", e?.message)); -} - -// "ready": resolves once there is any index — the local snapshot, or (none -// on disk) the published one, one download; the live poll only if that -// failed too, since a first electrum walk fetches every beacon transaction. -let firstFetch = null; +// "ready": resolves once there is any index. The pipe gets its grace period; +// then the own indexer (published snapshot, electrum as the last resort). async function whenReady() { - await warmed; - if (index) return true; - firstFetch = firstFetch || refreshFromPublishedSnapshot().finally(() => { firstFetch = null; }); - await firstFetch; - if (index) return true; - return pollAndMerge(); + await initP; + if (active() === "chain" && chain.index.size) return true; + if (own && own.index) return true; + const until = launchedAt + LAUNCH_GRACE_MS; + while (Date.now() < until && !chain.index.size) await new Promise((r) => setTimeout(r, 200)); + if (chain.index.size && active() === "chain") return true; + return startOwn(own ? ownReason : "no index yet").ready(); +} +async function poll() { + await initP; + if (own) return own.poll({ force: true }); + if (chain.pipeConnected) return chain.requestPoll(); + return false; // no live source yet: the health check starts the own indexer } port.on("message", async (e) => { @@ -312,19 +311,15 @@ port.on("message", async (e) => { const reply = (ok, error) => { if (m.id != null) port.postMessage({ type: "reply", id: m.id, ok: !!ok, error }); }; try { switch (m.type) { - case "init": - cfg = m; - await setTor(m.torPort); - warmed = warmFromSnapshot(); - break; - case "go": startFullWork(); break; - case "tor": await setTor(m.port); break; - case "stop-polling": if (pollTimer) { clearInterval(pollTimer); pollTimer = null; } break; + case "init": initP = init(m); await initP; break; + case "go": goReceived = true; await initP; if (own) own.start(); break; + case "tor": await setTor(m.port); if (own) own.setWebSocket(currentWS()); break; + case "stop-polling": if (own) own.stop(); if (chain) chain.stop(); break; case "ready": reply(await whenReady()); break; - // An on-demand poll is allowed before phase 2: it is the "this tab's - // name isn't in the snapshot" case, and costs one electrum round trip. - case "poll": await warmed; reply(await pollAndMerge()); break; - case "rebuild": await warmed; reply(await rebuild()); break; + // On-demand poll (a tab's name is not in the index): Ariadne's indexer + // when it is live, else the own one; allowed before "go". + case "poll": reply(await poll()); break; + case "rebuild": await initP; reply(await startOwn(ownReason || "rebuild requested").rebuild()); break; default: reply(false, "unknown request"); } } catch (err) { reply(false, err?.message || String(err)); } diff --git a/main.js b/main.js index 94e53c15..f3da2b93 100644 --- a/main.js +++ b/main.js @@ -1745,12 +1745,14 @@ async function fetchOpenSearch(href) { } // ---- BNS index: mirror of the indexer process -------------------------------- -// The index itself — snapshot warm-start, electrum delta poll, Sia snapshot -// refresh, electrum server discovery — runs in bns-indexer.js, a separate -// process (utilityProcess), so none of it competes with the browser's main -// thread. It boots in two phases: the local snapshot first (no network — what -// the first tab needs), then the network work once this side says "go", after -// the first page has loaded. This side keeps a read-only mirror of the name +// The index itself runs in bns-indexer.js, a separate process (utilityProcess), +// so none of it competes with the browser's main thread. Ariadne's Thread owns +// BNS indexing on the machine: that process reads Ariadne's indexer over its +// server-checked pipe, or Ariadne's local copies, and starts Theseus's own +// electrum indexer only while Ariadne is unhealthy or absent +// (DESIGN-bns-indexer-service.md). It boots in two phases: the local copies +// first (no network — what the first tab needs), then the network work once +// this side says "go", after the first page has loaded. This side keeps a read-only mirror of the name // map for the lookups that must answer synchronously (knownUnregistered, the // native-TLD set, error-page suggestions, Hermes reverse lookups). @@ -1762,6 +1764,12 @@ let sharedIndex = null; // Map name -> entry, mirrored from the indexer let indexBuiltAt = 0; // when the indexer last confirmed it is current (0 = snapshot only) let indexGen = 0; // bumps whenever the mirror's content changes const SNAPSHOT_BUNDLED = path.join(RES_DIR, "bns-name-snapshot.json"); +// The shared index modules (also Ariadne's): .mjs in the packaged resources, +// like resolver-web; the engine copies in dev. +const bnsSharedLib = (name) => (app.isPackaged ? path.join(RES_DIR, `${name}.mjs`) : path.join(__dirname, "..", "Argus", "src", "lib", `${name}.js`)); +// Where the names come from right now (Ariadne's pipe / its files / Theseus's +// own indexer), as the index process last reported it. Shown in Settings. +let bnsIndexStatus = null; let indexer = null, indexerStarts = 0, indexerSeq = 0, indexerGo = false, indexerStopping = false; const indexerPending = new Map(); const indexWaiters = []; @@ -1789,7 +1797,11 @@ function startIndexer() { startIndexer(); }, wait); }); - indexer.postMessage({ type: "init", resolverPath: RESOLVER, userData: app.getPath("userData"), bundledSnapshot: SNAPSHOT_BUNDLED, torPort: torState === "on" ? TOR_PORT : null }); + indexer.postMessage({ + type: "init", resolverPath: RESOLVER, + coreLib: bnsSharedLib("bns-index-core"), chainLib: bnsSharedLib("bns-source-chain"), pipeLib: bnsSharedLib("bns-pipe"), + userData: app.getPath("userData"), bundledSnapshot: SNAPSHOT_BUNDLED, torPort: torState === "on" ? TOR_PORT : null, + }); if (indexerGo) indexer.postMessage({ type: "go" }); } function onIndexerMessage(m) { @@ -1803,6 +1815,8 @@ function onIndexerMessage(m) { verifyQuickAnswers(); } else if (m.type === "fresh") { indexBuiltAt = m.builtAt; + } else if (m.type === "status") { + bnsIndexStatus = m; } else if (m.type === "reply") { const settle = indexerPending.get(m.id); if (settle) { indexerPending.delete(m.id); settle(!!m.ok); } @@ -6220,7 +6234,7 @@ function ariadneQueryState() { const shell = new Promise((resolve) => { const ps = spawn("powershell.exe", ["-NoProfile", "-NonInteractive", "-Command", "$svc=New-Object -ComObject Schedule.Service;$svc.Connect();$f=$svc.GetFolder('\\');" + - "foreach($n in 'BNS Resolver Daemon','BNS Sia Bridge'){try{$t=$f.GetTask($n);\"$n=$($t.State)\"}catch{\"$n=MISSING\"}};" + + "foreach($n in 'Ariadne BNS Indexer','BNS Resolver Daemon','BNS Sia Bridge'){try{$t=$f.GetTask($n);\"$n=$($t.State)\"}catch{\"$n=MISSING\"}};" + ARIADNE_REG_LOOKUP + "if ($r) { \"__VER__=$($r.DisplayVersion)\"; \"__UNINSTALL__=$($r.QuietUninstallString)\" }" ], { windowsHide: true }); @@ -6232,7 +6246,11 @@ function ariadneQueryState() { return Promise.all([shell, ariadneManifestFetch()]).then(([out, latest]) => { const lines = out.trim().split(/\r?\n/).filter(Boolean); const map = Object.fromEntries(lines.map((l) => { const i = l.lastIndexOf("="); return [l.slice(0, i), l.slice(i + 1)]; })); - const primary = map["BNS Resolver Daemon"]; // TASK_STATE: 1 disabled, 2 queued, 3 ready, 4 running + // TASK_STATE: 1 disabled, 2 queued, 3 ready, 4 running. Since Ariadne + // 0.2 its indexer is the process that matters to Theseus; the resolver + // only runs in All-browsers mode. + const indexerTask = map["Ariadne BNS Indexer"]; + const primary = indexerTask && indexerTask !== "MISSING" ? indexerTask : map["BNS Resolver Daemon"]; const installedVersion = map.__VER__ || null; const quietUninstall = map.__UNINSTALL__ || null; const latestVersion = latest ? latest.version : null; @@ -6244,7 +6262,12 @@ function ariadneQueryState() { // The "bundledVersion" field name is kept for renderer compatibility -- // it now carries the latest version advertised by silentmode.st's // releases manifest, not a version physically bundled with Theseus. - return { state, installedVersion, bundledVersion: latestVersion, canUpdate, hasUninstaller: !!quietUninstall }; + const resolverTask = map["BNS Resolver Daemon"]; + return { + state, installedVersion, bundledVersion: latestVersion, canUpdate, hasUninstaller: !!quietUninstall, + resolverRunning: resolverTask === "4" || resolverTask === "Running", + index: bnsIndexStatus, + }; }); } function cmpVersions(a, b) { diff --git a/package.json b/package.json index 40bdbc7e..b621b8a3 100644 --- a/package.json +++ b/package.json @@ -120,6 +120,18 @@ "from": "../Argus/src/lib/password-vault.js", "to": "password-vault.mjs" }, + { + "from": "../Argus/src/lib/bns-index-core.js", + "to": "bns-index-core.mjs" + }, + { + "from": "../Argus/src/lib/bns-source-chain.js", + "to": "bns-source-chain.mjs" + }, + { + "from": "../Argus/src/lib/bns-pipe.js", + "to": "bns-pipe.mjs" + }, { "from": "lib/hermes.js", "to": "lib/hermes.mjs" diff --git a/settings.html b/settings.html index 286081b9..c8e6238c 100644 --- a/settings.html +++ b/settings.html @@ -1733,6 +1733,21 @@ } else { arStatSub.innerHTML = `Not installed on this machine.${bundledSuffix}`; } + // Where Theseus's own BNS names come from right now: Ariadne's + // indexer (live), Ariadne's local copy, or Theseus's own indexer + // (the fallback while Ariadne's is not available). + const ix = r.index; + if (ix) { + const src = ix.source === "pipe" ? "Ariadne's indexer (live)" + : ix.source === "theseus-indexer" ? `Theseus's own indexer${ix.ownIndexer?.reason ? ` (${ix.ownIndexer.reason})` : ""}` + : ix.source === "own-copy" ? "Theseus's saved copy" + : String(ix.source || "").startsWith("local:") ? "a local snapshot copy" : "none yet"; + const line = document.createElement("div"); + line.className = "pmuted"; + line.style.marginTop = "4px"; + line.textContent = `Theseus reads names from ${src}${ix.ariadne?.paused ? " — Ariadne is in Economy, sync paused" : ""}.`; + arStatSub.appendChild(line); + } // Toggle state on both views. When not installed the toggle is // disabled — user has to hit Install first. if (!arBusy) {