diff --git a/DESIGN-bns-indexer-service.md b/DESIGN-bns-indexer-service.md new file mode 100644 index 00000000..115865e3 --- /dev/null +++ b/DESIGN-bns-indexer-service.md @@ -0,0 +1,346 @@ +# BNS indexing: Ariadne's Thread owns it, everyone reads it + +Status: design, revised 2026-10-03. + +Already implemented: +- the public raw snapshot, with two copies (see the server tier); +- Theseus's indexer process `bns-indexer.js`, which becomes the last-resort + fallback here; +- Theseus's quick single-name lookup while no index exists. + +## The model + +``` + server this machine readers + ────── ────────────────────────────────────────── ─────── + gateway index ─┬─► public Ariadne's Thread (scope: All users | Just me) + (electrum) │ snapshot ──► indexer ──writes──► local snapshot copies Theseus + │ ×2 copies (sync, may stop, (read-only for readers) ──┐ + │ read-only pause, crash) │ │ same + │ │ pipe (live, optional) │ │ source + └─► public BNS ▼ ▼ │ chain + indexer resolver (All browsers mode only) ◄──────────────┘ + /api/name/ DNS :53 + NRPT + gateway (All users) + (single names) PAC proxy + gateway (Just me) other browsers +``` + +Ariadne's Thread is the parent of everything BNS on a machine. It has two +processes with different jobs: + +- **The indexer** builds and maintains the index from the raw beacon + evidence and writes the local snapshot copies. It is heavy-ish (electrum + sync), and it is allowed to stop: Economy, "Launch at start" off, a crash, + an update. +- **The resolver** answers names: for other browsers in All browsers mode, + through system DNS or the per-user proxy. It is small and does no syncing + of its own beyond its fallbacks. + +**Resolution never depends on the indexer.** The resolver and Theseus read +names through the same fallback chain (below), in which the live indexer is +only the first and freshest source. If it is gone, the answers come from the +local snapshot copies, the public snapshots or the public BNS indexer. The +only thing lost is freshness, and only until a source with newer data +answers. + +## Tiers + +### Server (done) +- **Live index:** the gateway keeps it over electrum, and serves single names + at `/api/name/`. This is the **public BNS indexer**. Its answers carry + records and owner, interpreted by the operator; clients treat them as + provisional (see below). +- **Raw snapshot:** `cdn-name-snapshot` (VPS, every 10 min) publishes it as + **two public, read-only copies on two storage backends**: + - `https://dl.silentmode.st/bns-name-snapshot.json` (the dl vhost's disk) + - `https://navigate.st/bns/silentmode.bch/bns-name-snapshot.json` (the Sia + bucket, served by the gateway) + + Both are static, with CORS `*` and no credentials. Each contains the + history plus the verbose transactions, so clients rebuild the index from + the evidence themselves. A failed upload of the second copy is retried on + the next run. Clients try them in that order (`nameListUrl`, + `nameListMirrorUrl`). +- **Android** keeps using the interpreted `bns-name-index.json`. + +### The source chain (the resolver and Theseus alike) + +Fastest first. A lookup is answered by the first source that has the name; +the slower sources refresh in the background. + +1. **In memory:** whatever the reader has already loaded (normally all of + it). +2. **The live indexer over the pipe:** `snapshot` on connect, then + `subscribe` for changes. This is the freshest source when it is running. +3. **The local snapshot copies:** a disk read of milliseconds. There are two, + so a deleted, moved or half-written file still leaves one: + - Ariadne's: `C:\ProgramData\Ariadne\` (All users) or + `%LOCALAPPDATA%\Ariadne\` (Just me), written only by the indexer. + - The reader's own last-known-good copy. For Theseus that is its profile; + for the resolver it is beside Ariadne's file. +4. **The public snapshots,** either copy: one download of about 450 KB, + rebuilt locally from the transactions. It always runs in the background; + nothing waits for it. +5. **The public BNS indexer** (`/api/name/`): a single-name lookup of + about 0.2 s. Used when sources 1–3 have no data yet, or the name is + missing and the data is stale. The answer is **provisional**: + - It is served at once. + - It is compared with the verified index when that arrives, and on a + mismatch the tabs or cached answers are replaced. + - It is never used for the extension-publisher check, which waits for + verified data. +6. **Theseus only, last resort:** its own electrum indexer process + (`bns-indexer.js`), for a portable Theseus with no Ariadne and no CDN. + +**Privacy:** sources 4–5 only ever see BNS names. +- The resolver only receives queries for BNS TLDs (NRPT and the PAC file + route only those). +- Theseus only asks for names under known BCNR TLDs. +- An ordinary web host is never sent to the public indexer. + +### Ariadne's Thread: the indexer +- **One implementation** with Theseus's indexer logic (`bns-indexer.js`): + - warm start from the local snapshot; + - catch-up from the public snapshots; + - the electrum delta poll every 30 s; + - the index rebuilt from the beacon transactions, with `owner`; + - the TLD set from `tlds.bch`. + + This replaces the daemon's current warm start from the pre-resolved list, + which has no owners and an operator-interpreted index. +- **Local snapshot:** + - It uses the raw format, written atomically after every change, and the + previous file is kept as the second copy. + - In All users, **users may read it; only SYSTEM and Administrators may + write it.** The transactions are not checked against block headers, so + a user-writable copy would let any local process insert names or owners. + - `policy.json` stays user-writable, as today; it holds settings, not + data. +- **The pipe:** `\\.\pipe\ariadne-bns`, or `…-` for Just me. + - **Requests:** `hello` returns `{version, network, gen, builtAt, + snapshotPath}`; `snapshot` and `get ` return index data; `poll` + runs one delta poll on demand. + - **Push:** `subscribe` sends `{gen, changed}` on every change. + - **Server check:** a client checks that the server process is Ariadne's + indexer before trusting it (`GetNamedPipeServerProcessId`, then the + process image path under Program Files and, for All users, its owner + SYSTEM). Otherwise a process could take the pipe name while the indexer + is down and feed false owners to the extension-publisher check. A failed + check simply skips to source 3. + - **Not TCP:** a 127.0.0.1 port has no way to tell who is listening. + +### Ariadne's Thread: the resolver +- **Runs only in All browsers mode;** Theseus is its own resolver. It is + small: DNS, proxy and gateway listeners plus the source chain, with no + electrum. +- **On its own:** + - With the indexer down it keeps answering from the local copies. + - It fetches the public snapshot once an hour when the local copies are + older than that, holding the result in memory only, since the indexer + owns the files. + - It uses the public indexer for single misses. +- **If the resolver itself is down,** the task restarts it within a minute. + With NRPT routing the BNS TLDs to 127.0.0.1 there is no system fallback + for those names meanwhile (ICANN names are unaffected). Open question: a + public BNS DNS server as NRPT's second name server, and a public proxy as + the PAC file's second entry. + +### Ariadne's Thread: scope × mode + +**Scope** decides who it runs for: +- **All users** (the admin path): system processes for every account on the + OS. +- **Just me:** runs under one account, with no admin. + +**Mode** decides who it resolves for: +- **Theseus only.** +- **All browsers:** every browser on the machine, and for All users every app + as well. + +| | **Theseus only** | **All browsers** (default) | +|---|---|---| +| **All users** (admin, SYSTEM at boot) | indexer only | indexer + resolver: system DNS on :53 with NRPT rules, the local gateway on :80/:443, a machine-wide CA. Every app on the PC resolves BNS names | +| **Just me** (no admin, at logon) | indexer only | indexer + resolver: a PAC file set in the account's own Windows proxy settings (HKCU, no admin) points at a local proxy on a high port, which routes BNS names to the gateway and passes everything else through. The CA goes into the user's certificate store, which Windows confirms once | + +Notes on **Just me + All browsers**: +- Chrome, Edge and Brave follow the Windows proxy settings. Firefox follows + them when set to "use system proxy", its default, but needs the CA in its + own store, which Ariadne already handles for the machine CA. +- Programs that ignore the system proxy (some command-line tools, apps with + their own resolver) do not see BNS names. Covering them takes system DNS, + which needs admin. +- A PAC file is not DNS, so it also works when a browser uses its own + DNS-over-HTTPS, which bypasses the NRPT route on networks that tamper with + DNS. + +**Where the mode lives:** +- All users: `policy.json` in ProgramData. Just me: `policy.json` in the + user's profile. The field is `"mode": "theseus" | "all-browsers"`. +- Switching starts or stops the resolver and adds or removes the NRPT rules, + or the per-user proxy setting. +- **It never touches the indexer.** + +### Ariadne's Thread: lifetime and start-up + +Both processes are background processes with no window, independent of +Theseus. + +| | **All users** | **Just me** | +|---|---|---| +| Runs as | SYSTEM, scheduled tasks | the user, scheduled tasks (hidden) | +| Starts | at OS start, before anyone logs on | at that user's logon, the earliest without admin | +| Keeps running | until shutdown | until that user logs off | + +To start before logon, a user switches the scope to All users (one UAC +prompt). + +**Task settings, both tasks and both scopes:** +- Restart on failure (5 times, 1 min apart). +- No execution time limit. +- **`-AllowStartIfOnBatteries -DontStopIfGoingOnBatteries`.** Today's task + uses Windows' defaults, which skip the start on battery and stop the task + when a laptop unplugs. Power is a setting of Ariadne's (below), not + something Windows decides by killing the process. +- Below-normal process and I/O priority for the indexer. + +**"Launch at start" (on by default):** a switch in Ariadne's Thread settings, +mirrored in Theseus Settings › Plug-ins › Ariadne's Thread. +- **Scope of the switch:** it applies to the **indexer**. The resolver + follows the mode: in All browsers it always starts, because other browsers + depend on it and it is small. +- **Where it lives:** `policy.json`, `"startAtBoot": true | false`. The + running process enables or disables the trigger on its own task, so no + UAC prompt is needed in either scope. +- **On:** the indexer is up from OS start (All users) or logon (Just me), + whether or not Theseus is ever opened. +- **Off:** the indexer starts on demand when Theseus launches, by running the + task. For All users the task's security descriptor lets local users run it, + but not change it. Until then, names resolve from the source chain as + usual, in Theseus and in other browsers; only the freshness depends on the + indexer. +- **Turning it back on** starts the indexer at once as well as at the next + boot. + +**Power: "Performance" or "Economy"**, in Ariadne's Thread settings, +mirrored in Theseus. It lives in `policy.json`: `"power": "performance" | +"economy"`, plus `"economyWhen": "on-battery" | "low-battery"`. + +- **Performance (default):** the indexer syncs normally on battery. +- **Economy:** while the condition holds, the indexer **pauses its network + work**: the electrum poll, the published-snapshot refresh and server + discovery. + - Resolution is unaffected. Readers keep answering from memory and the + local copies, and the public indexer still covers single misses. + - When the condition clears, the indexer resumes with one immediate + catch-up poll. + - **The condition**, `economyWhen`: + - `on-battery`: running on battery, the old behaviour of Windows' default + task settings. + - `low-battery`: Windows' Battery Saver is on, which Windows turns on at + 20% by default. + - Pausing rather than stopping keeps the warm process and a cheap resume. + Stopping would be just as safe for resolution, since the readers do not + depend on the indexer. +- **Detection:** + - The indexer reads the power state every 60 s and on resume-from-sleep: AC + or battery and the Battery Saver flag, from `GetSystemPowerStatus` via a + small helper, or one long-lived PowerShell process, never one process per + check. + - Theseus's fallback indexer uses Electron's `powerMonitor`. +- **Desktops** (no battery) never enter Economy, and the switch is hidden. + +### Theseus +- **Launch:** the source chain above. In practice: Ariadne's local snapshot + file (milliseconds, parsed in Theseus's indexer process, never on the main + thread), then `subscribe` to the pipe for changes. +- **Its own copy:** Theseus keeps its own last-known-good copy in its + profile, so a missing Ariadne file still leaves one. +- **No index yet:** the quick single-name lookup (source 5) while the public + snapshot downloads in the background. +- **Never writes Ariadne's files,** and never starts, stops or reconfigures + the indexer. The one exception is running its task on demand when "Launch + at start" is off. Its Settings page shows Ariadne's status and edits its + `policy.json`. + +## Installer + +- **Ariadne's Thread is always installed** with Theseus. There is no "install + Ariadne?" question. +- **Setup asks two things on one page:** the scope and the mode. + +### Install scope + +| | **All users** (admin, default) | **Just me** (no admin) | +|---|---|---| +| Runs as | SYSTEM scheduled tasks, at boot | per-user scheduled tasks, at logon | +| Indexer + local snapshot | one per machine, `C:\ProgramData\Ariadne\` | one per account, `%LOCALAPPDATA%\Ariadne\` | +| Who may write the snapshot | SYSTEM and Administrators | the account (the same trust level as the Theseus profile) | +| Pipe | `\\.\pipe\ariadne-bns` | `\\.\pipe\ariadne-bns-` | +| All browsers via | system DNS + NRPT + gateway | per-user PAC proxy + gateway | + +Inno Setup provides the scope choice natively (`PrivilegesRequired=lowest` +with `PrivilegesRequiredOverridesAllowed=dialog`). + +**Pre-selection (decided 2026-10-03):** All users when the installing +account is an administrator; **Just me when it is not**, so a standard user +is never sent to a UAC prompt they cannot approve. Both can still pick the +other. + +### Mode + +- **All browsers (recommended, default):** every browser on this PC resolves + BNS names. With All users, so does every other app. +- **Theseus only:** BNS names resolve in Theseus; nothing else on the system + changes. + +### Changing either later + +From Theseus Settings › Plug-ins › Ariadne's Thread, or Ariadne's own +settings: + +- **Mode, Launch at start and Power:** no prompt, in either scope + (`policy.json`). +- **Scope,** in either direction: one UAC prompt, since it installs or + removes the SYSTEM tasks. The local snapshot moves to the new location. +- **Both present:** if a machine install and a per-user one exist at the same + time, readers use the machine one, and the per-user one stops itself. + +### Other installer behaviour + +- **Updates:** Ariadne keeps its own update lifecycle, chained by Theseus's + updater the way it chains the add-on updates. An indexer update never + interrupts resolution. +- **Uninstalling Theseus** asks whether to keep Ariadne's Thread. Keeping it + is the default in All-browsers mode. + +## Migration from today + +1. **Lift `bns-indexer.js` into a shared module,** and the source chain into + a shared reader module. Ariadne's two processes and Theseus use the same + code and the same tests. +2. **Split Ariadne's daemon** (today one process, `bnsd.js`) into: + - the indexer: raw snapshot, `owner`, `tlds.bch`, the electrum delta, the + two local copies with their ACL, the pipe, Power; + - the resolver: today's DNS, gateway and CA code on the shared reader, + plus the per-user PAC proxy for Just me. + + Its read-only `/api/*` HTTP endpoints move to the resolver, for the + gateway and the Theseus status panel. +3. **Theseus:** + - the source chain; + - its own snapshot copy; + - the pipe client with the server-process check; + - its electrum indexer demoted to the last resort. +4. **Installer:** Ariadne bundled again, with the scope and mode questions. +5. **Pages:** the tools page and theseus.x describe Ariadne's Thread as part + of Theseus, with All browsers as the default mode. + +## Open questions + +- **System-level fallback while the resolver restarts:** + - a public BNS DNS server as NRPT's second name server (All users); + - a public proxy as the PAC file's second entry (Just me). +- **Portable Theseus:** it never installs Ariadne, so it lives on its own + copy, the public sources and its own indexer. Should it offer to install + Ariadne? +- **macOS/Linux:** launchd or systemd units for the same two processes, once + Theseus ships there. diff --git a/bns-indexer.js b/bns-indexer.js new file mode 100644 index 00000000..b6d40aa8 --- /dev/null +++ b/bns-indexer.js @@ -0,0 +1,331 @@ +// BNS name indexer — runs in its own process (Electron utilityProcess), +// started by main.js, so the snapshot parsing, index builds and electrum +// traffic never compete with the browser's main thread. +// +// Boot is two-phase: +// 1. fast lane: load the local snapshot (user cache > bundled), build the +// index from it and publish it. No network. This is what the first tab +// needs, and it is all that runs while the browser is still starting. +// 2. full work, once main says "go" (its first page has loaded): electrum +// server discovery, the Sia snapshot refresh, and the 30 s delta poll. +// A lookup that misses the snapshot before then still gets a live +// delta poll on demand — only that, nothing else. +// +// Protocol (process.parentPort messages): +// main -> indexer { type: "init", resolverPath, userData, bundledSnapshot, torPort } +// { type: "go" } start phase 2 +// { type: "tor", port } electrum via Tor (null = direct) +// { type: "stop-polling" } +// { id, type: "poll" | "rebuild" | "ready" } replies { type: "reply", id, ok } +// indexer -> main { type: "index", builtAt, names: [[key, entry]...], network } +// { type: "fresh", builtAt } poll succeeded, nothing changed +// { type: "reply", id, ok, error? } +// +// The index entries are the resolver's own (records, owner, txid, height, …) +// built from the raw beacon transactions — the trust model is unchanged. +"use strict"; +const fs = require("fs"); +const path = require("path"); +const { pathToFileURL } = require("url"); +const WebSocket = require("ws"); + +const port = process.parentPort; +const log = (...a) => console.log("[bns]", ...a); + +// Same guard as main.js: ws turns "closed while still connecting" into an +// uncaught error event. Electrum servers that reset mid-handshake are routine. +{ + const close = WebSocket.prototype.close; + WebSocket.prototype.close = function (...args) { + if (this.readyState === WebSocket.CONNECTING && this.listenerCount("error") === 0) this.once("error", () => {}); + return close.apply(this, args); + }; +} +// A flaky network must not take the indexer down; main restarts it if it +// exits, but the index would go cold for a moment. +process.on("uncaughtException", (err) => { try { console.error("[bns] uncaught:", (err && err.stack) || err); } catch {} }); +process.on("unhandledRejection", (err) => { try { console.error("[bns] unhandled rejection:", (err && err.message) || err); } catch {} }); + +let cfg = null; +let R = null; +async function getResolver() { + if (!R) R = await import(pathToFileURL(cfg.resolverPath).href); + return R; +} + +// ---- Tor ------------------------------------------------------------------ +let torAgent = null; +async function setTor(torPort) { + if (!torPort) { torAgent = null; return; } + const { SocksProxyAgent } = await import("socks-proxy-agent"); + torAgent = new SocksProxyAgent(`socks5h://127.0.0.1:${torPort}`); +} +class TorWebSocket extends WebSocket { constructor(url, opts) { super(url, { agent: torAgent, ...opts }); } } +const currentWS = () => (torAgent ? TorWebSocket : WebSocket); + +// ---- electrum server pool: hardcoded seed + on-chain discovery, persisted ---- +// Bootstrap from the baked-in seed (with pinned IPs), then refresh from the +// on-chain ELECTRUM_LIST_NAME record so the pool can be rotated without a new +// build. The last discovered list is cached to disk and tried first next launch. +let electrumPool = null; +let lastElectrumRefresh = 0; +// Per-network cache files. Chipnet keeps the historical names so existing +// profiles are not invalidated; any other network gets its own files. +let networkId = "chipnet"; +const netSuffix = () => (networkId === "chipnet" ? "" : `.${networkId}`); +const electrumFile = () => path.join(cfg.userData, `electrum-servers${netSuffix()}.json`); +const snapshotUserPath = () => path.join(cfg.userData, `bns-name-snapshot${netSuffix()}.json`); +const serverKey = (s) => (typeof s === "string" ? s : s && s.url); +function mergeServers(preferred, rest) { + const seen = new Set(), out = []; + for (const s of [...(preferred || []), ...(rest || [])]) { + const k = serverKey(s); + if (k && !seen.has(k)) { seen.add(k); out.push(s); } + } + return out; +} +async function initElectrumPool() { + const r = await getResolver(); + const seed = r.ELECTRUM || r.CHIPNET_ELECTRUM; + let saved = []; + try { if (fs.existsSync(electrumFile())) saved = JSON.parse(fs.readFileSync(electrumFile(), "utf8")); } catch {} + electrumPool = mergeServers(saved, seed); // discovered first, seed always kept +} +async function refreshElectrumPool() { + try { + const { fetchElectrumServers } = await getResolver(); + const found = await fetchElectrumServers({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }); + if (found && found.length) { + electrumPool = mergeServers(found, electrumPool); + try { fs.writeFileSync(electrumFile(), JSON.stringify(found, null, 2)); } catch {} + } + } catch { /* list unpublished or unreachable — keep the current pool */ } +} +function maybeRefreshElectrum() { + if (!started) return; // phase 2 only + if (Date.now() - lastElectrumRefresh < 30 * 60 * 1000) return; + lastElectrumRefresh = Date.now(); + refreshElectrumPool(); +} + +// ---- index state + publishing --------------------------------------------- +let index = null; // Map name -> entry +let currentSnapshotState = null; // raw { beacon, history, txs, … } behind `index` +let lastSig = ""; +// Cheap fingerprint of the snapshot: an index rebuilt from the same history +// at the same heights is the same index, so main isn't sent a copy. +const sigOf = (snap) => `${snap.history.length}:${snap.history.reduce((a, h) => a + (Number(h.height) || 0), 0)}`; +function publish(idx, snap, reason) { + index = idx; + const sig = snap ? sigOf(snap) : `full:${Date.now()}`; + const builtAt = reason === "snapshot" ? 0 : Date.now(); // a snapshot is a floor, not a ceiling + if (sig === lastSig) { port.postMessage({ type: "fresh", builtAt }); return; } + lastSig = sig; + port.postMessage({ type: "index", builtAt, names: [...idx], network: networkId, reason }); +} + +function readSnapshotFrom(p) { + try { + if (!fs.existsSync(p)) return null; + const parsed = JSON.parse(fs.readFileSync(p, "utf8")); + if (!parsed || !Array.isArray(parsed.history)) return null; + return parsed; + } catch { return null; } +} + +// Phase 1: the local snapshot, nothing else. +async function warmFromSnapshot() { + const r = await getResolver(); + networkId = r.NETWORK?.id || "chipnet"; + if (!r.buildIndexFromSnapshot) return false; // an older resolver-web.js + const snap = readSnapshotFrom(snapshotUserPath()) || readSnapshotFrom(cfg.bundledSnapshot); + if (!snap) return false; + try { + const t0 = Date.now(); + const idx = r.buildIndexFromSnapshot({ snapshot: snap }); + currentSnapshotState = snap; + publish(idx, snap, "snapshot"); + log(`warm-started from snapshot: ${idx.size} names in ${Date.now() - t0} ms @ height=${snap.asOfHeight ?? "?"} root=${snap.root ?? "?"}`); + return true; + } catch (e) { + log("snapshot warm-start failed:", e.message); + return false; + } +} + +// ---- continuous background delta refresh ---------------------------------- +// One electrum connection per poll: fetch the beacon's history (one call), +// fetch only the tx bodies not already held, rebuild locally, persist. +const POLL_INTERVAL_MS = 30_000; +// Neither the electrum connect nor its WebSocket handshake has a timeout of +// its own; a server that accepts TCP and then stalls would wedge the loop. +const POLL_DEADLINE_MS = 45_000; +let pollInFlight = null, pollTimer = null; +async function pollAndMerge() { + if (pollInFlight) return pollInFlight; + let conn = null, deadline; + const work = (async () => { + const r = await getResolver(); + if (!r.connectElectrum || !r.BEACON_SCRIPTHASH || !r.buildIndexFromSnapshot) return rebuild(); + if (!electrumPool) await initElectrumPool(); + // Base state: memory > user cache > bundled > empty. The "empty" branch + // turns a first start without any snapshot into a full fetch. + const snap = currentSnapshotState + || readSnapshotFrom(snapshotUserPath()) + || readSnapshotFrom(cfg.bundledSnapshot) + || { beacon: r.BEACON_SCRIPTHASH, history: [], txs: {} }; + const el = conn = await r.connectElectrum({ electrum: electrumPool, WebSocket: currentWS(), directIP: true }); + try { + const freshHistory = await el.call("blockchain.scripthash.get_history", [r.BEACON_SCRIPTHASH]); + const known = new Set(snap.history.map((h) => h.tx_hash)); + // Merge fresh into snapshot history (dedup by tx_hash, keep fresh height — + // an event that was mempool at snapshot time now has a real height). + const merged = new Map(snap.history.map((h) => [h.tx_hash, h])); + const txs = { ...(snap.txs || {}) }; + let added = 0; + for (const h of freshHistory) { + if (!known.has(h.tx_hash)) { + try { txs[h.tx_hash] = await el.call("blockchain.transaction.get", [h.tx_hash, true]); added++; } + catch { /* unreadable — the reduction rules ignore missing txs */ } + } + merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height }); + } + const history = [...merged.values()]; + currentSnapshotState = { ...snap, beacon: r.BEACON_SCRIPTHASH, history, txs }; + const idx = r.buildIndexFromSnapshot({ snapshot: currentSnapshotState }); + publish(idx, currentSnapshotState, "poll"); + try { + fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); + fs.writeFileSync(snapshotUserPath(), JSON.stringify(currentSnapshotState)); + } catch { /* readonly userdata / disk full — skip */ } + if (added > 0) log(`delta-refresh: +${added} new tx${added === 1 ? "" : "s"} (total ${history.length}, index has ${idx.size} names)`); + return true; + } finally { try { el.close(); } catch {} } + })().catch(() => false); + const timeout = new Promise((resolve) => { + deadline = setTimeout(() => { try { conn?.close(); } catch {} resolve(false); }, POLL_DEADLINE_MS); + }); + const p = Promise.race([work, timeout]).finally(() => { + clearTimeout(deadline); + if (pollInFlight === p) pollInFlight = null; + maybeRefreshElectrum(); + }); + pollInFlight = p; + return p; +} + +// Full walk over electrum — only for a resolver without the delta primitives, +// or an explicit rebuild request. +let rebuilding = null; +function rebuild() { + if (rebuilding) return rebuilding; + rebuilding = (async () => { + const r = await getResolver(); + if (!electrumPool) await initElectrumPool(); + const idx = await r.buildIndex({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }); + publish(idx, null, "rebuild"); + return true; + })().catch(() => false).finally(() => { rebuilding = null; }); + return rebuilding; +} + +// The published raw snapshot (public, refreshed every 10 min on the VPS): +// one download that can carry a long-closed browser across days of beacon +// events, so the electrum poll after it only fetches what is newer still. +// Merged in now — not saved for the next start — and only ever adds: the +// index is rebuilt here from the transactions, as with any other source. +async function refreshFromPublishedSnapshot() { + try { + const r = await getResolver(); + // Two public copies (dl vhost, Sia via the gateway): first one that answers. + const urls = [r.NETWORK?.nameListUrl, r.NETWORK?.nameListMirrorUrl].filter(Boolean); + let pub = null; + for (const url of urls) { + try { + const res = await fetch(url, { redirect: "follow", signal: AbortSignal.timeout(15000) }); + if (!res.ok) { log(`published snapshot ${new URL(url).host}: HTTP ${res.status}`); continue; } + pub = await res.json(); + break; + } catch (e) { log(`published snapshot ${new URL(url).host}: ${e?.cause?.code || e?.message || e}`); } + } + if (!pub) return; + if (!pub || !Array.isArray(pub.history) || !pub.txs || (r.BEACON_SCRIPTHASH && pub.beacon !== r.BEACON_SCRIPTHASH)) return; + const base = currentSnapshotState || { beacon: r.BEACON_SCRIPTHASH, history: [], txs: {} }; + const merged = new Map(base.history.map((h) => [h.tx_hash, h])); + const txs = { ...(base.txs || {}) }; + let added = 0; + for (const h of pub.history) { + const mine = merged.get(h.tx_hash); + if (!pub.txs[h.tx_hash] && !txs[h.tx_hash]) continue; // no evidence, skip + if (!txs[h.tx_hash]) { txs[h.tx_hash] = pub.txs[h.tx_hash]; added++; } + // Keep a confirmed height over a mempool (0 / negative) one. + if (!mine || (!(Number(mine.height) > 0) && Number(h.height) > 0)) merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height }); + } + const next = { ...base, beacon: base.beacon || pub.beacon, history: [...merged.values()], txs }; + if (sigOf(next) === sigOf(base)) return; + currentSnapshotState = next; + publish(r.buildIndexFromSnapshot({ snapshot: next }), next, "published"); + try { + fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); + fs.writeFileSync(snapshotUserPath(), JSON.stringify(next)); + } catch {} + log(`published snapshot merged: +${added} tx${added === 1 ? "" : "s"} (total ${next.history.length}, ${index.size} names)`); + } catch (e) { log("published snapshot unavailable:", e?.message || e); } +} + +// ---- phases ----------------------------------------------------------------- +let warmed = null; // promise: phase 1 finished (index may still be null) +let started = false; // phase 2 running +function startFullWork() { + if (started) return; + started = true; + (async () => { + await warmed; + if (!electrumPool) await initElectrumPool(); + // Fastest catch-up first: one download of the published snapshot, then + // the electrum delta for whatever is newer than that. + await refreshFromPublishedSnapshot(); + lastElectrumRefresh = Date.now(); + refreshElectrumPool(); + // No snapshot at all (first start of a build without one): the poll's + // "empty" base state does the full fetch. + pollAndMerge(); + pollTimer = setInterval(() => pollAndMerge(), POLL_INTERVAL_MS); + })().catch((e) => log("start failed:", e?.message)); +} + +// "ready": resolves once there is any index — the local snapshot, or (none +// on disk) the published one, one download; the live poll only if that +// failed too, since a first electrum walk fetches every beacon transaction. +let firstFetch = null; +async function whenReady() { + await warmed; + if (index) return true; + firstFetch = firstFetch || refreshFromPublishedSnapshot().finally(() => { firstFetch = null; }); + await firstFetch; + if (index) return true; + return pollAndMerge(); +} + +port.on("message", async (e) => { + const m = e.data || {}; + const reply = (ok, error) => { if (m.id != null) port.postMessage({ type: "reply", id: m.id, ok: !!ok, error }); }; + try { + switch (m.type) { + case "init": + cfg = m; + await setTor(m.torPort); + warmed = warmFromSnapshot(); + break; + case "go": startFullWork(); break; + case "tor": await setTor(m.port); break; + case "stop-polling": if (pollTimer) { clearInterval(pollTimer); pollTimer = null; } break; + case "ready": reply(await whenReady()); break; + // An on-demand poll is allowed before phase 2: it is the "this tab's + // name isn't in the snapshot" case, and costs one electrum round trip. + case "poll": await warmed; reply(await pollAndMerge()); break; + case "rebuild": await warmed; reply(await rebuild()); break; + default: reply(false, "unknown request"); + } + } catch (err) { reply(false, err?.message || String(err)); } +}); diff --git a/main.js b/main.js index 77d3f271..f211b695 100644 --- a/main.js +++ b/main.js @@ -4,7 +4,7 @@ // h, Sia s3, direct ip, redirect u). Tabs, nav controls, a search box, a home // page, and optional Tor onion routing. No system daemon; the app is the trust // boundary. -const { app, BrowserWindow, WebContentsView, ipcMain, protocol, session, Menu, clipboard, nativeTheme, shell, dialog, net } = require("electron"); +const { app, BrowserWindow, WebContentsView, ipcMain, protocol, session, Menu, clipboard, nativeTheme, shell, dialog, net, utilityProcess } = require("electron"); const path = require("path"); const url = require("url"); const http = require("http"); @@ -908,14 +908,29 @@ function overrideFor(host, tld) { } // Cached BCNR-native TLD list from tlds.bch. Registered names under a native TLD // are NOT collision candidates (whole TLD belongs to BCNR); non-native = might collide. +// Persisted (bcnr-tlds.json) so a cold start with no index yet still knows +// which TLDs belong to BCNR — that decides which names may use the quick +// lookup (see coldLookup). let bcnrTlds = ["bch"]; -function isBcnrNativeTld(tld) { return bcnrTlds.includes(String(tld || "").toLowerCase()); } +let bcnrTldsLoaded = false; +const bcnrTldsFile = () => path.join(app.getPath("userData"), "bcnr-tlds.json"); +function loadKnownBcnrTlds() { + if (bcnrTldsLoaded) return; + bcnrTldsLoaded = true; + try { const l = JSON.parse(fs.readFileSync(bcnrTldsFile(), "utf8")); if (Array.isArray(l) && l.length) bcnrTlds = l.map((x) => String(x).toLowerCase()); } catch {} +} +function isBcnrNativeTld(tld) { loadKnownBcnrTlds(); return bcnrTlds.includes(String(tld || "").toLowerCase()); } function refreshBcnrTlds(index) { try { const raw = index?.get?.("tlds.bch")?.records?.tlds; if (typeof raw === "string") { const list = raw.split(/\s+/).filter(Boolean).map((s) => s.toLowerCase()); - if (list.length) bcnrTlds = list; + if (list.length) { + bcnrTldsLoaded = true; + const changed = list.join(" ") !== bcnrTlds.join(" "); + bcnrTlds = list; + if (changed) { try { fs.writeFileSync(bcnrTldsFile(), JSON.stringify(list)); } catch {} } + } } } catch {} } @@ -1350,26 +1365,27 @@ function overlayReady(view, timeoutMs = 4000) { // onto the main thread in the same ~300 ms, and the window sat blank with // a white toolbar strip until they drained. let chromeReadyDone = false; -let bnsWarm = null; // promise from the warmFromSnapshot() kicked off in whenReady function onChromeReady() { if (chromeReadyDone) return; chromeReadyDone = true; if (addonHost) addonHost.signalUiReady(); - // Tabs wait for the BNS warm-up so sharedIndex is set before the first - // restored tab navigates; the rest of the multi-source refresh follows - // (see the comment block above startBnsPolling for how the sources - // cooperate). - (bnsWarm || warmFromSnapshot().catch(() => null)).then(() => { - try { restoreTabs(); } - catch (e) { console.error("restoreTabs failed:", e?.message); try { createTab(); } catch {} } - for (const u of pendingTabUrls.splice(0)) { try { createTab(u); } catch {} } - startBnsPolling(); - ensureIndex().catch(() => {}); // fallback for first launch without a bundled snapshot - }); + // Tabs don't wait for the BNS index: restored tabs are dormant, and a tab + // that does navigate to a name waits in resolveHost for the indexer's + // snapshot (its first, network-free phase — normally already done). + try { restoreTabs(); } + catch (e) { console.error("restoreTabs failed:", e?.message); try { createTab(); } catch {} } + for (const u of pendingTabUrls.splice(0)) { try { createTab(u); } catch {} } + // The indexer's network phase (electrum sync, Sia refresh, polling) once + // the first page is up — or after 4 s, whichever comes first. + { + let went = false; + const go = () => { if (!went) { went = true; startBnsPolling(); } }; + setTimeout(go, 4000); + try { activeTab()?.view.webContents.once("did-stop-loading", () => setTimeout(go, 500)); } catch {} + } // Re-emit any pending update notice — harmless if nothing is pending. emitUpdateAvailable(); setTimeout(loadOverlays, 250); - refreshSnapshotFromSia().catch(() => {}); scheduleStartupUpdateCheck(); refreshRemoteHomeCards().catch(() => {}); } @@ -1492,6 +1508,7 @@ function torReady() { torState = "on"; torWsAgent = new SocksProxyAgent(`socks5h://127.0.0.1:${TOR_PORT}`); session.defaultSession.setProxy({ proxyRules: `socks5://127.0.0.1:${TOR_PORT}` }); + indexerTor(); applyWebRTCPolicy(); sendTor(); } @@ -1502,6 +1519,7 @@ let addonProxyOpts = null; function torOff() { torState = "off"; torWsAgent = null; session.defaultSession.setProxy(addonProxyOpts || { proxyRules: "" }); + indexerTor(); applyWebRTCPolicy(); sendTor(); } @@ -1618,315 +1636,182 @@ async function fetchOpenSearch(href) { } catch { return null; } } -// ---- electrum server pool: hardcoded seed + on-chain discovery, persisted ---- -// Bootstrap from the baked-in seed (with pinned IPs), then refresh from the -// on-chain ELECTRUM_LIST_NAME record so the pool can be rotated without a new -// build. The last discovered list is cached to disk and tried first next launch. -let electrumPool = null; -let lastElectrumRefresh = 0; -// Per-network cache files. Chipnet keeps the historical names so existing -// profiles are not invalidated; any other network gets its own files. -let resolverNetworkId = "chipnet"; -const netSuffix = () => (resolverNetworkId === "chipnet" ? "" : `.${resolverNetworkId}`); -const electrumFile = () => path.join(app.getPath("userData"), `electrum-servers${netSuffix()}.json`); -const serverKey = (s) => (typeof s === "string" ? s : s && s.url); -function mergeServers(preferred, rest) { - const seen = new Set(), out = []; - for (const s of [...(preferred || []), ...(rest || [])]) { - const k = serverKey(s); - if (k && !seen.has(k)) { seen.add(k); out.push(s); } - } - return out; -} -async function initElectrumPool() { - const R = await getResolver(); - // The resolver's network table decides the seed pool (BNS_NETWORK env; - // chipnet unless set). Older resolver builds only export CHIPNET_ELECTRUM. - resolverNetworkId = R.NETWORK?.id || "chipnet"; - const seed = R.ELECTRUM || R.CHIPNET_ELECTRUM; - let saved = []; - try { if (fs.existsSync(electrumFile())) saved = JSON.parse(fs.readFileSync(electrumFile(), "utf8")); } catch {} - electrumPool = mergeServers(saved, seed); // discovered first, seed always kept -} -async function refreshElectrumPool() { - try { - const { fetchElectrumServers } = await getResolver(); - const found = await fetchElectrumServers({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }); - if (found && found.length) { - electrumPool = mergeServers(found, electrumPool); - try { fs.writeFileSync(electrumFile(), JSON.stringify(found, null, 2)); } catch {} - } - } catch { /* list unpublished or unreachable — keep the current pool */ } -} -function maybeRefreshElectrum() { - if (Date.now() - lastElectrumRefresh < 30 * 60 * 1000) return; - lastElectrumRefresh = Date.now(); - refreshElectrumPool(); // fire-and-forget -} +// ---- BNS index: mirror of the indexer process -------------------------------- +// The index itself — snapshot warm-start, electrum delta poll, Sia snapshot +// refresh, electrum server discovery — runs in bns-indexer.js, a separate +// process (utilityProcess), so none of it competes with the browser's main +// thread. It boots in two phases: the local snapshot first (no network — what +// the first tab needs), then the network work once this side says "go", after +// the first page has loaded. This side keeps a read-only mirror of the name +// map for the lookups that must answer synchronously (knownUnregistered, the +// native-TLD set, error-page suggestions, Hermes reverse lookups). -// host -> { entry, host, gen }. `gen` is the indexBuiltAt of the index the -// entry came from; serveBns re-resolves when a newer index has landed, so an +// host -> { entry, host, gen }. `gen` is the indexGen of the mirror the entry +// came from; serveBns re-resolves when the mirror has changed since, so an // edited ip/s3/tls record, a transfer or an expiry shows up without a restart. const entries = new Map(); -// Cached chain index. Building it (connect + fetch every beacon tx) is the slow -// part, and it was happening on EVERY navigation. Build once, reuse for lookups, -// and refresh in the background — so .bch pages open near-instantly after the first. -let sharedIndex = null, indexBuiltAt = 0, indexBuilding = null; -const INDEX_TTL = 45_000; - -// ---- warm-start from a pre-fetched beacon snapshot ------------------------ -// -// The first ensureIndex() call walks the whole beacon over electrum — that's -// the "empty tab spinner" a user sees on cold start. The snapshot pipeline -// (Argus/src/publish-name-mirror.mjs) makes that walk skippable: an operator -// publishes the raw history+txs to Sia; every Theseus install carries a -// starter snapshot bundled at build time, then GETs a fresher one on boot. -// The warm sharedIndex is served immediately; the live buildIndex runs in the -// background to catch any events past the snapshot's asOfHeight. -// -// Sources tried in order: -// 1. app.getPath("userData")/bns-name-snapshot.json — the fresher copy -// written on our last successful Sia refresh (persisted across launches) -// 2. RES_DIR/bns-name-snapshot.json — the copy bundled with the build (stale -// by definition, but strictly better than "no index at all") -// Both are optional; if neither exists, ensureIndex() does what it always did -// and the user sees the same cold-start experience as before this change. +let sharedIndex = null; // Map name -> entry, mirrored from the indexer +let indexBuiltAt = 0; // when the indexer last confirmed it is current (0 = snapshot only) +let indexGen = 0; // bumps whenever the mirror's content changes const SNAPSHOT_BUNDLED = path.join(RES_DIR, "bns-name-snapshot.json"); -// The published snapshot's location is part of the network table; the constant -// below is only the fallback for a resolver build that predates the table. -const SIA_SNAPSHOT_URL_FALLBACK = "https://s3.silentmode.st:8600/bns/name-list.json"; -async function siaSnapshotUrl() { try { return (await getResolver()).NETWORK?.nameListUrl || SIA_SNAPSHOT_URL_FALLBACK; } catch { return SIA_SNAPSHOT_URL_FALLBACK; } } -function snapshotUserPath() { return path.join(app.getPath("userData"), `bns-name-snapshot${netSuffix()}.json`); } - -function readSnapshotFrom(p) { +let indexer = null, indexerStarts = 0, indexerSeq = 0, indexerGo = false, indexerStopping = false; +const indexerPending = new Map(); +const indexWaiters = []; +function startIndexer() { + if (indexer || indexerStopping) return; + indexerStarts++; try { - if (!fs.existsSync(p)) return null; - const parsed = JSON.parse(fs.readFileSync(p, "utf8")); - // A minimal shape check — buildIndexFromSnapshot will throw with a - // clear message on anything else, but we want to log which source - // was chosen for diagnostics. - if (!parsed || !Array.isArray(parsed.history)) return null; - return parsed; - } catch { return null; } + indexer = utilityProcess.fork(path.join(__dirname, "bns-indexer.js"), [], { serviceName: "Theseus BNS indexer", stdio: "pipe" }); + } catch (e) { console.warn("[bns] indexer failed to start:", e?.message); indexer = null; return; } + indexer.stdout?.on("data", (d) => { try { process.stdout.write(d); } catch {} }); + indexer.stderr?.on("data", (d) => { try { process.stderr.write(d); } catch {} }); + indexer.on("message", onIndexerMessage); + indexer.once("exit", (code) => { + indexer = null; + for (const settle of indexerPending.values()) settle(false); + indexerPending.clear(); + if (indexerStopping) return; + // The mirror stays usable meanwhile; only freshness is lost. + const wait = Math.min(30_000, 1000 * 2 ** Math.min(indexerStarts, 5)); + console.warn(`[bns] indexer exited (${code}); restarting in ${wait / 1000}s`); + setTimeout(startIndexer, wait); + }); + indexer.postMessage({ type: "init", resolverPath: RESOLVER, userData: app.getPath("userData"), bundledSnapshot: SNAPSHOT_BUNDLED, torPort: torState === "on" ? TOR_PORT : null }); + if (indexerGo) indexer.postMessage({ type: "go" }); } - -// In-memory copy of the raw snapshot state (`{beacon, history, txs, ...}`) -// that drives the sharedIndex. Kept alongside sharedIndex so the delta poll -// can merge new beacon events into it without re-reading from disk on every -// refresh. Written to disk after each successful merge — the user cache is -// always the most up-to-date snapshot this process knows about, so a restart -// resumes from where we left off instead of from the stale bundled copy. -let currentSnapshotState = null; - -async function warmFromSnapshot() { - if (sharedIndex) return sharedIndex; // already warm — nothing to do - const { buildIndexFromSnapshot } = await getResolver(); - if (!buildIndexFromSnapshot) return null; // running against an older resolver-web.js - const snap = readSnapshotFrom(snapshotUserPath()) || readSnapshotFrom(SNAPSHOT_BUNDLED); - if (!snap) return null; - try { - const idx = buildIndexFromSnapshot({ snapshot: snap }); - sharedIndex = idx; - currentSnapshotState = snap; - // Deliberately set indexBuiltAt to 0 so the first real navigation still - // triggers a live refresh — the snapshot is a floor, not a ceiling. - indexBuiltAt = 0; - refreshBcnrTlds(idx); - console.log(`[bns] warm-started from snapshot: ${idx.size} names @ height=${snap.asOfHeight ?? "?"} root=${snap.root ?? "?"}`); - return idx; - } catch (e) { - console.warn("[bns] snapshot warm-start failed:", e.message); - return null; +function onIndexerMessage(m) { + if (!m) return; + if (m.type === "index") { + sharedIndex = new Map(m.names); + indexBuiltAt = m.builtAt; + indexGen++; + refreshBcnrTlds(sharedIndex); + for (const w of indexWaiters.splice(0)) w(sharedIndex); + verifyQuickAnswers(); + } else if (m.type === "fresh") { + indexBuiltAt = m.builtAt; + } else if (m.type === "reply") { + const settle = indexerPending.get(m.id); + if (settle) { indexerPending.delete(m.id); settle(!!m.ok); } } } - -// ---- continuous background delta refresh -------------------------------- -// -// Every POLL_INTERVAL_MS the browser opens ONE electrum connection, fetches -// the beacon's current history (a single fast call), diffs it against the -// snapshot we already hold in memory, and only fetches the verbose tx bodies -// for the txids we don't have yet. Then we rebuild the index locally and -// persist the enlarged snapshot to disk. -// -// This turns "index refresh" from ~60 s of round-trips (fetch every verbose -// tx for the whole beacon) into ~1 s of round-trips per new event. And -// because it runs while the browser is idle, by the time the user actually -// types a name into the URL bar there is nothing to wait for. -// -// Sources conspiring for freshness: -// * warmFromSnapshot on boot — sharedIndex is warm before nav -// * this poll loop, every 30 s — keeps sharedIndex live and current -// * refreshSnapshotFromSia on boot — pulls the operator's published -// snapshot from Sia for the NEXT -// boot; if this browser was closed -// for a week, next launch skips -// days of catch-up -// * ensureIndex still exists — full-walk fallback for the case -// where the poll cannot connect -// (offline first launch, etc.) -const POLL_INTERVAL_MS = 30_000; -let pollInFlight = null; -let pollTimer = null; -let pollAttempts = 0, pollLastError = null; - -// Neither the electrum connect nor its WebSocket handshake has a timeout of -// its own; a server that accepts TCP and then stalls kept pollInFlight set -// forever, wedging the poll loop until restart. -const POLL_DEADLINE_MS = 45_000; -async function pollAndMerge() { - if (pollInFlight) return pollInFlight; - let conn = null, deadline; - const work = (async () => { - pollAttempts++; - try { - const R = await getResolver(); - if (!R.connectElectrum || !R.BEACON_SCRIPTHASH || !R.buildIndexFromSnapshot) { - // Older resolver-web without the delta primitives — nothing to do. - return null; - } - if (!electrumPool) await initElectrumPool(); - - // Base state: memory > user cache > bundled > empty. The "empty" branch - // is what turns the very first cold start (no bundled snapshot present - // because we shipped a build that predates snapshotting) into a full - // rebuild — mergeFreshHistory will fetch every tx. - let snap = currentSnapshotState - || readSnapshotFrom(snapshotUserPath()) - || readSnapshotFrom(SNAPSHOT_BUNDLED) - || { beacon: R.BEACON_SCRIPTHASH, history: [], txs: {} }; - - const el = conn = await R.connectElectrum({ - electrum: electrumPool, WebSocket: currentWS(), directIP: true, - }); - try { - const freshHistory = await el.call("blockchain.scripthash.get_history", [R.BEACON_SCRIPTHASH]); - const known = new Set(snap.history.map((h) => h.tx_hash)); - // Merge fresh into snapshot history (dedup by tx_hash, keep fresh height — - // an event that was mempool at snapshot time now has a real height). - const merged = new Map(snap.history.map((h) => [h.tx_hash, h])); - const txs = { ...(snap.txs || {}) }; - let added = 0; - for (const h of freshHistory) { - if (!known.has(h.tx_hash)) { - try { - txs[h.tx_hash] = await el.call("blockchain.transaction.get", [h.tx_hash, true]); - added++; - } catch { /* unreadable — the reduction rules ignore missing txs */ } - } - merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height }); - } - const history = [...merged.values()]; - currentSnapshotState = { ...snap, beacon: R.BEACON_SCRIPTHASH, history, txs }; - const idx = R.buildIndexFromSnapshot({ snapshot: currentSnapshotState }); - sharedIndex = idx; - indexBuiltAt = Date.now(); - pollLastError = null; - refreshBcnrTlds(idx); - // Persist for the next launch. Failure here is not fatal — worst case - // we redo this merge on the next start. - try { - fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); - fs.writeFileSync(snapshotUserPath(), JSON.stringify(currentSnapshotState)); - } catch { /* readonly userdata / disk full — skip */ } - if (added > 0) { - console.log(`[bns] delta-refresh: +${added} new tx${added === 1 ? "" : "s"} (total ${history.length}, index has ${idx.size} names)`); - } - } finally { try { el.close(); } catch {} } - } catch (e) { - pollLastError = e && e.message || String(e); - // Silent — the browser stays usable via sharedIndex (last-known-good) or - // the ensureIndex fallback on the next navigation. - } - })(); - const timeout = new Promise((resolve) => { - deadline = setTimeout(() => { - pollLastError = `poll timed out after ${POLL_DEADLINE_MS / 1000}s`; - try { conn?.close(); } catch {} - resolve(null); - }, POLL_DEADLINE_MS); +function indexerRequest(type, timeoutMs) { + return new Promise((resolve) => { + if (!indexer) startIndexer(); + if (!indexer) return resolve(false); + const id = ++indexerSeq; + const timer = setTimeout(() => { indexerPending.delete(id); resolve(false); }, timeoutMs); + indexerPending.set(id, (ok) => { clearTimeout(timer); resolve(ok); }); + indexer.postMessage({ id, type }); }); - const p = Promise.race([work, timeout]).finally(() => { - clearTimeout(deadline); - if (pollInFlight === p) pollInFlight = null; +} +// Phase 2 (electrum sync, Sia refresh, 30 s polling) — after the first page. +function startBnsPolling() { indexerGo = true; try { indexer?.postMessage({ type: "go" }); } catch {} } +function stopBnsPolling() { indexerStopping = true; try { indexer?.postMessage({ type: "stop-polling" }); } catch {} } +function indexerTor() { try { indexer?.postMessage({ type: "tor", port: torState === "on" ? TOR_PORT : null }); } catch {} } +function waitForIndex(timeoutMs) { + if (sharedIndex) return Promise.resolve(sharedIndex); + return new Promise((resolve) => { + const timer = setTimeout(() => resolve(sharedIndex), timeoutMs); + indexWaiters.push((idx) => { clearTimeout(timer); resolve(idx); }); }); - pollInFlight = p; - return p; } - -function startBnsPolling() { - if (pollTimer) return; - // Fire immediately so the boot warm-start gets a delta pass right away, in - // parallel with the Sia refresh and the ensureIndex fallback below. Then - // every POLL_INTERVAL_MS while the browser is running. - pollAndMerge().catch(() => {}); - pollTimer = setInterval(() => pollAndMerge().catch(() => {}), POLL_INTERVAL_MS); -} -function stopBnsPolling() { if (pollTimer) { clearInterval(pollTimer); pollTimer = null; } } - -// Fetch the latest published snapshot from Sia and persist it as the user -// copy — the next launch (or the next warmFromSnapshot call) picks it up. -// Fire-and-forget: failures are silent; the live buildIndex path is the -// authoritative catch-up. -async function refreshSnapshotFromSia() { - try { - const res = await fetch((await siaSnapshotUrl()), { redirect: "follow" }); - if (!res.ok) return; - const body = await res.text(); - const parsed = JSON.parse(body); - if (!parsed || !Array.isArray(parsed.history)) return; - try { fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true }); } catch {} - fs.writeFileSync(snapshotUserPath(), body); - console.log(`[bns] snapshot refreshed from Sia: ${parsed.history.length} beacon txs root=${parsed.root ?? "?"}`); - } catch { /* offline / Sia unreachable / bad JSON — the live path still works */ } -} - +// One live delta poll, on demand (callers bound the wait). +const pollAndMerge = () => indexerRequest("poll", 50_000); +// Any index at all — normally the snapshot, a few hundred ms after launch; +// with no snapshot, the first live poll. `force` asks for a full rebuild. async function ensureIndex(force = false) { - const { buildIndex } = await getResolver(); - if (!electrumPool) await initElectrumPool(); - if (!force && sharedIndex && Date.now() - indexBuiltAt < INDEX_TTL) return sharedIndex; - if (indexBuilding) return indexBuilding; // dedupe concurrent builds - indexBuilding = buildIndex({ WebSocket: currentWS(), directIP: true, electrum: electrumPool }) - .then((idx) => { sharedIndex = idx; indexBuiltAt = Date.now(); refreshBcnrTlds(idx); return idx; }) - .finally(() => { indexBuilding = null; }); - // When a stale index is served below nobody awaits the rebuild; a failed one - // (routine offline) would surface as an unhandled rejection. - indexBuilding.catch(() => {}); - // If we have a stale index, don't block on the rebuild — serve stale, refresh async. - return (sharedIndex && !force) ? sharedIndex : indexBuilding; + if (force) await indexerRequest("rebuild", 120_000); + else if (!sharedIndex) { indexerRequest("ready", 120_000); await waitForIndex(120_000); } + if (!sharedIndex) throw new Error("BNS index unavailable"); + return sharedIndex; } -async function resolveHost(host) { +// ---- quick lane: a cold start with no local index yet -------------------- +// Normally the index is there within milliseconds of launch (the local +// snapshot). When it isn't — first start, the copy deleted or moved — a name +// does not wait for the full download: one lookup of just that name on the +// gateway answers it (~0.2 s), the site opens, and the index arrives in the +// background (published snapshot, then the electrum check in the indexer). +// When it lands, every quick answer is compared with the verified index; a +// mismatch reloads the tabs showing that host from the verified record. +// Only names under a known BCNR TLD take this lane: an ordinary web host is +// never sent to the gateway — it waits briefly for the index, then goes to +// the web as before. Anything that must not act on an unverified answer +// (the extension-publisher check) passes { verified: true }. +const quickAnswers = new Map(); // host -> provisional entry +async function quickLookup(key) { + try { + const up = await Promise.race([ + contentFetch(`${GATEWAY}/api/name/${encodeURIComponent(key)}`, {}), + new Promise((_, reject) => setTimeout(() => reject(new Error("timeout")), 2500)), + ]); + if (up.status !== 200) return undefined; + const j = JSON.parse(up.buffer.toString("utf8")); + if (!j || j.name !== key) return undefined; + if (!j.registered || !j.records) return null; + return { name: key, records: j.records, owner: j.owner || null, category: j.category || null, provisional: true }; + } catch { return undefined; } // undefined = no answer; null = not registered +} +// Resolves to an entry / null from the quick lane, or undefined to use the index. +async function coldLookup(key) { + const indexP = waitForIndex(3000); + if (!isBcnrNativeTld(key.split(".").pop())) { await indexP; return undefined; } + const quickP = quickLookup(key); + const first = await Promise.race([indexP.then(() => "index"), quickP.then(() => "quick")]); + if (first === "index" && sharedIndex) return undefined; + const quick = await quickP; + if (quick === undefined) { await indexP; return undefined; } + return quick; +} +function verifyQuickAnswers() { + if (!quickAnswers.size || !resolver || !sharedIndex) return; + for (const [h, q] of quickAnswers) { + let v = null; try { v = sharedIndex.get(resolver.normalizeName(h)) ?? null; } catch {} + const same = v && JSON.stringify(v.records) === JSON.stringify(q.records) && (v.owner || null) === (q.owner || null); + if (same) continue; + console.warn(`[bns] quick lookup for ${h} differs from the verified index — reloading its tabs`); + for (const t of tabs) { + let th = ""; try { th = new URL(t.view.webContents.getURL()).hostname.toLowerCase(); } catch {} + if (th === h && t.url) navigateTab(t.id, t.url); + } + } + quickAnswers.clear(); +} +async function resolveHost(host, { verified = false } = {}) { const { normalizeName } = await getResolver(); let key; try { key = normalizeName(host); } catch { return null; } - // Prefer the warm sharedIndex — the poll loop keeps it live. If we don't - // have one yet (very cold start, snapshot missing AND poll hasn't landed - // yet), fall through to a full ensureIndex build. - let idx = sharedIndex || (await ensureIndex()); - let entry = idx.get(key) ?? null; - // Miss on a possibly-stale index → try a fast delta refresh (1 history + - // only-new-tx bodies), not a full walk. Only if we've had time for at least - // one poll to land (indexBuiltAt updated by both ensureIndex and the delta - // poll). If the delta path is unavailable (older resolver-web), fall back - // to a full rebuild — same behavior as before this change. - if (!entry && Date.now() - indexBuiltAt > 8_000) { - const R = await getResolver(); - if (R.connectElectrum && R.buildIndexFromSnapshot) { - // Every dotted host the web uses lands here on a miss, so the wait is - // bounded: with electrum unreachable or stalling, an ordinary website - // must not sit behind a TCP timeout per server. The poll carries on in - // the background and the next lookup sees its result. - await Promise.race([pollAndMerge(), new Promise((r) => setTimeout(r, 2500))]); - entry = sharedIndex?.get(key) ?? null; - } else { - idx = await ensureIndex(true); - entry = idx.get(key) ?? null; + const h = host.toLowerCase(); + if (!sharedIndex) { + indexerRequest("ready", 120_000); // local copy -> published snapshot -> electrum, in the background + if (verified) await waitForIndex(120_000); + else { + const quick = await coldLookup(key); + if (quick !== undefined) { + if (quick) { quickAnswers.set(h, quick); entries.set(h, { entry: quick, host: h, gen: -1 }); attachDnsRecords(quick); } + else entries.delete(h); + return quick; + } } } - if (entry) entries.set(host.toLowerCase(), { entry, host: host.toLowerCase(), gen: indexBuiltAt }); - else entries.delete(host.toLowerCase()); // no longer registered — stop serving the old record + // No index after all that (offline, CDN and electrum unreachable): BCNR is + // unreachable, so the host goes to the web, as documented above. + const idx = sharedIndex; if (!idx) return null; + let entry = idx.get(key) ?? null; + // Miss on a possibly-stale index → one delta poll in the indexer (1 history + // call + only-new-tx bodies), allowed even before its network phase has + // started: that is the "this tab's name isn't in the snapshot" case. Every + // dotted host the web uses lands here on a miss, so the wait is bounded — + // with electrum unreachable an ordinary website must not sit behind a TCP + // timeout per server; the poll carries on and the next lookup sees it. + if (!entry && Date.now() - indexBuiltAt > 8_000) { + await Promise.race([pollAndMerge(), new Promise((r) => setTimeout(r, 2500))]); + entry = sharedIndex?.get(key) ?? null; + } + if (entry) entries.set(h, { entry, host: h, gen: indexGen }); + else entries.delete(h); // no longer registered — stop serving the old record // Signed DNS records ride alongside the on-chain answer — started here, // never awaited (see attachDnsRecords). if (entry) attachDnsRecords(entry); - maybeRefreshElectrum(); return entry; } const MIME = { html: "text/html; charset=utf-8", htm: "text/html; charset=utf-8", css: "text/css", js: "text/javascript", @@ -1959,7 +1844,7 @@ async function serveBns(request) { let rec = entries.get(host); // A lookup that throws (resolver unavailable) keeps the last good record; // one that answers "not registered" has already evicted it. - if (!rec || rec.gen !== indexBuiltAt) { try { await resolveHost(host); rec = entries.get(host); } catch {} } + if (!rec || (rec.gen !== indexGen && !(rec.entry.provisional && !sharedIndex))) { try { await resolveHost(host); rec = entries.get(host); } catch {} } if (!rec) return new Response("NXDOMAIN: " + host, { status: 404, headers: { "content-type": "text/plain" } }); const r = rec.entry.records; // Subdomain inheritance: `checkers.game.x` collapses to `game.x` in the @@ -4834,7 +4719,7 @@ async function verifyPublisherEntry(entry) { try { const name = String(entry?.publisher || "").toLowerCase(); if (!name) return false; - const owner = (await resolveHost(name))?.owner; + const owner = (await resolveHost(name, { verified: true }))?.owner; if (!owner) { console.warn(`[addons] publisher ${name}: owner unknown to the local index`); return false; } const lib = await getPublisherSig(); const ok = lib.verifyPublisherEntry(entry, owner); @@ -7383,27 +7268,13 @@ if (!process.env.THESEUS_NO_AUTOSTART && !app.requestSingleInstanceLock()) { const appUrl = webapps.appUrlFromArgv(process.argv); if (appUrl && webapps.launch(appUrl)) { /* app window only */ } else createWindow(); - // Multi-source BNS warm-up so the first .bch page opens near-instantly and - // stays fresh for as long as the browser is running. Every source runs in - // parallel — none of them can block a navigation. Source 1 starts now - // (chrome.html's bookmark favicons are bns:// fetches, so the index has - // to be warm by the time the toolbar asks for them); 2–4 are kicked off - // from onChromeReady() once the toolbar has painted. - // 1) sync: load the on-disk snapshot (user cache > bundled). - // sharedIndex is set BEFORE the first restored tab navigates. - // 2) async, continuous: startBnsPolling() opens one electrum connection - // every 30 s, fetches the beacon history (one call), and only pulls - // the tx bodies we don't already have. Merges into currentSnapshotState - // and persists — so on every page navigation the sharedIndex is at - // most 30 s old with zero user-visible latency. - // 3) async, one-shot: refresh the on-disk snapshot from the operator's - // Sia mirror. Wins the NEXT boot, not this one — after a long idle - // period the browser resumes from a snapshot fresher than the poll - // could catch up on quickly. - // 4) fallback: ensureIndex() still exists for the very first launch - // where the bundled snapshot is absent AND the poll hasn't landed - // yet — a full-walk build. - bnsWarm = warmFromSnapshot().catch(() => null); + // BNS index: a separate process (bns-indexer.js). Its first phase — the + // local snapshot, no network — starts now, so the index is warm by the + // time the toolbar's bns:// bookmark favicons or the first tab ask for + // it. Its network phase (electrum sync every 30 s, Sia snapshot refresh, + // server discovery) is released from onChromeReady() after the first + // page has loaded. + startIndexer(); // Cheap update check: fetch the releases manifest and, if a newer // version is out, surface a chip in the toolbar. No auto-install — // clicking the chip opens the download URL. First check runs from diff --git a/package.json b/package.json index 92f620be..a84165f7 100644 --- a/package.json +++ b/package.json @@ -54,6 +54,7 @@ "THIRD-PARTY-NOTICES.md", "LICENSE", "main.js", + "bns-indexer.js", "preload.js", "chrome.html", "home.html",