Theseus 0.0.7: multi-source BNS with continuous delta refresh

The first .bch page now opens with zero user-visible latency and stays fresh
for as long as the browser runs. Four sources conspire in parallel — none of
them can block a navigation:

 1. Warm start from disk (sync)
    warmFromSnapshot loads the user-cache snapshot first, then falls back to
    the copy bundled with the build. sharedIndex is set BEFORE the first
    navigation can even fire.

 2. Continuous background delta refresh (every 30 s)
    startBnsPolling opens one electrum connection, fetches the beacon
    history (a single call), and only pulls verbose tx bodies for txids we
    don't already have — mergeFreshHistory-style. Merges into memory,
    rebuilds the index locally, persists the enlarged snapshot to disk.
    Turns a ~60 s full walk into ~1 s per new event.

 3. Sia snapshot pull on boot (one-shot, wins the NEXT boot)
    refreshSnapshotFromSia downloads the operator's published snapshot from
    s3.silentmode.st for the next launch. After a long idle period the
    browser resumes from that fresher snapshot instead of walking days of
    events.

 4. ensureIndex full-walk fallback
    Still runs on boot for the case where there's no bundled snapshot AND
    the poll hasn't landed yet — very first launch, offline install, etc.

resolveHost now prefers sharedIndex (which the poll keeps live) and, on a
miss with a stale index, triggers pollAndMerge (delta fetch) instead of
ensureIndex (full walk). The old full-walk fallback is only taken if the
delta primitives aren't available (running against an older resolver-web).

Argus/src/lib/resolver-web.js exports connectElectrum so the Theseus poll
can drive its own ad-hoc queries against the pool without duplicating the
Electrum wrapper.

Bundled starter snapshot refreshed: 73 beacon txs, root c37b859682…54e414ba.

Artifacts:
  dist-public/TheseusNavigator-Setup-0.0.7.exe    (95.4 MB)
    sha256 1b0dabcd2b13067aa1fd89c3271708b35ba22dfc7a620e248b35e0487b7a104f
  dist-public/TheseusNavigator-0.0.7-portable.exe (92.7 MB)
    sha256 706dc01b21a88d3c2fbc3eeced0cab6ace873630bf3edc2d40915ca5be5c8cae

main.js

@@ -849,6 +849,14 @@ function readSnapshotFrom(p) {
   } catch { return null; }
 }
 
+// In-memory copy of the raw snapshot state (`{beacon, history, txs, ...}`)
+// that drives the sharedIndex. Kept alongside sharedIndex so the delta poll
+// can merge new beacon events into it without re-reading from disk on every
+// refresh. Written to disk after each successful merge — the user cache is
+// always the most up-to-date snapshot this process knows about, so a restart
+// resumes from where we left off instead of from the stale bundled copy.
+let currentSnapshotState = null;
+
 async function warmFromSnapshot() {
   if (sharedIndex) return sharedIndex; // already warm — nothing to do
   const { buildIndexFromSnapshot } = await getResolver();
@@ -858,6 +866,7 @@ async function warmFromSnapshot() {
   try {
     const idx = buildIndexFromSnapshot({ snapshot: snap });
     sharedIndex = idx;
+    currentSnapshotState = snap;
     // Deliberately set indexBuiltAt to 0 so the first real navigation still
     // triggers a live refresh — the snapshot is a floor, not a ceiling.
     indexBuiltAt = 0;
@@ -870,6 +879,112 @@ async function warmFromSnapshot() {
   }
 }
 
+// ---- continuous background delta refresh --------------------------------
+//
+// Every POLL_INTERVAL_MS the browser opens ONE electrum connection, fetches
+// the beacon's current history (a single fast call), diffs it against the
+// snapshot we already hold in memory, and only fetches the verbose tx bodies
+// for the txids we don't have yet. Then we rebuild the index locally and
+// persist the enlarged snapshot to disk.
+//
+// This turns "index refresh" from ~60 s of round-trips (fetch every verbose
+// tx for the whole beacon) into ~1 s of round-trips per new event. And
+// because it runs while the browser is idle, by the time the user actually
+// types a name into the URL bar there is nothing to wait for.
+//
+// Sources conspiring for freshness:
+//   * warmFromSnapshot on boot           — sharedIndex is warm before nav
+//   * this poll loop, every 30 s          — keeps sharedIndex live and current
+//   * refreshSnapshotFromSia on boot      — pulls the operator's published
+//                                            snapshot from Sia for the NEXT
+//                                            boot; if this browser was closed
+//                                            for a week, next launch skips
+//                                            days of catch-up
+//   * ensureIndex still exists            — full-walk fallback for the case
+//                                            where the poll cannot connect
+//                                            (offline first launch, etc.)
+const POLL_INTERVAL_MS = 30_000;
+let pollInFlight = null;
+let pollTimer = null;
+let pollAttempts = 0, pollLastError = null;
+
+async function pollAndMerge() {
+  if (pollInFlight) return pollInFlight;
+  pollInFlight = (async () => {
+    pollAttempts++;
+    try {
+      const R = await getResolver();
+      if (!R.connectElectrum || !R.BEACON_SCRIPTHASH || !R.buildIndexFromSnapshot) {
+        // Older resolver-web without the delta primitives — nothing to do.
+        return null;
+      }
+      if (!electrumPool) await initElectrumPool();
+
+      // Base state: memory > user cache > bundled > empty. The "empty" branch
+      // is what turns the very first cold start (no bundled snapshot present
+      // because we shipped a build that predates snapshotting) into a full
+      // rebuild — mergeFreshHistory will fetch every tx.
+      let snap = currentSnapshotState
+        || readSnapshotFrom(snapshotUserPath())
+        || readSnapshotFrom(SNAPSHOT_BUNDLED)
+        || { beacon: R.BEACON_SCRIPTHASH, history: [], txs: {} };
+
+      const el = await R.connectElectrum({
+        electrum: electrumPool, WebSocket: currentWS(), directIP: true,
+      });
+      try {
+        const freshHistory = await el.call("blockchain.scripthash.get_history", [R.BEACON_SCRIPTHASH]);
+        const known = new Set(snap.history.map((h) => h.tx_hash));
+        // Merge fresh into snapshot history (dedup by tx_hash, keep fresh height —
+        // an event that was mempool at snapshot time now has a real height).
+        const merged = new Map(snap.history.map((h) => [h.tx_hash, h]));
+        const txs = { ...(snap.txs || {}) };
+        let added = 0;
+        for (const h of freshHistory) {
+          if (!known.has(h.tx_hash)) {
+            try {
+              txs[h.tx_hash] = await el.call("blockchain.transaction.get", [h.tx_hash, true]);
+              added++;
+            } catch { /* unreadable — the reduction rules ignore missing txs */ }
+          }
+          merged.set(h.tx_hash, { tx_hash: h.tx_hash, height: h.height });
+        }
+        const history = [...merged.values()];
+        currentSnapshotState = { ...snap, beacon: R.BEACON_SCRIPTHASH, history, txs };
+        const idx = R.buildIndexFromSnapshot({ snapshot: currentSnapshotState });
+        sharedIndex = idx;
+        indexBuiltAt = Date.now();
+        pollLastError = null;
+        refreshBcnrTlds(idx);
+        // Persist for the next launch. Failure here is not fatal — worst case
+        // we redo this merge on the next start.
+        try {
+          fs.mkdirSync(path.dirname(snapshotUserPath()), { recursive: true });
+          fs.writeFileSync(snapshotUserPath(), JSON.stringify(currentSnapshotState));
+        } catch { /* readonly userdata / disk full — skip */ }
+        if (added > 0) {
+          console.log(`[bns] delta-refresh: +${added} new tx${added === 1 ? "" : "s"} (total ${history.length}, index has ${idx.size} names)`);
+        }
+      } finally { try { el.close(); } catch {} }
+    } catch (e) {
+      pollLastError = e && e.message || String(e);
+      // Silent — the browser stays usable via sharedIndex (last-known-good) or
+      // the ensureIndex fallback on the next navigation.
+    } finally { pollInFlight = null; }
+  })();
+  return pollInFlight;
+}
+
+function startBnsPolling() {
+  if (pollTimer) return;
+  // Fire immediately so the boot warm-start gets a delta pass right away, in
+  // parallel with the Sia refresh and the ensureIndex fallback below. Then
+  // every POLL_INTERVAL_MS while the browser is running.
+  pollAndMerge().catch(() => {});
+  pollTimer = setInterval(() => pollAndMerge().catch(() => {}), POLL_INTERVAL_MS);
+}
+function stopBnsPolling() { if (pollTimer) { clearInterval(pollTimer); pollTimer = null; } }
+
 // Fetch the latest published snapshot from Sia and persist it as the user
 // copy — the next launch (or the next warmFromSnapshot call) picks it up.
 // Fire-and-forget: failures are silent; the live buildIndex path is the
@@ -901,10 +1016,26 @@ async function ensureIndex(force = false) {
 async function resolveHost(host) {
   const { normalizeName } = await getResolver();
   let key; try { key = normalizeName(host); } catch { return null; }
-  let idx = await ensureIndex();
+  // Prefer the warm sharedIndex — the poll loop keeps it live. If we don't
+  // have one yet (very cold start, snapshot missing AND poll hasn't landed
+  // yet), fall through to a full ensureIndex build.
+  let idx = sharedIndex || (await ensureIndex());
   let entry = idx.get(key) ?? null;
-  // Miss on a possibly-stale index → one fresh build (the name may be newly registered).
+  // Miss on a possibly-stale index → try a fast delta refresh (1 history +
-  if (!entry && Date.now() - indexBuiltAt > 8_000) { idx = await ensureIndex(true); entry = idx.get(key) ?? null; }
+  // only-new-tx bodies), not a full walk. Only if we've had time for at least
+  // one poll to land (indexBuiltAt updated by both ensureIndex and the delta
+  // poll). If the delta path is unavailable (older resolver-web), fall back
+  // to a full rebuild — same behavior as before this change.
+  if (!entry && Date.now() - indexBuiltAt > 8_000) {
+    const R = await getResolver();
+    if (R.connectElectrum && R.buildIndexFromSnapshot) {
+      await pollAndMerge();
+      entry = sharedIndex?.get(key) ?? null;
+    } else {
+      idx = await ensureIndex(true);
+      entry = idx.get(key) ?? null;
+    }
+  }
   if (entry) entries.set(host.toLowerCase(), { entry, host: host.toLowerCase() });
   maybeRefreshElectrum();
   return entry;
@@ -2330,14 +2461,27 @@ if (!process.env.THESEUS_NO_AUTOSTART) {
     protocol.handle("bns", serveBns);
     installDownloadTracker();
     createWindow();
-    // Two-stage warm-up so the first .bch page opens near-instantly:
+    // Multi-source BNS warm-up so the first .bch page opens near-instantly and
-    //   1) sync: load the on-disk snapshot (user cache preferred over bundled).
+    // stays fresh for as long as the browser is running. Every source runs in
-    //      sharedIndex is set before the first navigation can even fire.
+    // parallel — none of them can block a navigation.
-    //   2) async: rebuild live from electrum in the background (existing path),
+    //   1) sync: load the on-disk snapshot (user cache > bundled).
-    //      and refresh the on-disk snapshot from Sia for the NEXT launch.
+    //      sharedIndex is set BEFORE the first navigation can even fire.
+    //   2) async, continuous: startBnsPolling() opens one electrum connection
+    //      every 30 s, fetches the beacon history (one call), and only pulls
+    //      the tx bodies we don't already have. Merges into currentSnapshotState
+    //      and persists — so on every page navigation the sharedIndex is at
+    //      most 30 s old with zero user-visible latency.
+    //   3) async, one-shot: refresh the on-disk snapshot from the operator's
+    //      Sia mirror. Wins the NEXT boot, not this one — after a long idle
+    //      period the browser resumes from a snapshot fresher than the poll
+    //      could catch up on quickly.
+    //   4) fallback: ensureIndex() still exists for the very first launch
+    //      where the bundled snapshot is absent AND the poll hasn't landed
+    //      yet — a full-walk build.
     warmFromSnapshot().catch(() => {});
-    ensureIndex().catch(() => {}); // catches up past the snapshot's asOfHeight
+    startBnsPolling();
-    refreshSnapshotFromSia().catch(() => {}); // wins next boot, not this one
+    refreshSnapshotFromSia().catch(() => {});
+    ensureIndex().catch(() => {}); // fallback for first launch without a bundled snapshot
     // Cheap update check: fetch the releases manifest and, if a newer
     // version is out, surface a chip in the toolbar. No auto-install —
     // clicking the chip opens the download URL. Recheck every 6h so a
@@ -2352,6 +2496,7 @@ if (!process.env.THESEUS_NO_AUTOSTART) {
     // wipe the session file too so the next launch is genuinely blank.
     saveSession();
     stopTor();
+    stopBnsPolling();  // silence the background delta refresh before exit
     vaultState = null; // drop the in-memory vault key + purposeRoot
     try {
       await clearBrowsingData({

package.json

@@ -1,6 +1,6 @@
 {
   "name": "theseus-navigator",
-  "version": "0.0.6",
+  "version": "0.0.7",
   "description": "Theseus Navigator — a browser that follows the thread. By Silent Mode, a Deviant project.",
   "author": "Silent Mode",
   "main": "main.js",