theseus/DESIGN-bns-indexer-service.md

513 lines
26 KiB
Markdown
Raw Normal View History

# BNS indexing: Ariadne's Thread owns it, everyone reads it
Status: design, revised 2026-10-03.
Already implemented:
- the public raw snapshot, with two copies (see the server tier);
- Theseus's indexer process `bns-indexer.js`, which becomes the last-resort
fallback here;
- Theseus's quick single-name lookup while no index exists.
- Migration steps 1, 2 and 4 (Ariadne 0.2.0, not released yet): the shared
modules, the indexer/resolver split, the installer. See "Implementation
notes" at the end for where the code deviates from this text, and what
step 3 (the Theseus client) gets from the pipe.
## The model
```
server this machine readers
────── ────────────────────────────────────────── ───────
gateway index ─┬─► public Ariadne's Thread (scope: All users | Just me)
(electrum) │ snapshot ──► indexer ──writes──► local snapshot copies Theseus
│ ×2 copies (sync, may stop, (read-only for readers) ──┐
│ read-only pause, crash) │ │ same
│ │ pipe (live, optional) │ │ source
└─► public BNS ▼ ▼ │ chain
indexer resolver (All browsers mode only) ◄──────────────┘
/api/name/<n> DNS :53 + NRPT + gateway (All users)
(single names) PAC proxy + gateway (Just me) other browsers
```
Ariadne's Thread is the parent of everything BNS on a machine. It has two
processes with different jobs:
- **The indexer** builds and maintains the index from the raw beacon
evidence and writes the local snapshot copies. It is heavy-ish (electrum
sync), and it is allowed to stop: Economy, "Launch at start" off, a crash,
an update.
- **The resolver** answers names: for other browsers in All browsers mode,
through system DNS or the per-user proxy. It is small and does no syncing
of its own beyond its fallbacks.
**Resolution never depends on the indexer.** The resolver and Theseus read
names through the same fallback chain (below), in which the live indexer is
only the first and freshest source. If it is gone, the answers come from the
local snapshot copies, the public snapshots or the public BNS indexer. The
only thing lost is freshness, and only until a source with newer data
answers.
## Tiers
### Server (done)
- **Live index:** the gateway keeps it over electrum, and serves single names
at `/api/name/<n>`. This is the **public BNS indexer**. Its answers carry
records and owner, interpreted by the operator; clients treat them as
provisional (see below).
- **Raw snapshot:** `cdn-name-snapshot` (VPS, every 10 min) publishes it as
**two public, read-only copies on two storage backends**:
- `https://dl.silentmode.st/bns-name-snapshot.json` (the dl vhost's disk)
- `https://navigate.st/bns/silentmode.bch/bns-name-snapshot.json` (the Sia
bucket, served by the gateway)
Both are static, with CORS `*` and no credentials. Each contains the
history plus the verbose transactions, so clients rebuild the index from
the evidence themselves. A failed upload of the second copy is retried on
the next run. Clients try them in that order (`nameListUrl`,
`nameListMirrorUrl`).
- **Android** keeps using the interpreted `bns-name-index.json`.
### The source chain (the resolver and Theseus alike)
Fastest first. A lookup is answered by the first source that has the name;
the slower sources refresh in the background.
1. **In memory:** whatever the reader has already loaded (normally all of
it).
2. **The live indexer over the pipe:** `snapshot` on connect, then
`subscribe` for changes. This is the freshest source when it is running.
3. **The local snapshot copies:** a disk read of milliseconds. There are two,
so a deleted, moved or half-written file still leaves one:
- Ariadne's: `C:\ProgramData\Ariadne\` (All users) or
`%LOCALAPPDATA%\Ariadne\` (Just me), written only by the indexer.
- The reader's own last-known-good copy. For Theseus that is its profile;
for the resolver it is beside Ariadne's file.
4. **The public snapshots,** either copy: one download of about 450 KB,
rebuilt locally from the transactions. It always runs in the background;
nothing waits for it.
5. **The public BNS indexer** (`/api/name/<n>`): a single-name lookup of
about 0.2 s. Used when sources 1–3 have no data yet, or the name is
missing and the data is stale. The answer is **provisional**:
- It is served at once.
- It is compared with the verified index when that arrives, and on a
mismatch the tabs or cached answers are replaced.
- It is never used for the extension-publisher check, which waits for
verified data.
6. **Theseus only, last resort:** its own electrum indexer process
(`bns-indexer.js`), for a portable Theseus with no Ariadne and no CDN.
**Privacy:** sources 4–5 only ever see BNS names.
- The resolver only receives queries for BNS TLDs (NRPT and the PAC file
route only those).
- Theseus only asks for names under known BCNR TLDs.
- An ordinary web host is never sent to the public indexer.
### Ariadne's Thread: the indexer
- **One implementation** with Theseus's indexer logic (`bns-indexer.js`):
- warm start from the local snapshot;
- catch-up from the public snapshots;
- the electrum delta poll every 30 s;
- the index rebuilt from the beacon transactions, with `owner`;
- the TLD set from `tlds.bch`.
This replaces the daemon's current warm start from the pre-resolved list,
which has no owners and an operator-interpreted index.
- **Local snapshot:**
- It uses the raw format, written atomically after every change, and the
previous file is kept as the second copy.
- In All users, **users may read it; only SYSTEM and Administrators may
write it.** The transactions are not checked against block headers, so
a user-writable copy would let any local process insert names or owners.
- `policy.json` stays user-writable, as today; it holds settings, not
data.
- **The pipe:** `\\.\pipe\ariadne-bns`, or `…-<user SID>` for Just me.
- **Requests:** `hello` returns `{version, network, gen, builtAt,
snapshotPath}`; `snapshot` and `get <name>` return index data; `poll`
runs one delta poll on demand.
- **Push:** `subscribe` sends `{gen, changed}` on every change.
- **Server check:** a client checks that the server process is Ariadne's
indexer before trusting it (`GetNamedPipeServerProcessId`, then the
process image path under Program Files and, for All users, its owner
SYSTEM). Otherwise a process could take the pipe name while the indexer
is down and feed false owners to the extension-publisher check. A failed
check simply skips to source 3.
- **Not TCP:** a 127.0.0.1 port has no way to tell who is listening.
### Ariadne's Thread: the resolver
- **Runs only in All browsers mode;** Theseus is its own resolver. It is
small: DNS, proxy and gateway listeners plus the source chain, with no
electrum.
- **On its own:**
- With the indexer down it keeps answering from the local copies.
- It fetches the public snapshot once an hour when the local copies are
older than that, holding the result in memory only, since the indexer
owns the files.
- It uses the public indexer for single misses.
- **If the resolver itself is down,** the task restarts it within a minute.
With NRPT routing the BNS TLDs to 127.0.0.1 there is no system fallback
for those names meanwhile (ICANN names are unaffected). Open question: a
public BNS DNS server as NRPT's second name server, and a public proxy as
the PAC file's second entry.
### Ariadne's Thread: scope × mode
**Scope** decides who it runs for:
- **All users** (the admin path): system processes for every account on the
OS.
- **Just me:** runs under one account, with no admin.
**Mode** decides who it resolves for:
- **Theseus only.**
- **All browsers:** every browser on the machine, and for All users every app
as well.
| | **Theseus only** | **All browsers** (default) |
|---|---|---|
| **All users** (admin, SYSTEM at boot) | indexer only | indexer + resolver: system DNS on :53 with NRPT rules, the local gateway on :80/:443, a machine-wide CA. Every app on the PC resolves BNS names |
| **Just me** (no admin, at logon) | indexer only | indexer + resolver: a PAC file set in the account's own Windows proxy settings (HKCU, no admin) points at a local proxy on a high port, which routes BNS names to the gateway and passes everything else through. The CA goes into the user's certificate store, which Windows confirms once |
Notes on **Just me + All browsers**:
- Chrome, Edge and Brave follow the Windows proxy settings. Firefox follows
them when set to "use system proxy", its default, but needs the CA in its
own store, which Ariadne already handles for the machine CA.
- Programs that ignore the system proxy (some command-line tools, apps with
their own resolver) do not see BNS names. Covering them takes system DNS,
which needs admin.
- A PAC file is not DNS, so it also works when a browser uses its own
DNS-over-HTTPS, which bypasses the NRPT route on networks that tamper with
DNS.
**Where the mode lives:**
- All users: `policy.json` in ProgramData. Just me: `policy.json` in the
user's profile. The field is `"mode": "theseus" | "all-browsers"`.
- Switching starts or stops the resolver and adds or removes the NRPT rules,
or the per-user proxy setting.
- **It never touches the indexer.**
### Ariadne's Thread: lifetime and start-up
Both processes are background processes with no window, independent of
Theseus.
| | **All users** | **Just me** |
|---|---|---|
| Runs as | SYSTEM, scheduled tasks | the user, scheduled tasks (hidden) |
| Starts | at OS start, before anyone logs on | at that user's logon, the earliest without admin |
| Keeps running | until shutdown | until that user logs off |
To start before logon, a user switches the scope to All users (one UAC
prompt).
**Task settings, both tasks and both scopes:**
- Restart on failure (5 times, 1 min apart).
- No execution time limit.
- **`-AllowStartIfOnBatteries -DontStopIfGoingOnBatteries`.** Today's task
uses Windows' defaults, which skip the start on battery and stop the task
when a laptop unplugs. Power is a setting of Ariadne's (below), not
something Windows decides by killing the process.
- Below-normal process and I/O priority for the indexer.
**"Launch at start" (on by default):** a switch in Ariadne's Thread settings,
mirrored in Theseus Settings › Plug-ins › Ariadne's Thread.
- **Scope of the switch:** it applies to the **indexer**. The resolver
follows the mode: in All browsers it always starts, because other browsers
depend on it and it is small.
- **Where it lives:** `policy.json`, `"startAtBoot": true | false`. The
running process enables or disables the trigger on its own task, so no
UAC prompt is needed in either scope.
- **On:** the indexer is up from OS start (All users) or logon (Just me),
whether or not Theseus is ever opened.
- **Off:** the indexer starts on demand when Theseus launches, by running the
task. For All users the task's security descriptor lets local users run it,
but not change it. Until then, names resolve from the source chain as
usual, in Theseus and in other browsers; only the freshness depends on the
indexer.
- **Turning it back on** starts the indexer at once as well as at the next
boot.
**Power: "Performance" or "Economy"**, in Ariadne's Thread settings,
mirrored in Theseus. It lives in `policy.json`: `"power": "performance" |
"economy"`, plus `"economyWhen": "on-battery" | "low-battery"`.
- **Performance (default):** the indexer syncs normally on battery.
- **Economy:** while the condition holds, the indexer **pauses its network
work**: the electrum poll, the published-snapshot refresh and server
discovery.
- Resolution is unaffected. Readers keep answering from memory and the
local copies, and the public indexer still covers single misses.
- When the condition clears, the indexer resumes with one immediate
catch-up poll.
- **The condition**, `economyWhen`:
- `on-battery`: running on battery, the old behaviour of Windows' default
task settings.
- `low-battery`: Windows' Battery Saver is on, which Windows turns on at
20% by default.
- Pausing rather than stopping keeps the warm process and a cheap resume.
Stopping would be just as safe for resolution, since the readers do not
depend on the indexer.
- **Detection:**
- The indexer reads the power state every 60 s and on resume-from-sleep: AC
or battery and the Battery Saver flag, from `GetSystemPowerStatus` via a
small helper, or one long-lived PowerShell process, never one process per
check.
- Theseus's fallback indexer uses Electron's `powerMonitor`.
- **Desktops** (no battery) never enter Economy, and the switch is hidden.
### Theseus
Theseus has its own complete BNS resolver: the **index** (`bns-indexer.js`,
a utilityProcess) and **serving** (the `bns://` protocol handler: on-chain
`h`, Sia via the gateway, `ip` with the TLS pin, `p`, signed DNS, all under
Tor or the session proxy). Serving stays in Theseus always; it costs nothing
when not in use and is what makes `bns://` work with Tor. **The index is a
standby:**
- **While Ariadne's indexer is healthy,** Theseus does not start its own at
all. That saves about 60–75 MB and its own electrum traffic.
- **Unhealthy means any of:**
- the pipe does not answer at launch;
- the pipe fails the server-process check;
- no update has arrived within the heartbeat window (a few minutes);
- Ariadne's snapshot is older than a set limit while Ariadne reports itself
running.
- **Taking over:** Theseus starts its own indexer, warm-started from
Ariadne's last snapshot file, so there is no download and no cold sync.
Lookups meanwhile keep using the source chain; nothing waits on the switch.
- **Handing back:** once Ariadne has been healthy again for a minute or two,
Theseus stops its own indexer. The delay keeps a flapping service from
starting and stopping Theseus's indexer over and over.
- **Until the Ariadne pipe exists** (migration step 2), Theseus's indexer is
its only working source and runs always, as in 0.3.70.
- **Launch:** the source chain above. In practice: Ariadne's local snapshot
file (milliseconds, parsed in Theseus's indexer process, never on the main
thread), then `subscribe` to the pipe for changes.
- **Its own copy:** Theseus keeps its own last-known-good copy in its
profile, so a missing Ariadne file still leaves one.
- **No index yet:** the quick single-name lookup (source 5) while the public
snapshot downloads in the background.
- **Never writes Ariadne's files,** and never starts, stops or reconfigures
the indexer. The one exception is running its task on demand when "Launch
at start" is off. Its Settings page shows Ariadne's status and edits its
`policy.json`.
## Installer
- **Ariadne's Thread is always installed** with Theseus. There is no "install
Ariadne?" question.
- **Setup asks two things on one page:** the scope and the mode.
### Install scope
| | **All users** (admin, default) | **Just me** (no admin) |
|---|---|---|
| Runs as | SYSTEM scheduled tasks, at boot | per-user scheduled tasks, at logon |
| Indexer + local snapshot | one per machine, `C:\ProgramData\Ariadne\` | one per account, `%LOCALAPPDATA%\Ariadne\` |
| Who may write the snapshot | SYSTEM and Administrators | the account (the same trust level as the Theseus profile) |
| Pipe | `\\.\pipe\ariadne-bns` | `\\.\pipe\ariadne-bns-<user SID>` |
| All browsers via | system DNS + NRPT + gateway | per-user PAC proxy + gateway |
Inno Setup provides the scope choice natively (`PrivilegesRequired=lowest`
with `PrivilegesRequiredOverridesAllowed=dialog`).
**Pre-selection (decided 2026-10-03):** All users when the installing
account is an administrator; **Just me when it is not**, so a standard user
is never sent to a UAC prompt they cannot approve. Both can still pick the
other.
### Mode
- **All browsers (recommended, default):** every browser on this PC resolves
BNS names. With All users, so does every other app.
- **Theseus only:** BNS names resolve in Theseus; nothing else on the system
changes.
### Changing either later
From Theseus Settings › Plug-ins › Ariadne's Thread, or Ariadne's own
settings:
- **Mode, Launch at start and Power:** no prompt, in either scope
(`policy.json`).
- **Scope,** in either direction: one UAC prompt, since it installs or
removes the SYSTEM tasks. The local snapshot moves to the new location.
- **Both present:** if a machine install and a per-user one exist at the same
time, readers use the machine one, and the per-user one stops itself.
### Other installer behaviour
- **Updates:** Ariadne keeps its own update lifecycle, chained by Theseus's
updater the way it chains the add-on updates. An indexer update never
interrupts resolution.
- **Uninstalling Theseus** asks whether to keep Ariadne's Thread. Keeping it
is the default in All-browsers mode.
## Migration from today
1. **Lift `bns-indexer.js` into a shared module,** and the source chain into
a shared reader module. Ariadne's two processes and Theseus use the same
code and the same tests.
2. **Split Ariadne's daemon** (today one process, `bnsd.js`) into:
- the indexer: raw snapshot, `owner`, `tlds.bch`, the electrum delta, the
two local copies with their ACL, the pipe, Power;
- the resolver: today's DNS, gateway and CA code on the shared reader,
plus the per-user PAC proxy for Just me.
Its read-only `/api/*` HTTP endpoints move to the resolver, for the
gateway and the Theseus status panel.
3. **Theseus:**
- the source chain;
- its own snapshot copy;
- the pipe client with the server-process check;
- its electrum indexer demoted to the last resort.
4. **Installer:** Ariadne bundled again, with the scope and mode questions.
5. **Pages:** the tools page and theseus.x describe Ariadne's Thread as part
of Theseus, with All browsers as the default mode.
## Implementation notes (steps 1, 2, 4)
### Where the code is
- Shared, in `Argus/src/lib/` (Theseus already loads its engine from there):
`bns-index-core.js` (the indexer), `bns-source-chain.js` (the reader),
`bns-pipe.js` (protocol, server, checked client). Ariadne vendors them,
together with `resolver-web.js`, through
`AriadneResolver/ariadne-resolver/scripts/sync-shared.mjs` (`--check` fails
on drift).
- Ariadne: `src/daemon/indexer.js`, `src/daemon/resolver.js` (`bnsd.js` is
now a shim for task definitions from older installers),
`src/ctl/ariadne-ctl.js` (mode / power / start-at-boot / status), and
`src/helper/ariadne-helper.cs`, compiled at install by the in-box C#
compiler into `tools\ariadne-helper.exe` and `tools\ariadne-run.exe`.
### Deviations from the text above
- **Index folder:** the copies live in `<state>\index\`, not the state root.
`%ProgramData%\Ariadne\` inherits ProgramData's "Users may create files",
so a user could plant the snapshot there before the indexer first writes
it. `index\` has users-read / SYSTEM+Administrators-write, set by
install.ps1 and re-applied by the indexer at every start. It also holds
`bns-index-state.json` (gen, builtAt, paused, pid: when the state was last
confirmed, since the snapshot is only rewritten on change),
`electrum-servers.json` (a writable pool file would choose the evidence),
the resolver's own copy, and `machine-install.json`, the marker that makes
a Just-me install stand down.
- **Server check:** Node cannot reach a pipe's handle, so the client spawns
`ariadne-helper pipe <name> <scope> <imagePrefix>`. The helper connects,
checks the process on the other end of that same connection, prints one
JSON verdict line, then relays the pipe over its stdin/stdout. The checks:
- the image path is under the prefix;
- All users: the token user is SYSTEM, or, when a standard user may not
open a SYSTEM token, the server runs in session 0;
- Just me: the server runs as the same account.
Node's pipe server grants everyone the right to add instances
(`writableAll`), so every connection is checked, not only the first.
Without the exe the same C# source runs through PowerShell (~0.6 s
instead of ~0.1 s).
- **Restart:** Task Scheduler's "restart on failure" only covers a task that
fails to *start*. A node process that exits with an error leaves the task
"Ready" until the next boot, and that was also true of the old daemon.
`ariadne-run.exe` supervises both tasks: it restarts after 1 min, 5 tries
(the count resets after 10 min up), and exit 0 means a deliberate stop. It
holds node in a kill-on-close job, and it is windowless, so Just-me tasks
show no console.
- **Mode:** in Theseus-only mode the resolver task stays registered. The
resolver removes its routing and exits when it sees the mode, at start or
on a switch. Switching back has to run the task (`ariadne-ctl mode
all-browsers` does). Authenticated users may run, not change, the
All-users tasks.
- **No owners on `/api/*`:** the resolver's HTTP endpoints carry records but
not `owner`. Owner data only travels over the checked pipe.
- **Installer scope question:** Inno's built-in dialog always pre-selects "for
me only" under `PrivilegesRequired=lowest`. Setup therefore asks scope and
mode on its own page and relaunches itself with `/ALLUSERS` or
`/CURRENTUSER` when needed (`PrivilegesRequiredOverridesAllowed=commandline`).
- **Scope switch and the snapshot:** All users -> Just me copies the machine
snapshot. Just me -> All users does not: the per-user file is
user-writable, and the machine indexer must not start from it. It takes
the public snapshot instead, one download.
- **Just me + Firefox:** HKCU\Software\Policies is not writable without
admin, so no Firefox policy is set. Firefox trusts the user-store root
only through its own "use the operating system's certificates" setting.
Not yet verified.
### What the Theseus client (step 3) needs from the pipe
- **Name:** `\\.\pipe\ariadne-bns` (All users), else
`\\.\pipe\ariadne-bns-<own user SID>`. If both answer, use the machine
pipe.
- **Connecting:** through Ariadne's `tools\ariadne-helper.exe` with
`connectIndexerPipe` from `Argus/src/lib/bns-pipe.js`.
- `imagePrefix` is the folder of Ariadne's `runtime\node.exe`: the
`InstallLocation` from Ariadne's uninstall key, HKLM for All users, HKCU
for Just me.
- A rejected verdict (`{verified:false, reason}`) means: skip to the local
files.
- **Requests,** newline JSON `{id, op}` / `{id, ok, ...}`:
- `hello` returns `{version, network, gen, builtAt, snapshotPath, paused,
pid}`;
- `snapshot` returns `{network, gen, builtAt, tlds, names: [[key, entry]]}`;
the entries carry `owner`;
- `get {name}` returns `{gen, entry|null}`;
- `subscribe` returns `{gen}`;
- `poll` returns `{gen, polled}`, or `ok:false, error:"paused"` under
Economy. The indexer runs at most one on-demand poll per 5 s.
- **Pushes after `subscribe`:**
- `{event:"change", gen, changed, entries}`. `entries` is absent when more
than 500 names changed. Take `snapshot` again whenever `gen` skips.
- `{event:"fresh", gen, builtAt}` after each poll that changed nothing.
- `{event:"paused", paused}`.
- **Health:** a missing `fresh` is not by itself unhealthy. Under Economy
there are none, and Theseus must not take over then: that would undo
Economy. Use `hello.paused`, or the `paused` push. Unhealthy means:
- the pipe fails the check;
- `hello` gets no answer;
- or the indexer is not paused, yet `builtAt` has not moved for the
heartbeat window.
`bns-source-chain.js` already implements the reader side (heartbeat via
`hello` after 3 silent minutes).
- **Warm start for Theseus's own indexer** when taking over: `snapshotPath`
from `hello`, or `<state>\index\bns-name-snapshot.json` / `.prev.json`. Pass
them to `createIndexer({warmPaths})`.
- **Settings:** edit `policy.json` directly (`mode`, `power`, `economyWhen`,
`startAtBoot`), and run the task where `ariadne-ctl` does (mode ->
all-browsers, start-at-boot -> on). Task names:
- All users: `BNS Indexer`, `BNS Resolver Daemon` (unchanged, so 0.3.70's
panel still finds it);
- Just me: `BNS Indexer (<user>)`, `BNS Resolver (<user>)`.
- **Until step 3 ships:** Theseus 0.3.70's Ariadne panel only knows `BNS
Resolver Daemon`. In Theseus-only mode that task is Ready, not Running, so
the panel shows Ariadne as off.
### Bundling Ariadne into the Theseus installer (designed, not wired)
- **Payload:** the Ariadne setup exe, as an `extraResources` file of
Theseus's NSIS build, so the two keep separate version lifecycles.
- **Running it:** from NSIS after Theseus's own files are in place:
`AriadneResolver-Setup.exe /VERYSILENT /SUPPRESSMSGBOXES /NORESTART
/ALLUSERS|/CURRENTUSER /BROWSERMODE=<mode>`.
- Theseus's installer shows the two questions itself (one custom NSIS
page, the same pre-selection rule) and passes them on, so the user sees
one page, not two installers.
- An existing Ariadne keeps its scope unless the user changed it.
- **Elevation:**
- Theseus per-user + Ariadne All users: one UAC prompt, from Ariadne's
setup (`/ALLUSERS` respawns elevated).
- Theseus per-machine: already elevated, so no extra prompt.
- A declined prompt leaves Theseus installed and Ariadne absent. Theseus
then runs its own indexer (source 6) and offers the install again from
the Ariadne panel.
- **Updates:** Theseus's updater chains Ariadne's, as with the add-ons. The
manifest entry stays `ariadne-resolver` / `win-x64`, and the update passes
the installed scope (`/ALLUSERS` or `/CURRENTUSER`, read from which
uninstall key exists), so an update never asks the scope question.
- **Uninstalling Theseus** asks whether to keep Ariadne (default: keep in
All browsers mode) and runs Ariadne's `QuietUninstallString` only if not.
- **Portable Theseus** never bundles it (see open questions).
## Open questions
- **System-level fallback while the resolver restarts:**
- a public BNS DNS server as NRPT's second name server (All users);
- a public proxy as the PAC file's second entry (Just me).
- **Portable Theseus:** it never installs Ariadne, so it lives on its own
copy, the public sources and its own indexer. Should it offer to install
Ariadne?
- **macOS/Linux:** launchd or systemd units for the same two processes, once
Theseus ships there.