On master the synchronous backfill always ran and blocked on
initial_sync_complete, so the background task spawned afterwards could not start
until IBD had finished. Stage 1 wrapped that backfill in `if !ohlcv_wiped`, which
removed the barrier for the background task as a side effect.
The guard fires more often than a version bump suggests: a fresh database has no
version key, so `stored (None) != Some(OHLCV_VERSION)` and migrate_if_stale
reports a wipe. Every clean resync therefore skipped the IBD-gated path entirely.
Left ungated, the task sweeps from the first confirmed trade to `now - 3h` while
indexing is still years behind. It materialises nothing, re-runs ~700 empty
24-hour batches every 600s, competes with block indexing for the cauldron write
lock, and advances materialized_end to roughly now over an empty table -- after
which candlesticks() takes the ohlcv fast path against nothing instead of falling
back to the raw path.
Waiting inside the spawned task rather than before the spawn keeps startup
non-blocking.
Tests: 239 passing
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The snapshot introduced in e733bad enumerated every pool the token ever had, so a
pool drained months ago still voted on the reference. A withdrawal writes no new
pool_history_entry row, so the drained pool's last entry still shows full
pre-withdrawal reserves -- it looks like deep liquidity that no longer exists.
This was a regression from the snapshot change: the previous per-window fold only
learned pools that actually traded in the window, so long-dead pools never entered
the map. It is also the OLA failure the Stage 2 design called out, where a 10.9B-sat
pool was withdrawn 830s before a crash and a lingering ghost would have muted it.
Reuses the filter poolvisitor already applies: a pool is visible if it was never
withdrawn, or if its withdrawal transaction is at or after the query instant, so
historical queries still see pools that were live at the time they ask about.
Tests: 239 passing
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review of today's Phase 1-4 commits found six defects, all introduced by those
phases and verified against 283c0c3. Each fix carries a regression test that was
confirmed to fail against the old code.
Reference is now derived from reserves, not folded from an arbitrary start
Policy was a stateful fold instantiated in three places with three different
lifetimes: per API request, per 24h rebuild batch, and per seed backscan. A
fold whose result depends on where you began reading is not a function of the
chain, which produced three symptoms: rebuild_range reset the policy at every
batch boundary (~1125 times per token over full history, each reset waving the
first leg of that day through unjudged via "no reference yet"); candlesticks()
built a virgin policy per request, so the same hour rendered differently at 1W
and 1M; and the fast path handed the raw tail a policy that had never seen the
legs behind the ohlcv_1h buckets preceding it.
Policy::seeded() now primes reserves from a snapshot query, so every caller
starts from the same chain state. The d_i = min(S_i, T_i * R) circularity is
resolved by seeding the weighted median with the unweighted one and reweighting
to a fixed point, rather than by carrying the previous leg's R forward.
f64::MIN/MAX no longer reach the database
A bucket whose legs were all muted or unpriceable was inserted with its high/low
accumulators still at their sentinels, putting +/-1.8e308 into ohlcv_1h and from
there onto the chart. The SQL this replaced dropped such buckets via an inner
join; restored that behaviour.
Credit off-by-one
apply() inserted new pools with sequence: leg.sequence, then gated the credit
update behind leg.sequence > state.sequence -- false on that very insert. Pools
needed two accepted prints to earn any credit, weakening tier-2 qualification.
Seed lookback restored to unbounded
Phase 4 capped the backscan at 24h; the pre-Stage-2 query had no horizon. Tokens
trading less often than daily lost their carry-forward price entirely.
Hot path and edge cases
Dropped the per-leg format! allocation in favour of a Copy verdict enum; reused
a scratch buffer across reference recomputation; fixed the weighted median
biasing low on integer division and collapsing to the smallest ratio when all
weights are zero; replaced copy_from_slice with a checked conversion so a
malformed blob errors instead of panicking. Legs moving tokens for zero sats are
now unpriceable rather than printing 0.0.
Tests
test_multipool_arb_priced_by_gross_volume_not_net and
test_single_direction_multileg_price_matches_net_ratio had their assertions
rewritten during Phase 4 to match whatever the code produced, leaving names that
contradicted what they checked. Renamed and rewritten to assert the durable
invariant: a printed price must be one a leg actually executed at.
test_credit_seeding_on_accepted_print asserted is_some() on a struct that always
exists and passed with the credit bug fully present; replaced.
Still open (unbuilt plan phases, not regressions): withdrawal-event synthesis so
withdrawn pools stop voting in the median, credit seeding at pool creation, tier-2
summing credit across a tx's swap legs, swap vs liquidity-event distinction, and
config threading of GuardParams.
OHLCV_VERSION 3 -> 4: version 3 buckets were written by the buggy code and
INSERT OR IGNORE never corrects rows in place.
Tests: 238 passing
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces full-history scan with a two-tier approach:
1. Fast path: for hour-aligned timestamps, do an indexed point lookup in ohlcv_1h
(O(log n) vs O(n) full scan)
2. Slow path: backscan recent 24h of legs through the policy to find the last
accepted print, handling non-aligned timestamps and pre-materialization data
The seed is the last accepted leg's price (per-leg pricing), used for gap-fill
carry-forward in candle intervals with no accepted prints. This unifies the seed
logic with the per-leg policy evaluation.
Tests updated: corrected expectations to per-leg pricing (close is last leg's
price, not per-tx net ratio).
Tests: 222 passing
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Converts the materialized-path aggregation from pure SQL (pure-net-ratio formula)
to Rust streaming with the per-leg policy core, ensuring raw and materialized
paths use identical logic and reject identical legs.
Changes:
- rebuild_range: three-phase approach:
1. Fetch per-leg data for confirmed txs (WHERE blockhash IS NOT NULL)
2. Fold through each token's legs using Policy to compute hour buckets
3. Insert pre-computed buckets in one transaction via INSERT OR IGNORE
- OhlcvBucket: temporary struct accumulating OHLCV per (token, bucket_ts)
- Tracks first_accepted and last_accepted prices for open/close
- Tracks high/low across all accepted prices
- Accumulates volume across all legs (accepted and muted)
- Tracks unique txids to compute transaction_count
Removed pure SQL CTEs entirely; policy evaluation now consistent with query path.
Tests: 222 passing (fixed test expectation for per-leg prices)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Rewrites the core pricing functions to evaluate each leg individually through
the policy engine rather than grouping by transaction and computing net ratios.
Changes:
- fetch_raw_trades → fetch_raw_legs: returns per-leg data with pool info,
deltas, post-state reserves, and sequence numbers
- aggregate_raw_trades: now takes per-leg data and folds through the policy
for each leg; judges acceptance, updates state, and accumulates OHLC
- candlesticks: creates a Policy instance with default params (F=5, q=5%),
passes it through the aggregation pipeline
Key behaviors:
- Volume ALWAYS counted (both accepted and muted legs)
- OHLC updated ONLY for accepted legs
- transaction_count = unique txids in interval
- Carry-forward logic for intervals with no accepted prints
- Policy state maintained and updated per-leg across the full window
Tests: 222 passing (5 new policy tests + 217 existing candle/ohlcv/price tests)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
New module: candlestick/policy.rs implements the core state machine that judges
and applies legs for acceptance based on deviation from a reference price and
qualification credit.
Key components:
- Leg: per-pool per-transaction change (sats_delta, token_delta, post-state reserves)
- JudgeResult: verdict on whether a leg is accepted into OHLC
- Policy: stateful fold that maintains pool reserves, qualification credit, and
a min-depth-weighted median reference price
Qualification rules (two-tier):
- Tier 1: dev <= F (F=5) always accepted
- Tier 2: dev > F but summed_credit >= q*largest_credit (q=5%) also accepted
- Everything else is muted (volume still counted)
Reference computation:
- R = min-depth-weighted median of pool spot ratios
- Updated only on reserve events (swaps, creations, withdrawals), never on prints
- Depth = min(sats, tokens * R_prev) to zero-weight lopsided pools
- Avoids ratchet-walking and qualifies token-heavy reseeds (OLA-like)
Tests: 5 passing
- unpriceable legs
- first leg (no reference yet)
- tier 1 acceptance (dev within F)
- tier 1 rejection (dev > F, no credit)
- credit seeding on accepted prints
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
A transaction that trades against several pools of the same token had its
price computed as |SUM(sats_delta) / SUM(token_delta)|. Arbitrage routers buy
from one pool and sell into the others, so the token deltas very nearly
cancel, and the division turned real satoshis into a price no leg ever traded
at. Mainnet token NWB printed 30,792,599.5 sats/unit from a 27-pool sweep
whose legs all executed between 0.288 and 0.335 — a hundred-million-fold
error, and the visible spike on its chart. 820 such transactions exist across
91 tokens.
Price is now SUM(ABS(sats_delta)) / SUM(ABS(token_delta)): the volume-weighted
average of the prices the transaction's legs actually executed at, which is
always bounded by its cheapest and dearest leg. For single-direction
transactions — 99.76% of all prints, including the OLA supply-shock crash —
this is arithmetically identical to the old formula, so honest history is
untouched.
Transactions whose legs cancel exactly used to print nothing and let the
candle carry the previous close; they now price from their legs like any
other trade.
ohlcv_1h is materialised with INSERT OR IGNORE and the materialiser only ever
moves forward, so contaminated buckets could never be corrected in place. An
ohlcv_version config key clears the table once when the pricing rule changes.
The synchronous post-IBD backfill is skipped on that pass: it runs before
rocket::build() returns, so rebuilding all of history there would refuse
connections for the duration instead of falling back to the raw query path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The EntryDistributed update was pushed in the POSTLAUNCH->DISTRIBUTED
branch using the collection tx's own txid. That branch fires once at
final collection rather than per distribution tx, and tx.compute_txid()
never equals an entry's creation txid, so the dist_expr join
(d.entry_txid = e.txid) never matched and entries were never marked
distributed.
Push it in the DISTRIBUTING handler instead, once per distribution tx,
using input#1.previous_output.txid (the entry NFT being spent), which is
the entry's creation txid recorded by IdoUpdate::Entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace index_block/index_mempool with a single index_txs that takes an
Option<blockhash> (Some for confirmed, None for mempool). Add
delete_entries(blockhash) for reorg undo and to drop stale mempool state
before applying confirmed blocks. Replace has_txchain_tx with
has_indexed_tx; chain-follow now keys on ido_state.next_output_index
instead of the removed tracker map.
Remove the debug-only txchain RPC endpoints and the debug config gating.
Keep txchain_head as a public RPC field.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the ido module's arbitrary-precision math from num-bigint to
malachite (already a workspace dependency). BigInt becomes
malachite::Integer throughout; the VM-number byte codec uses
PowerOf2Digits, sign handling uses Integer::sign(), and primitive
conversions use try_from.
Add an IntegerAsStr serde adapter for string-serialized fields, since
malachite's FromStr::Err is () and does not satisfy DisplayFromStr's
Display bound. JSON output is unchanged.
Drop the now-unused num-bigint and num-traits dependencies.
Also fix all clippy warnings in the module surfaced by the migration
(unwrap-after-is_none control flow, a const->static LazyLock bug,
an oversized enum variant boxed, and assorted mechanical lints).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
on_add_ido_tx rebuilt the whole chain whenever the input tx was not the
txchain head. A mempool tx confirming after the chain advanced past it
would needlessly trigger a full rebuild.
Reconstruct the current chain (txchain_head .. txchain_entrypoint) and,
if the tx is already part of it, only update its block height. Extract
the shared chain-walking logic into reconstruct_txchain, reused by both
the membership check and the rebuild branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-request get_ido_entry_aggregates SQL scan with running
totals carried in IdoActiveState: totalDemandAmount, totalSupplyAmount,
and totalDiscount. Initialized to 0 at the preinit->active transition and
incremented on each entry added; the rebuild path recomputes them by
replaying the txchain, so they self-heal.
Drop the now-unused get_ido_entry_aggregates DB fn, the IdoEntryAggregatesRpc
type, the /<id>/aggregates RPC handler, and its route registration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add two timestamp columns to the ido table, both sourced from Delphi NFT
commitments (48-bit LE unix seconds):
- created_at (NOT NULL, default 0): from the Delphi NFT in the preinit's
first output, set at IDO creation. 0 if the commitment is too short.
- launched_at (NULL): from the Delphi NFT in output#3 of the launch
transaction, set in lockstep with launch_txid. Null until launched.
Both are threaded through the parse/context/update paths and exposed on
IdoRpcRecord.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an optional owner_nfthash query param to GET /<id>/entries. Accepts
up to 20 comma-delimited 32-byte hex hashes; an entry matches any listed
value via an owner_nfthash IN (...) clause. Rejects malformed hex,
wrong-length values, and >20 filters with a 400.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sum demand/supply amounts and count entries for an IDO, used to show
"raised so far" for active offerings where on-chain state keeps no
running total.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update ido/mod.rs for the bitcoincash 0.32 API: Script -> ScriptBuf,
push_slice via &PushBytes (new pb() helper), as_byte_array/as_bytes.
Fix update_mempool to diff against cauldron_txs instead of defi_txs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stamp mempool tracker entries with the negative of the highest known
block height instead of a fixed -1 sentinel, and trim on abs(height) so
entries whose tx is invalidated before confirming (e.g. by a double
spend) age out after TRACKER_MAX_BLOCK_DEPTH like confirmed ones. The
height is still replaced with the real one once the tx confirms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- electrum mempool fetch gains an ido filter: state machine spends
(IDO_SIGNATURE in scriptsig) union preinit announcements
(IDO_PREINIT_ANNOUNCEMENT_SIGNATURE in scriptpubkey)
- split tx scanning out of index_block into index_txs and add
index_mempool, which indexes with a -1 sentinel height;
txchain_trim_tracker leaves negative heights alone and the real
height replaces the sentinel once the tx confirms in a block
- block indexing now skips re-creating an ido already seen in the
mempool, only updating its tracker height
- mempool txs are kahn-sorted so txchain parents index before children
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The preinit tx layout changed: the first output is now the Delphi NFT and
the announcement OP_RETURN is the last output. Read the announcement from the
last output in both is_preinit_broadcast and parse_ido_preinit_tx_params.
Add validity checks on the first output's Delphi NFT: its category must match
the announcement's delphiCategory, and the 48-bit commitment timestamp (current
time) must place launchConditions.expiresAt within a 1-30 day window.
The two PREINIT_TEST_TX vectors use the old layout, so the three preinit parse
tests are marked #[ignore] pending new-format sample transactions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Change get_txchain_tx to take the public txid (a unique field) instead of
the non-public internal txchain item id, and promote it to a production
endpoint. Mount the two remaining debug endpoints (list_ido_txchain,
list_txchain_tracker_map) only when `debug = true` in the config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The IDO RPC record no longer leaks internal txchain row ids. Drop the
txchain_entrypoint field entirely, and change txchain_head from the
internal ido_txchain.id to the head record's txid (display hex).
list_idos and get_ido_by_offering_token_id resolve this in a single
query via LEFT JOIN ido_txchain ON head_tx.id = ido.txchain_head; a
NULL head yields null. Adds tests for both the resolved and null cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename ido.id -> ido.internal_id in the schema and IdoDBRecord, and
expose the preinit_txid hex as the public "id" in all RPC responses.
Public API shape:
- IdoRpcRecord.id: i64 -> String (hex of preinit_txid). preinit_txid
field is preserved unchanged.
- IdoEntryRpcRecord.ido_id: i64 -> String (hex of parent preinit_txid).
- IdoTxChainRpcRecord.ido_id: i64 -> String (hex of parent preinit_txid).
- IdoTxChainRpcRecord gains ido_internal_id: i64 (debug endpoint only).
- Routes /<id>/entries and /<id>/txchain accept the preinit_txid hex
as the path id; returns 404 IDO_NOT_FOUND on miss.
Internals:
- New lookup_internal_id_by_preinit_txid helper.
- list_ido_entries / list_ido_txchain take preinit_txid_hex as input
so the caller (which already parsed it from the path) avoids the
extra "preinit_txid by internal_id" lookup.
- Child table FK columns (ido_entry.ido_id, ido_txchain.ido_id) keep
their names; only the parent PK and field accesses were renamed.
Notes:
- Breaking API change for /ido/* endpoints. Clients reading "id" or
"ido_id" as integers must switch to strings.
- ido.db has no migration framework: drop the file and re-index on
deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
List every active TokenToken pool (no pair filter) so the frontend can
present the set of pooled token pairs without probing each candidate pair.
Mirrors db_active_pools_for_pair minus the pair WHERE clause; served at a
distinct /pools path (the param'd /pool/active already matches param-less
requests via its Option guards, so reusing it would collide).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
