This is the full developer documentation for Leani # Leani documentation > Build and operate application-specific Ethereum data processors with bounded, explicit state. Leani is an Ethereum indexing node you run in your own infrastructure. It backfills public history, follows the chain, and stores the results your application needs. Start with one small example: Run Leani [Install Leani](/docs/getting-started/install/), then [follow an Ethereum block](/docs/getting-started/) or [index contract events](/docs/getting-started/bounded-mainnet/). Connect an application [Query and subscribe with TypeScript](/docs/guides/query-then-follow/), then [persist changes to PostgreSQL](/docs/guides/postgres-consumer/). Build with the Rust library [Register your own processor, build a custom node, and query its output.](/docs/guides/custom-processor/) Each tutorial includes a matching configuration, commands, and expected output. The tutorials use a source checkout; Leani is also published as release archives, a Homebrew formula, and the npm SDK (see [published releases](/docs/getting-started/install/#published-releases)). Leani does not maintain EVM state. Arbitrary `eth_call`, native account balances, and transaction simulation need an execution provider. Read [known limitations](/docs/operations/preview-limitations/) before connecting a production application. The expandable sidebar groups hold detailed concepts, API and configuration reference, operations, and contributor material. Start with [how Leani works](/docs/concepts/how-leani-works/) when you want to understand what happens between a source block and your application output. # Install Leani > Build Leani from source, check the binary, and prepare the TypeScript SDK if you need it. ## Build from source The tutorials use a source checkout, which also holds their example configurations. To install a published release instead, see [published releases](#published-releases). You need Git, Rust installed through rustup, and your platform’s C/C++ build tools. The repository’s `rust-toolchain.toml` selects Rust 1.97.0 automatically. The first build downloads dependencies and can take several minutes. Build from source ```shell git clone --branch v0.1.1 https://github.com/smart-byte/leani.git cd leani cargo build --locked -p leani export LEANI_SOURCE="$PWD" export PATH="$LEANI_SOURCE/target/debug:$PATH" leani --version ``` Build the pinned workspace and use its binary in this terminal.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) Expect `leani 0.1.1`. If you already have the checkout, start at `cargo build --locked -p leani` in its root directory. `LEANI_SOURCE` points to the checkout; later tutorials use it to find example configs. The `PATH` setting makes `leani` available in this terminal. In each additional terminal, return to the checkout and run: ```bash export LEANI_SOURCE="$PWD" export PATH="$LEANI_SOURCE/target/debug:$PATH" ``` Continue with [your first Ethereum block](/docs/getting-started/). For a check without network access, run the [offline fixture](/docs/getting-started/offline-proof/). ## Prepare the TypeScript SDK The examples use Bun 1.2 or later. From the checkout: ```bash bun install --cwd packages/sdk --frozen-lockfile bun run --cwd packages/sdk build ``` Repository examples resolve `@smart-byte/leani-sdk` to the local source. To use the built package in your own application, run this from that application’s directory: ```bash bun add "$LEANI_SOURCE/packages/sdk" ``` Then follow the [query and subscription example](/docs/guides/query-then-follow/). For Rust applications, see [build a custom node](/docs/guides/custom-processor/). ## Published releases Install with `brew install smart-byte/tap/leani`, or choose a matching archive from the [GitHub releases](https://github.com/smart-byte/leani/releases). Verify the archive’s published checksum and provenance before running it, as [Verify a release](/docs/operations/verify-releases/) shows. Containers use `ghcr.io/smart-byte/leani:v`, such as `leani:v0.1.1`; pin the release’s digest for deployment. The Linux archives need glibc 2.39 or newer: Ubuntu 24.04, Debian 13, Fedora 40, RHEL 10 and its rebuilds, or a later release of these. On an older or musl-based distribution, run the container image or build from source. Select the `v0.1.1` binary release and install the matching SDK with `bun add @smart-byte/leani-sdk@0.1.1`. An install without the exact version resolves npm’s `latest` tag, which may not match your node. Keep binary, SDK, and configuration versions aligned with the release you selected. # Follow your first Ethereum block > Install Leani, read a real Ethereum block, and prepare a node for your application. [Install the current release](/docs/getting-started/install/) first. This example follows real Ethereum Mainnet blocks with no API key and no configuration file. Run it from a fresh directory: ```bash cd "$(mktemp -d)" ``` Follow Ethereum blocks ```shell leani subscribe blocks ``` Start the embedded P2P subscription and follow labelled live blocks.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) On first use, Leani asks independent public providers for a recent finalized checkpoint. At least two must agree before you can accept it. This is the starting trust anchor for subsequent verification. Peer discovery can take several minutes, even on a later run with cached state. Leani prints progress to stderr, then one line per block, shaped like: ```text 2026-09-07T12:11:59Z block=25925396 txs=381 gas=43.17M / 60.00M (72.0%) base_fee=0.069459109 gwei blobs=3 unattested ``` Your block numbers and values will differ. Leani keeps following new blocks until you press Ctrl-C. Add `--format json` for scripts. Read the finality label Rows say `unattested`, `attested` or `finalized`; see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes). Use `leani subscribe blocks --finality finalized` when you need only blocks Ethereum consensus has finalized. ## Prepare a node for your application When you want an HTTP API and SDK subscriptions, initialize a reusable node: Prepare a reusable block node ```shell mkdir leani-blocks && cd leani-blocks leani init blocks leani serve # In another terminal in this directory: # leani subscribe blocks --format json ``` Initialize a compact processor config and serve its API.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) Leave `serve` running. In another terminal, set up `leani` as described on the install page and change into the same `leani-blocks` directory. Then run: ```bash leani subscribe blocks --format json ``` It attaches to the running node. The API is at `http://127.0.0.1:18080` and JSON-RPC is at `http://127.0.0.1:18545`. Discover the configured processor with: ```bash curl -s http://127.0.0.1:18080/v1/capabilities ``` Expect a `block-summary` processor with `subscriptions: true`. Keep each node’s data directory separate; do not start two nodes against the same writable data. ## Choose your next example * [Index a small historical range](/docs/getting-started/bounded-mainnet/) to query decoded contract events. * [Query and follow with TypeScript](/docs/guides/query-then-follow/) to seed application state. * [Build a custom processor in Rust](/docs/guides/custom-processor/) to run your own transformation using the Leani library. For startup diagnostics and all CLI fields, see [block subscriptions](/docs/guides/live-block-subscriptions/). Read [known limitations](/docs/operations/preview-limitations/) before connecting production applications. # Index a bounded Mainnet range > Validate a finite Mainnet configuration, run one bounded backfill, and inspect exact processor coverage and query results. Index 1,000 Ethereum Mainnet blocks and query every Uniswap V2 pool-reserve observation in them, each stamped with its hour. This uses public Xatu data, needs no API key, and stops after a finite range. First [install from source](/docs/getting-started/install/) so `leani` and `LEANI_SOURCE` are available in your terminal. The commands below copy a complete configuration into a fresh temporary directory: Bounded Mainnet run ```shell LEANI_HISTORY_DIR="$(mktemp -d)" cp "$LEANI_SOURCE/config/modes/windowed.toml" "$LEANI_HISTORY_DIR/leani.toml" cd "$LEANI_HISTORY_DIR" leani doctor --json leani backfill \ --processor uniswap-v2-sync-30d --from-block 17000000 --to-block 17000999 leani serve ``` Use the checked-in configuration to index a finite Mainnet range.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) `doctor` should report `valid: true`. The backfill report should show continuous coverage from `17000000` through `17000999`. Leave `serve` running and query it from another terminal: Query indexed Uniswap events ```shell curl -s \ 'http://127.0.0.1:8080/v1/processors/uniswap-v2-sync-30d/collections/uniswap_v2.sync/entities?limit=3' curl -s \ 'http://127.0.0.1:8080/v1/processors/uniswap-v2-sync-30d/status' ``` Read decoded Mainnet events and their exact processor coverage from the local API.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) Expect one entity per `Sync` event in the range; `limit=3` returns only the first page. Follow `nextCursor` to read the rest. Entities are ordered by their keys, a transaction hash and log index, so sort by `blockNumber` and `logIndex` when drawing a time series. Each entity contains `values.reserve0` (USDC, six decimals) and `values.reserve1` (WETH, eighteen decimals). These are exact integer strings. `bucketStartTimestamp` names the hour of the event’s block, for grouping observations by hour; the processor computes no hourly aggregate or average price. The profile keeps a window of about 30 days of blocks and prunes older observations. ## Check coverage The status response’s `available` intervals should cover the entire requested range. A high `processedThrough` alone does not prove that earlier blocks were indexed. Xatu’s `finalized` label is a dataset assertion; this profile has independent finality verification disabled. Try an explicit range query, then one outside your indexed interval: ```bash curl -s 'http://127.0.0.1:8080/v1/processors/uniswap-v2-sync-30d/collections/uniswap_v2.sync/entities?fromBlock=17000000&toBlock=17000999' curl -s 'http://127.0.0.1:8080/v1/processors/uniswap-v2-sync-30d/collections/uniswap_v2.sync/entities?fromBlock=17001000&toBlock=17001999' ``` The first should return retained entities with complete requested coverage. The second should return `range_incomplete`, rather than an empty success. An indexed range whose output was subsequently pruned returns `output_not_retained`. ## Decode a different contract event The configuration’s `processors.settings` selects the contract address and Solidity event ABI. For a small WETH transfer example, use the complete config at `examples/weth-transfers/node.toml`: ```bash LEANI_EVENTS_DIR="$(mktemp -d)" cp "$LEANI_SOURCE/examples/weth-transfers/node.toml" "$LEANI_EVENTS_DIR/leani.toml" cd "$LEANI_EVENTS_DIR" leani doctor --json leani backfill --processor weth-transfers --from-block 17000000 --to-block 17000009 leani serve ``` This example uses API port 8081 so it can run beside the Uniswap node. Query: ```bash curl -s 'http://127.0.0.1:8081/v1/processors/weth-transfers/collections/weth.transfers/entities?limit=100' ``` Expect 205 transfers across three pages. Each entity’s `values` contains `from`, `to`, and `value`; WETH uses 18 decimals. Change the address, ABI, collection, instance name, and block range for your contract. Keep `history_mode = "on_demand"` for explicit backfills, and use a fresh data directory when changing processor identity or schema. Queries and subscriptions need different policies These examples retain query output with delivery disabled. To follow changes, use the matching node configuration in the [TypeScript guide](/docs/guides/query-then-follow/) or [PostgreSQL guide](/docs/guides/postgres-consumer/). For a transformation beyond event decoding, [build your own processor with the Rust library](/docs/guides/custom-processor/). # Query once, then follow live > Seed application state from an exact snapshot boundary, then resume the SSE stream without missing changes between the two operations. A normal query followed by a separate stream connection has a race: a change can commit after the query but before the stream starts. `queryAndFollow()` returns the snapshot and its opaque change boundary atomically. Passing that boundary to `subscribe({ after })` closes the gap. Unattested publication only `query-and-follow` answers `409 deferred_publication_snapshot` for a processor with `publish = "attested"` or `publish = "finalized"`. Its retained state already holds blocks whose changes are unpublished, and the stream never emits an undo for them, so a snapshot could seed rows no later event corrects. Follow such a processor’s change stream from its head instead. First follow the [source installation](/docs/getting-started/install/), including the SDK setup. Stop any other node using API port 8080 before this example. The supplied profile retains the latest USDC/WETH pool state and a bounded change stream; its public Xatu history needs no provider key: ```bash LEANI_SDK_DIR="$(mktemp -d)" cp "$LEANI_SOURCE/examples/sdk-stream/node.toml" "$LEANI_SDK_DIR/leani.toml" cd "$LEANI_SDK_DIR" leani doctor --json leani backfill --processor usdc-weth-latest --from-block 17000000 --to-block 17000999 leani serve ``` In another terminal, from the repository root: ```bash cd "$LEANI_SOURCE" bun examples/sdk-stream/index.ts ``` Expect `Loaded 1 pools`. The process then waits for more changes; this historical-only profile has no live source, so silence after the seed is expected. Press Ctrl-C to stop. The example keeps the price map in memory; for a durable destination, continue with [PostgreSQL](/docs/guides/postgres-consumer/). This historical-only profile lets you inspect the seed and resume boundary. For ongoing P2P updates, enable live execution and verified finality as described in the [Uniswap subscription guide](/docs/guides/uniswap-price-subscriptions/). Query once, then follow without a race ```typescript import { applyEntityChange, createLeaniClient, isLiveGapChange, LeaniError, type GenericSnapshotPage, type LeaniClient, type UniswapPoolPrice, } from "@smart-byte/leani-sdk"; // Run examples/sdk-stream/node.toml first (see the guide). export async function followPrices( leani: LeaniClient = createLeaniClient({ baseUrl: "http://127.0.0.1:8080" }), prices = new Map(), signal?: AbortSignal, ): Promise { const processor = "usdc-weth-latest"; const collection = "uniswap.pools.current"; const target = { put: (key: string, value: UniswapPoolPrice) => { prices.set(key, value); }, delete: (key: string) => { prices.delete(key); }, }; while (!signal?.aborted) { try { const snapshot = await leani.processors.queryAndFollow( processor, collection, { signal }, ); const seed = new Map(); try { let page: GenericSnapshotPage = snapshot; while (true) { for (const entity of page.data) seed.set(entity.key, entity.data); if (!page.nextCursor) break; page = await leani.processors.queryEntities( processor, collection, { cursor: page.nextCursor, signal }, ); } } finally { // Cleanup gets its own deadline even if the subscription was cancelled. await leani.processors.releaseSnapshot(processor, snapshot.snapshotId) .catch((error: unknown) => { if (!(error instanceof LeaniError && error.status === 404)) throw error; }); } prices.clear(); for (const [key, value] of seed) prices.set(key, value); console.log(`Loaded ${prices.size} pools`); for await (const change of leani.processors.subscribe( processor, { after: snapshot.boundaryCursor, signal }, )) { // Both apply and undo carry the entity mutation to perform. await applyEntityChange(target, change); if (change.operation === "finalized" && !isLiveGapChange(change)) { console.log("finalized through", change.data.throughBlock); } } } catch (error) { if (signal?.aborted) return; // ResetRequiredError from SSE is also a LeaniError with cursor_expired. if (!(error instanceof LeaniError) || !["cursor_expired", "query_snapshot_expired"].includes(error.code)) throw error; console.log("Retention moved; rebuilding the snapshot"); } } } if (import.meta.main) await followPrices(); ``` Seed local state at an exact boundary, resume SSE there, and handle apply, undo, and finalized events.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/sdk-stream/index.ts) 1. Follow every `nextCursor` to materialize the complete stable snapshot. 2. Release the snapshot when the local seed is committed. 3. Start the resumable stream at `boundaryCursor`. 4. Apply both `apply` and `undo`; an undo envelope carries the inverse entity mutation. 5. Treat `finalized` as an explicit transition and `reset_required` as a rebuild instruction. The SDK throws `ResetRequiredError` on an SSE reset; expired HTTP cursors throw `LeaniError` with `code: "cursor_expired"`. Never infer either transition from elapsed time. SSE resume is not a durable acknowledgement The stream cursor closes an in-process query/follow race and reconnects the live feed. For a destination that must fence Leani’s retention across process restarts, use a named [durable consumer](/docs/guides/durable-consumer/). The page `limit` only controls transport size. Creating the first page copies the entire matching collection under the store writer lock. Defaults cap each snapshot at 100,000 rows / 64 MiB and all outstanding snapshots together at 32 snapshots / 128 MiB (including key and row overhead). A request over the per-snapshot caps or outside coverage is rejected before it takes the writer lock. Physical store admission also applies. Release snapshots promptly; the node expires them after five minutes and cleans expired rows every 30 seconds. `query_snapshot_capacity` returns retryable HTTP 503 when aggregate admission is full. Very slow readers can outlive the retained stream and must rebuild, as the example does. # Consume changes into PostgreSQL > Run a complete TypeScript consumer, inspect committed rows, and verify cursor resume after restart. This example copies USDC/WETH processor changes into PostgreSQL. It commits each change and its cursor in one database transaction, then acknowledges Leani. Restarting the consumer resumes from its acknowledgement without duplicating committed rows. ## Start the matching Leani node Follow [query once, then follow](/docs/guides/query-then-follow/) through the backfill and `serve` commands. Keep that node running at `http://127.0.0.1:8080`. It must contain the `usdc-weth-latest` instance with delivery enabled. The earlier `windowed.toml` event example has delivery disabled and cannot be used for this consumer. You also need Bun and the local SDK setup from [Install Leani](/docs/getting-started/install/). ## Prepare PostgreSQL If you have PostgreSQL running, create an empty `leani_example` database and set `DATABASE_URL` to its connection string. Alternatively, with Docker running: ```bash docker run --name leani-example-postgres \ -e POSTGRES_PASSWORD=leani-example \ -e POSTGRES_DB=leani_example \ -p 127.0.0.1:54329:5432 -d postgres:17-alpine docker exec leani-example-postgres pg_isready -U postgres -d leani_example export DATABASE_URL='postgres://postgres:leani-example@127.0.0.1:54329/leani_example' ``` Wait until `pg_isready` reports that it is accepting connections. The container is an isolated example database; use your own credentials for other deployments. ## Run the consumer In the terminal with `DATABASE_URL`, run: ```bash cd "$LEANI_SOURCE" export LEANI_URL=http://127.0.0.1:8080 export LEANI_PROCESSOR=usdc-weth-latest export LEANI_CONSUMER=postgres-example export LEANI_CONSUMER_CREDENTIAL="$(openssl rand -hex 32)" bun examples/sdk-postgres/index.ts ``` The application creates its tables and registers the named consumer with the secret you supplied. Registration does not generate this secret. Preserve it when restarting or moving to another terminal; do not generate a different secret for the same consumer identity. Expect `Consuming usdc-weth-latest into PostgreSQL; press Ctrl-C to stop.` The historical example then drains the retained changes and waits for new ones. The runnable application consists of `examples/sdk-postgres/index.ts` and its imported helper, `examples/sdk-subscription/index.ts`. ## Inspect the result and restart Using `psql` with your connection string: ```bash psql "$DATABASE_URL" -c 'select count(*), max(sequence) from leani_applied_changes' psql "$DATABASE_URL" -c 'select processor, network, stream, sequence from leani_cursors' curl -s http://127.0.0.1:8080/v1/processors/usdc-weth-latest/consumers/postgres-example ``` For the 1,000-block example, the checked run committed 491 changes. PostgreSQL’s highest sequence should match Leani’s `acknowledgedSequence`, with `lagChanges` zero after draining. A change is a processor mutation or finality transition; it is not necessarily one row per Ethereum block. Press Ctrl-C in the consumer terminal. Run `bun examples/sdk-postgres/index.ts` again in that same terminal, keeping the exported identity and credential. The row count and cursor should stay unchanged until new changes arrive. ## Keep the transaction boundary The destination implementation is: Atomic PostgreSQL destination ```typescript class PostgresDestination implements Destination { constructor( private readonly sql: SQL, private readonly processor: string, private readonly network: string, ) {} async migrate(): Promise { // `stream` scopes sequences: a replaced node store numbers them anew. await this.sql` create table if not exists leani_applied_changes ( processor text not null, network text not null, stream text not null, sequence numeric(20, 0) not null, cursor text not null, operation text not null, kind text not null, entity_key text, payload jsonb, block_number bigint, primary key (processor, network, stream, sequence) ) `; await this.sql` create table if not exists leani_cursors ( processor text not null, network text not null, stream text not null, cursor text not null, sequence numeric(20, 0) not null, updated_at timestamptz not null default now(), primary key (processor, network, stream) ) `; } async transaction( stream: string, work: (transaction: DestinationTransaction) => Promise, ): Promise { await this.sql.begin(async (sql) => { await work({ apply: async (change: ChangeEnvelope) => { await sql` insert into leani_applied_changes ( processor, network, stream, sequence, cursor, operation, kind, entity_key, payload, block_number ) values ( ${this.processor}, ${this.network}, ${stream}, ${change.sequence}, ${change.cursor}, ${change.operation}, ${change.kind}, ${change.key}, ${JSON.stringify(change.data)}::jsonb, ${change.block?.number ?? null} ) on conflict (processor, network, stream, sequence) do nothing `; }, storeLeaniCursor: async (cursor: string, sequence: string) => { await sql` insert into leani_cursors ( processor, network, stream, cursor, sequence, updated_at ) values ( ${this.processor}, ${this.network}, ${stream}, ${cursor}, ${sequence}, now() ) on conflict (processor, network, stream) do update set cursor = excluded.cursor, sequence = excluded.sequence, updated_at = excluded.updated_at where leani_cursors.sequence <= excluded.sequence `; }, }); }); } } ``` Apply an idempotent change and persist its opaque cursor in one PostgreSQL transaction.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/sdk-postgres/index.ts) The `(processor, network, stream, sequence)` key makes replay idempotent. Sequences are unique only within one node store: a node whose store is replaced numbers its changes from the start again, and the consumer is created again there. `stream` names the durable consumer by its processor instance, ID, and creation time, as the node reports them, so those changes never collide with the rows already applied. Re-creating the consumer on the same store also changes `stream`, so the retained changes it receives again are applied again under the new value. The cursor is stored in the same transaction as the event. A failed acknowledgement after commit can cause redelivery; a committed event must remain safe to apply again. This example stores a change log. When building domain tables, apply both forward and inverse mutations to keep those tables correct through reorgs. Retention remains a node policy The query/follow example uses a bounded change window. A consumer that falls behind that window must rebuild. For a destination that must prevent pruning until it acknowledges, use `delivery.mode = "until_acknowledged"` with a configured required consumer; see [operating profiles](/docs/guides/choose-operating-profile/) and [durable delivery](/docs/concepts/durable-delivery/). The reusable [consumer loop](/docs/guides/durable-consumer/) explains leases, reset handling, and commit-before-ack in detail. Stop this example’s Docker database when finished with `docker stop leani-example-postgres`. # Build a custom node in Rust > Use the Leani Rust library to register a processor, index three Mainnet blocks, and query your own output. Use Leani as a Rust library when your application needs a transformation beyond its built-in processors. You supply a `Processor` and register its factory; Leani supplies acquisition, the CLI, storage, checkpoints, and delivery. The result is your own executable with the usual `doctor`, `backfill`, and `serve` commands. This example stores each block’s number and timestamp. It requests headers only and indexes three Mainnet blocks from public EraE archives. No API key or live peer connection is needed. Start with the [source installation](/docs/getting-started/install/) and keep `LEANI_SOURCE` pointing to the checkout. ## Run the complete example From the checkout, build the custom binary and test its processor: ```bash cargo build --locked -p leani-custom-example cargo test --locked -p leani-custom-example ``` Use a fresh working directory so the example owns its own database: ```bash export LEANI_CUSTOM_BIN="$LEANI_SOURCE/target/debug/leani-custom-example" LEANI_CUSTOM_DIR="$(mktemp -d)" cp "$LEANI_SOURCE/examples/custom-node/node.toml" "$LEANI_CUSTOM_DIR/leani.toml" cd "$LEANI_CUSTOM_DIR" "$LEANI_CUSTOM_BIN" doctor --json "$LEANI_CUSTOM_BIN" backfill \ --processor example-block-summary-local --from-block 19426589 --to-block 19426591 "$LEANI_CUSTOM_BIN" serve ``` Expect `valid: true` from `doctor` and continuous coverage of `19426589..19426591` after backfill. `serve` stays in the foreground. In a second terminal, query the custom route: ```bash curl -s http://127.0.0.1:9080/v1/processors/example-block-summary-local/query/19426589 ``` The response has the shape `{ "blockNumber": 19426589, "timestamp": 1710338159 }`. The timestamp comes from the block header. Query an unindexed block and the extension returns HTTP 404. Discover the extension through `/v1/capabilities` when your application does not know the configured instance name. Source trust EraE checks execution commitments against the archive material. This historical example relies on the public archive catalog; it does not claim independent consensus anchoring or follow the live head. ## Use the Rust library The package `leani` exports the node runtime and processor registry. The example’s Cargo dependency is a local path: ```toml [dependencies] leani = { path = "../../crates/node" } leani-primitives = { path = "../../crates/primitives" } leani-processor-api = { path = "../../crates/processor-api" } ``` Those paths are relative to `examples/custom-node/Cargo.toml`. Use paths to the same checkout in your own project. The `leani` node crate is not published, and a processor must implement the traits of the `leani-processor-api` that `leani` builds on, so take all three from the checkout, although `leani-primitives` and `leani-processor-api` are also on crates.io. The complete manifest also includes the serialization, Tokio, and Axum dependencies used below. The entry point registers the factory and hands command-line execution to `leani::run_with_registry`: Run Leani with your processor registry ```rust #[tokio::main] async fn main() -> std::process::ExitCode { let mut registry = ProcessorRegistry::new(); if let Err(error) = registry.register(BlockSummaryFactory) { eprintln!("error: {error}"); return Exit::Failure.into(); } match run_with_registry(registry).await { Ok(exit) => exit.into(), Err(error) => { eprintln!("error: {error:#}"); Exit::Failure.into() } } } ``` The leani Rust library supplies the CLI and node runtime.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/src/main.rs) `ProcessorRegistry::new()` starts empty: this binary contains the custom processor registered here. The stock `leani` executable will not recognize `example-block-summary`. Run the custom binary for this configuration. ## Transform a block The complete implementation is in `examples/custom-node/src/main.rs`. Its descriptor declares `Capability::Header`, the processor’s ID/version, its starting block, and its entity/change schemas. These methods implement the transformation: Map a block and commit its summary ```rust async fn map(&self, block: &BlockFrame) -> Result { self.descriptor.requirements[0] .validate_frame(block) .map_err(|error| ProcessorError::Input(error.to_owned()))?; let payload = serde_json::to_vec(&BlockSummary { block_number: block.block.number.0, timestamp: block.block.timestamp, }) .map_err(|error| ProcessorError::DeltaPayload(error.to_string()))?; Ok(EncodedDelta::new( &self.descriptor, block.chain_id, block.block, payload, )) } async fn reduce( &self, transaction: &mut dyn ReducerTransaction, cursor: &ProcessorCursor, delta: &EncodedDelta, ) -> Result { delta.validate(&self.descriptor)?; validate_cursor(cursor, delta)?; let key = delta.block.number.0.to_be_bytes().to_vec(); transaction .put(COLLECTION, key.clone(), delta.payload.clone()) .await?; let change = DomainChange { kind: self.change_kind.clone(), key, operation: ChangeOperation::Upsert, payload: delta.payload.clone(), }; transaction.emit(change.clone()).await?; Ok(DomainChanges { changes: vec![change], }) } ``` Processor methods from the complete custom-node example.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/src/main.rs) `map` validates its input and encodes a deterministic block summary. It must not depend on wall-clock time or external requests. `reduce` stores that summary under the block number and emits its change in the same transaction. Leani manages checkpoints and publication around that transaction. The example implements `change_json` and `entity_json` to expose typed JSON through delivery and the generic collection API. For your own transformation, change the summary type, mapping, and rendering together. Change the schema/version when the durable representation changes. ## Configure the processor The full `node.toml` supplies the archive source, budgets, and listeners. Its processor section is: Custom processor configuration ```toml [[processors]] id = "example-block-summary" instance = "example-block-summary-local" version = "1.0.0" history_mode = "on_demand" start_block = 19426589 [processors.state] mode = "durable" [processors.output] mode = "full" [processors.delivery] mode = "window" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "unfinalized" safety_blocks = 256 [processors.coverage] verification_segment_blocks = 8192 [processors.settings] change_kind = "example.block_summary" ``` The complete lifecycle policy for the block-summary processor instance.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/node.toml) `history_mode = "on_demand"` lets the explicit bounded `backfill` command own this run. `output.mode = "full"` retains all three summaries, and `delivery.mode = "window"` makes their changes available to subscribers. For a new experiment, use a new data directory. ## Add a typed query route The example’s optional query extension mounts `/{number}` underneath its instance’s query path. Its factory connects both the processor and extension: Custom processor query extension ```rust #[derive(Clone, Copy, Debug)] struct BlockSummaryFactory; #[derive(Clone, Copy, Debug)] struct BlockSummaryQueryExtension; #[allow(clippy::unnecessary_literal_bound)] impl QueryExtension for BlockSummaryQueryExtension { fn id(&self) -> &str { "block-summary-v1" } fn alias(&self) -> Option<&str> { Some("block-summaries") } fn routes(&self) -> Router { Router::new().route("/{number}", get(get_block_summary)) } } async fn get_block_summary( State(context): State, Path(number): Path, ) -> Result, ApiError> { let value = context .entity(COLLECTION, &number.to_be_bytes()) .await? .ok_or_else(|| ApiError::not_found("block summary is not indexed"))?; let summary = serde_json::from_slice(&value).map_err(|error| { ApiError::internal(&format!("stored block summary is invalid: {error}")) })?; Ok(Json(summary)) } #[allow(clippy::unnecessary_literal_bound)] impl ProcessorFactory for BlockSummaryFactory { fn id(&self) -> &str { "example-block-summary" } fn description(&self) -> &str { "Example block number and timestamp summaries" } fn create( &self, configured: &ProcessorConfig, _context: ProcessorFactoryContext, ) -> Result { let settings = configured .decode_settings::() .map_err(|error| ProcessorFactoryError::configuration(error.to_string()))?; Ok( ProcessorComponents::new(Arc::new(BlockSummaryProcessor::new(configured, settings)?)) .with_query_extension(Arc::new(BlockSummaryQueryExtension)), ) } } ``` A native processor factory attaching a typed, processor-owned query route.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/src/main.rs) With the example node still running, prepare the SDK using the install page, then run this from the checkout: ```bash bun examples/custom-node/client.ts ``` Discover and query a custom extension ```typescript import { createLeaniClient } from "@smart-byte/leani-sdk"; interface BlockSummary { blockNumber: number; timestamp: number; } const leani = createLeaniClient({ baseUrl: "http://127.0.0.1:9080" }); const capabilities = await leani.capabilities(); const extension = capabilities.queryExtensions.find( (candidate) => candidate.processor === "example-block-summary-local", ); if (!extension) throw new Error("block summary query extension is unavailable"); const summary = await leani.request( `${extension.basePath}/19426589`, ); console.log(summary); ``` Capability-driven TypeScript lookup of a processor-owned query route.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/client.ts) ## Move it into your application The example includes a standalone manifest with explicit dependencies. With your source checkout named `leani`, create a sibling application: ```bash cd "$LEANI_SOURCE/.." cargo new --bin my-leani-node cp "$LEANI_SOURCE/examples/custom-node/standalone.toml" my-leani-node/Cargo.toml cp "$LEANI_SOURCE/examples/custom-node/src/main.rs" my-leani-node/src/main.rs cp "$LEANI_SOURCE/examples/custom-node/node.toml" my-leani-node/node.toml cp "$LEANI_SOURCE/rust-toolchain.toml" my-leani-node/rust-toolchain.toml cd my-leani-node cargo build cargo test --locked ./target/debug/my-leani-node --config node.toml doctor --json ``` This application is outside Leani’s workspace. Its dependencies point to `../leani/crates/...`; adjust those paths if your checkout has another name. The compatibility patch in the standalone manifest matches the workspace’s upstream dependency patch. Commit the generated `Cargo.lock`, then use `cargo build --locked` for subsequent builds. Use your new binary with the same bounded `backfill` and `serve` commands from the beginning of this guide. Complete files: [standalone Cargo manifest](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/standalone.toml), [Rust implementation and tests](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/src/main.rs), and [node configuration](https://github.com/smart-byte/leani/blob/v0.1.1/examples/custom-node/node.toml). The implementation is compiled into your binary. For contract events an ABI describes, the [event decoder](/docs/getting-started/bounded-mainnet/) may be sufficient without Rust code. For custom aggregation or additional queries, continue with the [processor contract](/docs/contributing/processor-extension-contract/). Native processors run in the node process. Review their code and test their restart and reorg behavior before using them with production data. # Known limitations > Know Leani's current production boundaries, unsupported behavior, and explicit operator obligations before deploying it. The current release is intended for evaluation, processor development, reproducible backfills, and guarded application integrations with an independent fallback. * It is not a complete Ethereum archive node and does not maintain EVM state. Arbitrary `eth_call`, balances, storage, proofs, tracing, and state-diff RPC are not served by the EVM-free core. * Xatu rows are trusted public dataset input. Sparse EraE material verifies execution commitments but still depends on an external catalog until the planned consensus-root anchoring is complete. Source trust is exposed and must not be described as fully trustless history. * Raw RPC history is post-merge exact. Ommer-aware pre-merge exactness is deferred. * Only configured processor material and bounded recent/raw-history profiles are queryable. Pruned material must be reacquired. * The native browser SDK requires same-origin hosting or an application-owned reverse proxy with an explicit CORS policy, and the page’s origin in `api.allowed_origins`. * Mainnet multi-day soak, chaos, abrupt-power-loss, broad cross-source parity, and every advertised host sweep remain release gates for a production claim. * JSON-RPC has no built-in access control, and the API bearer is the only credential for admin routes. A bind beyond loopback needs the API bearer or an explicit `allow_unauthenticated_remote` opt-in; such deployments must still use a trusted network or authenticated reverse proxy and must not expose admin/debug endpoints publicly. Treat this page and the [release process](https://leani.dev/docs/contributing/releasing/) as the public scope boundary for the current release. # Capability honesty and the EVM-free boundary > Distinguish source capabilities, configured processor products, and the intentionally refused stateful Ethereum methods. Leani exposes what the running node can answer. Capability depends on all three layers: 1. the source can supply the required headers, bodies, receipts, logs, or consensus proof; 2. the processor or raw-history policy retained the required product; and 3. the query can locate the requested block or transaction exactly. That is why applications should inspect `/v1/capabilities`, processor status, and collection metadata instead of assuming every compiled route is usable for every configured node. ## What the EVM-free core can do With capable sources, Leani can serve blocks, transactions, receipts, logs, chain metadata, subscriptions, processor snapshots, generic collections, and processor-owned queries. Hash-only historical lookup also requires a retained locator or recent material. ## What it refuses Without execution state and an EVM, arbitrary `eth_call`, `eth_getBalance`, `eth_getCode`, `eth_getStorageAt`, `eth_estimateGas`, and debug/trace methods are not honest implementations. They fail explicitly. Derived balances or prices are processor products with documented scope; they are not substitutes for arbitrary historical Ethereum state. Keep an execution provider for unsupported methods and as an independent fallback. The exact matrix lives in [Ethereum JSON-RPC compatibility](/docs/reference/ethereum-json-rpc/). # Coverage, finality, and reorgs > Read Leani's exact coverage intervals, understand independent finality, and handle explicit apply, undo, and finalized transitions. Coverage answers “which block intervals did this processor commit?” Finality answers “what chain-confidence status does each committed interval have?” They are related but deliberately not collapsed into one boolean. ## Finality levels and publish modes Every record carries one level. A block only ever rises: unattested, attested, finalized. It is never published twice; each rise is a marker. | Level | Evidence | Typical delay after the slot starts | Can be undone | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | ------------------------------------------------------------------------------------------- | | `unattested` | At least two execution peers served the same header, it passes Leani’s header rules (base fee, gas limit, blob gas, slot timing, fork fields), and it extends the delivered chain within two slots of the newest attested head | \~2–5 s today; \~5–14 s after Glamsterdam | Yes: by a reorg, by an attested head on another block, or after 60 s without an attestation | | `attested` | At least 342 of 512 sync-committee members signed it (or a descendant), verified locally from a light-client update. After Glamsterdam the committee signs the next beacon block, which builds on this payload | \~15 s today; \~25–29 s after Glamsterdam | Yes, by a reorg | | `finalized` | Casper FFG finality, verified from a light-client finality update | \~13–19 min | No | `publish` sets the first level a processor’s consumers see: | `publish` | First level | Markers | Query-then-follow | | ---------------------- | ------------ | ---------------------------- | ------------------------------------------------- | | `unattested` (default) | `unattested` | `attested`, then `finalized` | Yes | | `attested` | `attested` | `finalized` | No: its state holds unpublished unattested blocks | | `finalized` | `finalized` | none | No | An Ethereum RPC’s `latest` corresponds to `unattested`; Leani’s JSON-RPC `latest` is the delivered tip at any level, and `safe` and `finalized` name the finalized head. Choose `attested` when a consumer must never act on a block that is later undone for want of an attestation. Late blocks — first served more than 7 s into their slot, the ones the network most often orphans — wait for their child or their attestation before they are published unattested. A bounded history job that only needs its final aggregate sets `[processors.delivery] emit = "job_result"` instead of a level. `finalized` reflects the node’s configured finality source. Embedded and native-P2P runs verify it locally through light-client proofs. Sources that serve finalized material under dataset trust (Xatu, EraE) carry the label under that source’s trust model; subscribing to them does not turn the label into an independent proof. Post-merge execution headers carry no seal, so any peer could serve a self-consistent block of its own after the chain tip. An `unattested` block therefore rests on several peers, Leani’s header rules and its link to the delivered chain, and stays within two slots of the newest attested head: an execution header that at least two thirds of the sync committee signed in a light-client optimistic update, which the finality source verifies with the header’s execution branch within the slot it appears. When a processor on the lane requires attested frames, the lane includes a block only when its header chain leads to an attested head. Without verified finality there is no attested head, so live following requires a finality source. If verified finality ever names another block at a height than the chain the node followed, neither block is finalized. When the node’s blocks there are unfinalized, for example because the lane stalled across a reorg, the network lanes restart: retained unfinalized blocks that neither link to the new finalized anchor nor are proven to be its ancestors are undone as a reorg and never finalized, as a branch reorged away while the node was down is at startup. Where parent links do not reach the anchor, execution headers from peers that link by parent hash down from it prove which retained blocks are its ancestors, and those are kept and finalized; the runbook’s [finality contradiction](/docs/operations/runbook/#finality-contradicts-the-followed-chain) section explains the proof and its fallback. The lanes halt and report not ready instead when the contradicted block is itself finalized, or when restarting twice did not resolve the contradiction.  In that example, `complete` is false because `113..119` is absent. A high `processedThrough` value does not hide the internal gap, and finalized coverage does not imply the unfinalized tail is absent. ## Filling history gaps Leani resumes from committed coverage, not from the last downloaded object. Retained material and configured archive sources are preferred when capable. Verified execution P2P can recover the remaining finalized gaps while the live lane continues following head. By default, omission of `sources.live.history_fallback_blocks` makes every gap at or after the processor start block eligible. Setting it creates an explicit suffix boundary. It is a policy limit, not a protocol limitation or a new default. A block-local live lane can open a gap of its own: after downtime it resumes at the newest finalized anchor, past blocks it never processed. Automatic history refills them. With `history_mode = "on_demand"`, the processor’s live stream announces them in a `system.live_gap.put` change instead, and the application requests them, as [Keep your application in sync](/docs/guides/keep-your-application-in-sync/) shows. A processor with `[processors.live_gap_fill]` fills a short range itself, from finalized history before it applies the next block, and announces only what it did not fill; see [Live gap fill](/docs/operations/configuration-guide/#live-gap-fill). Availability still matters Eligibility does not guarantee that peers retain very old bodies and receipts. Full-gap recovery is attempted; an incapable or unavailable source fails explicitly rather than converting missing history into empty output. ## Reorgs are delivery events For attested publication (`publish = "attested"`), downstream consumers can see `apply`, `undo`, and later `finalized` changes. Processor state, undo records, and published changes commit together. A consumer must implement these operations idempotently and must not infer finality from age alone. With `publish = "finalized"` the change stream carries only finalized `apply` records and `system.finality` markers. Unfinalized blocks are applied to processor state and journaled for undo, but their changes stay in the store until verified finality promotes them. Their bytes count against the delivery budget from the moment they are applied, so a slow consumer pauses the processor exactly as it would with unattested publication. For a processor with the default canonical delivery ordering, a block that is already finalized when it is applied, for example after gap recovery, first publishes the older blocks still waiting, so the stream stays in block order. A reorg of an unpublished block emits nothing downstream. Use processor status for exact intervals and `/v1/network/status` for the independent live/finality lane state. Readiness can be healthy while an older backfill gap is still being filled. ### Undo runs A reorg undoes its reverted blocks newest first, and so does the revert at a restart after a long outage. A consumer’s live stream delivers each reverted block’s `undo` changes as a batch of its own, so several reverted blocks arrive as a run of `undo` batches, newest first, before any replacement block’s `apply` batch. An application that keeps derived aggregates, such as leaderboards or rolling totals, beside its per-block state can rebuild them once per run instead of once per batch: 1. Apply each `undo` to the per-block state immediately, record in the same transaction that the aggregates need a rebuild, and acknowledge the batch as usual. 2. Rebuild the aggregates, and clear that record, when the run ends: before the first batch that is not an undo, at a heartbeat, or when the session closes. A run of N reverted blocks then costs one rebuild, not N. Because the record commits with the per-block state, an application that restarts in the middle of a run still finds the rebuild due. # History and live data sources > Understand how Leani selects, verifies, and combines archive, RPC, Beacon API, and native P2P sources. Schema rechecked against the upstream Xatu schema on 2026-08-04. Public availability remains operational evidence and must still be probed before a large backfill. ## 1. Recommended source portfolio | Source | Role | Strength | Main limitation | | --------------------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | EraE archives | Default historical backfill, tried first | Sparse native reader; complete block bodies and receipts with locally checked execution commitments | The mirror and its catalog remain the canonicality trust root; the public mirror stopped at block 24,518,655; receipts only from Byzantium | | Xatu public Parquet | Backfill of blocks eraE lacks, tried second | Free, columnar, about one day behind the chain, filter/project before retaining | Donated hosting; filtered rows are trusted data whose last logs cannot be proven complete; Ethereum mainnet only | | Ethereum P2P/Reth | Recent and live | No vendor key, follows head, bodies/receipts | Old history is thinning; consensus finality remains separate | | Beacon light client/CL | Canonicality/finality | Anchors post-Merge execution payloads | Adds a consensus component | | AWS public blockchain | Secondary Parquet source | Free public S3, daily updates | Validate schema completeness for each fork/use case | | Google Blockchain Analytics | Development/fallback SQL | Broad tables and easy SQL | Hosted dependency; first 1 TB/month free, then paid | | Portal Network | Future decentralized history/state | Designed for low-resource history access | Completeness/maturity must be tested | | JSON-RPC | Development/rare fallback | Broad compatibility | Reintroduces cost/trust; never default core | ### Choosing a history source | Use case | Source order | Why | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | Processors that read logs, such as event and transfer processors | eraE, then Xatu | eraE checks each block’s receipts root, so a missing log cannot pass; Xatu cannot prove that a block’s last logs are complete | | Processors that read only transactions, such as `rollup-txs` and `transaction-stats`, on a slow link | Xatu, then eraE | Xatu downloads only the columns a processor reads, about half of eraE’s bytes per block | | Blocks newer than the eraE mirror, which ends at block 24,518,655 (February 2026) | Xatu | The public eraE mirror publishes no later blocks; Xatu runs about a day behind the chain | | The last day and the chain head | Execution P2P | Neither archive holds these blocks yet | | Blocks no archive serves | Execution P2P, within `sources.live.history_fallback_blocks` | It checks every block against its header, but downloads old history slowly, at 40–55 blocks/s | | Blocks after Glamsterdam | Execution P2P | Xatu’s public tables lose the payload fields Leani needs, and the eraE mirror does not reach them yet | | Checking a range another source built | eraE | It recomputes each block’s transactions and receipts roots | The shipped profiles try eraE, then Xatu, and follow the head over execution P2P. [History source order](/docs/operations/configuration-guide/#history-source-order) explains how a backfill moves between sources, and [Sizing for bandwidth, memory, and disk](/docs/operations/configuration-guide/#sizing-for-bandwidth-memory-and-disk) gives measured speeds. ## 2. Field groups Processors declare the field groups they read. Sources declare the groups they fill. Filters choose rows (which transactions or logs); groups choose columns (which fields of those rows). Each group is a capability flag with the same name. `Senders`, `BlobFields`, `ReceiptOutcome` and `BlockSize` are new flags. Each group promises its fields only on components whose material is present: | Group | Promise | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Header` | `gas_limit` and `gas_used`. The base fee and blob gas fields come with the header but are not checked, because the checks are chain-agnostic. | | `Body` | Every transaction’s `to`, `nonce`, `gas_limit` and `value`, and the header’s `transaction_count`. An absent `to` means a contract creation, never “not supplied”. | | `Calldata` | Every transaction’s `input`. Empty calldata is empty bytes. | | `Senders` | Every transaction’s `from`. | | `BlobFields` | Every type-3 transaction’s blob versioned hashes (non-empty) and blob fee cap, and every type-3 receipt’s blob gas used. | | `ReceiptOutcome` | Every receipt’s `success`, `gas_used` and `effective_gas_price`, with one receipt per transaction (same index and type) when transactions are present. | | `BlockSize` | The header’s `size_bytes` and `transaction_count`. Whoever measures the block holds its body. | | `Logs` | Unchanged: the selected logs, with the log fields the requirement names. | Transaction identity (hash, type and index) and the block reference belong to no group, so they are always present. The bundles keep their earlier meaning: `Transactions` is `Body`, `Calldata`, `Senders`, `BlobFields` and `BlockSize`, and `Receipts` is `Logs` and `ReceiptOutcome`. A source’s plan states the groups its frames fill. The runtime refuses a plan that promises less than the request needs and moves the job to the next configured source. Every frame a processor receives is checked against the fields its declared groups promise, so a source that breaks a promise fails loudly instead of being misread. Stored data is checked when read. Retained raw history and local archives plan from the components they recorded, then check each frame’s fields as they read them. At the first frame that lacks a requested group’s fields, the read ends with a missing-material error for the rest of the chunk. The job then passes that range to the next source: a cache miss, never a job failure or a wrong value. ## 3. Xatu [Xatu Data](https://github.com/ethpandaops/xatu-data) is currently the strongest first backend. Properties: * public Parquet is available without access to EthPandaOps’ private ClickHouse; * canonical execution tables are partitioned in 1,000-block chunks; * canonical beacon tables are generally partitioned daily; * the repository is CC BY 4.0; * files can be queried directly or read through Arrow/ClickHouse/DataFusion. Relevant canonical execution tables include: * blocks and transactions; * receipts-derived logs; * traces; * contracts and address appearances; * balance diffs and reads; * ERC-20 and ERC-721 transfers; * native transfers; * nonce and storage diffs/reads. Relevant canonical beacon tables include: * execution block hash/number/base fee; * blob gas used and excess blob gas; * execution transactions with type, sender, blob gas fee cap, and blob hashes; * blob sidecars; * withdrawals. The implemented `XatuHistorySource` chooses one fail-closed physical projection from the requested groups: 1. blob fields or block sizes (`BlobFields` or `BlockSize`) use the type-3 blobs projection. The request must select type-3 transactions only, and may add only `Header`, `Body`, `Senders` and `ReceiptOutcome`. It reads `Body`’s nonce and value from `canonical_execution_transaction`. `Calldata` is refused there; 2. logs use the log projection, with address and all four topic positions. The logs table names a log’s transaction only by hash, so a log request that also selects transactions by type, sender, or recipient fails during planning; 3. any of `Body`, `Senders`, `Calldata` or `ReceiptOutcome` uses the transaction projection over `canonical_execution_transaction`. It always reads the identity and body columns (`block_number`, `transaction_index`, `transaction_hash`, `to_address`, `nonce`, `gas_limit`, `value`, and `transaction_type`). It adds `from_address` for `Senders` or a sender filter, `input` for `Calldata`, and `gas_used`, `gas_price` and `success` for `ReceiptOutcome`. Those rows become receipts without logs or raw bytes, with the effective gas price taken from `gas_price`. Xatu stores empty calldata as null, which is read as empty bytes; and 4. `Header` alone uses headers, read from the minimal execution and beacon block columns. Xatu serves `rollup-txs` history, which needs `Header`, `Body`, `Senders`, `Calldata` and `ReceiptOutcome`, scoped to the rule recipients. Requests for the full `Transactions` or `Receipts` bundles are refused by Xatu, because its projections cannot fill blob fields for every transaction or logs inside receipts. Such processors use P2P or eraE. Xatu history is Ethereum mainnet only. Its projections stamp every frame as chain 1, so the adapter refuses any Xatu network but `mainnet`, and configuration validation refuses a `xatu` history source on another chain. The current Xatu projections require Beacon execution payloads for parent hashes, so Mainnet requests must begin at block **15,537,394** (the Merge) or later. Earlier ranges fail during planning. A processor deployment/start block before the Merge does not make that range available from this adapter; use a source that explicitly advertises the required historical range. Parquet reads fetch only the byte ranges of the projected columns. Ranges at most 1 MiB apart are merged into one request, which also downloads the bytes between them, while the input budget covers those bytes; no larger gap is downloaded. Every requested byte, footers and merged gaps included, counts against the source budget’s `max_input_bytes` for the whole chunk, and is charged before it is requested. A chunk’s `fetched_bytes` reports those bytes, and its `projected_compressed_bytes` the selected column chunks alone. The reader holds one row group’s selected columns at a time, so an object whose largest such row group exceeds the budget’s `max_resident_bytes` is refused before any column is requested. At most the source budget’s `max_in_flight_requests` requests are in flight per object, and the node sets it from `budgets.source_concurrency`. Every request carries `If-Match` with the ETag the object had at HEAD, so an object rewritten during a read fails the chunk, which is retried against the new version, instead of mixing two versions. `If-Match` never matches a weak ETag, so an object without a strong ETag is refused as a schema drift: a backfill tries its next configured source, and fails when none can serve the range; `leani source probe xatu` reports each object’s ETag and flags such objects. A range response of another length, and a footer whose metadata range exceeds 64 MiB, are refused too. Public hosting can still retry or stall; source availability and throughput need dated operational evidence. Xatu publishes each partition some time after it closes, so the object for the newest blocks may not exist yet, and its reads get 404 Not Found. The source reports such a chunk as a range it does not cover, as eraE reports a range its catalog does not list yet, not as an unavailable source: a job moves on to its next configured source, and the node logs `xatu has not published this object yet` with the object’s location at debug level. Only a 404 is reported this way; other store failures, such as a 5xx response or a timeout, are not. Every immutable chunk encodes its projection kind, its requested groups, and its exact filters. `open` therefore cannot execute a different physical projection from the one that was planned. The source plan reports logical reader, table, selected columns, predicates, trust, derived status, and completeness. Unsupported combinations fail before object metadata or Parquet data is downloaded. Transaction aggregates select `canonical_execution_transaction` and read only the columns their groups need, so they read no receipt outcome unless they request `ReceiptOutcome`. Declarative events select `canonical_execution_logs` without beacon transactions, withdrawals, or blob fields. Both read the minimal execution and beacon block columns needed for timestamps, canonical hash, and parent identity. Each projected column has a declared encoding that fixes the Arrow types it may arrive as, and a column of any other type fails the object before a row is decoded, as a schema drift, which a backfill answers with its next configured source. Hashes, addresses, and byte strings are `0x`-prefixed hexadecimal text, or raw bytes only in a fixed-width binary column of exactly their size. The projections check what the dataset declares: * a block’s transaction rows, all of them, before any filter selects some, must hold every transaction the beacon payload counts, with indices `0..count`, so a missing row never passes for one the filter did not match. A block with a missing row is an incomplete range, which a backfill takes to its next configured source, as is a blob transaction the beacon payload lists without its execution row; * a block’s log rows, all of them, must carry log indices from 0 without a gap, and a block of at least eight transactions must have some. Xatu publishes no log count or bloom per block, so a block whose last logs are missing still passes, as does a smaller block missing all of them; * the blobs projection proves every block’s withdrawals, on which its execution size depends, from the global withdrawal index: between two slots with rows the indices must continue exactly, so a slot between them without rows had none. The nearest slots with rows before and after the chunk, in the daily objects it reads, prove its edges the same way, and a slot with 16 rows, the most a payload holds, lacks none. An edge they do not prove, a jump in the indices, such as a partly exported slot, or more than 16 rows in one payload leaves the chunk incomplete. Xatu exports no withdrawals root or count per block to check against; * logs may carry no topics (`LOG0`), whether the export leaves `topic0` null, empty, or NUL-padded; * blob receipts carry no logs, so blob frames declare their logs a partial projection, and a log requirement cannot be met through those receipts. Source-native transfer/trace/state-diff readers remain optional breadth work; their stronger derived-table semantics must be explicit before they can satisfy a processor capability. For blobs.money, combine: ```text canonical_beacon_block canonical_beacon_block_execution_transaction canonical_beacon_block_withdrawal canonical_execution_block canonical_execution_transaction ``` and use canonical execution logs only if rollup attribution later needs emitted events. Trust note: Xatu’s canonical tables are derived by an execution pipeline. Checksums verify downloaded files, not Ethereum execution. Full receipts can be checked against a receipt root; filtered transfer/log/state-diff rows cannot prove their completeness in isolation. Sources: * [Xatu repository and Parquet examples](https://github.com/ethpandaops/xatu-data) * [Canonical beacon schema](https://github.com/ethpandaops/xatu-data/blob/master/schema/canonical_beacon_.md) * [Canonical execution schema](https://github.com/ethpandaops/xatu-data/blob/master/schema/canonical_execution_.md) ## 4. EraE [EthPandaOps history endpoints](https://ethpandaops.io/data/history/) describe EraE as complete execution-layer history from genesis, covering pre- and post-Merge blocks and intended for client bootstrapping/verification. Implemented usage: 1. fetch and validate the checksum catalog; 2. read the e2store version and dynamic block index with bounded range requests; 3. fetch only the selected compressed headers, bodies, and receipts; 4. decode requested components and verify their transaction, receipt, ommer, withdrawal, and logs-bloom commitments; 5. map configured processors; 6. discard archive bytes; 7. checkpoint and continue. This gives bounded memory/network use, needs no temporary archive file, and avoids relying on old execution P2P peers. The reader supports HTTPS mirrors, local `file:` catalogs, and plain `http` mirrors on loopback, or elsewhere only with `allow_insecure_http = true` on the history source. It never follows a redirect. Compressed input windows are independent of the decoded frame buffer: buffering four frames does not limit an HTTP range to four blocks. A chunk starts with up to 32 blocks, then acquires up to 256 blocks per window, targeting at most 16 MiB of compressed input. Adjacent components share a byte-range request, and independent ranges are fetched concurrently up to the source’s in-flight request limit. When the resident budget permits, the next window downloads while the current one is decoded and consumed: * each batch’s byte ranges are checked against the remaining input budget before they are requested; * the current and prefetched compressed windows together must fit the resident budget; smaller budgets disable lookahead or shrink a window, and one large block may exceed the 16 MiB target but never the budget; * a block that exceeds the remaining prefetch room is read sequentially after the current compressed window is released; * the checksum catalog is read with a 16 MiB cap, and each range response with a cap at its requested length, as the bodies stream in; * an e2store record may not declare more bytes than its indexed extent holds; * a block’s compressed header, body, and receipt list are decompressed through one shared cap, the frame budget, at most 32 MiB; * each normalized frame is checked against the frame budget before it is queued, as the other sources check theirs. Snappy decoding, commitment checks, and normalization run on blocking workers so they do not stall the async network workers. Dropping a stream cancels its downloads and signals its decoding worker. Transaction requests use Alloy consensus’s native `secp256k1` recovery backend, retaining its signature validation checks. Header-only requests read only headers. Body requests verify the complete body. Bodies and receipts are read only as the requested groups need them, and senders are recovered only for `Senders` or a sender predicate. Acquired bodies also retain complete withdrawals for execution RPC and raw-history consumers, even without an explicit withdrawal request. Log-only requests can read headers and receipts without transaction bodies when neither transaction hashes nor transaction filters are needed. Receipt roots and logs blooms are still checked over every receipt before log filtering. Requested transaction, receipt, and log predicates are applied before material is built for downstream processors; transaction and log indices retain their original positions. A request with `allow_filtered = false` receives complete requested material. Errors, logs, and frame provenance show the mirror as `scheme://host[:port]/…/`, so credentials in its userinfo, query, or path are not printed. EraE serves headers and bodies from genesis, but receipts and logs only from Byzantium (block 4,370,000). Earlier receipts carry an intermediate state root instead of a status, which frames cannot represent, so a plan that needs receipts or logs before Byzantium fails with an error that says so. The checksum catalog is cached for 10 minutes. A plan that asks past it fetches it again once the catalog is 30 seconds old; each refetch that still lacks the range doubles that interval, up to the 10 minutes, and a refetch that brings new eras starts it over. Each refetch sends the `ETag` and `Last-Modified` the mirror last sent, so an unchanged catalog costs a 304. A chunk already planned opens from the cached catalog whatever its age. Validated dynamic indexes are shared by concurrent chunks and kept in an eight-entry cache keyed by object filename and catalog SHA-256. Compressed archive data is retained only for active streams. The direct mainnet catalog was populated on 2026-08-01 even though the rendered history webpage’s generated counter was stale. A live sparse probe decoded and verified post-Dencun block `19,426,589` from a 967 MB source object, retaining 111,181 bytes of normalized frame data. Catalog continuity, mirror behavior, and sustained throughput still require release evidence. A later conformance run verified all 300 blocks in `19,426,589..=19,426,888` (123,856,697 normalized in-memory bytes) without retaining the 967 MB EraE object. Exporting canonical byte arrays as pretty JSON was intentionally not retained: it expanded past 2 GB. Operational backfills map and commit bounded frames directly; raw JSON is a diagnostic format, not a storage format. The mirror and its checksum catalog are the trust root. The catalog advertises each full object’s SHA-256; sparse mode records it as the object’s version, not as a checksum, because it never downloads the whole object to recompute it. Local commitment checks establish internal Ethereum execution consistency for acquired components: transaction, receipt, ommer, withdrawal, and logs-bloom commitments, and the parent hashes within an object. Checks for omitted bodies or receipts are reported as `NotChecked`. The block hash is computed from the archived header itself, so frames report `header_hash` as not checked, and `withdrawals_root` as unavailable before Shanghai. Canonicality follows the mirror’s catalog until it is anchored through independently verified consensus material. ### EraE versus Xatu The shipped profiles list eraE first and Xatu second. A backfill tries them in that order for each gap and passes a range the first cannot serve to the next (see [History source order](/docs/operations/configuration-guide/#history-source-order)). EraE comes first because each block it serves is complete by construction: the reader recomputes the transactions root and the receipts root from the bodies and receipts it reads and compares them with the block’s header. A missing transaction or log cannot pass. A new event processor also needs no new public table, and the format is the one execution clients import history from. Xatu comes second, and serves what eraE cannot: * the public eraE mirror publishes no blocks after its last catalog update, so Xatu, about one day behind the chain, serves later history; * it is columnar, so a narrow projection, such as a transaction processor that reads a few columns over a long range, transfers fewer bytes per block than eraE, which reads whole bodies and receipts; * it does not publish a log count or bloom per block, nor the transaction root of a filtered selection. Leani counts every transaction row of a block against the beacon payload before filtering and requires log rows without a gap, so a missing transaction or an inner log is caught; a block whose last logs are missing is not. A block of at least eight transactions without any log row counts as missing its logs. Every check that fails passes the range to the next source. On the host measured in [Sizing for bandwidth, memory, and disk](/docs/operations/configuration-guide/#sizing-for-bandwidth-memory-and-disk), with the shipped settings, Xatu ran `rollup-txs` about twice as fast as eraE and `blobs-money` about one and a half times as fast; eraE downloaded about twice the bytes per block and used about three times the CPU, for its root checks. Which is faster on another host depends on the processor’s projection and on the host’s bandwidth. Neither source anchors a block’s hash to consensus: eraE trusts its mirror’s checksum catalog, and Xatu its publisher. EraE files may carry header proofs against the beacon chain’s historical summaries; Leani does not verify them yet. EraE contains no EVM call traces or state diffs. After Glamsterdam, Xatu’s public beacon tables no longer carry execution payload fields, on which Leani’s Xatu projections depend for transaction counts and parent hashes; the Xatu source needs an update before blocks after the fork can be served from it. ## 5. Ethereum P2P, Reth, and SHiNode [SHiNode](https://github.com/vicnaum/shinode) demonstrates that Reth/devp2p can retrieve headers, transaction bodies, receipts, and logs without EVM state and follow the head. SHiNode context from the research snapshot: * Rust, MIT/Apache-2.0; * current documented release v0.3.0; * claims over 1,000 blocks/second and roughly six hours for full Mainnet on a modern SSD; * stores roughly 250 GB in events-only mode or under 800 GB with transactions; * implements `eth_chainId`, `eth_blockNumber`, limited `eth_getBlockByNumber`, and `eth_getLogs`; * its own transaction receipt RPC and WebSocket subscriptions were planned, not implemented at the snapshot date; * calldata is not currently retained; * receipt verification and consensus-layer integration are planned; * its README explicitly warns against production use; * it already recognizes that EIP-4444 makes pure P2P history unreliable and lists archive/Portal support as future work. Leani should assess SHiNode as a source implementation, not inherit its storage policy. Our P2P path should map and discard old raw material rather than sealing permanent event shards. SHiNode is active enough to be valuable upstream work, but its small maintainer base and production warning mean the fork must be independently tested and audited. The preferred implementation boundary has changed since SHiNode’s original design. [Reth 2.2](https://github.com/paradigmxyz/reth/releases) exposes a `ReceiptsClient` trait and P2P receipt downloading. Reth also documents individually usable crates, including networking APIs, eth-wire messages, and downloaders in its [repository layout](https://github.com/paradigmxyz/reth/blob/main/docs/repo/layout.md). Therefore: 1. build `source-p2p` directly on the smallest upstream Reth crate set; 2. keep Reth types and version churn inside that adapter; 3. use SHiNode as a reference for scheduling, peer behavior, and failure tests; 4. contribute a missing general hook upstream where practical; 5. use a narrow isolated fork only if the boundary spike proves it necessary. This avoids adopting SHiNode’s permanent event-shard store, incomplete RPC surface, and release cadence while preserving the important stateless-history idea. The independent node now implements exact receipt/block RPC reconstruction, post-commit WebSocket subscriptions, and a bounded recent raw-material window; the bullets above describe SHiNode, not this repository. ## 6. P2P history expiry [EIP-4444](https://eips.ethereum.org/EIPS/eip-4444) specifies that clients should not serve block bodies and receipts older than one year over execution P2P and may prune them. The [Ethereum Foundation’s partial history expiry announcement](https://blog.ethereum.org/2025/07/08/partial-history-exp) states that execution clients support removing pre-Merge history. Consequences: * P2P is the best live/recent input, not the only historical source. * A full-genesis P2P sync may work opportunistically but is not a dependable product contract. * Archive files, public datasets, Portal, and independent mirrors must be interchangeable. Leani therefore places its finalized execution-P2P history adapter after retained history, Xatu, EraE, and configured archives. By default it may plan every gap at or after the processor’s configured start block. An explicitly configured `history_fallback_blocks` value limits that eligibility to a suffix ending at the finalized anchor. It verifies the complete header ancestry to that anchor, checks body and receipt commitments, streams only processor-requested normalized fields, and resumes material acquisition from durable processor coverage. Live requests share the manager but have dispatch priority over queued history requests. The requested field groups also narrow what history acquires. A history request that needs no receipt group (`Receipts`, `ReceiptOutcome` or `Logs`) and no withdrawals downloads block bodies only. Senders are recovered only for `Senders` or a sender filter. Live lanes are unchanged. An on-demand job keeps the sources it starts with. A job whose range ends in the eligible blocks therefore does not start until the node’s finalized P2P anchor covers its last block, typically seconds after the node starts; without `history_fallback_blocks`, that is every job. It waits two minutes at most: a node without such an anchor by then, for example while verified finality is unavailable, starts the job without the P2P fallback, where a recent range fails visibly. A job that ends before the eligible suffix starts at once. This is a resilient fallback, not a cheaper archive transport. A recent receipt-complete block can carry hundreds of kilobytes before devp2p Snappy, and public peers may be full, slow, or missing old history. Keep the persistent node identity and scored peer cache across restarts; use archive sources for predictable large backfills and P2P when those sources lag or fail. ## 7. Portal Network The [Portal Network documentation](https://ethereum.org/developers/docs/networking-layer/portal-network/) targets lightweight access to: * beacon light-client data; * headers; * block bodies; * receipts; * eventually state and transaction lookup. It is a natural future backend because its goal aligns with this project. It should be introduced after measuring: * content availability for random historical blocks; * retrieval throughput for sequential backfills; * proof/canonicality semantics; * operational reliability; * Rust client integration. Portal is not the MVP’s only source. ## 8. AWS public blockchain data [AWS Public Blockchain Data](https://registry.opendata.aws/aws-public-blockchain/) provides free, date-partitioned Parquet in public S3 and states that new data is delivered daily. Use it as: * an independent comparison source; * a fallback for conventional block/transaction/log/trace data; * a potential secondary mirror. Before using it for a processor, inspect a current partition rather than assuming its schema has every recent fork field. Earlier inspection for this project found blob-specific coverage less complete than Xatu. ## 9. Google Blockchain Analytics and Ethereum ETL [Google Blockchain Analytics](https://docs.cloud.google.com/blockchain-analytics/docs) exposes Ethereum blocks, transactions, receipts, logs, decoded events, token transfers, traces, and account-state-oriented tables in BigQuery. [Pricing](https://cloud.google.com/blockchain-analytics/pricing) follows BigQuery and includes the first 1 TB of processed query data per month. It is useful for: * SQL validation; * one-off research; * filling a missing backend capability; * cross-source correctness tests. It is not the fully free/self-sovereign production core. [Ethereum ETL](https://github.com/blockchain-etl/ethereum-etl) is extraction/transformation tooling, not itself a free source of Ethereum history. Its schema and decoders are useful references, but running it traditionally still requires a node or RPC. The project now documents blob header and transaction fields, so its schemas can help normalize alternative datasets. ## 10. Firehose, Substreams, and subgraphs [Firehose data flow](https://firehose.streamingfast.io/architecture/data-flow) accepts protobuf blocks from a reader process and serves a cursor-based historical/live stream. [Firehose storage](https://firehose.streamingfast.io/firehose/architecture/data-storage) normally retains one-block and merged 100-block files. Substreams adds parallel Wasm transformation and module-output caches. Relevance: * useful protobuf/cursor conventions; * potential ecosystem compatibility; * a SHiNode/EraE adapter can theoretically emit `FIRE BLOCK`; * event-only processors need only headers/transactions/receipts/logs. Conflict: * normal Firehose durability keeps raw chain data; * Ethereum Firehose blocks often include call traces and state changes unavailable from receipt-only sources; * running all Firehose/Substreams services is operationally heavier than the desired default. The project should implement a compatibility experiment after the core normalized stream works, not start by deploying a full Firehose stack. Traditional subgraphs are another possible processor/output compatibility layer. Event handlers and topic filters fit the receipt-only core. Subgraphs that use contract calls do not: [The Graph’s documentation](https://thegraph.com/docs/en/subgraphs/developing/creating/advanced/) shows that mappings can declare `eth_call`, which requires EVM state at the indexed block. A future adapter could emit entity/database changes or feed an event-only Graph mapping, but “subgraph compatible” must carry the same capability restrictions as Substreams. ## 11. Source-selection policy For each requested range: 1. filter sources by required capabilities; 2. reject sources whose lag/finality violates the processor policy; 3. prefer projection/predicate pushdown for specialized backfills; 4. prefer root-verifiable full data when trust policy requires it; 5. minimize expected downloaded bytes, then latency; 6. schedule an overlapping source at transitions; 7. record source and verification per committed range; 8. fail over at chunk boundaries when possible. Live execution data and finality are planned separately. A Reth P2P peer can supply a valid body and receipts for a branch; it does not by itself prove the post-Merge canonical/finalized branch. The initial finality adapter consumes verified beacon light-client updates through an untrusted Beacon API endpoint. The self-contained profile instead discovers consensus peers directly and uses the standard light-client bootstrap/update/finality req/resp protocols. Both transports share the same local proof verifier and explicit weak-subjectivity checkpoint boundary. A raw-history job retries a range that only lagging sources lack, those with a non-zero expected lag, such as an EraE catalog that has not caught up. It keeps their reasons in the job’s `last_error` while it waits, until it commits a segment, and fails once it has gone without progress for four times the longest expected lag, and at least 24 hours. That time counts from the newest segment the job retained, or else from its creation, both kept in the catalog, so restarting the node does not start it again; a new job under the same ID counts from its own creation. A transport failure, such as an unreachable mirror, also keeps the job waiting, for at most 72 hours without progress, or the lag bound when that is longer; then the job fails with the reasons in `last_error`. Those hours include time the node was down or the job paused at its storage limit, so either bound fails a job only once the running node has itself seen the range fail for an hour without committing a segment; after a restart the sources get that hour again, and frames a source delivers without a commit do not restart it. A source whose advertised range does not cover the range, such as a local archive of older blocks, is passed over. A range missing inside the advertised range of a source without expected lag, such as a gap in a local archive, fails the job at once, and so does a terminal error, such as a schema drift, from any source that could serve it, which another source’s lag no longer hides. Retained raw history is itself a source. The node verifies a segment’s whole-file BLAKE3 checksum once per process, when it writes the segment or when it first reads it after a restart, and each block read checks its own record’s checksum against the segment’s seek directory; segment reads never run on the async runtime’s threads. A segment that fails either check leaves the catalog, so its blocks read as missing and another source serves them; its file moves to `raw-history/quarantine/`. Each raw-history job that owned it records why in `last_error`, and acquires the blocks again: a running job next, and a completed one once it is back in the queue. A segment whose file is missing when the node starts leaves the catalog the same way, unless most files are missing, which the node takes as a segment directory not yet available and keeps their rows; a file that disappears while the node runs fails its reads, which fall back to other sources, until the file returns. A job adopting retained segments skips one whose file is gone, and removes its row once nothing owns it. Jobs whose ranges overlap share segments: before each acquisition a job adopts the compatible segments other jobs retained, and a job that publishes a segment another job has just published adopts that one, once it verifies, instead of failing. An operator must be able to choose: * `fast/trusted`: Xatu filtered tables; * `verify/full`: EraE or full P2P receipts; * `independent`: multiple sources and sampled/full cross-checking. # Durable changes and acknowledgements > Understand named consumers, opaque cursors, leases, pruning fences, snapshot handoffs, redelivery, and reset-required recovery. Leani delivery is deliberately at-least-once. Exactly-once application effects come from an application transaction, not from pretending an HTTP response and a database commit are atomic. Delivery sequence. History commits coverage through block 26,088,923. Live block #26,088,924 arrives, and its parent hash 0xee28…2bfb30 equals the hash of block 26,088,923, so the handoff is exact. Its header chain leads to an attested head and the core commits the branch. Leani publishes change seq 283 to your database, which applies it idempotently, persists the cursor and sequence, and commits in one transaction before acknowledging. The acknowledgement allows pruning up to seq 283. Real blocks, hashes, and sequences from a recorded leani 0.1.0-rc.1 Mainnet session · interval and timing illustrative ```text Leani change ─> begin destination transaction ├─ apply idempotently ├─ persist opaque cursor + sequence └─ commit ─> acknowledge Leani ``` If the acknowledgement response is lost, the same change may arrive again. The destination’s sequence key makes the mutation idempotent, and its committed cursor proves what it already applied. Leani’s acknowledgement separately tells the node what delivery history a required consumer still fences from pruning. ## Leases and reset requirements A durable consumer has a stable ID, a role, an acknowledgement, and a finite lease. One created through the consumer API also has a credential, which its delivery calls carry, and a backfill subscription’s consumer can have one; one declared in the node configuration has none. An acknowledgement cannot pass what the node delivered to that consumer. Active SDK iterators renew their lease. If a best-effort consumer falls behind retained history, the stream emits `reset_required` with coverage hints rather than skipping invisibly. ## Query then follow A consistent snapshot is held across pagination and returns the boundary cursor needed to continue with changes after the snapshot. Release the snapshot when finished. This closes the race between “last row queried” and “first live event received” without freezing ingestion. See [Consume changes durably](/docs/guides/durable-consumer/) or the complete [PostgreSQL adapter](/docs/guides/postgres-consumer/). # How Leani works > Follow one block from a capable Ethereum source through validation, deterministic processing, durable products, and application delivery. Leani is an application-specific Ethereum indexing node. It does not synchronize Ethereum’s execution state and then ask what your application needs. It declares the processor’s material requirements first, obtains only sources capable of satisfying them, and retains only the products selected by lifecycle policy. 1. [01 acquirehistory and live head, one exact handoff](/docs/concepts/data-sources/) 2. [02 validatecommitments, canonicality, provenance](/docs/concepts/coverage-finality-and-reorgs/) 3. [03 reducedeterministic processors, ordered state](/docs/reference/processor-contracts/) 4. [04 deliverquery, stream, acknowledge, prune](/docs/concepts/durable-delivery/)  Acquire by capability A planner chooses a retained, archive, RPC, or P2P source only when it declares all material required for that request. Validate before publish Hashes, commitments, ancestry, canonicality, and finality provenance are checked at the appropriate trust boundary. Reduce deterministically Block-local work can run concurrently. Stateful deltas reduce in canonical order with undo and checkpoint behavior defined by the processor. Retain by policy State, output, artifacts, delivery, checkpoints, undo, and raw history have separate policies and budgets. The intended storage shape is: ```text disk = application products + processor state and checkpoints + a bounded recent/reorg window + one bounded transient input chunk + optional indexes explicitly enabled by the operator ``` State-minimized, not stateless A token balance, latest pool price, or running aggregate is application state and can be durable. Leani avoids retaining the EVM state trie and unrelated chain history; it does not pretend useful indexes require no state. Continue with [coverage and finality](/docs/concepts/coverage-finality-and-reorgs/) and [storage lifecycles](/docs/concepts/storage-products-and-lifecycles/). # Storage products and lifecycles > Choose independent retention policies for processor state, query output, artifacts, delivery, checkpoints, undo, and raw history. “Retention” is not one switch. Each product answers a different recovery or query need and has its own budget. | Product | What it is for | Typical policy | | ----------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | State | Mutable reducer state needed for future blocks | durable or checkpointed | | Output | Entities served by generic or processor-owned queries | latest, window, full, or none | | Artifacts | Compact immutable results reusable for replay/export | bounded retention or none | | Delivery | Ordered change history for external consumers | best effort, until acknowledged, or none | | Checkpoints | Recovery points for processor state | automatic, at most once per 1,000 finalized blocks and when a historical job completes, with a finite keep count | | Undo | Unfinalized branch replacement | every unfinalized block, plus a safety depth past finality | | Raw history | Reusable normalized source material | explicitly enabled, segmented, and bounded | The policies are orthogonal. An externalized processor can retain durable state and an acknowledgement-fenced change log while keeping no query output. A latest-value processor can retain one materialized entity with no delivery log. ## Hard ceilings and backpressure Process memory, temporary material, mapped deltas, SQLite size, artifacts, and delivery history have operator ceilings. A limit action is part of the contract: pause work, expire a best-effort consumer with reset required, or reject growth. Leani must not silently discard required delivery history to remain “healthy.” Start from intent, not disk size Choose what must survive a restart, what must be queryable, and what an external sink already owns. Then select a checked profile and size its budgets. See [Choose an operating profile](/docs/guides/choose-operating-profile/). # Choose your first consumer > Choose a runnable TypeScript snapshot or PostgreSQL destination example. Start with one complete example, including its matching node configuration: * **Keep a price map in memory:** [query once, then follow](/docs/guides/query-then-follow/). It reads a stable snapshot, applies subsequent changes, and handles a reset. * **Keep application data across restarts:** [consume into PostgreSQL](/docs/guides/postgres-consumer/). It creates the destination tables, commits changes and their cursor together, then acknowledges Leani. The guide includes a restart check. The historical `windowed.toml` quickstart retains query results with delivery disabled. Use the configuration supplied by the consumer example; attaching a consumer to that query-only node is unsupported. A durable consumer is a named destination with a cursor and a lease. Its acknowledgement tells Leani what the destination has committed. For the full protocol and a reusable destination interface, see [consume changes durably](/docs/guides/durable-consumer/). # Prove the release offline > Exercise Leani's runtime, processor, delivery, coverage, checkpoint, and SQLite invariants without credentials or network access. The fixture is the fastest honest proof that an installed binary can process and verify data. It uses deterministic blocks but runs the production storage, processor, change-delivery, coverage, checkpoint, and integrity paths. Offline fixture proof ```shell LEANI_FIXTURE_DIR="$(mktemp -d)" leani e2e fixture \ --data-dir "$LEANI_FIXTURE_DIR" \ --blocks 128 \ --report "$LEANI_FIXTURE_DIR/report.json" ``` Exercise the installed binary without an Ethereum provider.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/fixture-quickstart/snippets.sh) 1. Use a new data directory; the fixture refuses to overwrite an existing one. 2. Inspect the JSON report for `status: "passed"`, continuous coverage, the expected change count, and `databaseVerified: true`. 3. Keep the report with the version and checksum you tested. 4. When using this diagnostic path, move to a [bounded Mainnet range](/docs/getting-started/bounded-mainnet/) after the local proof passes. Fixture evidence is not Mainnet evidence This proves packaging and deterministic runtime behavior. It does not prove upstream availability, peer quality, finality freshness, or real-source throughput. # Offline and container diagnostics > Diagnose an installation offline and exercise the local container image. Use this page when you need to distinguish an installation problem from an upstream network problem. For your first run, start with [Install Leani](/docs/getting-started/install/) and [follow a block](/docs/getting-started/). ## Native The [offline fixture](/docs/getting-started/offline-proof/) uses deterministic blocks with the production runtime and SQLite store. It reports 128 changes, continuous coverage, and database integrity without requiring a network. ## Docker ```bash docker build --tag leani:local . docker volume create leani-quickstart docker run --rm \ --volume leani-quickstart:/var/lib/leani \ leani:local \ e2e fixture \ --data-dir /var/lib/leani/fixture \ --blocks 128 \ --report /var/lib/leani/fixture/report.json ``` The runtime image runs as UID/GID 10001 and the named volume remains available for inspecting `report.json` and `leani.sqlite`. Remove it explicitly when finished: ```bash docker volume rm leani-quickstart ``` The image also contains all six lifecycle examples: ```bash docker run --rm leani:local \ doctor --config /etc/leani/examples/externalized.toml --json ``` The image’s default command serves `/etc/leani/node.toml`, which protects the API with a bearer token from `LEANI_API_TOKEN` and refuses to start without one of at least 16 printable ASCII characters without spaces: ```bash export LEANI_API_TOKEN="$(openssl rand -hex 32)" docker run --rm --env LEANI_API_TOKEN \ --publish 127.0.0.1:8080:8080 \ --volume leani-data:/var/lib/leani \ leani:local ``` Send it as `Authorization: Bearer $LEANI_API_TOKEN` to every route except the operational ones: `/health/*`, `/metrics`, `/v1/network/status`, and `/debug/network`. See the [runbook](/docs/operations/runbook/) for the rest of the container profile. ## Bounded public event example After the offline check passes, follow [Index a bounded Mainnet range](/docs/getting-started/bounded-mainnet/). It includes a matching source configuration, decoded event queries, and coverage checks. ## Durable external consumer For a complete destination setup, use [Consume changes into PostgreSQL](/docs/guides/postgres-consumer/). The guide specifies its node profile, database, registration secret, and restart check. # Use Leani with AI agents > Install the Leani agent skill, point agents at the llms.txt docs, and script the CLI safely. Coding agents such as Claude Code and Codex can run Leani for you. Give them the Leani skill, which teaches the commands, the trust and finality rules, and how to configure processors, and point them at the machine-readable docs. ## Install the skill One command installs it for Codex, Claude Code, Cursor, and other agents that read [Agent Skills](https://agentskills.io): ```bash npx skills add smart-byte/leani -g ``` `-g` installs it for your user, so every project sees it; leave it out to install into the current project only. The command installs the skill from `main`. To match an older `leani` binary, install from its release tag instead, such as `npx skills add https://github.com/smart-byte/leani/tree/v0.1.0-rc.6/skills/leani -g`. Agents with plugin marketplaces can install it as a plugin instead. In Codex: ```bash codex plugin marketplace add smart-byte/leani codex plugin add leani@leani ``` In Claude Code: ```text /plugin marketplace add smart-byte/leani /plugin install leani@leani ``` The `leani` binary carries the same guide for its own release: `leani guide` prints the overview and `leani guide processors` the processor reference. An agent without the skill can read it there, and one with a newer skill than the binary should prefer it. Start a new agent session after installing. Then name Leani in the request, such as “use leani to count the distinct addresses that deposited into Lido in the last 7 days”. The name alone is ambiguous on the web, so without the skill, write `leani.dev` to give the agent a site to read. ## Questions become processors The skill teaches agents to answer an Ethereum data question in this order: 1. Use a built-in processor when one produces the data, such as `evm-events` for decoded contract events. 2. Otherwise write a [custom processor](/docs/guides/custom-processor/). This is the normal way to extend Leani: it is needed for aggregates such as distinct counts or sums, for transaction values rather than logs, and for state across contracts. 3. Backfill the range, check coverage, and query the result. ## Give agents the docs The site publishes its documentation as plain text for language models: * [`/llms.txt`](https://leani.dev/llms.txt): a short briefing and an index of the files below. * One file per section, such as [`/_llms-txt/reference.txt`](https://leani.dev/_llms-txt/reference.txt). * [`/llms-full.txt`](https://leani.dev/llms-full.txt): everything in one file. ## What agents should know about the CLI The skill covers these rules; they matter for any script too: * `leani subscribe --json` prints NDJSON; progress and logs go to stderr. * `subscribe` follows until interrupted. Use `--once --timeout 5m` for one result: `--once` returns the first row that meets `--finality` (default `unattested`; see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes)). * Checkpoint trust needs a person’s approval. Without a terminal, `init` and `subscribe` stop until you rerun them with `--yes`; an agent should show you the checkpoint first. * Exit codes tell outcomes apart without parsing stderr; see the [command-line reference](/docs/reference/cli/#exit-codes). # Backfill while following live > Run independent historical and live lanes, fill the exact committed gap, and hand off without overlap or omission. Leani does not make a large historical scan block head following. The live lane starts independently, commits canonical head work, and can recover a short live gap. Historical jobs fill missing committed intervals for each processor.  1. Run `doctor` and inspect which historical sources satisfy the processor’s declared material shape. 2. Start `serve` with live execution and finality configured. Live readiness is independent of older processor coverage. 3. For `history_control = "node_owned"`, let the node schedule missing ranges. For `application_subscriptions`, create bounded backfill subscriptions from the application and consume their independent history lanes. Request the ranges the live stream announces in `system.live_gap.put` changes too, those the live lane does not fill itself; see [Keep your application in sync](/docs/guides/keep-your-application-in-sync/). 4. Observe exact coverage intervals, not only `processedThrough`. 5. Accept the handoff only when parent hash, canonical ancestry, and processor coverage meet at the boundary. ## The default gap policy If `sources.live.history_fallback_blocks` is omitted, P2P fallback is eligible to fill every history gap at or after the processor start block. Retained or archive sources are selected first when capable. Configure a numeric limit only when you intentionally want to restrict P2P recovery to a finalized suffix. A failed job is terminal until acted upon Terminal failed backfill jobs are not retried forever. Inspect the recorded error and correct its cause. Then retry a failed subscription with `POST /admin/v1/backfill-subscriptions/{id}/retry`, which resumes from its committed progress, or delete the job and create it again. Completion is published only after all overlapping ownership and coverage invariants are satisfied. For application-owned work, each history lane has its own lease and acknowledgement. A live acknowledgement never advances a history lane, and one history lane reconnects without resetting its siblings. ## Run a history subscription in one call `runBackfill()` from `@smart-byte/leani-sdk/backfill` takes one application-owned subscription through its lifecycle: it creates the subscription, applies and acknowledges each history batch, acknowledges the completion, and deletes the subscription. It resolves once the subscription has ended and been deleted, so run it beside the live lane rather than before it, and catch its rejection: an abort or a failure keeps the subscription, so the same request can run again once the cause is fixed. ```ts import { createBackfillSubscriptionClient, runBackfill, } from "@smart-byte/leani-sdk/backfill"; const leani = createBackfillSubscriptionClient({ baseUrl: "http://127.0.0.1:18080" }); const result = await runBackfill( leani, { processor: "blobs-production", fromBlock: 19_426_589, toBlock: 19_436_588, idempotencyKey: "blobs-api:19426589-19436588", consumer: { id: "blobs-api", role: "required", leaseTtlSeconds: 300 }, }, { // Your destination's write. It must be idempotent: a batch applied but // not acknowledged is delivered again. onBatch: (batch) => saveChanges(batch.events), }, ); console.log(`history ${result.outcome}`); ``` The idempotency key makes a rerun after a crash resume the same subscription after its last acknowledged batch. The request must be identical, consumer lease and credential included: the node refuses a different request under the same key. After a lost connection, it backs off and opens the stream again, unless the subscription ended meanwhile. A failed subscription is retried from its committed progress up to `maxRetries` times, then its error is thrown, leaving it failed; pass `onFailure: "discard"` to delete it instead, or `"throw"` to keep it failed and handle the error yourself. # blobs.money integration case study > See how a real application splits historical indexing, live following, query serving, and fallback responsibilities. The blobs domain is the first processor, correctness corpus, and performance benchmark for Leani. blobs.money now contains a native sidecar consumer, application-owned PostgreSQL gap reconciliation, a local compatibility RPC path, and indexed WETH/USDC price consumption. This is pre-launch integration, not a completed production cutover: shadow parity, failure recovery, staging, and rollback evidence remain required before removing the managed-provider fallback. ## 1. Why it is the right first processor The current blobs.money backend needs block headers, full type-3 transaction envelopes, and transaction receipts. Its persistent product data does not require arbitrary contract state, internal traces, or EVM execution. The retained legacy RPC path uses a managed provider for: * live block-number notifications over WebSocket; * `eth_getBlockByNumber` with transactions; * `eth_getBlockReceipts`; * occasional individual transaction receipt fallback; * current block number and recovery/catch-up; * a separate `eth_config` call for current/future blob parameters. The application keeps that legacy path behind a runtime source toggle while the Leani integration is validated. Its private implementation details are not part of Leani’s public contract. ## 2. Inputs and outputs ### Input mapping | blobs.money input | Candidate historical source | Candidate live source | | ------------------------------ | --------------------------------------------------------- | ---------------------- | | number/hash/parent/timestamp | Xatu beacon/execution block | P2P header + CL | | size, gas used/limit, base fee | Xatu execution/beacon block plus tx sizes and withdrawals | P2P header/body | | blob gas used/excess blob gas | Xatu canonical beacon block | P2P post-Dencun header | | tx hash/from/type/blob hashes | Xatu beacon execution tx | P2P block body | | tx gas used/status/blob gas | Xatu canonical execution tx | P2P receipts | | finalized canonical block | Xatu canonical tables | beacon finality | | blob fork schedule | checked-in exact chain config | same local schedule | Xatu’s `block_total_bytes` is consensus-block size and is retained only as `consensus_size_bytes`. The execution `size` used by blobs.money is derived as the exact fork-aware RLP length from execution header fields, transaction encodings, and withdrawals. Unknown future header formats fail closed. ### Existing useful outputs `blocks`: * number/hash/timestamp/size; * blob count/gas/excess/base fee; * execution base fee/gas used/gas limit/burn; * blob burn and reserve fee; * transaction count; * target/max blobs and transform version. `blob_transactions`: * transaction hash; * block number; * sender; * blob count; * execution, blob, and total burn. The processor stores source-neutral blobs entities in Leani’s embedded output store and exposes them through the v1 API/change stream. A comparison adapter maps those values into the shapes of the existing `blocks` and `blob_transactions` tables for parity testing. It does not write the application database during the core build. The blob base fee and blob burn are the protocol’s: EIP-4844’s fee for the block’s excess blob gas under the fork active at its timestamp. The EIP-7918 reserve price is reported separately as the reserve fee. blobs.money’s current formula instead uses the larger of the two from Fusaka on, so its base fee, blob burn, and transaction burns differ from Leani’s wherever the reserve binds, and the parity report lists those fields until blobs.money adopts the protocol fee. ## 3. Historical processor Implemented flow: 1. Read canonical beacon blocks for Dencun-to-target dates. 2. Read beacon execution transactions, filtered to type 3 or non-empty blob hashes. 3. Read canonical execution transactions for matching hashes to obtain gas used and success. 4. Normalize into `BlobsBlockInput`. 5. Run the existing tested fee formulas or port them exactly to Rust. 6. Apply the node’s embedded blobs block and transaction entities. 7. Store a range checkpoint with source URLs/checksums and processor version. 8. Discard downloaded/streamed Parquet material. No blob contents or sidecars are required for current blobs.money metrics. Sidecars remain an optional processor capability. ## 4. Live processor For each new execution block: 1. receive/observe the candidate header; 2. fetch the full body and receipts over P2P; 3. verify header hash, transaction root, receipt root, and parent; 4. map type-3 transactions; 5. apply the blobs output transaction atomically; 6. publish an unattested update, and mark it attested once the sync committee signs it; 7. retain raw material and undo through the reorg window; 8. mark output finalized when the consensus source finalizes it; 9. prune raw material/undo according to policy. The node replaces the current WebSocket block notification plus subsequent HTTP block/receipt calls with one internally coordinated stream. ## 5. Hot/cold handoff Example startup: 1. **t0:** with target finalized block B, start P2P capture at B − 128 and the Xatu backfill at Dencun. 2. **t1:** when the Xatu mapper and reducer reach B − 128, compare hashes over the overlap. 3. **t2:** consume the buffered P2P deltas through the current head and switch the reducer to live.  Leani is tested independently through its API, SDK, RPC, and exported parity reports. The blobs.money repository owns the separate PostgreSQL transaction, range-reconciliation, deployment, and rollback concerns. [Keep your application in sync](/docs/guides/keep-your-application-in-sync/) walks through that reconciliation loop. There are two integration paths: 1. **Compatibility RPC:** point the current ethers/fetch and WebSocket code at the node’s local JSON-RPC. This proves method parity with minimal application change but still transports and transforms raw blocks in blobs.money. 2. **Native consumer:** consume already-derived atomic blobs block events, commit application rows, and then advance the Leani-owned durable acknowledgement. Replays across that boundary are idempotent. The second path is implemented and is the intended primary integration. The first remains a compatibility, development, and emergency-fallback path until production evidence supports a cutover. blobs.money currently carries a local client; it should adopt the published `@smart-byte/leani-sdk` package once their contracts are identical. ## 6. `eth_config` The compatibility path can use `eth_config` to discover current/future blob schedules. This traffic is tiny compared with blocks and receipts. The node now maintains a versioned Mainnet schedule with exact activation timestamps and fork hashes, and serves the final EIP-7910 response locally. Future changes must be reconciled against protocol announcements and client chain specs before release. The native blobs processor carries the checked schedule in its versioned contract. ## 7. Reference validation and production cutover Before production cutover, compare Leani output with a read-only export of the existing pipeline: * compare every `blocks` field; * compare every blob transaction and sender; * investigate current rows with missing/fallback receipts; * compare rollup attribution; * verify no gap at restart and source handoff; * inject reorg fixtures; * record source lag and finality. Use these cutover stages: 1. shadow writes to comparison tables; 2. report differences without serving new data; 3. make leani the historical backfill source; 4. validate the local JSON-RPC compatibility path; 5. run the native SDK stream into shadow tables; 6. make the SDK stream the live primary; add an application-owned PostgreSQL adapter only if the integration proves it necessary; 7. disable Alchemy block/receipt/head usage; 8. keep emergency fallback explicit, metered, and off by default; 9. remove `full_blocks` only in a separately approved recoverable migration. ## 8. Acceptance criteria * Full Dencun-to-head backfill without paid RPC. * Exact parity with the current formulas and schema over the agreed range, except the reserve-clamped blob fee and burns described in section 2. * No permanent raw block/receipt storage. * Live head lag normally within two blocks. * Finalized cursor exposed separately. * Restart, source outage, and shallow reorg tested. * Native API, SDK, JSON-RPC, and blobs.money integration fixtures pass. * Generic Bun SDK consumer can resume from a cursor and handle apply/undo/finality. Production integration has separate acceptance criteria: * production shadow parity; * rollback tested; * Alchemy CU attributable to block, receipt, and head monitoring reduced to zero in normal operation; * remaining external requests itemized; * any `full_blocks` removal explicitly approved and recoverable. # Choose an operating profile > Select a checked lifecycle profile based on the product your application must retain, query, deliver, or externalize. Start from the ownership boundary, not from a vague “small” or “full” mode. Every checked profile is a complete configuration that `leani doctor` validates. | Profile | Leani owns | Best fit | | ---------------- | ----------------------------------------------------------------- | --------------------------------------------------- | | `head-only` | Current state plus a bounded best-effort live log | Ephemeral head follower that can rebuild | | `aggregate-only` | Reducer state and latest aggregate | Counters or totals without retained events | | `windowed` | State plus a block/row/age/byte-bounded query window | Recent event and dashboard queries | | `externalized` | State plus acknowledgement-fenced delivery until the sink commits | PostgreSQL or another application-owned store | | `job-result` | One bounded result and delivery contract for a finite job | Reports and batch outcomes | | `full` | Complete configured output collection | Small, intentionally permanent application datasets | The `windowed` profile keeps one entity per decoded event, without `key_fields`. That keeps its `evm-events` processor block-local: on-demand backfills and materialization jobs accept its ranges in any order, and its finalized coverage compacts. Keyed output makes the processor ordered, which refuses materialization jobs and keeps one coverage row per applied block. Copy the closest file from `config/modes` to `./leani.toml`, change its `data_dir`, processor identity, start block, source credentials, and budgets, then run: ```bash leani doctor --json ``` ## Questions that choose the policy * Must processor state survive restart? * Does the application query Leani, or does it own the serving database? * Can a slow consumer be reset, or must it fence pruning? * Is old output reconstructible from a source or compact artifact? * How much unfinalized undo depth matches the publication policy? * Which limit action should pause work instead of discarding data? Profiles are examples, budgets are obligations The checked defaults demonstrate valid combinations; they are not universal production sizing. Measure the processor’s real output and delivery shape, then set explicit memory, temporary disk, SQLite, artifact, and spool ceilings. # Consume changes durably > Persist application state and an opaque Leani cursor atomically, then acknowledge only after commit. The cursor in a Leani change envelope is an opaque resume and acknowledgement token. Store it beside the application state derived from that change. The destination transaction is the durable boundary; the later Leani acknowledgement advances retention and delivery state. 1. Create a stable named consumer once, or inspect the existing consumer after a restart. 2. Poll retained changes under a lease. 3. Apply one change idempotently and store its cursor in the same destination transaction. Key both by the consumer’s stream as well as the sequence: a node whose store is replaced numbers its changes from the start again. 4. Commit the destination transaction. 5. Acknowledge that cursor to Leani. If this request is lost, accept the same change again and rely on idempotent application. Durable SDK consumer ```typescript import { LeaniError, createLeaniClient, isResetRequired, type ChangeEnvelope, type DurableConsumer, } from "@smart-byte/leani-sdk"; export interface DestinationTransaction { /** Must be idempotent because a committed change can be redelivered. */ apply(change: ChangeEnvelope): Promise; /** Store this beside application state in the same transaction. */ storeLeaniCursor(cursor: string, sequence: string): Promise; } export interface Destination { /** * `stream` names the durable consumer in the node's store. Sequences are * unique only within it: a node whose store is replaced numbers its * changes anew, and its consumer is created again. */ transaction( stream: string, work: (transaction: DestinationTransaction) => Promise, ): Promise; } export interface ConsumerOptions { baseUrl: string; processor: string; consumer: string; credential: string; leaseTtlSeconds?: number; signal?: AbortSignal; } export async function consumeDurably( destination: Destination, options: ConsumerOptions, ): Promise { const leani = createLeaniClient({ baseUrl: options.baseUrl }); const consumers = leani.processors.consumers; let consumer: DurableConsumer; try { consumer = await consumers.inspect(options.processor, options.consumer, { signal: options.signal, }); } catch (error) { if (!(error instanceof LeaniError) || error.status !== 404) throw error; // Only an until_acknowledged stream accepts a required consumer that // fences pruning; a window stream prunes by its own limits. consumer = await consumers.create(options.processor, { id: options.consumer, role: "best_effort", start: { position: "earliest_retained" }, leaseTtlSeconds: options.leaseTtlSeconds ?? 60, credential: options.credential, signal: options.signal, }); } // The consumer's creation identifies the store it lives in. const stream = `${consumer.processorInstance}/${consumer.id}@${consumer.createdAtUnixMs}`; while (!options.signal?.aborted) { const page = await consumers.changes( options.processor, options.consumer, options.credential, { limit: 256, signal: options.signal }, ); if (page.data.length === 0) { await consumers.renew( options.processor, options.consumer, options.credential, { signal: options.signal }, ); await new Promise((resolve) => setTimeout(resolve, 500)); continue; } for (const change of page.data) { if (isResetRequired(change)) { throw new Error("retained history no longer covers this consumer; rebuild from a snapshot"); } // Keep the lease alive while a fetched page is being applied. A slow // destination must not rely on the next acknowledgement for renewal. await consumers.renew( options.processor, options.consumer, options.credential, { signal: options.signal }, ); await destination.transaction(stream, async (transaction) => { await transaction.apply(change); await transaction.storeLeaniCursor(change.cursor, change.sequence); }); // Acknowledge only after application state and its cursor commit together. await consumers.acknowledge( options.processor, options.consumer, options.credential, change.cursor, { signal: options.signal }, ); } } } ``` Commit application state and its Leani cursor atomically before acknowledging.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/sdk-subscription/index.ts) Never acknowledge before your commit An early acknowledgement can make Leani eligible to discard history your application has not durably applied. A failed acknowledgement after commit is safe: the event may be delivered again, so application writes must be idempotent. The consumer’s credential goes with every call, and Leani accepts an acknowledgement only for a change it delivered to that consumer. This loop uses the generic consumer routes, which serve processors with `canonical` delivery ordering. A processor with a split live stream (`block_versioned_idempotent`, such as `blobs-money`) fences renewal and acknowledgement by streaming session: read it with the SDK’s `streamLive()` or `subscribe()` from `@smart-byte/leani-sdk/backfill`, and acknowledge each batch’s boundary. # Query a node with JSON-RPC > Use curl or viem with the RPC endpoint of a prepared block node. First prepare a node with `leani init blocks` and `leani serve` as described in [Getting started](/docs/getting-started/). Its JSON-RPC endpoint is `http://127.0.0.1:18545`; its separate native SDK API is on port 18080. ## Check the chain ```bash curl -s http://127.0.0.1:18545 \ -H 'Content-Type: application/json' \ --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' ``` Expect `{"jsonrpc":"2.0","id":1,"result":"0x1"}` for Ethereum Mainnet. In a Bun application, install viem with `bun add viem`, save the following as `watch.ts`, and run `bun watch.ts`: ```ts import { createPublicClient, http } from "viem"; const eth = createPublicClient({ transport: http("http://127.0.0.1:18545"), }); console.log(await eth.getChainId()); // 1 ``` ## Requesting blocks, receipts, or state Available data depends on the node’s configured sources and retained material. The block-summary feed requests headers and bodies without receipts. It also accepts Xatu’s dataset-declared header and transaction projections, checking that all body indices and the payload transaction count agree. Historical logs can come from Xatu with on-demand history enabled. A historical receipts request needs a source that supplies complete receipts; the filtered Xatu event quickstart alone cannot satisfy it and returns `no_viable_historical_source`. The full historical configuration examples use RPC port 8545. Use the `rpc_http` listener shown by `leani doctor --json` for your selected profile. Do not infer the port or supported data from a different tutorial. See the [RPC method matrix](/docs/reference/ethereum-json-rpc/) before moving an existing application. Arbitrary `eth_call`, account-state queries, and execution simulation are outside the EVM-free core. # Keep your application in sync > Reconcile your own copy of a processor's output by diffing coverage on every connection, filling announced live gaps, and writing idempotently. An application that keeps a processor’s output in its own database owns that copy, and reconciling it is application logic. Leani makes the loop short: it tells the application what it covers and which blocks the live lane skipped. This guide applies to processors with `history_control = "application_subscriptions"` and `history_mode = "on_demand"`, whose history only the application requests. A node-owned on-demand processor announces skipped blocks the same way, but its operator requests them as `POST /admin/v1/materialization-jobs`. ## Where blocks go missing * After downtime or a long outage, the live lane resumes at the newest verified finalized anchor. It skips the blocks between the processor’s cursor and that anchor. Automatic history refills them; on-demand history leaves them to the application. With `[processors.live_gap_fill]` (see [Live gap fill](/docs/operations/configuration-guide/#live-gap-fill)) the lane fills a short range itself before it applies the next block, and the live stream delivers those blocks as `live_recovery` batches. It announces only what it did not fill: a longer range, a range history cannot serve yet, and the rest of a fill that failed or ran out of time. * At startup, retained unfinalized blocks are reverted as a reorg only when they are off the finalized chain, or when the node cannot prove that they are on it: without execution peers within a minute, or after an outage longer than about a day. The application receives their `undo` changes, and the blocks become part of the skipped range. After a restart that follows a long outage, and on reorgs, several blocks can be reverted in a row: one `undo` batch per block, newest first. Rebuild derived data once per such run, as [Undo runs](/docs/concepts/coverage-finality-and-reorgs/#undo-runs) describes. * On the application’s side, a crash between its commit and its acknowledgement redelivers a batch, and a failed backfill subscription stops until acted upon. ## The loop 1. **Diff coverage on every live session.** The live stream’s `hello` carries the processor’s `coverage` as the session starts. Holes in `coverage.available`, from the first block the application keeps through `processedThrough`, are blocks the processor never processed. Before its first block, everything from the processor’s start up to the finalized head is history to request. `streamLive()` exposes each session’s `hello` but does not reconnect: after a failure the SDK marks retryable, open a new session with backoff, which checks coverage again. `subscribe()` reconnects on its own and does not expose the `hello`. Keep this diff when the processor fills live gaps: it still finds the holes below the processor’s cursor, which no fill reaches, and the range of a fill that failed. 2. **Keep announced gaps with the batch.** Before the live lane applies a block past its cursor’s successor, the live stream delivers a `finalized` change of kind `system.live_gap.put` with `data: { fromBlock, toBlock, reason: "not_filled" }`, in a batch of its own. On a stream that keeps records until they are acknowledged, it survives reconnects. Store the range together with the batch’s rows, then acknowledge the batch. In the SDK, `isLiveGapChange` narrows the change, and `applyEntityChange` ignores it. A processor that fills live gaps announces only what it did not fill, so keep handling notices. 3. **Request the kept ranges.** `liveGapBackfill(range, request)` from `@smart-byte/leani-sdk/backfill` builds the `fill_missing` subscription with the idempotency key `live-gap:{consumer}:{fromBlock}-{toBlock}`. Creating it again for the same range returns the same subscription; a hole that history has partly filled meanwhile is a new range, so remember what you already requested. Drop a range once its subscription exists. When a request fails, keep the range and try again later: a failing request must never hold the live lane back. 4. **Consume the history lanes** beside the live lane, as [Backfill while following live](/docs/guides/backfill-while-following-live/) describes; each has its own lease and acknowledgement. 5. **Act on failed subscriptions.** A failed subscription stays failed: inspect its `lastError` and correct the cause. Then retry it with `POST /admin/v1/backfill-subscriptions/{id}/retry` (SDK `retry(id)`), which resumes from its committed progress. To abandon it instead, delete it, adding `?discardUnacknowledged=true` (SDK `delete(id, { discardUnacknowledged: true })`) when the consumer will not read the rest. 6. **Write idempotently.** Live, history, and redelivered batches can overlap and arrive out of block order. Write each block’s rows as an upsert keyed by block, so that a block arriving twice, or from history after live, changes nothing. Deduplicate batches by cursor, not by block: block `n` can come as a notice batch and then as its own batch. A `live_recovery` batch can repeat a block that a history subscription delivers, such as one the application requested from its coverage diff while the lane filled the same range. A notice can also be followed by `live_recovery` batches for the blocks it announced, when the lane retries a fill after it held the block that follows, or after a crash or a restart of its live lane between the notice and that block. Keep an application in sync ```typescript import { applyEntityChange, isLiveGapChange, LeaniError, type EntityChangeTarget, type ProcessorCoverage, } from "@smart-byte/leani-sdk"; import { createBackfillSubscriptionClient, liveGapBackfill, type BackfillRange, type BackfillStatus, } from "@smart-byte/leani-sdk/backfill"; /** * Blocks from `from` that the processor lacks, through the last block it * processed. Before its first block, that is everything up to the finalized * head. */ export function coverageHoles(coverage: ProcessorCoverage, from: number): BackfillRange[] { const through = coverage.processedThrough ?? coverage.chainFinalizedHead?.number; if (through === undefined) return []; const holes: BackfillRange[] = []; let next = Math.max(from, coverage.configuredStartBlock); for (const { fromBlock, toBlock } of coverage.available) { if (fromBlock > next) holes.push({ fromBlock: next, toBlock: Math.min(fromBlock - 1, through) }); next = Math.max(next, toBlock + 1); } if (next <= through) holes.push({ fromBlock: next, toBlock: through }); return holes.filter((hole) => hole.fromBlock <= hole.toBlock); } /** An idempotent destination that also keeps the ranges it still has to request. */ export interface SyncedDestination extends EntityChangeTarget { /** Keep a range to request, together with the rows of the batch that announced it. */ rememberGap(range: BackfillRange): Promise; /** The ranges kept and not yet requested. */ gaps(): Promise; /** Drop a range once a subscription delivers it. */ forgetGap(range: BackfillRange): Promise; } export interface SyncOptions { baseUrl: string; processor: string; consumer: string; credential: string; /** The first block the destination keeps. */ fromBlock: number; /** * Consume each new subscription's history lane, beside the live lane: * start `runBackfill()` from `@smart-byte/leani-sdk/backfill` here with * `liveGapBackfill(subscription, …)` and the same processor and consumer, * lease TTL and credential included. Do not await it, which would hold the * live lane back, but catch its rejection. `keepInSync` forgets the range * once this returns, so until a run resolves `completed` the range is the * application's: keep its request and run it again after a rejection or a * restart, once the cause is fixed. An abort or a failure keeps the * subscription. */ onBackfill(subscription: BackfillStatus): void | Promise; signal?: AbortSignal; } /** * Follow a processor's live lane into a destination, and request as history * every block the processor lacks: the holes in the coverage each session * starts with, and the blocks the live lane announces it skipped. */ export async function keepInSync( destination: SyncedDestination, options: SyncOptions, ): Promise { const leani = createBackfillSubscriptionClient({ baseUrl: options.baseUrl }); const requested: BackfillRange[] = []; const request = async () => { for (const range of await destination.gaps()) { const inFlight = requested.some( (earlier) => earlier.fromBlock <= range.fromBlock && range.toBlock <= earlier.toBlock, ); if (!inFlight) { try { await options.onBackfill(await leani.create(liveGapBackfill(range, { processor: options.processor, consumer: { id: options.consumer, role: "required", leaseTtlSeconds: 300, credential: options.credential, }, signal: options.signal, }))); } catch (error) { // Keep the range and try again after the next batch: a request // that fails must not hold the live lane back. console.warn(`request for blocks ${range.fromBlock}-${range.toBlock} failed`, error); return; } requested.push(range); } await destination.forgetGap(range); } }; let delayMs = 1_000; while (!options.signal?.aborted) { try { const session = await leani.streamLive(options.processor, options.consumer, { credential: options.credential, signal: options.signal, }); delayMs = 1_000; try { for (const hole of coverageHoles(session.hello.coverage, options.fromBlock)) { await destination.rememberGap(hole); } await request(); for await (const record of session) { if (record.type !== "batch") continue; for (const change of record.changes) { if (isLiveGapChange(change)) await destination.rememberGap(change.data); else await applyEntityChange(destination, change); } await session.acknowledge(record.acknowledgeableCursor); await request(); } } finally { await session.close(); } } catch (error) { if (options.signal?.aborted) return; // An outage ends only this session: reconnect, which redelivers what // was not acknowledged and checks coverage again. A replaced session // can still hold the lease for a moment. Anything else, such as a // rejected credential, is the caller's. const transient = error instanceof LeaniError && (error.retryable || error.code === "consumer_session_active" || error.code === "consumer_session_lost"); if (!transient) throw error; console.warn(`live session failed; reconnecting in ${delayMs} ms`, error); await new Promise((resolve) => setTimeout(resolve, delayMs)); delayMs = Math.min(delayMs * 2, 30_000); } } } ``` Follow the live lane and request every block the processor lacks as history.[Verified source](https://github.com/smart-byte/leani/blob/v0.1.1/examples/sdk-subscription/live-sync.ts) # Follow Ethereum blocks from the CLI > Print one verified summary for every new Ethereum block from an embedded runtime or a prepared Leani node. The built-in `block-summary` processor is the smallest continuous Leani demo. It requests verified execution headers and bodies, so it can report the exact transaction count without waiting for a particular contract event. It does not download receipts. Run it without configuring or starting a node: ```bash leani subscribe blocks ``` First output can take several minutes while Leani finds serving peers and validates a current block. The newest block is printed once the live lane accepts it, followed by one line per new block: ```text 2026-09-01T09:14:35Z block=25881412 txs=187 gas=32.47M / 60.00M (54.1%) base_fee=0.143592817 gwei blobs=6 unattested ``` The summary contains the block number, hash and parent, timestamp, transaction count, encoded block size, gas limit and usage, base fee, blob gas, excess blob gas, and finality. Leani verifies the transaction and withdrawals roots from the body while keeping receipt acquisition out of this latency-sensitive path. Use newline-delimited JSON for scripts: ```bash leani subscribe blocks --format json | jq '{block: .blockNumber, transactions: .transactionCount, gas: .gasUsed, baseFee: .baseFeePerGas}' ``` Rows say `unattested`, `attested` or `finalized`, and JSON reports the same `finality`; see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes). Use `--finality attested` or `--finality finalized` when the first result must have that level. Attached output follows the node’s configured source trust; a dataset’s finality label does not become a cryptographic proof simply by subscribing to it. Against a node that publishes unattested blocks, an attached `--finality attested` or `--finality finalized` subscription prints each block once the node’s marker for that level covers it. During a reorg, each reverted block prints again with `"operation": "undo"` (an `undo` suffix in the terminal), however late the undo arrives. The undo of a block the subscription saw but did not print, such as an older block of the embedded startup scan, stays hidden. The undo of a block it never saw, because it was applied before the run started, prints as a `leani.subscription-undo.v1` row with its `blockNumber`, `blockHash`, and change `key` while it is fresh. JSON output mixes such rows, and the gap rows described below, with `leani.block-summary.v1` rows, so scripts should dispatch on each row’s `schema`. `--once` exits after the first summary that meets `--finality`, so by default after the first `unattested` row. A cold start can take minutes; bound it with `--timeout`, which exits 124 when nothing matched in time. An undo or gap row never ends `--once`: ```bash leani subscribe blocks --json --once --timeout 5m ``` The default `unattested` level is the fastest first output, which is useful for startup timing: ```bash time leani subscribe blocks --once ``` ## Attach to a prepared node Create the compact block-summary configuration once: ```bash mkdir leani-block-demo cd leani-block-demo leani init blocks ``` The generated product configuration is intentionally small: ```toml config_version = 1 network = "ethereum-mainnet" data_dir = "./data" [finality] checkpoint = "0x..." checkpoint_slot = 15100000 endpoints = [ "https://ethereum-beacon-api.publicnode.com/", "https://lodestar-mainnet.chainsafe.io/", ] [blocks] [api] bind = "127.0.0.1:18080" ``` Start the node in one terminal: ```bash leani serve ``` Then subscribe in another terminal from the same directory: ```bash leani subscribe blocks ``` Auto mode discovers the local config, detects the running API, reads a stable latest-block snapshot, and follows its resumable change stream. If no node is reachable, the same command starts the embedded runtime. Use `--mode client` or `--mode embedded` only when you need to force one behavior. With `--endpoint URL`, client and auto mode attach to that node without reading a local configuration. The subscription prints nothing until the stream’s `hello` shows an Ethereum mainnet node (chain 1) serving `block-summary` under the requested `--processor`, and it checks every reconnect again. A node for another chain or processor, or another service answering at `api.bind`, ends the command with an error instead of printing its stream. A stream error the node marks `retryable` reconnects from the last cursor. `--finality finalized` replays the node’s retained change log, so a block that was not yet final when the subscription connected still prints once finality covers it, however late that is; blocks already final at that point are skipped. The subscription holds at most 1,024 unfinalized blocks. If the node stops reporting finality for longer, the oldest held blocks are dropped. Finalized output then has a hole there, reported as one row for the dropped range right before the next finalized row: ```json {"schema":"leani.subscription-gap.v1","protocol":"blocks","operation":"gap","fromBlock":25881412,"toBlock":25881420,"timestamp":1788254075,"reason":"finality_gate_full"} ``` The terminal prints a `gap` line instead. If the node pruned the changes of blocks that were not yet final before the subscription connected, those blocks are reported the same way with `"reason": "change_log_pruned"`. A gap row never ends `--once`. Reset just the embedded block feed when measuring a true cold start: ```bash leani reset subscription blocks --yes ``` The configured node and embedded subscription use separate state directories, so both can be tested from the same project directory. A subscription reset only removes a directory `leani subscribe` marked as subscription state and no node has taken over since (see [the Uniswap guide](/docs/guides/uniswap-price-subscriptions/#cli)), and `leani reset all` refuses while an embedded subscription is running. # Configure native Ethereum P2P > Run Leani's persistent execution P2P manager and independently anchored consensus finality without a managed execution RPC. Leani embeds an execution P2P manager for headers, bodies, receipts, discovery, and live subscriptions. Consensus finality is a separate trust boundary. A healthy execution peer set cannot assert finalized Ethereum ancestry by itself. 1. Start from `config/example.toml` and replace the weak-subjectivity checkpoint and finalized beacon slot with a recent value obtained independently. 2. Give the node a durable data directory. Its execution node key and bounded peer cache survive restarts. Warm startup immediately hedges a small set of recently proven body servers against fresh DNS and Discv discovery, then progressively admits the broader cache. 3. Expose the configured discovery/listener ports and allow outbound execution and consensus traffic. 4. Run `doctor --json` before `serve`. 5. Inspect `/v1/network/status`, `/debug/network`, and Prometheus metrics for execution peers, live subscription usability, consensus peers, finality freshness, and history recovery separately. ## One manager, two priorities Live and historical work share one process-wide manager and admission gate. Live requests have priority; bounded history chunks yield rather than replacing the manager or discarding healthy peer reputation. Body and receipt batch sizes adapt independently after partial or failed responses. ## History fallback P2P history is used after capable retained/archive sources. Without an explicit `history_fallback_blocks` value, every gap from the processor start is eligible. This enables a theoretical full P2P sync, subject to peer retention and material availability. Setting the value opts into a bounded finalized suffix. Do not conflate transport with finality Execution material is commitment- and ancestry-checked against the configured consensus anchor. Keep checkpoint provenance, finality freshness, and peer health visible as distinct operational signals. # Live Uniswap prices from the CLI and SDK > Stream verified Uniswap V3 prices from a prepared node or a standalone Leani process. Leani’s `subscribe` command turns the built-in Uniswap V3 observation processor into a small live-price utility. It follows Ethereum directly and prints only matching pool updates to stdout; lifecycle and peer messages go to stderr, so stdout is safe to pipe. ## CLI After installing the binary, subscribe to the ETH/USDC 0.05% Mainnet pool: ```bash leani subscribe uniswap-v3 ETH/USDC ``` That exact command is also the standalone demo. Auto mode first checks for a reachable node from the explicit endpoint or local configuration; when there is none, it starts the embedded runtime. Use `--mode embedded` only when a reachable local node should intentionally be ignored. The first run asks three independent checkpoint providers for a recent finalized weak-subjectivity checkpoint. More than half of them, so at least two of the three, must return the exact same slot and block root, and no responding provider may report another root at that slot. Leani then shows every provider’s answer and asks you to accept the agreed one. Providers must use HTTPS, except on loopback, and their redirects are not followed. Execution then follows native P2P peers, while later Beacon API responses are proof-checked from that agreed root. For an intentional non-interactive invocation, use `--yes`. To reproduce a completely cold embedded start, reset the reconstructible state for the exact protocol, market set, and finality mode, then subscribe again: ```bash leani reset subscription uniswap-v3 ETH/USDC leani subscribe uniswap-v3 ETH/USDC ``` The reset prints the exact hashed subscription directory and asks for confirmation; `--yes` is available for an intentional scripted reset. It removes only that embedded subscription’s checkpoint, peer cache, P2P identity, and SQLite state. It does not touch `leani.toml` or prepared-node state. Stop a matching embedded subscriber before resetting it. For a finalized subscription, pass `--finality finalized` to both commands. The reset only removes a directory that `leani subscribe` marked as its own state with a `.leani-subscription` file: every directory it creates, and a subscription’s default directory from an earlier release once that subscription runs again. A node that takes over a marked directory, such as `leani serve` with that `data_dir`, removes the marker. The reset refuses any other directory, such as a node’s `data_dir` passed as `--data-dir`. An explicit `--data-dir` from an earlier release that already held state is not marked; delete it yourself once you have checked that it holds only subscription state. To test the binary with no retained node or subscription state at all, stop every Leani process that uses the configured data directory and run: ```bash leani reset all ``` This removes the entire resolved runtime data directory, including the node database and WAL, raw history, processor artifacts, checkpoints, execution-peer cache, P2P identity, and every embedded subscription. It preserves `leani.toml` and the binary. The command rejects broad targets that contain the working directory or configuration; use `--yes` only for intentional scripted cold-start tests. It first takes the lock of every embedded subscription under the data directory and refuses, deleting nothing, while one is running. After the light client has verified a newer finality anchor and its consensus-committed execution hash, Leani persists that locally verified anchor. Ordinary restarts reuse it without contacting the checkpoint providers while it remains fresh. The transport can also be fully peer-to-peer after bootstrap: ```bash leani subscribe uniswap-v3 ETH/USDC --finality-source p2p ``` P2P finality still needs an initial trust root. It reuses the local verified anchor when available; otherwise it performs the same checkpoint quorum once, then obtains and verifies light-client updates over consensus P2P instead of a Beacon API. Override the provider set with repeatable `--checkpoint-url` flags (or `LEANI_CHECKPOINT_URLS`) and the threshold with `--checkpoint-quorum`. The default pretty stream looks like this: ```text 2026-08-28T19:22:11Z ETH/USDC 2439.03084279 volume=0.601207569812525017 ETH block=25855798 unattested ``` Volume is the absolute swap amount in the first, or base, token of the market. For example, `ETH/USDC` reports ETH volume while `WBTC/ETH` reports WBTC volume. One embedded runtime can filter several pools at once: ```bash leani subscribe uniswap-v3 ETH/USDC ETH/USDT WBTC/ETH ``` Use the stable `leani.market-price.v2` NDJSON contract for scripts. Its `baseVolume` string uses exact base-token decimal units: ```bash leani subscribe uniswap-v3 ETH/USDC --format json | jq --unbuffered 'select(.operation == "apply") | [.market, .price, .baseVolume] | @tsv' ``` `--format raw` exposes the underlying change envelope, and `--once` exits after one matching update. `--finality finalized` builds an embedded finalized-only processor that prints a price only after Ethereum consensus has finalized its block. The default `unattested` mode emits head updates and marks undo records when a block is undone (see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes)). An undo repeats the row of the observation it reverts, with `"operation": "undo"`, however late it arrives; in raw mode it is the undo change envelope, whose `data` is `null` or the previous value, never the reverted one. An undo of a change the subscription saw but did not print, such as another pool’s, stays hidden. An undo of a change it never saw, because it was applied before the run started, is a `leani.subscription-undo.v1` row naming the reverted `blockNumber`, `blockHash`, and change `key` while it is fresh; attached to a node with more pools, such a row can name another pool’s change. The JSON stream also carries `leani.subscription-gap.v1` rows (see [the block feed](/docs/guides/live-block-subscriptions/#attach-to-a-prepared-node)), so scripts should dispatch on each row’s `schema`. `--once` never exits on an undo or gap row. Embedded mode does not bind API or RPC ports, activate unrelated processors, or start a historical backfill. It starts from an independently verified finalized execution anchor and connects native execution P2P. Output from the configured processor lane says `unattested`, `attested` or `finalized`; use `--finality finalized` to wait for a block Ethereum consensus has finalized. Finalized output still waits for the fully joined path. Its small SQLite store, locally verified checkpoint, and peer cache live under the platform data directory; use `--data-dir PATH` to select an explicit location. The durable lane of a completely cold run can take longer because public peer discovery and the finalized-to-head catch-up are network dependent; subsequent runs reuse verified canonical material and the peer cache. If a local configuration exists, embedded fallback seeds its private peer cache from the configured node context, but it never copies the node’s P2P identity or shares its writable database. Once verified startup is complete, Leani prints the newest still-fresh observation it acquired during that startup—one per requested market—instead of silently discarding those prices. It then continues with new swaps. Consequently, embedded `--once` returns the newest fresh startup price when one exists. When a node is already running from a project-local `leani.toml`, run the same command from another terminal in that directory: ```bash # One-time setup leani init uniswap-v3 ETH/USDC # Terminal 1 leani serve # Terminal 2 leani subscribe uniswap-v3 ETH/USDC ``` Auto mode discovers the reachable `api.bind` in `./leani.toml` and attaches; otherwise it starts the embedded runtime. Use `--mode client --endpoint URL` when attaching to a remote node should be mandatory rather than automatic; with `--endpoint`, neither mode reads a local configuration. The client takes a head cursor, reads the node’s query-only latest observation for an immediate still-fresh price, then opens SSE and applies the same timestamp freshness gate as embedded mode, so retained history and in-progress node catch-up are not presented as live prices. Before opening the stream, it reads the processor’s immutable pool catalog and rejects any requested market that the node does not process. Running-node subscriptions never mutate processor scope; all clients reuse the configured durable stream. Nothing prints until the stream’s `hello` shows an Ethereum mainnet node (chain 1) serving `uniswap-observations` under the requested `--processor`, and every reconnect checks it again. A node for another chain or processor, or a service that is not a Leani node answering at `api.bind`, ends the command with an error instead of printing its stream. A stream error the node marks `retryable` reconnects from the last cursor. An attached `--finality finalized` subscription replays the node’s retained change log and prints each observation once the node’s finality marker covers its block, however late finality arrives, skipping blocks that were already final when it connected; the freshness gate applies only to unattested and attested output. See [the block feed](/docs/guides/live-block-subscriptions/#attach-to-a-prepared-node) for the gap row it prints when the node stops reporting finality. The initializer fetches a current majority checkpoint quorum and pins it in a compact configuration. It asks before accepting a new trust root and refuses to overwrite an existing file. `endpoints` holds the managed PublicNode and Lodestar Beacon API pool plus every responding checkpoint provider that also serves the Beacon light-client API, without duplicates. `leani --config demo.toml init ...` writes to a different path; `--yes` is available for an intentional non-interactive setup. A generated ETH/USDC file has this shape (the root and slot are live values): ```toml config_version = 1 network = "ethereum-mainnet" data_dir = "./data" [finality] checkpoint = "0x..." checkpoint_slot = 15100000 endpoints = [ "https://ethereum-beacon-api.publicnode.com/", "https://lodestar-mainnet.chainsafe.io/", ] [uniswap] markets = ["ETH/USDC"] [api] bind = "127.0.0.1:18080" ``` The compact document expands to the standard local-node profile: native execution P2P with an opportunistic minimum of one peer and a 16-peer healthy target, locally verified Beacon finality over the `endpoints` transport pool, which `leani init` fills with the ordinary PublicNode (`https://ethereum-beacon-api.publicnode.com/`) and Lodestar (`https://lodestar-mainnet.chainsafe.io/`) Beacon API endpoints and the responding checkpoint providers that serve the Beacon light-client API, Xatu history fallback, bounded 512 MiB memory and 2 GiB temporary-disk budgets, on-demand Uniswap history, unattested publication, undo records kept until 256 blocks past finality, 64 MiB/24 hour delivery retention, ephemeral P2P listener ports, RPC on `18545`/`18546`, and the native API on `18080`. Pool addresses, fee tiers, and the earliest processor block come from the same built-in market catalog as `subscribe`: `ETH/USDC` starts at its pool’s creation block, 12,376,729, and `ETH/USDT` and `WBTC/ETH` at the Uniswap V3 factory deployment block, 12,369,621, which no pool predates. The start block is part of the processor’s identity, so a release that moves one makes the node refuse the state it kept and name the ways to move on (see the changelog). Operators who need to tune any of these can use the full configuration schema instead. The pinned checkpoint is the trust root; Beacon API endpoints only transport untrusted consensus data and never provide execution blocks or Uniswap observations. Leani queries those transports concurrently and verifies their responses locally. After the first verified response it waits about two seconds for the others, then follows the highest finalized slot that the configured agreement threshold of endpoints has reached or passed; with the compact default of one, the most recent verified finality wins, and finality never moves backwards. The node needs the pinned checkpoint only for its first start: it then keeps the newest verified finality anchor in `data_dir` and restarts from it, so the checkpoint does not expire while the node keeps verifying finality (see the runbook’s [checkpoint lifecycle](/docs/operations/runbook/#checkpoint-lifecycle)). ## TypeScript SDK The node API deliberately publishes exact raw pool state. This example derives USDC per ETH with `bigint`, preserving the V3 ratio until the final decimal rendering: ```ts import { createLeaniClient } from "@smart-byte/leani-sdk"; const leani = createLeaniClient({ baseUrl: "http://127.0.0.1:18080" }); const Q192 = 1n << 192n; function decimalRatio(numerator: bigint, denominator: bigint, places = 8) { const scale = 10n ** BigInt(places); const rounded = (numerator * scale + denominator / 2n) / denominator; const digits = rounded.toString().padStart(places + 1, "0"); return `${digits.slice(0, -places)}.${digits.slice(-places)}`; } function usdcPerEth(sqrtPriceX96: string) { const sqrt = BigInt(sqrtPriceX96); // Pool token0 is USDC (6 decimals), token1 is WETH (18 decimals). return decimalRatio(Q192 * 10n ** 18n, sqrt * sqrt * 10n ** 6n); } function tokenAmount(rawAmount: string, decimals: number) { const signed = BigInt(rawAmount); const amount = signed < 0n ? -signed : signed; const digits = amount.toString().padStart(decimals + 1, "0"); const integer = digits.slice(0, -decimals); const fraction = digits.slice(-decimals).replace(/0+$/, ""); return fraction ? `${integer}.${fraction}` : integer; } for await (const change of leani.uniswap.subscribe()) { if ( change.operation === "apply" && change.data?.pool.toLowerCase() === "0x88e6a0c2ddd26feeb64f039a2c41296fcb3f5640" && change.data.sqrtPriceX96 && change.data.amount1 ) { console.log({ price: usdcPerEth(change.data.sqrtPriceX96), baseVolume: tokenAmount(change.data.amount1, 18), }); } } ``` The CLI uses the same integer formulas. It does not convert `sqrtPriceX96` or swap amounts to floating-point numbers. ## Built-in Mainnet markets | Market | Fee tier | Pool | | -------- | -------- | -------------------------------------------- | | ETH/USDC | 0.05% | `0x88e6A0c2dDD26FEEb64F039a2c41296FcB3f5640` | | ETH/USDT | 0.30% | `0x4e68Ccd3E89f51C3074ca5072bbAC773960dFa36` | | WBTC/ETH | 0.30% | `0xCBCdF9626bC03E24f779434178A73a0B4bad62eD` | # Configure a node > Compose data sources, processor identities, lifecycle policies, API settings, credentials, and operating profiles. Leani reads strict, versioned TOML. Unknown fields are errors. Relative paths in a configuration file, such as `data_dir` or an archive source’s `manifest`, are relative to that file; path flags on the command line stay relative to the working directory. Version 1 has two first-class document shapes: a compact processor-oriented setup with sane network defaults, and the fully explicit advanced schema. Both become the same validated runtime configuration. The machine-readable contract is rendered in the [configuration field reference](https://leani.dev/docs/reference/configuration/); six complete advanced lifecycle profiles are checked under `config/modes`. For a continuous block-following demo, generate the compact form and a current trusted checkpoint together: ```bash leani init blocks leani doctor ``` For a local Uniswap node, select one or more markets instead: ```bash leani init uniswap-v3 ETH/USDC leani doctor ``` `init` requires a strict majority of the configured checkpoint providers, at least two of the three defaults, to agree on one finalized slot and root, and fails closed if a responding provider reports another root at that slot. It shows every provider’s answer, asks before accepting the checkpoint, writes `leani.toml`, and never overwrites an existing file. A global `--config PATH` selects another output path. The node keeps its state in `data`, beside the configuration file, unless `--data-dir` names another directory; like any path flag, a relative `--data-dir` is relative to the working directory. The generated document contains only the choices an operator normally needs: ```toml config_version = 1 network = "ethereum-mainnet" data_dir = "./data" [finality] checkpoint = "0x..." checkpoint_slot = 15100000 endpoints = [ "https://ethereum-beacon-api.publicnode.com/", "https://lodestar-mainnet.chainsafe.io/", ] [blocks] [api] bind = "127.0.0.1:18080" ``` The checkpoint and slot above are illustrative; the generator writes the live quorum result. The checkpoint is the trust root. PublicNode (`https://ethereum-beacon-api.publicnode.com/`) and Lodestar (`https://lodestar-mainnet.chainsafe.io/`) are ordinary, untrusted Beacon API transports in the compact profile’s managed transport pool; they do not provide execution blocks or Uniswap data. `init` writes that pool plus every responding checkpoint provider that also serves the Beacon light-client API, without duplicates. Leani queries the pool concurrently and verifies every response locally from the pinned checkpoint. After the first verified response it waits about two seconds for the others, then follows the highest finalized slot that `minimum_agreement` endpoints have reached or passed. With the compact profile’s default agreement of one, the most recent verified finality wins, and a stalled transport does not hold up node startup. Listing endpoints in the compact document replaces the managed defaults, which lets operators use only self-hosted transports. Compact Ethereum Mainnet configurations expand to: | Concern | Default | | --------- | ----------------------------------------------------------------------------------------- | | execution | native P2P, one-peer availability floor, 16-peer healthy target, 100-peer cap | | history | public Xatu source, on-demand processor scheduling | | processor | selected built-in feed, unattested publication, checkpointed state, 256-block undo safety | | resources | 512 MiB memory, 2 GiB temporary disk, four source and mapper tasks | | retention | full query output; 64 MiB or 24 hours of change delivery | | listeners | ephemeral P2P ports, RPC `127.0.0.1:18545`/`18546`, API `127.0.0.1:18080` | The reviewed expansion is checked in as [`config/defaults/ethereum-mainnet.toml`](https://github.com/smart-byte/leani/blob/main/config/defaults/ethereum-mainnet.toml). Use the advanced schema below when any expanded field needs an override. Validate a file without opening its database or any network connection: ```bash cp config/modes/windowed.toml leani.toml leani doctor --json ``` Without `--config`, Leani uses `./leani.toml`. Set `LEANI_CONFIG` when a service or shell should consistently use a configuration elsewhere. An explicit `--config` remains the highest-precedence override. Compact documents default operational tuning but never the chain identity, checkpoint trust root, or selected processor targets. Advanced documents keep every choice explicit. ## Advanced root fields | Field | Type | Meaning | | ------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `config_version` | integer | Must be `1`. | | `data_dir` | path | Writable node state. Use a distinct directory per deployment or test. A relative path is relative to the directory of the configuration file, not the working directory. | | `chain` | table | Portable name and non-zero EIP-155 `chain_id`. | | `artifact_storage` | table | Physical backend and bounded compaction policy for retained processor artifacts. | | `budgets` | table | Hard memory, temporary disk, pending-delta, recent-input, and concurrency limits. | | `sources` | table | Priority-ordered history sources and the optional live source. | | `finality` | table | `consensus_p2p`, `beacon_api`, or `disabled`. | | `processors` | array | One immutable processor kind/instance contract per entry. | | `rpc` | table | HTTP/WS binds, browser origins, request and response limits, and bounded historical lookup behavior. | | `api` | table | Native API bind, bearer-token environment variable, allowed hosts and browser origins, and delivery batching. | `budgets.recent_raw_soft_bytes` must not exceed `budgets.recent_raw_hard_bytes`. All byte budgets and concurrency values are positive. Each finality advance prunes finalized recent frames toward the soft limit. Live ingestion enforces the hard limit as it retains each frame: at the limit it waits, with live readiness down, until finality prunes (see the operations runbook’s storage pressure section). Two budgets bound a history source read’s input. `budgets.temporary_disk_bytes` bounds what one read may acquire over its life: the objects, byte ranges, and responses it fetches, and the frames it builds from them. A read that would acquire more fails with an `input_bytes` budget error, and a local archive object larger than it is refused before it is read. `budgets.memory_bytes` bounds the raw input one read holds in memory at once: one line of a local archive object, which it verifies whole and then reads a line at a time; the selected columns of one Xatu Parquet row group; one batch of EraE byte ranges, including a prefetched window when both fit the budget; one window of execution P2P frames; or one retained raw-history record. EraE normally targets 16 MiB compressed windows independently of the decoded frame buffer, and shrinks them when the resident budget is smaller. Its frames may be at most `memory_bytes`, or 32 MiB when that is less. A read that would hold more fails with a `resident_bytes` budget error naming `budgets.memory_bytes`, and a backfill then tries its next configured source. A historical RPC request, which holds everything it returns, may acquire and hold at most `memory_bytes` divided by `budgets.source_concurrency`. Raw-history jobs acquire one segment’s worth per read, `raw_history.maximum_segment_logical_bytes`. The bound is per read, not for the node. At once, the node runs up to `budgets.history_pipeline.maximum_active_chunks` backfill reads, one archive reconciliation read, the live lane, and one read per running raw-history job, and each may hold up to `memory_bytes`. What a read decodes is not counted here: decoded frames and mapped deltas have the `budgets.history_material` and pipeline budgets below, and `memory_bytes` is not a process RSS limit. On mainnet, a Xatu read of a 2023–24 transaction object, 1,000 blocks around blocks 18,000,000 to 19,430,000, holds about 80–110 MiB of compressed columns and decodes about 230–340 MiB. Keep `memory_bytes` above the largest of those, as the shipped 512 MiB is, and plan RAM for about `maximum_active_chunks` × 450 MiB during Xatu backfills, 3.6 GiB at the default 8. The allocator keeps memory a finished read freed for the next, so resident memory after a burst of reads stays near its peak: set a container’s memory limit for that peak, not for the budgets. On Linux, glibc gives each thread that allocates its own 64 MiB arena and returns little of what they free, so a long-running node grows: an rc.4 node on a 20-thread host held 5.7 GB, 4.7 GB of it in 57 such arenas. Over a 100,000-block `blobs-money` backfill from Xatu, glibc reached 1.5 GiB and kept it. The container image preloads jemalloc, which returns freed memory: the same backfill held 711 MiB and fell to 130 MiB when it finished. `deploy/leani.service` sets `MALLOC_ARENA_MAX=2`, which held it at 0.8–1.0 GiB. Wherever you run the binary yourself, set `MALLOC_ARENA_MAX=2` or preload jemalloc (Debian’s `libjemalloc2` with `LD_PRELOAD=libjemalloc.so.2`). `[budgets.history_material]` is the sole budget for shared immutable source frames and acquisition reorder buffers. Chunks read ahead of the one a job is reading reserve memory only up to half of its `memory_bytes`, so size it to at least twice the largest expected frame. The half is checked as each frame is reserved: a chunk that another job joins, or that its reading job pauses and leaves while others wait for it, becomes read-ahead with the frames it already holds, so read-ahead can hold more than half. `[budgets.history_pipeline]` is separate: `maximum_active_chunks` caps physical history reads node-wide and `maximum_mapped_bytes` caps processor-owned mapped deltas across every active job. A job reads only its own planned chunks, so a backfill subscription smaller than `maximum_active_chunks` source chunks leaves read slots idle; the node logs a hint with a size that fills them, once per processor, and `leani_history_material_active_acquisitions` against `leani_history_material_active_acquisitions_limit` shows the slots in use. Its nested `commit` table flushes a contiguous SQLite transaction at the first configured block, domain-change, stable-encoded-byte, or elapsed-time limit: by default 128 blocks, 10,000 changes, 16 MiB, or 50 ms. A batch publishes its blocks’ delivery changes in the same transaction. Blocks not yet finalized, or that the live lane already applied or holds, commit one at a time, as do the blocks of a batch whose reducer reads windowed output or rewrites it unchanged. A source error or invalid frame ends a batch after the blocks before it commit. `target_writer_hold`, 250 ms by default, controls the adaptive block limit: the runtime halves the current limit when observed p95 historical writer hold time exceeds the target and cautiously grows it again below half the target. A hold includes the commit’s WAL sync, which a smaller batch does not shorten, so on a disk whose syncs are slow a low target shrinks batches to a block or two, and every commit rewrites the same pages for little data. On a Hetzner EX44 (RAID 1 over two NVMe drives, about 8 ms per sync), the former 20 ms target ran a `rollup-txs` backfill from Xatu at 53 blocks/s and wrote 104 KiB per block; the 250 ms target ran it at 799 blocks/s and wrote 11 KiB per block. A job holds at most `maximum_buffered_frames_per_acquisition` frames of `[budgets.history_material]`, 128 by default, that it has not committed, so a longer `maximum_delay` makes acquisition wait instead of filling larger batches: a one-second delay ran the same backfill at 127 blocks/s. A single mapped delta or single-block commit that exceeds its byte limit fails immediately because waiting cannot make that item fit. A backfill that reuses retained recent frames reads them in batches of at most `commit.maximum_blocks` blocks and `maximum_mapped_bytes` encoded bytes, each holding one active-chunk slot like a physical history read. `[budgets.store].maximum_physical_bytes` is the emergency node-store admission bound shared by SQLite, its WAL, and configured processor-artifact segments. It is deliberately not nested under delivery: artifacts, materialized views, correctness metadata, and delivery share the same physical capacity. It measures allocated file bytes. Stores use SQLite incremental auto-vacuum, so pruning returns pages to the filesystem and admission recovers; a store created before that needs one `leani db compact` (see the operations runbook). `[budgets.artifacts]` independently caps immutable finalized artifact bytes and unfinalized-block candidate bytes awaiting finality. A transaction that would exceed either logical limit rolls back without advancing processor coverage. Finality is the exception: when the retained limit is full, finality still advances, the finalized candidates stay pending with a logged warning, and a later finality advance promotes them once there is room. A processor’s `window` policy may prune its own instance below a tighter bound; the node-wide artifact budget still protects against the aggregate of many processors. `[artifact_storage] backend = "sqlite"` retains those immutable artifacts as KV rows. The opt-in `tiered_segments` backend uses SQLite as the durable write buffer and continuously compacts complete `segment_target_blocks` ranges into checksummed seekable files under `data_dir/processor-artifacts`. Compression, single-item/segment hard limits, compaction cadence, and maximum segments per cycle are explicit. Only processors using `artifacts.mode = "full"` are eligible in the first tiered implementation; rolling windows remain in SQLite. Segment bytes do not receive a second budget: they count against the same `[budgets.store].maximum_physical_bytes` ceiling. `[raw_history]` keeps finalized source frames in checksummed segment files under `data_dir/raw-history`, beside their own catalog. `maximum_logical_bytes` caps the frames retained, and `maximum_physical_bytes` the bytes on disk: segments, the partial files of segments being written, the catalog and its write-ahead log, and quarantined files. Before a segment is written, its maximum size and the catalog rows it will add are reserved: a fixed 64 KiB, and 512 bytes per locator row, one per block with block-hash locators and 256 per block with transaction-hash locators. A segment with more transactions still publishes its extra rows, so the catalog can exceed `maximum_physical_bytes` by that one segment’s extra rows, which the next segment’s admission counts. Reads of `maximum_segment_logical_bytes`, one segment’s worth, feed a raw-history job. A store already over either budget, for example after lowering it, still opens and serves its history, but admits no new segment until it is back under: its jobs pause at the storage limit, or fail when created with `segment.onLimit: fail`. A segment that fails verification moves to `data_dir/raw-history/quarantine`, which keeps at most 1 GiB, or one segment of `maximum_segment_physical_bytes` when that is larger: past that, the longest-quarantined files are deleted first, but never the newest (see the operations runbook). `[budgets.delivery]` supplies the shared delivery envelope above independent per-stream limits. `maximum_retained_bytes` caps exact delivery payload bytes across live and history streams. `maximum_history_retained_bytes` is a lower, non-borrowable history ceiling so stalled backfills cannot consume live-lane headroom. The two values must satisfy `history <= retained`. Reaching a shared bound backpressures only the committing lane; ACK, pruning, queries, networking, and unrelated processors remain available. The backfill subscriptions of a replaced processor instance keep counting against the history ceiling until they are deleted (see the runbook’s [Processor rebuild or rollback](/docs/operations/runbook/#processor-rebuild-or-rollback)). ## Sources and finality `[[sources.history]]` requires `id`, `kind`, `priority`, and `trust`. Supported kinds are `xatu`, `era_e`, and `archive`. `xatu` reads public Parquet history; validation refuses `parquet`, which no source reads. `archive` requires `manifest`; `era_e` alone accepts an `https` or `file` `endpoint` ending in `/`. A plain `http` endpoint is accepted on loopback, or elsewhere only with `allow_insecure_http = true` on that source, which accepts a mirror whose catalog and archive bytes anyone on the path can replace. `xatu` sources are Ethereum mainnet only: another `chain.chain_id` fails validation. ### History source order A backfill splits its range into gaps of missing coverage and tries the configured history sources for each gap in `priority` order, lowest first. A source that fails on part of a gap passes only those blocks to the next source: a dataset hole, from its first to its last block in the chunk read, or the chunk whose read failed. It keeps the blocks before them, and the first source takes the gap back after them. A source that has not published the blocks, whose schema changed, or that fails again before committing a block passes the rest of the gap instead, and one that cannot fill the processor’s field groups is skipped for the job. Each backfill retries up to three times the number of sources, at least five times, before it fails. When every history source fails and the live source is execution P2P, P2P can serve the gap within `sources.live.history_fallback_blocks`. The shipped profiles list eraE at priority 10 and Xatu at priority 20: | Source | Serves | Checks each block for | Notes | | ----------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | `era_e` | Whole blocks and receipts up to its mirror’s last catalog update | Transactions and receipts roots against the header | The public mirror’s catalog ended at block 24,518,655 (2026-02-26) when this was written | | `xatu` | Columnar projections, about one day behind the chain | Transaction rows against the beacon payload count; log rows without a gap | Cannot prove a block’s last logs complete; Ethereum mainnet only | | P2P (live source) | Recent and live blocks from peers | Commitments of every header, body, and receipt | Slowest for old history; peers may no longer serve very old blocks | So for blocks the eraE mirror covers, a backfill reads verified complete blocks; for later blocks, Xatu; and P2P fills what neither serves. Put Xatu first when bandwidth, not completeness, limits a backfill: its projections transfer fewer bytes per block for processors that read a few columns. ### Sizing for bandwidth, memory, and disk Measured with this release on a Hetzner EX44 (Intel i5-13500, 20 threads, 64 GB, RAID 1 over two NVMe drives, 1 Gbit/s) that other workloads share, 25,000-block backfills with one consumer and `MALLOC_ARENA_MAX=2`: | Backfill | Source | Downloaded per block | 4 reads, 2 chunks | 8 reads, 4 chunks | | ----------------------- | ------ | -------------------- | ------------------------- | ------------------------- | | `rollup-txs`, Feb 2026 | eraE | 122 KiB | 250–300 blocks/s, 170 MiB | 250–330 blocks/s, 280 MiB | | `blobs-money`, Feb 2026 | eraE | 122 KiB | 280 blocks/s, 160 MiB | 250–300 blocks/s, 250 MiB | | `rollup-txs`, Feb 2026 | Xatu | 61 KiB | 550–560 blocks/s, 230 MiB | 730–780 blocks/s, 340 MiB | | `blobs-money`, Feb 2026 | Xatu | 72 KiB | 410–420 blocks/s, 560 MiB | 500–570 blocks/s, 730 MiB | | `rollup-txs`, Oct 2026 | Xatu | 56 KiB | 490 blocks/s, 290 MiB | 620–890 blocks/s, 420 MiB | | `blobs-money`, Oct 2026 | Xatu | 46 KiB | 570 blocks/s, 420 MiB | 470–650 blocks/s, 620 MiB | “4 reads, 2 chunks” is `source_concurrency = 4` and `maximum_active_chunks = 2`, as `deploy/container.toml` ships; the other shipped profiles read with 8 and keep the built-in 8 chunks. Each cell gives the speed and the peak memory. Ranges span repeated runs, which differed by up to a third with the archives’ throughput and the shared host’s load. With jemalloc, which the container image preloads, peak memory stayed within a fifth of these and fell back once a backfill finished. Execution P2P served 5,000 recent blocks at 43–55 blocks/s. More reads and chunks made most Xatu backfills faster, at more memory, and left eraE unchanged. An eraE backfill is limited by the frames it may hold uncommitted: with `maximum_buffered_frames_per_acquisition = 512` and `memory_bytes = "512MiB"` in `[budgets.history_material]`, eraE backfills ran at 380–410 blocks/s, a quarter to a half faster, at the same peak memory. Profiles: * **Default:** the shipped profiles, which try eraE before Xatu and read up to 8 chunks at once with 8 reads each. * **Faster eraE backfills:** set those two `[budgets.history_material]` values. Concurrent backfills share that budget, so each can then hold more of it. * **Less memory:** set `maximum_active_chunks = 4`, the “8 reads, 4 chunks” column. Xatu backfills then hold about half as much, and execution P2P backfills fetch about half as much at once. * **Limited memory**, such as a container under 1 GiB: use 4 reads and 2 chunks. eraE needs the least memory. * **Limited bandwidth:** a backfill downloads blocks per second times bytes per block, about 35 MiB/s from eraE at 300 blocks/s and 25–50 MiB/s from Xatu at the speeds above. On a slower link, the link sets the speed: at 100 Mbit/s, about 12 MiB/s, eraE reaches about 100 blocks/s and Xatu 170–270. There, list Xatu first for processors that read only transactions, such as `rollup-txs` and `transaction-stats`, and keep eraE first for processors that read logs, which Xatu cannot prove complete. * **Slow disk syncs**, such as network-attached volumes: keep `target_writer_hold` at 250 ms or raise it. A lower target shrinks batches (see [Advanced root fields](#advanced-root-fields)). Xatu additionally accepts three acquisition-tuning controls. `chunk_blocks` sets the generic header, transaction, and log span, while `blobs_chunk_blocks` sets the wider span used by the five-table blobs join. Both must be non-zero multiples of Xatu’s physical 1,000-block execution partition. `batch_rows` controls the bounded Arrow/Parquet decode batch and is independent of history-source concurrency. The measured defaults are 1,000, 8,000, and 8,192 respectively. Increasing a span can reduce repeated daily object reads, but it also increases per-chunk memory and is not universally faster; tune it per projection rather than as a global block-size knob. `[sources.live] kind = "p2p"` enables the shared persistent Reth peer manager. It requires verified `beacon_api` or `consensus_p2p` finality. Stable production ports normally use `listener_port = 30303`, `discovery_port = 30303`, and `discv5_port = 30304`. Discv4 and Discv5 require distinct fixed UDP ports; zero selects an isolated ephemeral port for that protocol. `stall_timeout_seconds`, ten minutes by default and six minutes to 30 days, bounds how long the live lane may stay not ready while verified finality is ready, within one run of the network lanes: after it, the node exits with status 1 so that its process supervisor restarts it. A wait at the recent-frame hard limit does not count. Reth owns the bounded dial queue and receives TCP/RLPx failures directly. A terminal failure immediately releases its slot to another eligible peer, while Reth’s per-peer backoff prevents rapid redials. Newly authenticated DNS/Discv candidates are added to that same queue. `max_concurrent_dials` remains the hard simultaneous dial ceiling; `peer_refill_interval_ms` is only a short safety-net poll for newly added candidates, not a network request rate. `minimum_peers` is the connected-session startup gate. A peer need not finish capability qualification before requests may use it: every response is still commitment-checked, and qualification ranks proven peers first. Background qualification continues concurrently until `body_serving_peer_target` distinct peers have served a commitment-valid body at the current target; the default target is four. Request health is tracked independently for headers, bodies, and receipts. A timeout or incomplete response cools only that material lane, so a peer that cannot serve one processor’s body request remains available to processors that can use its headers or receipts. Commitment-invalid data still bans the peer. `bootstrap_dns_tree` may point at an optional EIP-1459 `enrtree://` peer-hint tree. The public key embedded in that URL authenticates its ENRs. It does not make those peers trusted sources: Leani still commitment-checks every header, body, and receipt. This is the deployment hook for a future Leani/community pool of execution nodes prepared to accept indexing traffic; compact and advanced configurations leave it unset by default. Finality checkpoints are bootstrap trust anchors, not evergreen config defaults. The first start needs a checkpoint at most 14 days old; later starts bootstrap from the newest verified anchor the node persisted (see the runbook’s [checkpoint lifecycle](/docs/operations/runbook/#checkpoint-lifecycle)). Enabled finality requires a recent 32-byte checkpoint root other than all zeros. `consensus_p2p` also requires its non-zero finalized beacon slot and a `finality.minimum_peers` of at most 24, the consensus peers dialed at once. `beacon_api` requires one or more endpoints, each transport listed once, and `minimum_agreement` within their count. Disabling finality is suitable for bounded finalized-dataset backfills, not verified head following: the live lane includes a block only once its ancestry reaches an execution header the sync committee attested, which the finality source verifies each slot, so validation refuses `sources.live.kind = "p2p"` with `finality.kind = "disabled"`. ## Processor identity and publication Every modern entry declares: ```toml [[processors]] id = "evm-events" # registered processor kind instance = "weth-transfer-v1" # stable operator-selected identity version = "1.1.0" start_block = 12965000 ``` Processor code, semantic version, settings, start point, and source identity are part of an instance’s immutable identity. The node refuses a change to any of them under an `instance` its store holds (`processor instance … conflicts with its stored descriptor`), before it upgrades an older store, so it never mutates an existing cursor namespace. Give the changed processor a new `instance` ID: it indexes from its `start_block` beside the old instance, whose rows stay in the store. Settle the old instance’s backfill subscriptions before and after the switch, as the runbook’s [Processor rebuild or rollback](/docs/operations/runbook/#processor-rebuild-or-rollback) describes. Lifecycle policies, such as retention and delivery limits, are not part of the identity and can change in place. Publication values are: * `unattested` (default): publish apply/undo/finality transitions from the first level the live lane delivers, and keep an unfinalized undo window. * `attested`: publish apply/undo/finality transitions from `attested` on, and keep an unfinalized undo window. * `finalized`: publish only finalized `apply` records. Unfinalized changes wait in the store for verified finality, count against the delivery budget meanwhile, and undo is never published. `[processors.delivery] emit = "job_result"` replaces `publish` for a bounded backfill: it publishes its terminal aggregate and then becomes complete/reclaimable. Deletion remains explicit. `[processors.settings]` belongs to the selected factory and is decoded strictly. `doctor` constructs every configured processor, so malformed ABI, address, pool, or custom settings fail before source or store I/O. ### Live gap fill After downtime the live lane resumes at the newest verified finalized anchor, so a block-local processor skips the blocks between its cursor and that anchor. Automatic history refills them. With `history_mode = "on_demand"` the lane only announces them, in a `system.live_gap.put` change, and whoever requests history fills them. `[processors.live_gap_fill]` lets the lane fill a short range itself: ```toml [[processors]] id = "blobs-money" instance = "blobs-production" history_control = "application_subscriptions" history_mode = "on_demand" # ... [processors.delivery] mode = "until_acknowledged" on_limit = "pause" # the default, and the only value a fill accepts # ... [processors.live_gap_fill] max_blocks = 7200 # fill a skipped range of up to about a day; announce longer ones ``` Before the lane applies the block that follows a skipped range, live or replayed at startup, it recovers the range from the processor’s history sources, 128 blocks at a time, and applies each block, oldest first, as a finalized `apply` with `originKind: "live_recovery"`. Then it applies the block itself. The live stream stays contiguous and in block order, announces nothing for the range, and the processor’s coverage holds the filled blocks as finalized. The [change protocol](/docs/reference/native-api-and-delivery/#42-operations) describes what a consumer sees, and [Keep your application in sync](/docs/guides/keep-your-application-in-sync/) what an application still does. `max_blocks` is required and takes 1 to 7,200, about a day of slots. The table has no other setting. Validation refuses it for a processor that is not `history_mode = "on_demand"`, because automatic history refills skipped blocks, and for an ordered processor, such as `transaction-stats` or `uniswap-latest`, which holds later blocks instead of skipping them. It also requires `[processors.delivery] on_limit = "pause"`, which is the default: validation refuses `fail` and `expire_and_reset`, because a fill’s burst of blocks could fail the lane, or reset its consumers, where announcing the blocks would not. It is allowed under both `history_control` values. The lane announces a range as it does without the table, and fetches nothing, when the range is longer than `max_blocks`, when no history source serves finalized blocks, or when the range is not finalized yet. The limit is all or nothing, because blocks filled at the top of a range could not link to the cursor. A fill that starts stops at the first block it cannot fill, and the lane announces from there. That happens when history fails or serves too few blocks; when a block is not finalized, does not link by parent hash to the one before it, is not the canonical block at its height, or lacks material the processor requires; and when the time limit passes. The limit is five minutes and is fixed. The fill starts no request after it and gives up a request still running at it, so a fill runs for at most five minutes plus applying the blocks of its last request. A filled block is as trusted as the same block delivered by a `fill_missing` subscription, which uses the same sources and verification policy. The filled blocks link by parent hash from the cursor’s block through the range and, when the fill completes, to the block that follows. The sources are the processor’s history sources, in the order a history request tries them: retained raw history (`[raw_history]`, when enabled), the configured `[[sources.history]]` by `priority`, and last the execution P2P bridge, within `sources.live.history_fallback_blocks`. The bridge is available to a fill during the replay that follows a restart, too. Bandwidth depends on the source. A dataset source such as Xatu serves only the material the processor needs, about a kilobyte per block for `blobs-money`. Execution P2P transfers whole bodies and receipts, on the order of a megabyte per block on Mainnet today, so a few thousand blocks are gigabytes. Configure a dataset source for a processor that sets the table. Requests run one after another and do not coalesce, so a fill is slower than a pipelined backfill subscription over the same range. `sources.live.kind = "disabled"` leaves no live lane, so the table does nothing, and `require_retained_input = true` limits a fill to retained raw history, which usually does not hold the range: the lane announces it. The live lane is shared. While a fill runs, it takes no other block: every processor’s live lane waits, the head that JSON-RPC reports stops advancing, and WebSocket subscriptions publish nothing new. That happens after the live lane was not running for a while, and `max_blocks` and the time limit bound it. Several processors with the table fill one after another, and each fetches the blocks again. Filled blocks are ordinary delivery records under the processor’s delivery policy, and count against `delivery.max_bytes` and the node-wide `[budgets.delivery]` and `[budgets.store]` limits. A lane that reaches one of those limits mid-fill stops there. It holds the block after the range and announces nothing. While the store would refuse that block, because the lane is paused above its stream’s resume mark (90 % of `max_bytes`), a node-wide budget has no room for it, or the lane is paused at a storage limit while storage stays above its low-water mark, the lane fetches no history for the fill, as long as the block after the range is among the retained frames. Once finality prunes that frame during a long hold, the lane recovers it from history on every live block, as any paused lane recovers its first unapplied block. After a restart of the live lanes, a lane held at a node-wide budget can fetch once more before it waits again. The lane continues the fill once the store would take the block: once consumers acknowledge and pruning brings its stream down to the resume mark, once pruning any stream gives a node-wide budget room, or once storage has room. The unfilled rest is recomputed from the processor’s cursor on resume, so a history commit that an application’s subscription lands above the cursor meanwhile can shorten what the lane fills or announces. That is rare, and idempotent writes and the application’s coverage diff cover it. A single filled block larger than the stream’s `max_bytes` fails the lane, as an equally large live block would: raise `max_bytes` or remove the table. A fill also makes a pause at a delivery or storage limit more likely, and a held block’s delta counts against the shared `[budgets] pending_delta_bytes`; when it does not fit, the lane fails (`live_gap_marker_exceeds_pending_delta_budget`) until the budget is raised and the lane reset. The [runbook](/docs/operations/runbook/#crash-or-interrupted-commit) lists the log lines and the recovery. The table is not part of the instance’s identity, so add, change, or remove it in place, as with the lifecycle policies below. A release that predates the table refuses a configuration that has it, so remove it before rolling back. ## Orthogonal lifecycle policies Every processor has six independent lifecycle policies. The pre-launch `retention` shorthand has been removed so artifact, output, delivery, state, checkpoint, and undo semantics cannot be inferred from one overloaded setting. ### State `[processors.state] mode` is `durable` or `checkpointed`. The store persists processor state in both modes, and `[processors.checkpoint]` alone decides whether recovery checkpoints are taken: `checkpointed` requires `[processors.checkpoint] mode = "automatic"`. `ephemeral` is rejected, because no mode discards processor state. ### Artifacts `[processors.artifacts] mode` is `none`, `window`, or `full`. Artifacts are immutable finalized map results used for bulk scan, export, and compatible replay. They are independent of queryable output and delivery records; the default is `none`. `window` requires exactly one `[processors.artifacts.window]` limit: `max_blocks`, `max_age`, or `max_bytes`. Full retention is removed only by an explicit artifact deletion operation. ### Output `[processors.output] mode` is: * `none`: retain no queryable entity material; * `latest`: retain the current entity value per key; * `window`: retain the declared block, wall-clock age, row, or byte window; * `full`: retain all selected queryable output. `window` requires a `[processors.output.window]` table with exactly one limit: `max_blocks`, `max_age`, `max_rows`, or `max_bytes`. Validation refuses `[processors.output] finalized_only`, which was never applied; to publish only finalized blocks, set `publish = "finalized"`. ### Delivery `[processors.delivery] mode` is `best_effort`, `window`, or `until_acknowledged`. Byte sizes accept integers or `B`, `KiB`, `MiB`, `GiB`, `KB`, `MB`, and `GB` strings. Durations accept integer seconds or `s`, `m`, `h`, and `d` strings. `until_acknowledged` requires at least one required `[[processors.delivery.consumers]]`, each with a `lease_ttl` from one second to 100 years. Required consumers protect pruning until they are explicitly expired/reset; a temporarily lapsed lease does not silently advance the watermark. `on_limit` is `pause`, `fail`, or `expire_and_reset`. Paused instances automatically resume after acknowledgement/pruning moves below the low-water mark. Pruning is cumulative and block-aligned: ```toml [processors.delivery.pruning] interval = "30s" minimum_batch_blocks = 64 minimum_batch_changes = 10000 maximum_delete_changes = 10000 retain_finalized_blocks = 256 retain_acknowledged_age = "1h" ``` `max_age` and `retain_acknowledged_age` use durable node wall-clock enqueue/acknowledgement time, not Ethereum block timestamps, so the changes a backfill enqueues count as new. A pass deletes at most `maximum_delete_changes`. A pass that deletes that many is followed at once by the next, and a stream that pauses at its limit is pruned at once, without waiting for `interval`. ### Checkpoint and undo `[processors.checkpoint] mode` is `none` or `automatic`. Automatic mode checkpoints processor state at most once per 1,000 finalized blocks, and again when a historical job completes, and keeps the newest positive `keep` count. A restore repairs state only when the processor’s cursor sits exactly on a checkpoint: a cadence block or the end of a completed job, even if finality reached that block later. A processor between checkpoints cannot be restored from one; restore a full-store backup or rebuild the processor. Portable savepoints are exports, created and deleted explicitly through the API; the node cannot restore them, and automatic checkpoint pruning does not cover them. Each processor instance keeps at most 16, and each new one counts against the physical store budget, which answers 507 when it does not fit. Editing lifecycle policies, for example to raise a limit, leaves existing checkpoints and savepoints valid. `[processors.undo] mode` is `none` or `unfinalized`, and `unattested` and `attested` publication require `unfinalized`. Unfinalized blocks always keep their undo records, so a reorg can revert them in either mode. Once finality passes a block, `unfinalized` keeps its record for another `safety_blocks` blocks and `none` deletes it at once. Blocks applied as already finalized record no undo. ## API and credentials Set `api.bearer_token_env` to the name of an environment variable, never a token value. When configured, that one bearer protects native read, stream, consumer, and management routes on the API listener; operational health, metrics, and network-dashboard routes remain unauthenticated. `serve` fails to start when the variable is unset, holds fewer than 16 characters, or holds anything but printable ASCII without spaces; use a random value such as `openssl rand -hex 32`. Clients send `Authorization: Bearer `, with the scheme in any case, and the node compares tokens in constant time. A consumer created through the API carries its own credential, and a backfill subscription’s consumer can; a new one holds 32 to 512 printable ASCII characters without spaces, such as `openssl rand -hex 32`. Every consumer-scoped call for a consumer with a credential, its streams included, then needs it in `x-leani-consumer-credential` besides the bearer, and gets 403 without it, so one client cannot move another consumer’s cursor. Listing, inspecting, and revoking consumers stay administrative: only the bearer protects them, so an operator can revoke a consumer whose credential is lost. Consumers declared under `[[processors.delivery.consumers]]` have no credential and need only the bearer. The store keeps credential hashes and session-token keys under a random per-store secret, which backups carry with the store. A backfill subscription’s credential also enters its idempotency identity, an unkeyed hash of the whole request, so use random credentials. The native API, HTTP RPC, and WebSocket RPC binds must all differ. RPC has no built-in authentication. A bind beyond loopback, such as `0.0.0.0`, fails validation unless it is protected or explicitly opted in: | Listener | Beyond loopback requires | | ------------------------------ | -------------------------------------------------------------------- | | `api.bind` | `api.bearer_token_env`, or `api.allow_unauthenticated_remote = true` | | `rpc.http_bind`, `rpc.ws_bind` | `rpc.allow_unauthenticated_remote = true` | Opt in only behind a trusted network or an authenticating proxy. Put remote listeners behind a TLS/authenticating proxy and restrict the unauthenticated operational routes to a monitoring network. The compact configuration has no bearer setting, so its `[api] bind` stays on loopback; use the advanced schema to serve beyond it. ### Browser access The listeners refuse most requests a web page could make through a visitor’s browser: * The native API answers a `Host` that is an IP address, `localhost`, or a name in `api.allowed_hosts`, and answers any other name with 421. A page that points its own name at 127.0.0.1 still sends that name; an address cannot be repointed, so health probes and scrapers that call the node by IP keep working. List every name clients use to reach the API, such as a compose service name or the name a reverse proxy forwards, without a scheme or port: `allowed_hosts = ["leani", "indexer.internal"]`. * Browsers send an `Origin` header on CORS requests, such as `fetch()` calls to another origin, event streams, and WebSocket upgrades, and on POST and DELETE requests. They send none on navigations or on no-cors GET and HEAD requests, such as an image or script load. Both listeners accept requests without an `Origin`, and requests from loopback origins on any port. Any other origin gets 403. A browser application served from another origin, including one behind the same reverse proxy, needs its origin in `api.allowed_origins` or `rpc.allowed_origins`, such as `["https://app.example"]`. The node sets no CORS headers; a cross-origin application still needs a proxy that adds them and allows the `content-type` and `x-leani-request` request headers. * The native API refuses, with 403 `cross_site_request`, a GET or HEAD without an `Origin` that the browser’s Fetch Metadata marks as coming from another site’s page (`Sec-Fetch-Site: cross-site` or `same-site`) and not as a navigation (`Sec-Fetch-Mode` other than `navigate`). A consumer’s live or backfill stream route refuses such a request even as a navigation, since opening the stream takes the consumer’s session lease. Current browsers send Fetch Metadata only to potentially trustworthy URLs, which are HTTPS, loopback addresses, and `localhost`; a node reached over plain HTTP by another name or address, such as a LAN IP with `allow_unauthenticated_remote`, gets none, and neither refusal applies there. The SDK and curl send none either, and pass. * A native API POST or DELETE needs `content-type: application/json` or the header `x-leani-request: 1`, and gets 415 otherwise. JSON-RPC calls need `content-type: application/json`. The SDK sends both. A cross-site page can send neither without a CORS preflight, which the node never grants. * Only `POST /v1/processors/{processor}/collections/{collection}/entities` creates a query snapshot. A GET without a cursor reads current output without one. What still reaches the native API from a page: navigations from another site, including into a frame, to routes other than consumer streams, and GET or HEAD requests without Fetch Metadata: from browsers without it, or to a node served over plain HTTP by a name or address other than loopback or `localhost`. The page cannot read the responses, but without Fetch Metadata a GET that opens a consumer’s live or backfill stream takes that consumer’s session lease. A consumer’s credential keeps pages from opening its stream, and `api.bearer_token_env` keeps them from opening any; a browser adds neither header on its own. When upgrading a relative `data_dir`, Leani refuses to start if its former working-directory location contains a database or embedded subscriptions while the location beside the configuration holds none: it would otherwise start on an empty store. Move the old state, or set an absolute path to select the intended store. When both locations hold state, it uses the one beside the configuration and warns, naming both. Equivalent paths and symlink aliases of the same directory count as one. ### Historical CLI and node backfill `leani backfill --processor INSTANCE --from-block N --to-block M` uses the same history source assembly, configured pipeline, material coordinator and verified P2P fallback as node jobs. `--processor` defaults to the configuration’s only processor. When live P2P is enabled, the standalone command verifies the configured finality anchor and opens its own persistent peer pool. It checks the finalized upper bound and honors `sources.live.history_fallback_blocks`. No live listeners or processors are started. Disabled P2P and retained-input-only policies remain in force. To use a running node and its existing peer pool: ```bash leani backfill --endpoint http://127.0.0.1:18080 \ --processor INSTANCE --from-block N --to-block M ``` The command sends an authenticated materialization request and waits for its status. `LEANI_API_TOKEN` supplies the token without putting it in shell arguments. The token travels in an `Authorization` header, so across hosts use HTTPS, for example through a TLS-terminating reverse proxy; plain HTTP suits loopback or a private container network, and the command warns when it sends the token over plain HTTP beyond loopback. An optional endpoint path prefix is preserved. Ctrl-C cancels the submitted job. Without `--endpoint`, stop any process holding the data directory before starting standalone backfill. Ctrl-C cancels a standalone run, so the next `serve` does not resume it; a second Ctrl-C exits at once. A standalone job is named by processor, chain and range. When an earlier run of the same range recorded another verification policy, for example before `raw_history` was enabled, the store refuses the new input as a different job. Delete the earlier one with `DELETE /admin/v1/materialization-jobs/{id}`, or submit through `--endpoint`, which uses a fresh key per submission. Xatu exports have table-specific coverage. In particular, a recent beacon file does not prove that the execution-transaction file needed for receipts has been published. Missing or incomplete partitions can fall back to the verified P2P bridge; dataset-only configurations need another available history source. Finality or peer failures are reported rather than silently removing the configured bridge. ### JSON-RPC limits Each limit has a `rpc` setting: | Setting | Default | When exceeded | | ---------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max_batch_requests` | 100 | the batch is refused whole with one `-32600` error | | `max_response_bytes` | `16MiB` | each call past the limit is answered with `-32005` | | `max_log_results` | 10000 | `eth_getLogs` fails with `-32005` and a smaller block range to retry | | `max_log_addresses` | 1000 | the filter fails with `-32602` | | `max_log_topic_alternatives` | 1000 | the filter fails with `-32602` | | `max_subscriptions_per_connection` | 128 | `eth_subscribe` fails with `-32005` | | `max_websocket_connections` | 256 | the WebSocket upgrade gets HTTP 503 | | `max_outbound_bytes` | `64MiB` | queued notifications of all WebSocket connections; when full, the connection holding the most is closed with `1013` | | `max_subscription_event_bytes` | `16MiB` | a WebSocket connection whose subscriptions produce more notification bytes for one chain event is closed with `1008`, and one that leaves more unread with `1013` | A closed WebSocket connection releases its slot and its subscriptions. Notifications are queued one at a time, so while earlier ones are still unsent, a connection can be closed with `1013` before one event’s notifications pass the limit that would close it with `1008`. Each connection holds at most `max_subscription_event_bytes` of queued notifications and one response to a call; it reads its next call only once that response is sent, so a slow reader slows only itself. A connection that keeps more than a quarter of its notification room queued for 30 seconds is closed with `1013`: it reads slower than its subscriptions produce. All WebSocket listeners also share `max_outbound_bytes` of queued or sending notifications (64 MiB by default). When that runs out, the connection holding the most of it is closed with `1013`, not the one that notifies next; responses never use it. A send may take 30 seconds plus the time its payload needs at 64 KiB/s before the connection is closed. These are encoded-payload limits, not process RSS limits: at the defaults, responses add at most `max_websocket_connections` × `max_response_bytes`. `max_outbound_bytes` must be at least `max_subscription_event_bytes` and at most 4,294,967,295 bytes. `rpc.transaction_locator` is deprecated and ignored: nothing reads it. The node still accepts it, with a warning in its log and in `leani doctor`; remove it. ### Delivery batching History and live delivery use independent transport limits under `api.delivery.history_batches` and `api.delivery.live_batches`. The defaults are byte-first (`4MiB` history, `64KiB` live), with hard encoded-byte, event, processed-block, and maximum-delay bounds. History subscriptions may request smaller effective limits; those values are persisted with the subscription and returned by its status endpoint. `maximum_buffered_batches` and `maximum_buffered_bytes` independently cap the per-connection encoder queue; backpressure stops the encoder producer before either bound is exceeded. SDK delivery sessions refuse an NDJSON record over 65 MiB with a non-retryable `invalid_response` error. A record carries up to `maximum_encoded_bytes` of changes plus its own fields, so for SDK clients keep each lane’s `maximum_encoded_bytes` at or below 64 MiB; raising it, and `maximum_buffered_bytes` with it, beyond that lets a record exceed what the SDK accepts. Opening a history stream may request only stricter byte/event/block/delay limits through query parameters; these are connection-local and do not mutate the durable subscription. A live stream, including one the SDK’s `streamLive()` opens, takes no such parameters and uses `live_batches` as configured. The SDK’s `latency`, `balanced`, and `throughput` profiles are convenience functions that expand to those concrete numbers and are resent after reconnect. `compression = "gzip"` is negotiated through `Accept-Encoding`; the response is one continuous gzip member with an explicit sync flush after every NDJSON record, so batches and heartbeats are observable immediately by streaming decoders. Set it to `"none"` to disable compression for that lane. ## Pre-launch schema policy The project is not live yet and has no compatibility contract for existing consumers, config files, or databases. Configuration, SDK, API, and SQLite schemas may be replaced when that produces a cleaner final design; development databases should be recreated after an announced incompatible change. Durable delivery and processor encodings remain explicitly versioned so upgrade-safe replay can become a launch requirement without changing the public cursor model. # Operations runbook > Deploy, observe, back up, restore, compact, and recover a Leani node using its explicit health and storage contracts. ## Deployment profiles `deploy/container.toml` is a safe historical-only profile. It starts the query API plus HTTP and WebSocket RPC on all container interfaces, but Docker Compose publishes them only on loopback. It does not claim live or finality readiness. Its processor uses on-demand history, so nothing is indexed until you run a bounded `backfill` against the same data volume. `serve` holds the data directory’s lock, and a backfill beside it fails with `runtime data directory /var/lib/leani is already in use by another Leani process`, so stop the service first. `docker compose run` mounts the service’s volume, and its arguments replace the image’s `serve` command, so name the profile with `--config`: ```bash docker compose stop leani docker compose run --rm leani backfill --config /etc/leani/node.toml \ --processor blobs-money --from-block 19426589 --to-block 19427588 docker compose up -d leani ``` Keep `LEANI_API_TOKEN` exported for these commands too: Compose resolves it for every command. The processor starts at block 19,426,589 and is block-local, so later ranges may follow in any order. Then query the range. Subscriptions remain quiet until a live source commits events. A live-enabled node with a verified finalized head also accepts the CLI over its native API, without stopping it: ```bash leani backfill --endpoint http://127.0.0.1:18080 \ --processor blobs-money --from-block 19426589 --to-block 19427588 ``` The standalone form attaches the same verified P2P history fallback when `sources.live.kind = "p2p"`; the historical-only container profile above leaves P2P disabled. Use a live-enabled configuration when missing Xatu partitions must be recovered from peers. The profile protects the API with a bearer token from `LEANI_API_TOKEN`: `serve` fails to start without one of at least 16 printable ASCII characters without spaces, so export it, for example `export LEANI_API_TOKEN="$(openssl rand -hex 32)"`, before `docker compose up`, which stops with an error when it is unset, or pass it with `docker run --env LEANI_API_TOKEN`. The container’s health check calls `/health/live`, which needs no token. The profile opts JSON-RPC, which has no authentication, into its all-interface binds with `rpc.allow_unauthenticated_remote`; keep its ports on loopback or behind an authenticating proxy. The API answers only `Host` names in `api.allowed_hosts`, besides `localhost` and IP addresses; the profile lists the compose service name `leani`, so other services can call `http://leani:8080`. To add the names your clients use, mount a copy of the profile with them over `/etc/leani/node.toml`. The profile runs `blobs-money` as the instance `blobs-container-1-6`. A new processor version that changes the processor’s identity takes a new instance ID, because the store refuses another version under an instance it holds. Version 0.1.0-rc.1 ran `blobs-money` 1.4.0 as `blobs-container`, and rc.2 to rc.8 ran 1.5.0 as `blobs-container-1-5`, so on an earlier volume the new image starts beside that instance and leaves it inert: nothing indexes or serves it, but its rows stay in the volume and count toward `[budgets.store] maximum_physical_bytes`. This upgrade is itself an instance replacement: backfill subscriptions on the earlier instance show `processorConfigured: false` and keep counting against `[budgets.delivery] maximum_history_retained_bytes` until you settle them as step 2 of [Processor rebuild or rollback](#processor-rebuild-or-rollback) describes. For a clean slate, remove the volume with `docker compose down -v` before `docker compose up --build`, or restore a snapshot of the volume. The profile runs at most two history chunks at once (`[budgets.history_pipeline] maximum_active_chunks = 2`): a Xatu backfill holds about 450 MiB per active chunk, so two fit its 1 GiB `memory_bytes` budget and leave the rest of the compose file’s 2 GiB container limit to the node. The container’s root filesystem is read-only. The image points `SQLITE_TMPDIR` and `TMPDIR` at `/tmp/leani`, where SQLite spills statement journals and sorts that outgrow memory, and the compose file mounts a tmpfs there that the node’s user can write. Without a writable temporary directory, large statements fail with `disk I/O error` (SQLite code 6410), such as a finality advance over many blocks or a peer store refresh; mount one when you run the image another way. For a direct binary deployment: 1. install the binary as `/usr/local/bin/leani`; 2. create an `leani` system user; 3. place a validated config at `/etc/leani/node.toml`; 4. install `deploy/leani.service`; 5. give the service user write access only to the configured data directory. Keep bearer tokens in `/etc/leani/environment`, not in TOML. Public exposure should go through a TLS reverse proxy with request and connection limits. ## Health and metrics * `GET /health/live` checks that the process serves requests and SQLite answers a trivial query. It reads no tables, so it stays fast on large stores. * `GET /health/ready` additionally checks every required live/finality actor from in-memory readiness state. * `GET /debug/network` serves the built-in P2P and coverage dashboard. * `GET /v1/network/status` returns the dashboard’s machine-readable snapshot. * `GET /metrics` returns bounded-cardinality Prometheus text metrics. Store-wide and per-processor row and byte counts are recounted at most every 10 seconds and can lag by that much. `/v1/network/status` shares the per-processor counts. * `GET /v1/processors/{id}/status` reports coverage independently of process health. A historical-only node may be ready while a requested range is incomplete; the range query still fails with `range_incomplete` unless the caller opts into partial data. A live profile remains not ready until its independently anchored live and finality actors report ready. Live readiness also drops while retained recent frames sit at `budgets.recent_raw_hard_bytes`: live ingestion waits there, logging `recent_storage_full`, until finality prunes older frames (see [Storage pressure](#storage-pressure)). A parked processor lane does not affect readiness (see [Parked processor live lane](#parked-processor-live-lane)). With `sources.live.kind = "p2p"` and either supported finality backend, `serve` verifies the configured weak-subjectivity checkpoint, seeds the finalized execution anchor, starts one shared P2P subscription, and launches cold backfills for every processor that has a capable history source. A source failure drops readiness immediately and the supervisor retries with bounded exponential backoff (see [Shutdown and supervision](#shutdown-and-supervision)). The live lane delivers `unattested` blocks at most two slots past the newest attested head: the execution header of a light-client optimistic update that at least two thirds of the sync committee signed, which the finality source verifies within the slot it appears (see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes)). When a processor requires attested frames, it includes blocks only up to that head. When no new attested head arrives for four slots, for example while the finality transports are down or sync-committee participation stays below two thirds, the lane reports itself disconnected and live readiness drops until one does. Configuration refuses `sources.live.kind = "p2p"` with `finality.kind = "disabled"`, because nothing would verify those heads; use `sources.live.kind = "disabled"` for finalized-dataset backfills. When live readiness drops this way while finality stays ready, run `leani source probe finality`. Each Beacon endpoint whose finality verified reports its `attested_head`, or in `attested_head_error` why it gave none: it served no optimistic update, its update anchors no head, or its update failed verification. The lane needs one endpoint that serves `/eth/v1/beacon/light_client/optimistic_update`; replace endpoints that serve none. With `finality.kind = "consensus_p2p"`, peers must serve the `light_client_optimistic_update` request; the debug log names each peer that served none. The network view reports the process-wide execution P2P manager. Live, history, and probe requests share its identity, discovery state, peer pool, request gate, and Reth reputation state. Each row includes: * its current phase and active/stopped state; * connected and known-peer counts; * the requested block range and attempt count; * how recently it changed; and * the most recent bounded error message. The view deliberately omits peer identities and retains only 32 stopped manager records. It also reports cumulative material requests by outcome and the time spent waiting for local peer-capacity slots. Physical peer sessions are counted by establishment and disconnect, with disconnects grouped by Reth’s protocol-level reason. A high queue wait is a local scheduling signal; a timeout occurs only after dispatch capacity was obtained. Prometheus exposes the same state under the `leani_p2p_*` metric prefix. The execution identity is stored as `execution-p2p-secret` in `data_dir`. Treat it as node identity material: keep it across restarts, restrict its permissions, and never publish it. `execution-network.sqlite` is a disposable, bounded operational store capped by `sources.live.peer_store_max_entries`. It contains both broad discovery candidates and Leani’s independently observed service evidence: verified header/body/receipt success times, highest served block, smoothed response latency, last failure classification/time, fork compatibility, and latest anchor qualification. Reth’s known-peer snapshots and new service observations are committed by a dedicated background writer, so SQLite I/O never stops the network-manager event loop. Incremental upserts retain candidates absent from one short or failed run; indexed quality ordering decides which records survive the bound. Standalone feeds merge the SQLite evidence across sibling local Mainnet contexts, so a body- or receipt-serving peer learned by one processor is preferred by another without sharing processor data or the local P2P identity secret. The peer store is reconstructible and intentionally separate from the authoritative processor database and its backups. Warm startup uses two cache tiers. A small hot tier contains recent body-serving peers whose last verified body success is newer than their last recorded failure, with IPv4 `/16` and IPv6 `/32` diversity preferred. Those peers are dialed immediately. All other records stay in the broad tier and enter Reth’s dialer in `body_serving_peer_target`-sized batches at the configured peer refill interval. The segment-correct EIP-1459 DNS seeder and Reth’s native Discv4/Discv5 discovery run concurrently from the beginning. Reth owns the only Discv4 service and the bounded dial scheduler. `/v1/network/status` groups candidate admissions, established sessions, and qualification outcomes by `cached_hot`, `cached_broad`, `dns_tree`, `trusted`, or `discv4_or_5`. Prometheus exports the same origin labels through `leani_p2p_peer_candidates_admitted_total`, `leani_p2p_peer_sessions_by_origin_total`, and `leani_p2p_peer_qualifications_total`. Use these counters when comparing cold and warm startup instead of treating cache size as evidence of usefulness. Every new ETH session is immediately tested against the current independently verified execution anchor. Leani qualifies several sessions concurrently until the configured body-serving target is ready, then one at a time. Qualification is background work at normal priority throughout: it never uses the request slots reserved for the live lane. Qualification ranks proven serving peers first but does not exclude a newly connected peer: each material response is independently verified before use, so fresh peers remain immediate fallbacks. Proven serving peers are tried before untested discovery records on the next start. `sources.live.material_request_concurrency` bounds simultaneous physical requests together with the global source budget and connected-peer capacity. Every execution P2P request shares one process-wide limit: the larger of `material_request_concurrency` and `history_header_request_concurrency`, 32 by default. History, probes, and peer qualification leave four of those slots free for the live lane (half the limit when it is below eight), and a waiting live request gets the next free slot first. Each source acquisition’s `max_in_flight_requests` and the four-request per-peer pipeline usually impose a lower effective limit. More concurrency helps only when the peer pool and link have spare capacity; it also multiplies decoded response buffers. Treat `budgets.source_concurrency` and `budgets.history_pipeline.maximum_active_chunks` as memory-affecting settings and tune them together while watching RSS and the material high-water metrics. `material_request_blocks` sets the blocks per body or receipt request; the evidence-backed default is eight. A peer that stops a reply at its response size limit serves only the first of them: those are kept and the rest is requested again. A failed request is retried block by block without narrowing other requests. Raise it only with repeated real-source evidence for the deployment’s peer and bandwidth profile. For address/topic-filtered log processors, P2P checks the verified header bloom before requesting receipts. A negative avoids the receipt bundle; a positive still downloads all receipts for that block because the ETH protocol has no server-filtered receipt request. Bodies are fetched only when the descriptor’s log identity requires transaction hash. Therefore processor requirements, pool activity, bloom false positives, and receipt density can matter more than the configured concurrency ceiling. For an inbound-capable deployment, set `sources.live.listener_port` and `sources.live.discovery_port` to `30303`, set `sources.live.discv5_port` to a distinct fixed UDP port such as `30304`, expose all three sockets, leave `enable_discv5 = true`, and configure `sources.live.nat` for the environment. Zero ports remain useful for isolated tests. Inbound reachability improves diversity but is not required for verification. `sources.live.trusted_peers` may contain stable operator/community `enode://` seeds that supplement public discovery without turning the pool into an allowlist. `sources.live.minimum_peers` is the hard connected-session floor before execution requests may start and should normally remain `1`. Capability qualification does not gate startup: it ranks proven peers first while commitment validation remains mandatory for every response. `body_serving_peer_target` controls how many peers Leani concurrently proves can serve commitment-valid bodies; reaching it does not delay startup after the minimum is available. `preferred_peers` is a soft operational target and never blocks ingestion. The Reth manager continues filling connections beyond that target in the background up to `max_outbound_peers`. Header, body, and receipt failures are isolated to their material lane. A transient miss therefore cools that lane without disconnecting a session that may still be useful to another processor; cryptographically invalid material continues to ban the peer globally. If the pool remains at zero connected peers for `sources.live.peer_recovery_timeout_seconds`, the node rebuilds the complete execution network manager and restarts Discv4, Discv5, and DNS bootstrap. This recovers stale sockets after host sleep and retries a failed cold-start bootstrap without operator action. The default is five minutes: discovery and outbound dialing continue throughout that window, allowing Reth’s per-peer backoff and candidate rotation to work before the underlying sockets are recycled. The mainnet DNS root is also resubmitted every 15 seconds so a transient resolver failure during the first lookup does not wait for the whole-manager watchdog. Each rebuild logs `building the execution P2P network manager` and then `built the execution P2P network manager` or, at warn level, `failed to build the execution P2P network manager` with its error. While the live lane reconnects, it logs each failed attempt at warn level in `the live lane could not reconnect to the execution network; retrying`. A fixed Discv5 port can stay busy for up to a minute after a rebuild, while the ended manager’s last discovery lookup finishes; the lane keeps retrying meanwhile. Processor `coverage.complete` means only that the stored range from the configured start through `processedThrough` has no internal gaps. It does not mean that `processedThrough` has reached the live source. The network status adds `storedRangeContiguous`, `syncTargetBlock`, and `blocksRemaining`; use those fields together with `readiness.liveReady` when diagnosing catch-up. `syncTargetBlock` is derived from the live session’s `observedHeadBlock`: the newest header the live lane has validated, never a head a peer only claimed. It reports validated progress, so while the node catches up it trails the network head. `fromBlock` and `toBlock` describe only its current material request. A speculative request for the block after the observed head therefore does not create artificial processor lag. `blocksRemaining` counts every missing block from the configured start through the observed head, including internal gaps; it is not merely the distance between the highest stored block and that head. `readiness.ready` means the required live and finality network lanes are usable. It can be true while a processor backfills older gaps, so the dashboard reports that combined `live + backfill` state explicitly. The `network.supervisor` object reports `running`, `backing_off`, or `stopped`, including its bounded last error, cumulative failure count, and retry countdown. A zero active-session count during `backing_off` is a supervised retry, not an idle healthy node. The live/archive overlap may map the same canonical block at different finality levels. The runtime validates all finality-only checksum variants, keeps the strongest stored finality, and removes the redundant pending delta. Any difference that cannot be reproduced by changing finality still fails closed as a real transform or source conflict. At a restart, startup reconciliation instead deletes every pending delta for a block the processor already applied, without comparing it, before opening any network lane, so archive availability or a slow P2P catch-up cannot delay cleanup; its warning counts them in `discarded_pending_deltas`. `processors[].pendingDeltas` exposes any durable rows that remain afterward. `finality.kind = "consensus_p2p"` is the self-contained profile. Configure both the independently obtained checkpoint root and its finalized beacon slot. The node uses signed embedded mainnet bootnode ENRs unless `finality.bootnodes` overrides them, binds discv5 to `finality.discovery_port` (`0` selects an ephemeral port), performs the mandatory Status exchange, and locally verifies bootstrap, sync-committee, BLS, finality, and execution-payload proofs. Permit outbound TCP and UDP; an inbound UDP mapping improves discovery but is not a trust requirement. `finality.kind = "beacon_api"` remains available when HTTP consensus transports are operationally preferable. Configure `finality.minimum_agreement` above one when independent transports are available. Each poll waits about two seconds after the first verified endpoint for the others, then follows the highest finalized slot that `minimum_agreement` endpoints have reached or passed, so a lagging endpoint neither holds finality back nor drops readiness at an epoch transition. Finality never moves backwards, and two verified endpoints that report different roots at one slot fail closed. Endpoints that name one transport twice, for example with and without a trailing slash, fail validation because each would count toward agreement. See [Checkpoint lifecycle](#checkpoint-lifecycle) for how the configured checkpoint is used and when it must be refreshed. On-demand RPC shares the configured source-concurrency ceiling. The node divides its memory and temporary-disk budgets across those request slots, enforces a one-minute request timeout and the advertised maximum log range, tries an alternate capable source only before returning a response, and never stores fetched frames in the recent cache. Put additional connection and request-rate limits at the reverse proxy when exposing this potentially expensive path. Suggested alerts: * liveness failure: page immediately; * required live/finality readiness false for 5 minutes: page; * processor head stops advancing for 3 expected block intervals: warn; * pending-delta or outbox bytes exceed 80% of configured budget: warn; * database size approaches the filesystem limit: warn at 70%, page at 90%; * finalized cursor does not advance for 20 minutes: warn; * `leani_finality_anchor_expiry_seconds` below three days: warn; * any finalized contradiction or store-integrity failure: stop publication and page immediately. Import `observability/grafana/leani.json` for the release dashboard and load `observability/prometheus/alerts.yaml` into the Prometheus-compatible rule evaluator. The checked rules cover missing scrapes, required-component readiness, stalled processor/finality cursors, a non-draining outbox, and a finality anchor close to its age limit. The stalled-cursor rules apply only where the profile requires the live source (`leani_live_required`) or finality (`leani_finality_required`), so a historical-only node whose backfill has finished does not alert. They join processor series to those gauges on the scrape target’s `job` and `instance` labels. Keep `honor_labels` disabled, the default, for the Leani scrape job: Prometheus then stores each processor’s own `instance` label as `exported_instance`. Filesystem percentage alerts remain the host operator’s responsibility because the node exports its database/recent-cache byte counts but not host capacity. ## Shutdown and supervision `serve` stops at Ctrl-C or SIGTERM. Open change streams (SSE) and consumer delivery streams (NDJSON) end after a whole event or record, and JSON-RPC WebSocket connections get a `1001` going-away close frame; clients reconnect from their last cursor as after any closed connection. The node then waits at most 10 seconds for open connections and its background work, and at most 5 more for blocking work such as a file write, abandons whatever is still open, and exits 0. A second signal during the first 10 seconds exits at once with status 130; startup reconciliation repairs a commit it interrupted (see [Crash or interrupted commit](#crash-or-interrupted-commit)). `deploy/leani.service` (`TimeoutStopSec`) and `compose.yaml` (`stop_grace_period`) allow 30 seconds for the stop; give another container setup more than Docker’s default 10 seconds. Background work runs in supervised tasks: the durable backfill and raw-history schedulers, the network lane supervisor, artifact compaction, query snapshot cleanup, and each processor’s delivery pruning and coverage compaction. Each runs until the shutdown. A task that stops before it, by returning, failing, or panicking, shuts the node down: the log names the task in `a background task ended; shutting the node down` at error level, and the node exits with status 1 after the same bounded shutdown. The systemd unit (`Restart=on-failure`) and `compose.yaml` (`restart: unless-stopped`) start it again. A persistent execution P2P source that cannot be built, for example for an invalid `sources.live.nat`, stops the node the same way. The network lane supervisor restarts failed runs of the network lanes itself, and drops readiness whenever the lanes do not run, a panic included. A failed run restarts after 1 second, and each further failure doubles the wait up to a minute; after a run that stayed up for at least a minute, the wait starts at 1 second again. A run that ends waits for its cold backfills to stop before the next one starts; every 10 seconds of that wait, the log names the ones still running in `cancelled automatic cold backfills are still running` at warn level. When the lanes halt (see [Finality contradicts the followed chain](#finality-contradicts-the-followed-chain)), they stay stopped and not ready while the node keeps serving what it stored. A live lane that stays not ready while verified finality is ready for `sources.live.stall_timeout_seconds`, ten minutes by default, within one run of the lanes is treated as stalled. Usually its execution P2P network is wedged, which a lane restart cannot heal; it can also be a finality endpoint that stops sending attested heads, which a restart may not fix. The lanes then stop, the log says `the live lane stalled while verified finality stayed ready; the node exits so that it restarts` at error level, and the node exits with status 1 for systemd or Docker to start it again. While finality is not ready either, the network itself is likely down and the node keeps waiting, and a wait at the recent-frame hard limit does not count (see [Storage pressure](#storage-pressure)). A run that fails quickly, as when another process holds the P2P ports or `persistent_retries = false` gives up, never reaches the timeout: the supervisor restarts it with backoff and logs each failure. Run the node under a supervisor that restarts it, or raise the timeout, up to 30 days. ## Checkpoint lifecycle `finality.checkpoint`, with `finality.checkpoint_slot`, is the trust root, and it is needed only for the first start. That start verifies a light-client bootstrap for the checkpoint, which must be at most 14 days (336 hours) older than its slot. From then on, the finality source keeps one bootstrapped light client, one per endpoint for `beacon_api`, and advances it with every finality update, plus the sync-committee update when a period ends. Nothing is bootstrapped again, and no age check runs while the node keeps running. A `beacon_api` endpoint that was down at start, or bootstraps slowly, bootstraps later from the newest anchor the endpoints agreed on, so it can still join after the configured checkpoint has aged out. Its bootstrap keeps running across polls instead of restarting each time. The node persists the newest verified finalized anchor as `finality-anchor.json` in `data_dir`. It writes a temporary file, syncs it to disk, and renames it over the old one, so a crash leaves either the previous anchor or the new one. The file records the anchor’s beacon slot and root, its execution block number and hash, the configured checkpoint it was verified from, and a format `version`. It holds no secrets. Only anchors at an epoch’s first slot are written, because Beacon nodes serve bootstraps for epoch-boundary blocks, so the file trails finality by about one epoch. An older anchor never replaces a newer one. Every start, and every restart of the network lanes, bootstraps from the persisted anchor instead of the configured checkpoint when both hold: * the anchor was verified from the configured checkpoint, so it is newer by construction; and * the anchor is at most 14 days old. An anchor verified from another checkpoint is never used: changing `finality.checkpoint` re-anchors the node on the new root, and the node logs a warning naming both roots. `leani doctor` reports the mismatch too. The one exception is an embedded subscription (`leani subscribe` without a running node) that reuses a cached checkpoint, `checkpoint.json`, whose trust is `locally_verified`: an anchor the subscription verified itself on an earlier run. A persisted anchor at a later slot replaces it. A checkpoint from a provider quorum follows the rule above, whether the subscription has just accepted it or reuses it from the cache with trust `provider_quorum`. The bootstrap is still verified against the anchor’s block root, and the configured checkpoint’s own age is not checked. A missing, corrupt, or unreadable file, or an anchor past the limit, falls back to the configured checkpoint with a warning. So does a persisted anchor whose bootstrap no endpoint or peer serves. Startup fails only when the configured checkpoint is unusable too, for example because it is older than 14 days. `leani source probe finality` and the benchmark’s history probe start from the persisted anchor by the same rules, but never write the file. The persisted anchor stops advancing while verified finality stalls, while the node is stopped, and while writing the file fails. A failed write is logged and counted, and never fails finality itself. Watch these: * `leani_finality_anchor_age_seconds` is the age of the newest verified anchor; * `leani_finality_anchor_expiry_seconds` is the time left before a restart can no longer bootstrap from it, and turns negative once it has passed; * `leani_finality_anchor_write_failures_total` counts failed writes of `finality-anchor.json`: while it grows, a restart may start from an older anchor than these gauges show; * the `LeaniFinalityAnchorExpiring` alert fires when fewer than three days remain; * `leani doctor` warns when the anchor a restart would use expires within three days: the persisted anchor, or the configured checkpoint if none applies. To refresh the trust root, take a recent finalized checkpoint root and slot from independent providers, set `finality.checkpoint` and `finality.checkpoint_slot`, and restart: the persisted anchor verified from the old root is then ignored. `leani reset all` removes the file with the rest of the runtime state. Errors, logs, and probe reports show endpoint and provider URLs as `scheme://host[:port]`, with `/…` in place of any path, because providers put API keys in userinfo, query strings, and paths. Request errors add the API path, such as `eth/v1/beacon/genesis`. Endpoints that would show alike carry their index, such as `finality.endpoints[1]`. ## Backup, restore, and compact Stop the node and any embedded subscription or standalone backfill using this data directory before running `leani db` commands (including `inspect`, `verify`, and `backup`). They take the same `.leani.lock` as other writers, and every one but `backup` opens the store with migrations enabled, which upgrades a store from an earlier release. Use the API and `/metrics` for inspection while the node is running. Create a consistent backup after stopping those writers: ```bash leani db backup /backups/leani-$(date +%s).sqlite \ --config /etc/leani/node.toml ``` `db backup` reads the store through a read-only connection and never upgrades it: the backup keeps the store’s schema, so the backup of a store from an earlier release restores with that release’s binary. It refuses a store whose schema is newer than the binary supports, and a data directory without a store. That command is intentionally SQLite-only. When any processor artifact segments exist, as with `artifact_storage.backend = "tiered_segments"`, it fails before writing a backup rather than producing an incomplete restore. Until a versioned bundle-backup command ships, stop the node and snapshot the complete `data_dir` (SQLite database/WAL plus `processor-artifacts`) or use an atomic filesystem/volume snapshot. Verify the backup using an otherwise identical config whose `data_dir` points to an isolated directory, with the binary that will open it: a newer binary upgrades the copy. Never overwrite the active database. Stop the service, move the active database aside with its `leani.sqlite-wal` and `leani.sqlite-shm` files, since SQLite would replay a leftover WAL onto the restored copy, copy the verified backup into place, then start the previous compatible binary. Retain the moved database until coverage, processor counts, cursors, and subscriptions are verified. `db compact` checkpoints the WAL and vacuums the active database; immutable artifact segments need no rewrite. The rewrite also turns on SQLite incremental auto-vacuum, which new stores have from creation, so pages freed by pruning return to the filesystem. A store created before auto-vacuum was enabled logs a warning at every start and never shrinks below its high-water mark; run `db compact` on it once. Like every `leani db` command, it runs only while the node is stopped, and it needs free disk for a second copy of the database, because the vacuum rewrites the whole database. `db prune-changes` is lease-aware; expired consumers must perform the documented snapshot/reset flow. ## Failure runbooks ### History schema or source failure Stop the affected backfill, keep the last committed coverage, run the source probe, and compare its schema/source plan with the last benchmark report. Switch to another capable configured source at the missing range boundary. Never mark failed chunks complete. ### P2P outage or invalid material Readiness must become false when the live session is unavailable. Open `/debug/network` or inspect `/v1/network/status` before treating every zero-peer message as a live outage. Preserve the recent and undo windows, stop unfinalized publication before a hard storage limit, and compare local queue wait with peer timeouts. The production source keeps one persistent broad peer pool, penalizes invalid material through Reth, retries without discarding healthy connections, and caps recovery backoff. Invalid commitments are permanent evidence against that response, not retryable data. Archive lag is not a live outage. The reconciliation cursor remains at the first unverified finalized batch and tries interchangeable history sources again. A durable `failed` archive reconciliation is a correctness incident: preserve the database and source object, stop publication, and compare the processor version/configuration and block identity before resuming. Reconciliation compares only exact finalized coverage, which keeps each block’s identity and mapped-delta checksum. It skips blocks nothing covers, and blocks that coverage compaction reduced to proof segments, logging the latter once as `archive reconciliation cannot compare blocks compaction reduced to proof segments; they stay unaudited`. Compaction keeps the blocks after a processor’s first verified hot/cold handoff exact until a verified reconciliation passes them, unless they lie more than 65,536 blocks below finality, so only an audit stalled that long loses blocks to compaction. ### Finality outage or disagreement Continue only unattested or attested publication. Do not advance or prune the finalized cursor. A disagreement or contradiction is fail-closed and requires operator review of checkpoint provenance and independent endpoints. A single bad transport is not an outage: consensus P2P bans a peer whose light-client material fails verification, asks the next one, and neither asks nor dials it again, across reconnects, until the lanes restart. Stale updates are retried, so neither ends the finality stream. An exhausted peer set logs a warning at most every five minutes. ### Network fork this release does not support Each release verifies the consensus forks it knows; Glamsterdam’s Gloas is not among them yet. Once mainnet activates a fork a release does not know, finality stays unavailable and the live lane stops, fail-closed: consensus peers answer with a fork digest the node does not know, which it logs once as `consensus peers serve a fork this release cannot verify`, and a Beacon API endpoint fails with an error naming the fork, such as ``serves `gloas` light-client data``. No peer is banned for it. Upgrade to a release that supports the fork before its mainnet activation. Upgraded within 14 days, the node resumes from its persisted finality anchor; later, it needs a fresh `finality.checkpoint`. ### Live lane disconnected during a finality apply Symptom: right after finality advances, the live lane reports itself disconnected with `no sync-committee-attested execution head above block … for … seconds`, live readiness drops, and no block is included, although the finality sources are healthy. It is most likely every 1,000 finalized blocks, when each processor with automatic checkpoints copies its whole state into a recovery checkpoint, and with large processor states. Cause: both finality sources verify and publish attested heads only while the node reads their finality stream, and the node applies each verified finality update, checkpoints included, before it reads the next. An apply that takes longer than about 36 seconds leaves the live lane without a new attested head for its 48-second grace, four slots. What to expect: nothing is lost or reverted. When the apply completes, the next attested head arrives, the lane catches up on the blocks produced meanwhile, and readiness returns. This is a known limitation of this release; a later release publishes attested heads independently of finality applies. ### Finality contradicts the followed chain When verified finality names another block at a height than the retained canonical chain holds, or a retained block below the finalized block does not link to it, the node neither finalizes either chain nor defers forever. A contradiction with retained *unfinalized* blocks, for example after the lane stalled across a reorg of an attested head, heals itself. The node logs `verified finality contradicts retained unfinalized blocks; restarting the network lanes to revert them` as a warning and restarts the network lanes after the supervisor’s backoff. The restart reverts only the retained unfinalized blocks that are not on the newly verified finalized chain, and undoes processor coverage of them as a reorg, logging `retained unfinalized blocks do not link to the verified finalized anchor; reverting them as a reorg`. Those blocks are never finalized; the cold backfill and the live lane fetch the finalized chain again. Every start, and every restart of the network lanes, decides which retained blocks are on that chain the same way. Where parent links from them do not reach the anchor, as after a contradiction or an outage longer than the finality lag, the node proves the blocks on the anchor’s chain with execution headers from its peers that link by parent hash down from the anchor, keeps and finalizes them, and logs `proved retained unfinalized blocks against the verified finalized anchor` with the blocks kept and reverted. When it cannot prove them within 60 seconds, for example without execution peers, it logs `could not prove retained unfinalized blocks against the verified finalized anchor; reverting them` as a warning; when the proof would span more than 8,192 blocks, `retained unfinalized blocks are too far below the verified finalized anchor to prove; reverting them`. It then reverts every retained unfinalized block that does not link to the anchor. The network lanes halt instead when the contradiction is with *finalized* history, which no revert repairs, or when the same unfinalized contradiction comes back a third time before finality moves to another block, so two restarts did not repair it. The node then logs `verified finality contradicts the retained canonical chain; network lanes halt until the node restarts` at error level: readiness stays false, `network.supervisor` reports `stopped` with the contradiction as its last error, and nothing retries it. The API keeps serving what is stored. 1. Preserve the database and logs. Check the checkpoint provenance and the finality transports: a halt means the node followed another chain than the one consensus finalized and could not revert it, or a trust root is wrong. 2. If the error says `verified finality contradicts finalized canonical history`, finalized history in the store is wrong and no revert repairs it: rebuild in a new `data_dir`. 3. If it says the contradiction `recurred after 2 restarts of the network lanes`, restart the node once the finality transports check out: startup reverts the unfinalized blocks as above. If it halts the same way again, rebuild in a new `data_dir`. ### Storage pressure Inspect per-processor attribution with `db inspect` and `/metrics`. Prune change history only through `db prune-changes`, respecting active leases. Never evict rollback data required by unfinalized blocks. Add capacity or stop unfinalized advancement before the hard limit. If the database file stays at its high-water mark after pruning, check the startup log for the auto-vacuum warning, stop the node, and run `db compact` once. Retained recent frames are the reorg and replay input, so the node never drops an unfinalized one to make room. When they reach `budgets.recent_raw_hard_bytes`, live ingestion stops before the next block, live readiness drops, and the log reports `recent_storage_full`. Ingestion resumes once a finality advance prunes older frames. After 15 minutes at the limit the live lane fails, and the supervisor restarts the network lanes from a fresh finalized anchor, which lets finality prune again; the live stall timeout does not count this wait. A node that keeps reaching the limit needs working finality, or a hard limit above the `rpc.minimum_recent_blocks` window plus the unfinalized tail. A processor’s delivery log is pruned in the background by its `delivery.pruning` policy, oldest change first: finalized changes older than its window, and, while the log exceeds its byte limit or the processor is paused at it, older finalized changes whatever their age, never one a required consumer has not acknowledged. A block-local processor’s history and live blocks share its log, so a live change still inside the window can precede older history. Pruning by age stops at that change. Pruning at the limit does not, since the window of a paused processor does not move; a best-effort consumer that had not read the change gets `reset_required`. A paused processor resumes once the log is below 90 percent of its limit. Embedded subscriptions (`leani subscribe` without a node) prune the same way while they run, so their built-in processors keep within 64 MiB and 24 hours of changes. Retained raw history has its own `[raw_history]` budgets. A raw-history store over either of them, for example after lowering one, still opens and serves its segments, but admits no new segment. A job created with `segment.onLimit: pause` waits at the storage limit, with the budget named in `lastError`, and resumes once the node runs with a larger budget; one created with `fail` fails. ### Retained raw-history segment failure The node verifies each retained segment’s whole-file checksum the first time it reads the segment after starting, and each block’s record checksum on every read. At startup it only checks that each segment file exists with its closed length, and logs what it recovered. A segment that fails never stops the node: it leaves the catalog, so its blocks read as missing and RPC and backfills use another source, its file moves to `data_dir/raw-history/quarantine` with a `.corrupt` suffix, and the node logs a warning. A segment whose file is missing at startup leaves the catalog the same way. A closed file no catalog row claims, left by a crash before its publication committed, moves to the quarantine with an `.orphan` suffix. 1. Check the raw-history jobs: each job that owned the segment names it and its blocks in `lastError`. A running job acquires those blocks again, and a completed one returns to the queue and does too. A failed or cancelled job keeps the error it ended with. 2. Check the disk and filesystem before trusting the node with more writes: a corrupt segment usually means a failing disk. 3. The quarantine keeps at most 1 GiB, or one segment of `raw_history.maximum_segment_physical_bytes` when that is larger. Past that, the files quarantined longest ago are deleted first, at startup and after each new quarantine, but the newest is always kept. Copy a file elsewhere to keep it for inspection; delete the others whenever you like. ### Retained raw-history segments unavailable When more than half of the catalogued segment files are missing at startup, as when the volume holding `data_dir/raw-history/segments` is not mounted yet or a restore is still running, the node takes the directory as unavailable rather than emptied. It keeps every catalog row, logs a warning, and counts them as `unavailable_segments` in its startup recovery log. A segment file that disappears while the node runs is treated the same way. Reads of those segments fail and fall back to other sources, and the jobs that own them do not acquire them again. Do not create raw-history jobs until the directory is back: a new job drops the rows of the missing segments nothing owns, and may replace others with segments it acquires itself. 1. Mount or restore the directory, then restart the node: the segments whose rows were kept serve again. A returning file whose row a new job dropped meanwhile is quarantined as an orphan. 2. If the files are gone for good, delete every raw-history job that owns them, cancelling running ones first, then create each again with the same request. The new job skips a segment whose file is gone, removes its catalog row once nothing owns it, which also frees its share of the budgets, and acquires the blocks from its sources. ### Parked processor live lane A failure that belongs to one processor parks only that processor’s live lane; shared ingestion, finality, and every other processor continue. This holds for every processor mode and delivery ordering. `/v1/processors/{id}/status` reports the lane as `paused` or `failed`. `/v1/network/status` adds its `runState`, `pauseReason`, and `liveGap` (the first unapplied block), and `leani_processor_run_state` carries the state. A paused lane resumes by itself. A failed lane is frozen: it commits no live block, and finality does not advance it. It is still mapped for a reorg’s replacement blocks, only to move its first unapplied block onto the new branch. Its first failure stands: a later failure does not change its reason, and a later park does not move its first unapplied block. Only `processor_finality_conflict` replaces the reason. A failed lane waits for an operator: fix the cause, then call `POST /admin/v1/processors/{id}/lanes/live/reset` with the header `x-leani-request: 1`, which works for every delivery ordering and pauses the lane so that it replays from its first unapplied block. A lane that fails again on the same block parks again; it does not stop the node. A reorg never leaves a parked lane’s first unapplied block on a block the chain no longer has. A reorg that replaces blocks at or below it moves it down to the first replacement block, so the lane replays the whole new branch. A reorg without a replacement branch moves it onto the new tip. When the lane has applied that tip, a paused lane resumes at once, and a failed lane’s first unapplied block moves onto the new tip’s successor once that block arrives. If the lane never applied the new tip, such as the finalized anchor the node seeds after downtime, a paused block-local lane’s gap simply completes, while an ordered lane waits there for that block from history. A failed block-local lane keeps its first unapplied block on that tip, so after a reset it waits for a history source to serve it (`finalized_gap_waiting_for_history_source`), a block it does not need: make sure a history source can serve the finalized anchor, and the lane applies it and resumes by itself. A first unapplied block found off the canonical chain, as an earlier version could leave one, moves back to the last canonical block the lane applied; the lane then replays from there instead of skipping blocks or failing. | Reason | State | Resumes when | | --------------------------------------------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `delivery_spool_hard_limit`, `physical_store_hard_limit`, `artifact_store_hard_limit`, `pending_delta_hard_limit` | paused | Consumers acknowledge or storage frees up. | | `unfinalized_gap_waiting_for_finality` | paused | Finality reaches the first unapplied block, which then comes from the history source. No retained frame can serve that block, for example because it was retained without this processor’s material. | | `finalized_gap_waiting_for_history_source`, `finalized_gap_recovery_unavailable`, `finalized_gap_recovery_incomplete` | paused | A history source can serve the finalized gap. | | `operator_reset_pending_replay` | paused | The replay after an operator reset reaches the live tip. | | `startup_reconciliation_replay` | paused | The replay of retained frames that [startup reconciliation](#crash-or-interrupted-commit) started reaches the live tip, normally before live blocks resume. | | `hot_cold_handoff_failed` | paused | The processor’s automatic cold backfill or its overlap verification failed; the handoff is recorded as failed. A block-local lane replays and resumes at once. An ordered lane resumes once a later backfill passes its gap. A start’s backfill runs only to the finalized head, so restart the node once `chainFinalizedHead` in `/v1/network/status` has reached the lane’s `liveGap`; an earlier restart leaves the lane paused. | | `processor_live_reduce_failed` | failed | Reset after fixing the processor or its input. Its reducer, or the store’s check of what the reducer wrote, rejected the block. | | `processor_live_mapping_failed` | failed | Reset after fixing the processor or its input. Mapping the block, or checking a pending delta’s finality variants, failed. | | `processor_live_delta_conflict` | failed | Investigate before resetting: a pending delta for an applied block carries different content, so the processor’s transform is not deterministic or its inputs differed. The applied block stands. Restart the node, whose [startup reconciliation](#crash-or-interrupted-commit) deletes that pending delta, then reset the lane. | | `single_block_exceeds_delivery_limit`, `single_delta_exceeds_pending_delta_limit`, `live_gap_marker_exceeds_pending_delta_budget` | failed | Reset after raising the limit. | | `live_gap_canonical_identity_changed` and the other `finalized_gap_recovery_*` reasons | failed | Reset after checking the history source against the canonical chain. For an ordered lane, `finalized_gap_recovery_canonical_mismatch` can also mean that the block history returned does not descend from the lane’s own last applied block, which is then off the canonical chain: restart the node, whose [startup reconciliation](#crash-or-interrupted-commit) undoes an unfinalized off-chain block, then reset. The lane then waits in `operator_reset_pending_replay` until the automatic cold backfill of the next start fills the undone heights, so restart the node once more; an on-demand processor waits for a backfill request instead. A finalized off-chain block needs a rebuild. | | `processor_finality_conflict` | failed | Rebuild the processor as a replacement instance; the reset route refuses this lane with `409 live_lane_requires_rebuild`. The same instance cannot be rebuilt in place today: configure a replacement instance, with a new processor version or configuration, and move to it as in [Processor rebuild or rollback](#processor-rebuild-or-rollback). Its coverage holds a finalized block hash at another height, or another block at a finalized height, so it contradicts the canonical chain, and finality stops advancing it. This reason replaces any earlier failure reason. | ### Crash or interrupted commit A live block commits in steps: the node first retains its frame as the new canonical block, then each processor applies it in a transaction of its own. A reorg likewise switches the canonical chain first and then undoes each processor’s reverted blocks. When an apply reaches a processor’s delivery limit, the store pauses or fails the lane first and the runtime then records its first unapplied block. A crash, a kill, the embedded subscription runtime’s shutdown abort, or a network-lane restart can stop between those steps. That leaves a processor behind the canonical chain or on a reverted branch, a pending delta that nothing applies, or a stopped lane with no record of where it stopped. Every network-lane start therefore runs startup reconciliation for each processor before any lane opens: 1. It undoes the processor’s unfinalized blocks that are not the canonical block at their height, newest first, and publishes their undo changes as the reorg would have. 2. It deletes pending deltas the processor can never apply: those below its start, for a block that is not canonical, and for a block it already applied. For a block-local processor that includes any below its first unapplied block, such as one an interrupted cold-backfill commit left; the backfill maps that block again. 3. It pauses a running lane that trails the retained canonical frames at its first unapplied one, with reason `startup_reconciliation_replay`, and replays those frames through the normal gap path before live blocks resume. A lane that has applied nothing yet, such as a newly configured block-local processor, replays the retained frames too. A block-local lane whose next block falls in a hole in the retained frames, such as blocks never retained over downtime, replays from the first retained frame above the hole; its cold backfill covers the hole. 4. A lane the store paused or failed at its delivery limit, but whose first unapplied block was never recorded, gets that block recorded and keeps its state and reason. A paused lane replays from there once consumers acknowledge and delivery capacity frees up, instead of skipping the blocks it missed, and a failed lane can be reset. The recorded block carries no required delivery size, so for `single_block_exceeds_delivery_limit` the reset route cannot refuse a limit that is still too small; the replay then fails the lane again on the same block. The store can also pause or fail a lane at its delivery limit while the lane is at the live tip, for example in its cold backfill’s commit on the lane’s live stream. That records no first unapplied block either, and at a restart reconciliation has no retained block to record yet. The live lane handles it with its next block: a paused lane commits that block as usual, so the store’s refusal parks the lane there, or applies it once capacity has freed up. A failed lane records that block, which a block-local lane has not applied or which follows an ordered lane’s last applied block, so a reset replays from there. A failed ordered lane still behind its history records nothing, and its history cannot apply while it is failed. Startup reconciliation records its first unapplied block, the one after its cursor, only while that block is among the retained frames. Once finality has pruned it, nothing records it: the reset route refuses the lane with `400 invalid_request`, since it has no durable live gap, and only rebuilding the processor as a replacement instance recovers it, as in [Processor rebuild or rollback](#processor-rebuild-or-rollback). A paused or failed lane whose first unapplied block is already recorded replays from there as usual, and a lane failed for another reason, such as `processor_finality_conflict`, is left alone. Reconciliation replays nothing below the retained frames. A processor’s automatic cold backfill fills those blocks up to the finalized anchor; an on-demand processor waits for a backfill request. Before its live lane applies a block past those blocks, an on-demand block-local processor logs “the live lane skips blocks this processor never processed” at warn level, with `from_block`, `to_block`, and `announced`, and its live stream announces the range in a `system.live_gap.put` change for the application to request. A startup replay from the first retained frame above a hole announces it the same way. A processor with `[processors.live_gap_fill]` (see [Live gap fill](/docs/operations/configuration-guide/#live-gap-fill)) fills a range of at most its `max_blocks` from history first, before it applies that block. The lane logs `filling the blocks the live lane skips from history` at info level, with `from_block`, `to_block`, and `blocks`, and `filled the blocks the live lane skipped` when it is done, with `applied`, `duplicates` (blocks a history subscription had applied), and `elapsed_ms`. Every live lane waits while a fill runs, at most five minutes plus applying the blocks of its last request: a request still running at five minutes is given up. `elapsed_ms` is how long the node held all of them. A longer range logs `the live lane skips more blocks than live_gap_fill.max_blocks; announcing them` at info level, and announces as above. A fill that cannot start or finish, because no history source serves the range, the range is not finalized yet, a source fails or serves a block that does not link, or the time is up, logs `could not fill the blocks the live lane skips; announcing the rest` at warn level, with `filled_blocks` and the `reason`. The stream then announces from the first block it did not fill, and the warning above follows. A lane that reaches a delivery or storage limit mid-fill stops there. While the store would refuse its next block, at the stream’s delivery limit, at a node-wide `[budgets.delivery]` or `[budgets.store]` budget, or at a storage limit while storage stays above its low-water mark, the lane fetches no history for the fill, as long as the block after the range is among the retained frames. Once finality prunes that frame during a long hold, the lane recovers it from history on every live block, as any paused lane recovers its first unapplied block. After a restart of the live lanes, a lane held at a node-wide budget can fetch once more before it waits again. Either way it reports `paused` with that reason (see [Parked processor live lane](#parked-processor-live-lane)), announces nothing, and continues the fill once the store would take the block: once consumers acknowledge and pruning makes room in its stream or in the node-wide budget, or once storage frees up. A single filled block larger than the stream’s `max_bytes` fails the lane with `single_block_exceeds_delivery_limit`, as an equally large live block would: raise `max_bytes` or remove the table, then reset the lane. A fill also makes a pause at a delivery or storage limit more likely, and a held block’s pending delta counts against the shared `[budgets] pending_delta_bytes`. When that budget cannot take it, the lane fails with `live_gap_marker_exceeds_pending_delta_budget`: raise the budget, then reset the lane. Each processor that needed a repair logs one warning, “startup reconciliation repaired processor state against the canonical chain”, with the fields `undone_blocks`, `highest_undone_block`, `discarded_pending_deltas`, and `replay_from_block`. The info line “startup reconciliation completed” then reports, per processor, the blocks applied and reverted and the pending deltas left. A consistent store is left untouched, so the warning appears only after an interruption or for a lane that trails the retained frames. No operator action is needed. ### SQLite failure Stop writers, run `db verify`, preserve the database and WAL for diagnosis, and restore a previously verified backup into a new path. Do not run a destructive downgrade or hand-edit schema metadata. ### Expired stream cursor The API returns `410 cursor_expired` or a terminal `reset_required` SSE event. Query a fresh snapshot, atomically replace consumer state, then resume from the returned retained cursor. ### Processor rebuild or rollback Processor code/config identities create separate immutable instances. Backfill the replacement instance in parallel, compare coverage and outputs, then move consumers. Rollback selects the prior binary/store pair or restores its backup; it never mutates a newer instance in place. The old instance’s backfill subscriptions do not move. Their retained records count against `[budgets.delivery] maximum_history_retained_bytes` until they are deleted, and once the node no longer configures the old instance, their stream and acknowledgement routes refuse (normally `404 not_found`), so their consumers can neither read nor acknowledge them. Settle them around the switch: 1. Before the configuration drops the old instance, let each of its history subscriptions finish, with its consumer acknowledging the `backfill_complete` record, or cancel it. Delete each once its consumer has acknowledged what it was delivered. 2. After the node restarts without the old instance, list the subscriptions with `GET /admin/v1/backfill-subscriptions`: `processorConfigured: false` marks those of an instance the node does not configure. That can be temporary, as when a restart’s configuration leaves an instance out by mistake: configure it again instead. If the old instance will not return, cancel each of its subscriptions with `POST /admin/v1/backfill-subscriptions/{id}/cancel` and delete it with `DELETE /admin/v1/backfill-subscriptions/{id}?discardUnacknowledged=true`, which discards its unacknowledged records. Both requests need the header `x-leani-request: 1`. The SDK’s backfill client makes the same calls with `list()`, `cancel(id)`, and `delete(id, { discardUnacknowledged: true })`. A store upgrades to the binary’s schema on the first start that opens it, and the upgrade cannot be undone. A start that the store refuses for a processor under an existing instance ID (`processor instance … conflicts with its stored descriptor`) refuses before that upgrade, so the store stays usable by the binary that wrote it. Store migrations are forward only: rolling an upgraded store back means restoring the backup taken before its upgrade. # Trust and threat model > Identify the assets, trust boundaries, controls, and residual risks of a Leani deployment. ## Protected assets * canonical processor state and its undo history; * finality/checkpoint provenance; * API credentials and source URLs; * host memory, disk, file descriptors, and outbound bandwidth; * durable consumer cursors and sink delivery state. ## Trust boundaries Public datasets, Beacon/HTTP endpoints, and consensus P2P peers are untrusted transports. A source descriptor declares which completeness claims are dataset-trusted and which commitments were cryptographically verified. Execution P2P peers are Byzantine. Processor input requirements reject missing, partial, or failed material before mapping. The native API and JSON-RPC endpoints accept hostile input. SQLite and the configured data directory are trusted local state. No dynamic native plugins are loaded. Statically linked native processors are fully trusted code: they run in the node process with its CPU, memory, filesystem, and network privileges. Their descriptors constrain data selection and retention but are not a sandbox or a security boundary. Only build processor crates from reviewed source; deploy untrusted extension logic only after a sandboxed runtime exists. ## Controls * strict TOML with unknown-field rejection and localhost defaults; * explicit source capabilities, trust, range, and byte/concurrency budgets; * checksummed, path-safe local archive objects; * block/header/transaction/receipt commitment checks on P2P material; * local checkpoint, sync-committee, BLS, finality-branch, and execution-payload proof verification for both finality transports; * attested blocks anchored to attested heads: within each slot, the finality source verifies a light-client optimistic update, signed by at least 342 of the 512 sync-committee members, and its execution branch. Execution P2P requests the attested header and the header chain below it by hash, down from the attested hash; a peer that serves another header for a requested hash is banned; * unattested blocks checked before any attestation: at least two peers serve the same header, it passes reth’s consensus header rules and Leani’s slot rules on the source’s chain spec, it links to the delivered tip, and it is at most two slots past the newest attested head. A block whose header fails a rule waits for an attested head; no peer is penalised for it. Only the finality source can publish attested heads; the execution source only reads them. Only anchored headers earn peer rewards or clear withheld-header strikes, and only a mismatch anchored that way can reset the live lane; * hash-checked finality: a finalized block must be the retained canonical block at its height, only retained blocks that link to it are promoted, and a processor’s coverage at that height must hold its hash, or another block’s coverage there fails the processor’s lane. Neither chain is finalized on a contradiction. One with retained unfinalized blocks restarts the network lanes; one with finalized history, or the same one recurring a third time before finality moves on, halts them. Every start undoes, as a reorg, retained unfinalized blocks that neither link to the verified finalized anchor nor are proven to be its ancestors. Where parent links do not reach the anchor, execution headers from peers, hash-linked down from it, decide which retained unfinalized blocks are its ancestors, and those are kept and finalized. Peers can delay that proof, and a start that does not complete it within a minute undoes the blocks, but they cannot forge it; * atomic processor state, undo, coverage, change-log, and outbox commits; * opaque checksummed cursors bound to store, chain, processor, and version; * bounded API pages, request bodies, SSE batches, and source queues; * bounded JSON-RPC work: at most 100 calls per batch, 16 MiB per response, 10,000 `eth_getLogs` results, 1,000 filter addresses and topic alternatives, 128 subscriptions per WebSocket connection, and 256 connections, each configurable; * browser isolation for loopback listeners: the native API answers a `Host` that is an IP address, `localhost`, or a name in `api.allowed_hosts` (421 otherwise), so a page that points its own name at 127.0.0.1 is refused; both listeners refuse a browser `Origin` that is neither loopback nor allowed (403), on streams and WebSocket upgrades too; the native API also refuses a GET or HEAD that another site’s page sends without an `Origin` other than as a navigation, such as an image load, when the browser marks it with Fetch Metadata (403); mutations need a JSON body or `x-leani-request: 1`, and JSON-RPC calls `application/json` (415 otherwise), which a cross-site page cannot send without a preflight the node never grants; only POST creates query snapshots; and a consumer’s live or backfill stream route refuses any request that Fetch Metadata marks as cross-site or same-site without an allowed `Origin`, navigations into frames and tabs included (403); * a native API bind beyond loopback requires a bearer token, and a JSON-RPC bind beyond loopback an explicit opt-in; startup rejects tokens shorter than 16 characters or other than printable ASCII without spaces, tokens are compared in constant time, and CLI help never prints the values of `LEANI_API_TOKEN` or `LEANI_CHECKPOINT_URLS`; * optional bearer authentication with redacted debug output; * per-consumer credentials: a consumer created with one needs it on every consumer-scoped delivery route, streams included, besides any bearer token (403 otherwise); it is compared in constant time, stored as a BLAKE3 hash keyed with a random per-store secret, scoped to that processor instance, stream, and consumer identity, and a new one holds at least 32 printable ASCII characters. A backfill subscription’s credential also enters the subscription’s idempotency identity, an unkeyed BLAKE3 hash of the whole request. Credentials isolate applications on the delivery routes only: listing, inspecting, and revoking a consumer are administrative operations that only the bearer token protects; * consumer session tokens authenticated by a BLAKE3 MAC keyed with the per-store secret, which is drawn once from the operating system’s random source and never leaves the store; * acknowledgements bounded by what each consumer was delivered, by a page read or a stream batch, and on live streams by block boundaries; the generic consumer routes refuse consumers of split live streams, whose sessions fence renewal and acknowledgement; * no CORS layer; administrative routes share the native API listener and its bearer authentication, so binding it beyond localhost exposes those routes as well; * non-root, read-only container profile with all capabilities dropped; * fail-closed unsupported RPC responses instead of synthesized partial state. ## Residual risks * `trusted_dataset` sources can omit predicate-matching rows without a cryptographic completeness proof; * event-derived ERC-20 balances do not model rebases or non-standard hidden balance changes; * the Beacon API finality profile still depends on configured transports even though proofs are checked locally; * `attested` rests on the sync committee: at least two thirds of it signing a header that is not on the canonical chain would make that header an attested head. An attested head is not final, so honest reorgs still remove attested blocks; a peer that follows another branch lacks the attested block and is cooled down, not banned. Attested heads need no agreement between Beacon endpoints, since each carries the committee’s signature; two different ones at one slot are ignored; * `unattested` blocks can be forged only by two colluding peers whose block satisfies every header rule, and only for two slots: an attested head on another block, or 60 s without one, undoes them. They can also be undone honestly, when the network orphans a late payload; see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes); * live following at `attested` adds one to two slots of latency, and stops, reporting not ready, while the finality source verifies no attested head; unattested delivery waits too once the newest attested head is more than two slots old; * the embedded CLI no longer prints unverified peer previews; * native consensus and execution P2P peer availability can prevent timely progress, and a stale or incorrectly sourced weak-subjectivity checkpoint can select the wrong consensus history; * bearer tokens do not provide transport encryption; * two kinds of browser GET still reach the native API without the bearer token: navigations from another site’s page to routes other than consumer streams, including into a frame, and subresource loads that carry no Fetch Metadata. Browsers send Fetch Metadata only to potentially trustworthy URLs, which are HTTPS, loopback addresses, and `localhost`, so a node reached over plain HTTP by another name or address, such as a LAN IP with `allow_unauthenticated_remote`, gets none from current browsers either. The page cannot read either response, and neither creates a query snapshot. Without Fetch Metadata, such a GET, a navigation included, can still open the live or backfill stream of a consumer without a credential and take its session lease, which locks the consumer out while the connection stays open. A bearer token, or the consumer’s own credential, prevents it, since a browser adds neither on its own. Consumers declared in the node configuration have no credential. `allow_unauthenticated_remote` and the `allowed_hosts` and `allowed_origins` lists widen what reaches a listener; keep them to what clients need; * consumer credentials do not guard the administrative routes: whoever holds the bearer token, or any client that reaches the API when none is configured, can list, inspect, and revoke every consumer, as it can reset a live lane; * the per-store secret lives in the store: anyone who can read the database or one of its backups can test guesses of consumer credentials offline and make session tokens, and a backfill subscription’s idempotency identity, an unkeyed hash of its request, lets them test guesses of that subscription’s credential without the secret. Credentials created before the 32-character minimum keep working until their consumer is re-created; * a malicious or compromised native processor can exhaust resources, disclose node secrets, corrupt process memory through unsafe dependencies, or emit plausible but false output; * resource limits reduce denial-of-service impact but do not replace an edge proxy for public internet deployment. Production exposure requires TLS, independent finality transports, filesystem quotas, monitoring, backups, and a completed fault-injected soak. # Verify a release > Check Leani archives, container images, SBOMs, and packages against their published checksums and build provenance before deploying them. Each release is built once from a tagged commit and published without being rebuilt. Check what you download against that build before deploying it. ## Release archives The [GitHub release](https://github.com/smart-byte/leani/releases) carries the native archives, `SHA256SUMS`, the CycloneDX SBOMs including the node’s `leani.cdx.json`, and the `THIRD_PARTY_LICENSES.txt` dependency notices. Check a downloaded archive against the checksum list and its build provenance: ```bash sha256sum --check --ignore-missing SHA256SUMS gh attestation verify leani-v0.1.0-x86_64-unknown-linux-gnu.tar.gz --repo smart-byte/leani ``` The attestation names the release tag, its commit, and the publication run, which checked that the archive came from a successful preparation run of that commit. ## Container images Pin a release’s image digest for deployment. The publication job attests the multiarchitecture image and each architecture’s image. It binds each architecture’s CycloneDX SBOM to that image’s digest instead of attaching it to the GitHub release, so the SBOM can be verified against the exact image. The attestations are stored with the images in GHCR as well: ```bash gh attestation verify oci://ghcr.io/smart-byte/leani:v0.1.0 --repo smart-byte/leani gh attestation verify oci://ghcr.io/smart-byte/leani@sha256: --repo smart-byte/leani gh attestation verify oci://ghcr.io/smart-byte/leani@sha256: \ --repo smart-byte/leani --predicate-type https://cyclonedx.org/bom ``` ## Packages `@smart-byte/leani-sdk` is published to npm with provenance from the repository’s `sdk-release.yml` workflow; `npm audit signatures` checks it in a project that installed it. The four Rust authoring libraries are published to crates.io through trusted publishing from `crates-release.yml`. ## Release channels Prerelease versions are marked as GitHub prereleases and do not replace the latest stable release. Until the first stable release, each SDK prerelease takes npm’s `latest` tag, so an unversioned install resolves to the newest prerelease. From the first stable release on, stable versions take `latest` and prereleases take `next`. The Homebrew formula follows the same rule: the release coordinator opens a reviewed tap PR once a release’s archives are public. # Normalized frame archive > Reference the deterministic, checksummed archive format used to move normalized Leani frame segments. Status: implemented interchange format v1. The `archive` history source lets the node backfill from local, checksummed objects without retaining raw chain history. It is deliberately source-neutral: an Amazon Parquet, Xatu, EraE, BigQuery, or custom adapter can project its input to normalized `BlockFrame` JSON Lines and then use the same runtime and processors. Example history configuration: ```toml [[sources.history]] id = "offline-frames" kind = "archive" priority = 5 trust = "trusted_dataset" manifest = "./archives/mainnet/manifest.json" ``` A relative `manifest` path is relative to the directory of the configuration file, not the working directory the node starts in. Manifest: ```json { "format_version": 1, "id": "mainnet-public-data-2026-08-01", "chain_id": 1, "schema_version": "normalized-frame-jsonl.v1", "capabilities": ["Header", "Transactions", "Receipts", "Logs"], "complete_capabilities": ["Header", "Transactions", "Receipts", "Logs"], "finality": "Finalized", "objects": [ { "from_block": 19426589, "to_block": 19427588, "path": "19426589-19427588.jsonl", "blake3": "lowercase-64-hex-digits", "bytes": 1234567 } ] } ``` Each non-empty object line is one JSON-encoded `BlockFrame`. Objects must: * use simple relative paths (no absolute paths or `..`); * have the exact declared byte length and BLAKE3 digest; * contain exactly one frame per block in their declared range; * be ordered, contiguous, and parent-linked; * match the manifest chain, finality, and declared material capabilities; * be reached without crossing a symbolic link below the manifest’s directory. Reads enforce the normal source byte, frame, and frame-count budgets, and hold one line at a time. The adapter verifies the complete object’s length and BLAKE3 digest before it yields any frame; a length or checksum mismatch fails the read then. It then streams the frames from the same open file, hashing it again as it goes, and adds immutable object identity to their provenance. A structural failure, such as a truncated frame, a block out of order, missing, or repeated, a parent mismatch, a schema mismatch, or a chain or finality mismatch, fails the read as corrupt where it is found, after the frames before it may already have been applied; a later source or a rerun resumes after them. An object edited in place between the two passes fails with the checksum error only after its frames. Frames report only the checks the archive made: the object’s BLAKE3 digest, and the parent link to the previous frame of the same object. A frame’s own verification claims and consensus anchor are dropped, and the trust of its earlier provenance is capped at `trusted_dataset`. The archive is a trusted-dataset source, not automatically a cryptographic raw history source. An EraE adapter must additionally decode raw bodies/receipts, verify their commitment roots, and anchor canonicality before advertising `verified_material`. Backfill any configured processor: ```text leani backfill \ --processor erc20-balances \ --from-block 12345678 \ --to-block 12346677 ``` The runtime records coverage and discards input frames after reduction. Only processor output, undo/change journals, and job metadata remain in SQLite. An ordered processor such as `erc20-balances` applies its history contiguously from `start_block` upward, so `--from` must be the block after the last one it applied, or its `start_block` when it has applied none; the command refuses any other start and names the block to use. When `rpc.historical_mode = "on_demand"`, the same source can answer bounded historical RPC without importing frames into SQLite. Exact block and receipt responses additionally require complete canonical header RLP, EIP-2718 transaction bytes, typed receipt bytes, senders, receipt gas fields, and withdrawals whenever the header commits to a withdrawals root. The manifest must declare those capabilities complete. A blobs-only Xatu projection is correctly rejected for generic RPC because its predicate-filtered transaction material is not a complete block body. # Brand assets > Download the Leani mark, wordmark, avatar, and share card, with the rules, colors, and fonts that go with them. Use these files when you write about Leani, link to it, or list it in a directory or app store. | Asset | Size | Use it for | | ------------------------------------------------- | ------------- | ---------------------------------------------------------------- | | [Wordmark on dark](/leani-wordmark-on-dark.svg) | SVG | Headers, slides, and diagrams on dark backgrounds | | [Wordmark on light](/leani-wordmark-on-light.svg) | SVG | Headers, slides, and diagrams on light backgrounds | | [Mark on dark](/leani-mark-on-dark.svg) | SVG | Small spaces on dark backgrounds, where the wordmark doesn’t fit | | [Mark on light](/leani-mark-on-light.svg) | SVG | Small spaces on light backgrounds | | [App icon](/leani-mark.svg) | SVG | The mark on its dark tile, for app icons and tiles | | [Avatar](/leani-social-avatar.png) | 1024×1024 PNG | Profile pictures, app icons, directory listings | | [Share card](/og-image.png) | 1200×630 PNG | Link previews; the site’s pages already use it |   ## The mark The mark is a lowercase i that leans 12°, one degree for each second of an Ethereum slot. Its dot is a square block: the one block an application keeps. The wordmark spells `leani` in a geometric lowercase and ends in the same i. * Keep the 12° lean, the square dot, and the proportions. Don’t redraw the wordmark in a font. * Use the dark-background files on dark surfaces and the light-background files on light ones. Only the app icon and avatar carry a tile. * Leave clear space around the mark at least as wide as its dot. * Below 24 pixels, use the mark rather than the wordmark. The favicon is a heavier cut of the mark for 16 and 32 pixels. * Write the name as Leani in prose. ## The epoch field The supporting pattern is one epoch’s 32 slots: a 6×6 grid of dots without its corners. One slot can be kept as a green block. The share card and the site use it as background texture and to show data being reduced. * Use the field behind headlines and illustrations, never inside the mark or next to it. * Don’t use it below 48 pixels or behind running text. ## Colors and type | Role | Color | | --------------------------------- | --------- | | Background | `#0b0f0d` | | Text | `#e6f0ea` | | Accent green | `#3ee07f` | | Accent green on light backgrounds | `#12a150` | | Text on light backgrounds | `#0b0f0d` | Body text uses Inter; code and commands use JetBrains Mono. The wordmark is drawn, not typeset. ## Source files The SVGs live in `site/src/assets/brand`. Run `bun run brand:update` in `site/` to publish them and to regenerate the favicons and avatar from the app icon. The share card is rendered from `site/scripts/og-card.html`; its header comment has the command. # Command-line interface > Generated Leani command, option, default, and value inventory from the Rust Clap definition. The tracked CLI fixture is rendered by a Rust test from the actual Clap command tree. The Bun-only site consumes that reviewed fixture without compiling Leani. ## Exit codes | Code | Meaning | | ---- | -------------------------------------------------------------------------------------------------------------------- | | 0 | Success. A `subscribe` stopped by a signal or `--timeout` also exits 0, unless `--once` had not printed a match yet. | | 1 | Runtime failure. | | 2 | Usage error: an unknown command, flag, or value. | | 3 | `doctor` found the configuration unreadable or invalid. | | 124 | `subscribe --once` reached `--timeout` before a match. | | 130 | `subscribe --once` was stopped by SIGINT or SIGTERM before a match, or a second signal cut a shutdown short. | ## Commands `leani`Low-storage, source-interchangeable Ethereum indexing node `Usage: leani [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani doctor`Validate and describe configuration without mutating state `Usage: leani doctor [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--json` | Emit machine-readable JSON | `false` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani serve`Start configured sources, processors, stores, and serving endpoints `Usage: leani serve [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani guide`Print the guide for agents and scripts that matches this binary `Usage: leani guide [OPTIONS] [TOPIC]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `topic` | Section to print \[default: the overview] | `processors` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani init`Create a compact configuration for a built-in processor `Usage: leani init [OPTIONS] [TARGET]...` | Argument | Purpose | Default / values | | --------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `protocol` | Processor preset to configure | `blocks \| uniswap-v3` | | `targets` | Optional protocol targets; Uniswap requires one or more markets such as \`ETH/USDC\` | `—` | | `--data-dir` | Data directory for the generated configuration \[default: \`data\` beside it] | `—` | | `--checkpoint-url` | Checkpoint provider used for weak-subjectivity quorum (repeatable or comma-delimited) | `https://ethereum-beacon-api.publicnode.com/, https://mainnet.checkpoint.sigp.io/, https://beaconstate-mainnet.chainsafe.io/` | | `--checkpoint-quorum` | Number of independent checkpoint providers that must agree | `2` | | `--yes` | Accept a new checkpoint provider quorum without interactive confirmation | `false` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani subscribe`Stream live processor output from an embedded runtime or a running node `Usage: leani subscribe [OPTIONS] [TARGET]...` | Argument | Purpose | Default / values | | --------------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `protocol` | Built-in subscription feed to activate | `blocks \| uniswap-v3` | | `targets` | Optional feed targets; Uniswap requires one or more markets such as \`ETH/USDC\` | `—` | | `--format` | Rendering contract for stdout | `pretty` | | `--json` | Shorthand for \`--format json\` | `false` | | `--mode` | Where the subscription processor should run | `auto` | | `--endpoint` | Base URL of an already running Leani node | `—` | | `--processor` | Processor instance or unambiguous processor kind on a running node | `—` | | `--token` | Optional bearer token for a running node | `—` | | `--finality` | Lowest level that counts as a result: \`unattested\` (default), \`attested\`, or \`finalized\` | `unattested` | | `--finality-source` | Finality transport for the embedded runtime | `auto` | | `--checkpoint-url` | Checkpoint provider used for weak-subjectivity quorum (repeatable or comma-delimited) | `https://ethereum-beacon-api.publicnode.com/, https://mainnet.checkpoint.sigp.io/, https://beaconstate-mainnet.chainsafe.io/` | | `--checkpoint-quorum` | Number of independent checkpoint providers that must agree | `2` | | `--yes` | Accept a new checkpoint provider quorum without interactive confirmation | `false` | | `--data-dir` | Persistent directory for the lightweight embedded runtime | `—` | | `--once` | Exit after the first matching update | `false` | | `--timeout` | Stop after this long, e.g. \`90s\`, \`5m\`, or \`300\` (seconds); with \`--once\`, exit 124 if nothing matched | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani reset`Reset reconstructible local runtime state for cold-start testing `Usage: leani reset [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani reset all`Delete all node and embedded-subscription state in the resolved data directory `Usage: leani reset all [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | --------------------------------------------------------------------------- | ---------------- | | `--data-dir` | Runtime data directory to reset instead of the configured/default directory | `—` | | `--yes` | Reset without interactive confirmation | `false` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani reset subscription`Delete reconstructible state for one embedded market subscription `Usage: leani reset subscription [OPTIONS] [TARGET]...` | Argument | Purpose | Default / values | | -------------- | ---------------------------------------------------------------------------------------------- | ---------------------- | | `protocol` | Built-in subscription feed whose embedded state should be reset | `blocks \| uniswap-v3` | | `targets` | The exact targets used by \`leani subscribe\`; empty for \`blocks\` | `—` | | `--finality` | Finality of the subscription state being reset (\`unattested\`, \`attested\` or \`finalized\`) | `unattested` | | `--data-dir` | Exact embedded subscription directory to reset | `—` | | `--yes` | Reset without interactive confirmation | `false` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani reset help`Print this message or the help of the given subcommand(s) `Usage: leani reset help [COMMAND]` `leani reset help all`Delete all node and embedded-subscription state in the resolved data directory `Usage: leani reset help all` `leani reset help subscription`Delete reconstructible state for one embedded market subscription `Usage: leani reset help subscription` `leani reset help help`Print this message or the help of the given subcommand(s) `Usage: leani reset help help` `leani backfill`Process a historical block range and then exit `Usage: leani backfill [OPTIONS] --from-block --to-block ` | Argument | Purpose | Default / values | | -------------- | ------------------------------------------------------------------------------- | ---------------- | | `--processor` | Configured processor instance or kind \[default: the only configured processor] | `—` | | `--from-block` | First block of the range, inclusive | `—` | | `--to-block` | Last block of the range, inclusive | `—` | | `--endpoint` | Submit to an already running node, preserving an optional URL prefix | `—` | | `--token` | Bearer token for a running node | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source`Inspect or probe configured data sources `Usage: leani source [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe`Probe a source without changing processor state `Usage: leani source probe [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe xatu`Inspect the configured Xatu catalog `Usage: leani source probe xatu [OPTIONS] --from-block --to-block ` | Argument | Purpose | Default / values | | --------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------- | | `--network` | Xatu network catalog; only mainnet is supported | `mainnet` | | `--from-block` | First execution block in the inclusive probe range | `—` | | `--to-block` | Last execution block in the inclusive probe range | `—` | | `--processor` | Processor projection whose required tables should be inspected | `blobs-money` | | `--from-date` | First beacon date, required to inspect daily beacon tables | `—` | | `--to-date` | Last beacon date, required to inspect daily beacon tables | `—` | | `--concurrency` | Maximum simultaneous footer requests | `4` | | `--max-input-bytes` | Maximum bytes the projection fetches in all, Parquet footers and merged gaps included, and the most it holds at once | `268435456` | | `--batch-rows` | Arrow rows decoded in one bounded batch | `8192` | | `--report` | Optional JSON report destination | `—` | | `--projection-output` | Optional destination for normalized projected frames | `—` | | `--expected-export` | Read-only blobs.money compatibility export to compare field by field | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe p2p`Exercise the recent execution P2P boundary `Usage: leani source probe p2p [OPTIONS] --from-block --to-block ` | Argument | Purpose | Default / values | | --------------------------- | --------------------------------------------------------------------- | ---------------- | | `--from-block` | First execution block in the inclusive probe range | `—` | | `--to-block` | Last execution block in the inclusive probe range (at most 64 blocks) | `—` | | `--expected-tip` | Consensus-verified hash expected for \`to-block\` | `—` | | `--minimum-peers` | Minimum connected peers before issuing requests | `3` | | `--peer-wait-seconds` | Maximum time to wait for peers | `300` | | `--request-timeout-seconds` | Timeout applied independently to each request | `30` | | `--retries` | Maximum attempts per header, body, or receipt request | `5` | | `--retry-backoff-seconds` | Pause between request attempts while the peer pool changes | `1` | | `--max-input-bytes` | Maximum normalized bytes accepted for the complete range | `268435456` | | `--report` | Optional destination for a JSON summary | `—` | | `--output` | Optional destination for complete normalized frames | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe erae`Exercise sparse execution-history archive reads `Usage: leani source probe erae [OPTIONS] --from-block --to-block ` | Argument | Purpose | Default / values | | ------------------- | -------------------------------------------------------------------------------------- | ---------------- | | `--from-block` | First execution block in the inclusive probe range | `—` | | `--to-block` | Last execution block in the inclusive probe range | `—` | | `--endpoint` | Override the public mainnet archive base URL with an https, file, or loopback http URL | `—` | | `--max-input-bytes` | Maximum sparse archive and normalized bytes | `268435456` | | `--report` | Optional destination for a JSON summary | `—` | | `--output` | Optional destination for complete normalized frames | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe finality`Exercise the configured finality boundary `Usage: leani source probe finality [OPTIONS]` | Argument | Purpose | Default / values | | --------------------- | -------------------------------------------------------------------------- | ---------------- | | `--checkpoint` | Override the configured weak-subjectivity checkpoint root | `—` | | `--checkpoint-slot` | Override the finalized beacon slot associated with the checkpoint | `—` | | `--endpoint` | Override configured Beacon API endpoints (repeatable) | `—` | | `--minimum-agreement` | Override the number of endpoints that must return the same verified anchor | `—` | | `--report` | Optional JSON report destination | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani source probe help`Print this message or the help of the given subcommand(s) `Usage: leani source probe help [COMMAND]` `leani source probe help xatu`Inspect the configured Xatu catalog `Usage: leani source probe help xatu` `leani source probe help p2p`Exercise the recent execution P2P boundary `Usage: leani source probe help p2p` `leani source probe help erae`Exercise sparse execution-history archive reads `Usage: leani source probe help erae` `leani source probe help finality`Exercise the configured finality boundary `Usage: leani source probe help finality` `leani source probe help help`Print this message or the help of the given subcommand(s) `Usage: leani source probe help help` `leani source help`Print this message or the help of the given subcommand(s) `Usage: leani source help [COMMAND]` `leani source help probe`Probe a source without changing processor state `Usage: leani source help probe [COMMAND]` `leani source help probe xatu`Inspect the configured Xatu catalog `Usage: leani source help probe xatu` `leani source help probe p2p`Exercise the recent execution P2P boundary `Usage: leani source help probe p2p` `leani source help probe erae`Exercise sparse execution-history archive reads `Usage: leani source help probe erae` `leani source help probe finality`Exercise the configured finality boundary `Usage: leani source help probe finality` `leani source help help`Print this message or the help of the given subcommand(s) `Usage: leani source help help` `leani conformance`Compare normalized source exports or render exact RPC response vectors `Usage: leani conformance [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani conformance frames`Compare canonical material in two normalized frame exports `Usage: leani conformance frames [OPTIONS] --left-source --left --right-source --right --capability ` | Argument | Purpose | Default / values | | ---------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `--left-source` | | `—` | | `--left` | | `—` | | `--right-source` | | `—` | | `--right` | | `—` | | `--capability` | Canonical material to compare (repeatable or comma-delimited) | `header \| body \| transactions \| calldata \| receipts \| logs \| withdrawals \| blob-sidecars \| traces \| state-diffs \| consensus-finality` | | `--report` | Optional JSON report destination | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani conformance processor`Compare deterministic processor deltas from two source exports `Usage: leani conformance processor [OPTIONS] --left-source --left --right-source --right ` | Argument | Purpose | Default / values | | ---------------- | ----------------------------------------- | ---------------- | | `--processor` | Configured built-in processor ID | `blobs-money` | | `--left-source` | | `—` | | `--left` | | `—` | | `--right-source` | | `—` | | `--right` | | `—` | | `--report` | Optional JSON report destination | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani conformance rpc`Render exact Ethereum block/receipt responses from normalized frames `Usage: leani conformance rpc [OPTIONS] --frames ` | Argument | Purpose | Default / values | | -------------- | ---------------------------------------------------------- | ---------------- | | `--frames` | Normalized frame export produced by a source probe | `—` | | `--expected` | Optional reference snapshot array from an execution client | `—` | | `--output` | Optional destination for the reconstructed snapshots | `—` | | `--report` | Optional differential JSON report destination | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani conformance help`Print this message or the help of the given subcommand(s) `Usage: leani conformance help [COMMAND]` `leani conformance help frames`Compare canonical material in two normalized frame exports `Usage: leani conformance help frames` `leani conformance help processor`Compare deterministic processor deltas from two source exports `Usage: leani conformance help processor` `leani conformance help rpc`Render exact Ethereum block/receipt responses from normalized frames `Usage: leani conformance help rpc` `leani conformance help help`Print this message or the help of the given subcommand(s) `Usage: leani conformance help help` `leani benchmark`Run a reproducible historical throughput and storage measurement `Usage: leani benchmark [OPTIONS] [COMMAND]` | Argument | Purpose | Default / values | | -------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--mode` | Isolated stage or full pipeline to measure | `process` | | `--profile` | Product storage/delivery policy measured by the selected stage | `acquire-discard \| raw-only \| materialized \| compact-artifact \| compact-artifact-segment \| compact-artifact-tiered \| raw-artifact \| raw-materialized \| raw-artifact-materialized \| externalized \| raw-externalized` | | `--destination` | Destination path used by delivery modes | `rust-http` | | `--postgres-schema` | \`PostgreSQL\` schema exercised by the SDK destination | `generic-event-log` | | `--corpus` | Deterministic material shape | `blobs-like` | | `--blocks` | Number of generated finalized blocks | `10000` | | `--seed` | Stable corpus seed recorded in the report | `1` | | `--chunk-blocks` | Physical source chunk size in blocks | `2048` | | `--artifact-segment-blocks` | Blocks per immutable artifact segment | `8192` | | `--artifact-segment-compression` | Compression used by immutable artifact segments | `snappy` | | `--artifact-compaction-interval-ms` | Background artifact compaction cadence in milliseconds | `100` | | `--artifact-compaction-maximum-segments-per-cycle` | Maximum artifact segments published per background cycle | `4` | | `--warmups` | Unreported warm-up runs | `1` | | `--runs` | Measured runs retained in the report | `3` | | `--sample-interval-ms` | Resource sample interval in milliseconds | `200` | | `--consumer-delay-ms` | Optional fixed delay after each destination batch | `0` | | `--consumer-reconnect-every-batches` | Reopen the SDK/PostgreSQL stream after every N acknowledged batches (zero disables) | `0` | | `--consumer-drop-ack-response-once` | Simulate losing the first successful SDK/PostgreSQL acknowledgement response | `false` | | `--concurrent-live-blocks` | Finalized live blocks committed while history is acquired and delivered | `0` | | `--live-block-interval-ms` | Delay between synthetic live commits in milliseconds | `10` | | `--mapper-concurrency` | Parallel processor map tasks | `4` | | `--maximum-active-chunks` | Node-wide concurrently active physical source chunks | `4` | | `--maximum-mapped-bytes` | Node-wide mapped-delta byte budget | `134217728` | | `--commit-maximum-blocks` | Maximum blocks per atomic \`SQLite\` history commit | `128` | | `--commit-maximum-changes` | Maximum domain changes per atomic \`SQLite\` history commit | `10000` | | `--commit-maximum-encoded-bytes` | Maximum stable-encoded change bytes per history commit | `16777216` | | `--commit-maximum-delay-ms` | Flush delay for a partially filled history commit | `50` | | `--commit-target-writer-hold-ms` | Target p95 \`SQLite\` writer hold time for adaptive history commits | `20` | | `--delivery-target-encoded-bytes` | Target uncompressed encoded bytes per delivered history batch | `4194304` | | `--delivery-maximum-encoded-bytes` | Hard maximum uncompressed encoded bytes per delivery batch | `16777216` | | `--delivery-maximum-events` | Hard maximum domain events per delivery batch | `20000` | | `--delivery-maximum-processed-blocks` | Hard maximum processed blocks represented by one delivery batch | `8192` | | `--delivery-maximum-delay-ms` | Flush delay for a partially filled delivery batch | `50` | | `--delivery-maximum-buffered-batches` | Maximum encoded batches queued per connection | `4` | | `--delivery-maximum-buffered-bytes` | Maximum encoded bytes queued per connection | `67108864` | | `--delivery-compression` | Delivery transport compression | `gzip` | | `--report` | Optional JSON report destination | `—` | | `--samples-report` | Optional immutable NDJSON destination for high-frequency samples | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani benchmark sweep`Run or resume a manifest-defined candidate sweep `Usage: leani benchmark sweep [OPTIONS] --manifest --output-directory ` | Argument | Purpose | Default / values | | -------------------- | ---------------------------------------------------------------------- | ---------------- | | `--manifest` | Versioned JSON sweep manifest | `—` | | `--output-directory` | Directory containing immutable candidate reports and the sweep summary | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani benchmark real-source`Measure a pinned Mainnet range through real history sources, processor, HTTP delivery, acknowledgement, and pruning `Usage: leani benchmark real-source [OPTIONS] --from-block --to-block --data-dir --report ` | Argument | Purpose | Default / values | | -------------------------- | -------------------------------------------------------------------- | ------------------- | | `--processor` | Configured processor kind or instance | `blobs-money` | | `--source-policy` | Historical source selection policy | `history-portfolio` | | `--from-block` | First finalized execution block in the inclusive range | `—` | | `--to-block` | Last finalized execution block in the inclusive range | `—` | | `--data-dir` | Fresh benchmark directory. Existing node state is rejected | `—` | | `--consumer-delay-ms` | Optional delay before acknowledging every delivered batch | `0` | | `--timeout-seconds` | Overall wall-clock limit for acquisition, processing, and delivery | `7200` | | `--sample-interval-ms` | Resource/progress sampling interval | `250` | | `--source-concurrency` | Override configured historical source acquisition concurrency | `—` | | `--mapper-concurrency` | Override configured processor mapper concurrency | `—` | | `--maximum-active-chunks` | Override the shared historical coordinator active-chunk ceiling | `—` | | `--expected-output-digest` | Optional expected BLAKE3 digest of canonical delivered domain events | `—` | | `--delivery-compression` | Delivery transport compression. Batch sizing comes from node config | `gzip` | | `--report` | Versioned JSON evidence report | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani benchmark help`Print this message or the help of the given subcommand(s) `Usage: leani benchmark help [COMMAND]` `leani benchmark help sweep`Run or resume a manifest-defined candidate sweep `Usage: leani benchmark help sweep` `leani benchmark help real-source`Measure a pinned Mainnet range through real history sources, processor, HTTP delivery, acknowledgement, and pruning `Usage: leani benchmark help real-source` `leani benchmark help help`Print this message or the help of the given subcommand(s) `Usage: leani benchmark help help` `leani e2e`Run explicit end-to-end verification gates `Usage: leani e2e [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani e2e fixture`Run a deterministic bounded source/runtime/SQLite fixture with no network `Usage: leani e2e fixture [OPTIONS] --data-dir ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--data-dir` | Fresh directory for the fixture database | `—` | | `--blocks` | Number of deterministic finalized blocks | `128` | | `--report` | Optional JSON evidence report | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani e2e mainnet`Backfill and follow Ethereum Mainnet using production source adapters `Usage: leani e2e mainnet [OPTIONS] --from-block --data-dir ` | Argument | Purpose | Default / values | | ------------------------- | ---------------------------------------------------------------------- | ------------------------- | | `--processor` | Configured processor to verify | `blobs-money` | | `--from-block` | First historical block in the required continuous coverage range | `—` | | `--data-dir` | Fresh data directory. Existing state requires \`--resume\` | `—` | | `--resume` | Reuse an existing E2E data directory after an interrupted run | `false` | | `--minimum-follow-blocks` | Blocks that must be followed beyond the initial finalized anchor | `16` | | `--stable-seconds` | Seconds all convergence and head-freshness assertions must remain true | `300` | | `--max-head-age-seconds` | Maximum permitted age of the latest retained execution block | `60` | | `--timeout-seconds` | Overall wall-clock limit for the staging run | `7200` | | `--report` | JSON evidence report written on both success and failure | `mainnet-e2e-report.json` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani e2e help`Print this message or the help of the given subcommand(s) `Usage: leani e2e help [COMMAND]` `leani e2e help fixture`Run a deterministic bounded source/runtime/SQLite fixture with no network `Usage: leani e2e help fixture` `leani e2e help mainnet`Backfill and follow Ethereum Mainnet using production source adapters `Usage: leani e2e help mainnet` `leani e2e help help`Print this message or the help of the given subcommand(s) `Usage: leani e2e help help` `leani db`Inspect and maintain the durable store `Usage: leani db [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db inspect` `Usage: leani db inspect [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db verify` `Usage: leani db verify [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db backup` `Usage: leani db backup [OPTIONS] ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `destination` | | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db compact` `Usage: leani db compact [OPTIONS]` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db prune-changes`Prune durable changes older than a sequence while respecting active consumer leases `Usage: leani db prune-changes [OPTIONS] --processor --before ` | Argument | Purpose | Default / values | | -------------- | ----------------------------------------- | ---------------- | | `--processor` | | `—` | | `--before` | | `—` | | `--config` | Path to the versioned TOML configuration | `—` | | `--log-format` | Log rendering format | `pretty` | | `--log-filter` | Tracing filter, e.g. \`info,leani=debug\` | `—` | | `--help` | Print help (see more with '--help') | `—` | | `--version` | Print version | `—` | `leani db help`Print this message or the help of the given subcommand(s) `Usage: leani db help [COMMAND]` `leani db help inspect` `Usage: leani db help inspect` `leani db help verify` `Usage: leani db help verify` `leani db help backup` `Usage: leani db help backup` `leani db help compact` `Usage: leani db help compact` `leani db help prune-changes`Prune durable changes older than a sequence while respecting active consumer leases `Usage: leani db help prune-changes` `leani db help help`Print this message or the help of the given subcommand(s) `Usage: leani db help help` `leani help`Print this message or the help of the given subcommand(s) `Usage: leani help [COMMAND]` `leani help doctor`Validate and describe configuration without mutating state `Usage: leani help doctor` `leani help serve`Start configured sources, processors, stores, and serving endpoints `Usage: leani help serve` `leani help guide`Print the guide for agents and scripts that matches this binary `Usage: leani help guide` `leani help init`Create a compact configuration for a built-in processor `Usage: leani help init` `leani help subscribe`Stream live processor output from an embedded runtime or a running node `Usage: leani help subscribe` `leani help reset`Reset reconstructible local runtime state for cold-start testing `Usage: leani help reset [COMMAND]` `leani help reset all`Delete all node and embedded-subscription state in the resolved data directory `Usage: leani help reset all` `leani help reset subscription`Delete reconstructible state for one embedded market subscription `Usage: leani help reset subscription` `leani help backfill`Process a historical block range and then exit `Usage: leani help backfill` `leani help source`Inspect or probe configured data sources `Usage: leani help source [COMMAND]` `leani help source probe`Probe a source without changing processor state `Usage: leani help source probe [COMMAND]` `leani help source probe xatu`Inspect the configured Xatu catalog `Usage: leani help source probe xatu` `leani help source probe p2p`Exercise the recent execution P2P boundary `Usage: leani help source probe p2p` `leani help source probe erae`Exercise sparse execution-history archive reads `Usage: leani help source probe erae` `leani help source probe finality`Exercise the configured finality boundary `Usage: leani help source probe finality` `leani help conformance`Compare normalized source exports or render exact RPC response vectors `Usage: leani help conformance [COMMAND]` `leani help conformance frames`Compare canonical material in two normalized frame exports `Usage: leani help conformance frames` `leani help conformance processor`Compare deterministic processor deltas from two source exports `Usage: leani help conformance processor` `leani help conformance rpc`Render exact Ethereum block/receipt responses from normalized frames `Usage: leani help conformance rpc` `leani help benchmark`Run a reproducible historical throughput and storage measurement `Usage: leani help benchmark [COMMAND]` `leani help benchmark sweep`Run or resume a manifest-defined candidate sweep `Usage: leani help benchmark sweep` `leani help benchmark real-source`Measure a pinned Mainnet range through real history sources, processor, HTTP delivery, acknowledgement, and pruning `Usage: leani help benchmark real-source` `leani help e2e`Run explicit end-to-end verification gates `Usage: leani help e2e [COMMAND]` `leani help e2e fixture`Run a deterministic bounded source/runtime/SQLite fixture with no network `Usage: leani help e2e fixture` `leani help e2e mainnet`Backfill and follow Ethereum Mainnet using production source adapters `Usage: leani help e2e mainnet` `leani help db`Inspect and maintain the durable store `Usage: leani help db [COMMAND]` `leani help db inspect` `Usage: leani help db inspect` `leani help db verify` `Usage: leani help db verify` `leani help db backup` `Usage: leani help db backup` `leani help db compact` `Usage: leani help db compact` `leani help db prune-changes`Prune durable changes older than a sequence while respecting active consumer leases `Usage: leani help db prune-changes` `leani help help`Print this message or the help of the given subcommand(s) `Usage: leani help help` # Configuration fields > Generated field and constraint inventory for Leani configuration schema version 1. This inventory is generated from `config/schema-v1.json`. “Required here” means required by the immediate containing object; optional objects can still contain required members when present. Validate the assembled file JSON Schema catches structural mistakes. `leani doctor` additionally applies cross-field, processor-owned, source, retention, and budget validation. ## `config_version` | Field | Type / values | Required here | Constraints | | ---------------- | ------------- | ------------- | ----------- | | `config_version` | `1` | yes | `—` | ## `network` | Field | Type / values | Required here | Constraints | | --------- | -------------------- | ------------- | ----------- | | `network` | `"ethereum-mainnet"` | yes | `—` | ## `data_dir` | Field | Type / values | Required here | Constraints | | ---------- | ----------------------------------------------------------------------------------------------------- | ------------- | ------------------------------------ | | `data_dir` | `string`Writable node state. A relative path is relative to the directory of this configuration file. | yes | `{"default":"./data","minLength":1}` | ## `finality` | Field | Type / values | Required here | Constraints | | ---------------------------- | ----------------------------------------- | ------------- | ----------------------------------- | | `finality` | `starterFinality` | yes | `—` | | `finality.checkpoint` | `string` | yes | `{"pattern":"^0x[0-9a-fA-F]{64}$"}` | | `finality.checkpoint_slot` | `positiveInteger` | yes | `{"minimum":1}` | | `finality.endpoints` | `array` | yes | `{"minItems":1}` | | `finality.kind` | `beacon_api \| consensus_p2p \| disabled` | yes | `—` | | `finality.minimum_agreement` | `positiveInteger` | no | `{"minimum":1}` | | `finality.bootnodes` | `array` | no | `—` | | `finality.minimum_peers` | `nonNegativeInteger` | no | `{"minimum":0}` | | `finality.discovery_port` | `integer` | no | `{"minimum":0,"maximum":65535}` | ## `blocks` | Field | Type / values | Required here | Constraints | | -------- | --------------- | ------------- | ----------- | | `blocks` | `starterBlocks` | no | `—` | ## `uniswap` | Field | Type / values | Required here | Constraints | | ----------------- | ----------------------------------------- | ------------- | ---------------- | | `uniswap` | `starterUniswap` | no | `—` | | `uniswap.markets` | `array` | yes | `{"minItems":1}` | ## `api` | Field | Type / values | Required here | Constraints | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------------------------- | | `api` | `starterApi` | yes | `—` | | `api.bind` | `socketAddress`IPv4 or bracketed IPv6 socket address with port | yes | `{"default":"127.0.0.1:18080"}` | | `api.bearer_token_env` | `string` | no | `{"minLength":1}` | | `api.allow_unauthenticated_remote` | `boolean`Serve the API on a non-loopback bind without bearer\_token\_env. Only behind an authenticating proxy. | no | `{"default":false}` | | `api.allowed_hosts` | `array`Host names, besides localhost, that requests may address. Any other Host name gets HTTP 421; IP addresses always pass. | no | `{"default":[]}` | | `api.allowed_origins` | `allowedOrigins`Browser origins, besides loopback ones on any port, that may call the API. Any other Origin gets HTTP 403. | no | `{"default":[]}` | | `api.delivery` | `object` | no | `—` | | `api.delivery.history_batches` | `deliveryBatch` | no | `—` | | `api.delivery.history_batches.target_encoded_bytes` | `humanBytes` | yes | `—` | | `api.delivery.history_batches.maximum_encoded_bytes` | `humanBytes` | yes | `—` | | `api.delivery.history_batches.maximum_events` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.history_batches.maximum_processed_blocks` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.history_batches.maximum_delay` | `humanMilliseconds` | yes | `—` | | `api.delivery.history_batches.maximum_buffered_batches` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.history_batches.maximum_buffered_bytes` | `humanBytes` | yes | `—` | | `api.delivery.history_batches.compression` | `none \| gzip` | no | `—` | | `api.delivery.live_batches` | `deliveryBatch` | no | `—` | | `api.delivery.live_batches.target_encoded_bytes` | `humanBytes` | yes | `—` | | `api.delivery.live_batches.maximum_encoded_bytes` | `humanBytes` | yes | `—` | | `api.delivery.live_batches.maximum_events` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.live_batches.maximum_processed_blocks` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.live_batches.maximum_delay` | `humanMilliseconds` | yes | `—` | | `api.delivery.live_batches.maximum_buffered_batches` | `positiveInteger` | yes | `{"minimum":1}` | | `api.delivery.live_batches.maximum_buffered_bytes` | `humanBytes` | yes | `—` | | `api.delivery.live_batches.compression` | `none \| gzip` | no | `—` | ## `chain` | Field | Type / values | Required here | Constraints | | ------------------- | -------------------- | ------------- | ----------------- | | `chain` | `chain` | yes | `—` | | `chain.name` | `string` | yes | `{"minLength":1}` | | `chain.chain_id` | `positiveInteger` | yes | `{"minimum":1}` | | `chain.merge_block` | `nonNegativeInteger` | no | `{"minimum":0}` | ## `raw_history` | Field | Type / values | Required here | Constraints | | -------------------------------------------- | ----------------- | ------------- | --------------- | | `raw_history` | `rawHistory` | no | `—` | | `raw_history.enabled` | `boolean` | no | `—` | | `raw_history.maximum_logical_bytes` | `humanBytes` | no | `—` | | `raw_history.maximum_physical_bytes` | `humanBytes` | no | `—` | | `raw_history.maximum_frame_logical_bytes` | `humanBytes` | no | `—` | | `raw_history.maximum_segment_logical_bytes` | `humanBytes` | no | `—` | | `raw_history.maximum_segment_physical_bytes` | `humanBytes` | no | `—` | | `raw_history.reader_connections` | `positiveInteger` | no | `{"minimum":1}` | | `raw_history.maximum_source_frames` | `positiveInteger` | no | `{"minimum":1}` | | `raw_history.maximum_buffered_frames` | `positiveInteger` | no | `{"minimum":1}` | ## `artifact_storage` | Field | Type / values | Required here | Constraints | | ------------------------------------------------- | --------------------------- | ------------- | --------------- | | `artifact_storage` | `artifactStorage` | no | `—` | | `artifact_storage.backend` | `sqlite \| tiered_segments` | no | `—` | | `artifact_storage.segment_target_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `artifact_storage.compression` | `none \| snappy \| deflate` | no | `—` | | `artifact_storage.maximum_artifact_logical_bytes` | `humanBytes` | no | `—` | | `artifact_storage.maximum_segment_logical_bytes` | `humanBytes` | no | `—` | | `artifact_storage.maximum_segment_physical_bytes` | `humanBytes` | no | `—` | | `artifact_storage.compaction_interval` | `humanMilliseconds` | no | `—` | | `artifact_storage.maximum_segments_per_cycle` | `positiveInteger` | no | `{"minimum":1}` | ## `budgets` | Field | Type / values | Required here | Constraints | | ------------------------------------------------------------------ | -------------------------------- | ------------- | --------------- | | `budgets` | `budgets` | yes | `—` | | `budgets.memory_bytes` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.temporary_disk_bytes` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.pending_delta_bytes` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.recent_raw_soft_bytes` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.recent_raw_hard_bytes` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.source_concurrency` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.mapper_concurrency` | `positiveInteger` | yes | `{"minimum":1}` | | `budgets.history_material` | `historyMaterialBudget` | no | `—` | | `budgets.history_material.mode` | `disabled \| observe \| enabled` | no | `—` | | `budgets.history_material.memory_bytes` | `humanBytes` | no | `—` | | `budgets.history_material.maximum_buffered_frames_per_acquisition` | `positiveInteger` | no | `{"minimum":1}` | | `budgets.history_material.minimum_physical_chunk_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `budgets.history_material.maximum_overfetch_ratio` | `number` | no | `{"minimum":1}` | | `budgets.history_pipeline` | `historyPipelineBudget` | no | `—` | | `budgets.history_pipeline.maximum_active_chunks` | `positiveInteger` | no | `{"minimum":1}` | | `budgets.history_pipeline.maximum_mapped_bytes` | `humanBytes` | no | `—` | | `budgets.history_pipeline.commit` | `historyCommitBudget` | no | `—` | | `budgets.history_pipeline.commit.maximum_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `budgets.history_pipeline.commit.maximum_changes` | `positiveInteger` | no | `{"minimum":1}` | | `budgets.history_pipeline.commit.maximum_encoded_bytes` | `humanBytes` | no | `—` | | `budgets.history_pipeline.commit.maximum_delay` | `humanMilliseconds` | no | `—` | | `budgets.history_pipeline.commit.target_writer_hold` | `humanMilliseconds` | no | `—` | | `budgets.store` | `storeBudget` | no | `—` | | `budgets.store.maximum_physical_bytes` | `humanBytes` | no | `—` | | `budgets.artifacts` | `artifactBudget` | no | `—` | | `budgets.artifacts.maximum_retained_bytes` | `humanBytes` | no | `—` | | `budgets.artifacts.maximum_pending_bytes` | `humanBytes` | no | `—` | | `budgets.delivery` | `deliveryBudget` | no | `—` | | `budgets.delivery.maximum_retained_bytes` | `humanBytes` | no | `—` | | `budgets.delivery.maximum_history_retained_bytes` | `humanBytes` | no | `—` | ## `sources` | Field | Type / values | Required here | Constraints | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------- | | `sources` | `sources` | yes | `—` | | `sources.history` | `array` | yes | `{"minItems":1}` | | `sources.history[].id` | `string` | yes | `{"minLength":1}` | | `sources.history[].kind` | `xatu \| era_e \| archive` | yes | `—` | | `sources.history[].priority` | `integer` | yes | `{"minimum":0,"maximum":65535}` | | `sources.history[].trust` | `trusted_dataset \| verified_material` | yes | `—` | | `sources.history[].chunk_blocks` | `positiveInteger`Xatu only: blocks per acquisition chunk for headers, transactions, and logs, a multiple of Xatu's 1,000-block partitions. | no | `{"minimum":1,"default":1000,"multipleOf":1000}` | | `sources.history[].blobs_chunk_blocks` | `positiveInteger`Xatu only: blocks per acquisition chunk for the blobs projection, a multiple of 1,000. | no | `{"minimum":1,"default":8000,"multipleOf":1000}` | | `sources.history[].batch_rows` | `positiveInteger`Xatu only: Parquet rows decoded per bounded Arrow batch. | no | `{"minimum":1,"default":8192}` | | `sources.history[].manifest` | `string`Archive only: the archive manifest. A relative path is relative to the directory of this configuration file. | no | `—` | | `sources.history[].endpoint` | `string` | no | `{"format":"uri"}` | | `sources.history[].allow_insecure_http` | `boolean` | no | `—` | | `sources.live` | `liveSource` | yes | `—` | | `sources.live.kind` | `p2p \| disabled` | yes | `—` | | `sources.live.minimum_peers` | `nonNegativeInteger` | yes | `{"minimum":0}` | | `sources.live.body_serving_peer_target` | `nonNegativeInteger` | no | `{"minimum":0}` | | `sources.live.preferred_peers` | `nonNegativeInteger` | no | `{"minimum":0}` | | `sources.live.max_outbound_peers` | `nonNegativeInteger` | no | `{"minimum":0}` | | `sources.live.max_concurrent_dials` | `nonNegativeInteger` | no | `{"minimum":0}` | | `sources.live.listener_port` | `integer` | no | `{"minimum":0,"maximum":65535}` | | `sources.live.discovery_port` | `integer` | no | `{"minimum":0,"maximum":65535}` | | `sources.live.discv5_port` | `integer` | no | `{"minimum":0,"maximum":65535}` | | `sources.live.enable_discv5` | `boolean` | no | `—` | | `sources.live.nat` | `string` | no | `—` | | `sources.live.trusted_peers` | `array` | no | `—` | | `sources.live.bootstrap_dns_tree` | `string`Optional authenticated EIP-1459 execution peer-hint tree. It affects discovery only; execution material remains protocol-verified locally. | no | `{"pattern":"^enrtree://"}` | | `sources.live.peer_refill_interval_ms` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.peer_recovery_timeout_seconds` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.stall_timeout_seconds` | `integer`Exit the node, so that its process supervisor restarts it, after the live lane stays not ready this long while verified finality is ready. | no | `{"default":600,"minimum":360,"maximum":2592000}` | | `sources.live.material_request_concurrency` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.material_request_blocks` | `integer` | no | `{"minimum":1,"maximum":16}` | | `sources.live.request_timeout_seconds` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.request_retries` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.request_retry_backoff_ms` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.history_header_request_concurrency` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.history_header_request_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.persistent_retries` | `boolean` | no | `—` | | `sources.live.retry_backoff_max_seconds` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.peer_store_flush_seconds` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.peer_store_max_entries` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.archive_reconciliation_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.archive_reconciliation_interval_seconds` | `positiveInteger` | no | `{"minimum":1}` | | `sources.live.history_fallback_blocks` | `positiveInteger`Optional maximum finalized suffix eligible for execution-P2P history fallback. When omitted, every gap at or after the processor start block is eligible. | no | `{"minimum":1}` | | `sources.live.handoff_overlap_blocks` | `positiveInteger` | no | `{"minimum":1}` | ## `processors` | Field | Type / values | Required here | Constraints | | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | --------------------------------- | | `processors` | `array` | yes | `{"minItems":1}` | | `processors[].id` | `string` | yes | `{"minLength":1}` | | `processors[].instance` | `string` | yes | `{"minLength":1}` | | `processors[].version` | `string` | yes | `{"minLength":1}` | | `processors[].history_control` | `node_owned \| application_subscriptions` | no | `—` | | `processors[].history_mode` | `automatic \| on_demand` | no | `—` | | `processors[].require_retained_input` | `boolean` | no | `—` | | `processors[].start_block` | `nonNegativeInteger` | yes | `{"minimum":0}` | | `processors[].publish` | `unattested \| attested \| finalized`First chain-confidence level the processor's consumers see. | no | `{"default":"unattested"}` | | `processors[].state` | `statePolicy` | yes | `—` | | `processors[].state.mode` | `durable \| checkpointed` | yes | `—` | | `processors[].artifacts` | `artifactPolicy` | no | `—` | | `processors[].artifacts.mode` | `none \| window \| full`Retention of immutable finalized map results, independent of queryable output and delivery. | no | `{"default":"none"}` | | `processors[].artifacts.window` | `artifactWindow`Required in window mode, and only valid there. | no | `—` | | `processors[].artifacts.window.max_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `processors[].artifacts.window.max_age` | `humanDuration` | no | `—` | | `processors[].artifacts.window.max_bytes` | `humanBytes` | no | `—` | | `processors[].output` | `outputPolicy` | yes | `—` | | `processors[].output.mode` | `none \| latest \| window \| full` | yes | `—` | | `processors[].output.window` | `outputWindow` | no | `—` | | `processors[].output.window.max_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `processors[].output.window.max_age` | `humanDuration` | no | `—` | | `processors[].output.window.max_rows` | `positiveInteger` | no | `{"minimum":1}` | | `processors[].output.window.max_bytes` | `humanBytes` | no | `—` | | `processors[].delivery` | `deliveryPolicy` | yes | `—` | | `processors[].delivery.mode` | `none \| best_effort \| window \| until_acknowledged` | yes | `—` | | `processors[].delivery.emit` | `changes \| job_result` | no | `{"default":"changes"}` | | `processors[].delivery.max_bytes` | `humanBytes` | no | `—` | | `processors[].delivery.max_age` | `humanDuration` | no | `—` | | `processors[].delivery.on_limit` | `pause \| fail \| expire_and_reset` | no | `—` | | `processors[].delivery.pruning` | `deliveryPruning` | no | `—` | | `processors[].delivery.pruning.interval` | `humanDuration` | yes | `—` | | `processors[].delivery.pruning.minimum_batch_blocks` | `positiveInteger` | yes | `{"minimum":1}` | | `processors[].delivery.pruning.minimum_batch_changes` | `positiveInteger` | yes | `{"minimum":1}` | | `processors[].delivery.pruning.maximum_delete_changes` | `positiveInteger` | yes | `{"minimum":1}` | | `processors[].delivery.pruning.retain_finalized_blocks` | `positiveInteger` | yes | `{"minimum":1}` | | `processors[].delivery.pruning.retain_acknowledged_age` | `humanDuration` | yes | `—` | | `processors[].delivery.consumers` | `array` | no | `—` | | `processors[].delivery.consumers[].id` | `string` | yes | `{"minLength":1,"maxLength":128}` | | `processors[].delivery.consumers[].required` | `boolean` | yes | `—` | | `processors[].delivery.consumers[].lease_ttl` | `humanDuration`From one second to 100 years. | yes | `—` | | `processors[].checkpoint` | `checkpointPolicy` | yes | `—` | | `processors[].checkpoint.mode` | `none \| automatic` | yes | `—` | | `processors[].checkpoint.keep` | `nonNegativeInteger` | yes | `{"minimum":0}` | | `processors[].undo` | `undoPolicy` | yes | `—` | | `processors[].undo.mode` | `none \| unfinalized` | yes | `—` | | `processors[].undo.safety_blocks` | `nonNegativeInteger` | yes | `{"minimum":0}` | | `processors[].coverage` | `processorCoverage` | no | `—` | | `processors[].coverage.verification_segment_blocks` | `positiveInteger` | no | `{"minimum":1}` | | `processors[].live_gap_fill` | `processorLiveGapFill`Fill the blocks a block-local live lane skips, such as after a restart at a later finalized anchor, from history before it applies the next block, instead of only announcing them. Requires history\_mode = "on\_demand" and delivery.on\_limit = "pause". Every live lane waits while a fill runs. | no | `—` | | `processors[].live_gap_fill.max_blocks` | `integer`Largest skipped range the lane fills; it announces a longer one, and the rest of a fill that fails or runs longer than five minutes. | yes | `{"minimum":1,"maximum":7200}` | | `processors[].settings` | `object`Strictly decoded by the selected processor factory | no | `—` | ## `rpc` | Field | Type / values | Required here | Constraints | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | ------------------------------- | | `rpc` | `rpc` | yes | `—` | | `rpc.http_bind` | `socketAddress`IPv4 or bracketed IPv6 socket address with port | yes | `—` | | `rpc.ws_bind` | `socketAddress`IPv4 or bracketed IPv6 socket address with port | yes | `—` | | `rpc.historical_mode` | `on_demand \| disabled` | yes | `—` | | `rpc.minimum_recent_blocks` | `positiveInteger` | yes | `{"minimum":1}` | | `rpc.allow_unauthenticated_remote` | `boolean`Serve JSON-RPC, which has no authentication, on a non-loopback http\_bind or ws\_bind. Only behind a trusted network or authenticating proxy. | no | `{"default":false}` | | `rpc.allowed_origins` | `allowedOrigins`Browser origins, besides loopback ones on any port, that may call JSON-RPC over HTTP or WebSocket. Any other Origin gets HTTP 403. | no | `{"default":[]}` | | `rpc.max_batch_requests` | `positiveInteger`Most calls in one batch; a longer batch is refused whole with -32600. | no | `{"minimum":1,"default":100}` | | `rpc.max_response_bytes` | `humanBytes`Most encoded bytes of one response or batch response; each call past it is answered with -32005. | no | `{"default":"16MiB"}` | | `rpc.max_log_results` | `positiveInteger`Most logs one eth\_getLogs call returns; more fail with -32005 and a smaller block range to retry. | no | `{"minimum":1,"default":10000}` | | `rpc.max_log_addresses` | `positiveInteger`Most addresses in one log filter; more fail with -32602. | no | `{"minimum":1,"default":1000}` | | `rpc.max_log_topic_alternatives` | `positiveInteger`Most alternatives at one topic position of a log filter; more fail with -32602. | no | `{"minimum":1,"default":1000}` | | `rpc.max_subscriptions_per_connection` | `positiveInteger`Most live subscriptions on one WebSocket connection; more fail with -32005. | no | `{"minimum":1,"default":128}` | | `rpc.max_websocket_connections` | `positiveInteger`Most open WebSocket connections; a further upgrade gets HTTP 503. | no | `{"minimum":1,"default":256}` | | `rpc.max_outbound_bytes` | `humanBytes`Queued subscription notification bytes shared across WebSocket connections and listeners; at least max\_subscription\_event\_bytes. | no | `{"default":"64MiB"}` | | `rpc.max_subscription_event_bytes` | `humanBytes`Most encoded bytes of subscription notifications one chain event may produce for one WebSocket connection, and that a connection may leave unread. A connection past the first is closed with 1008, past the second with 1013. | no | `{"default":"16MiB"}` | # Ethereum JSON-RPC compatibility > See exactly which Ethereum JSON-RPC methods Leani implements, refuses, or exposes only through native APIs. Status: metadata, bounded recent-material HTTP, and live WebSocket profiles are implemented. Canonical header/transaction/receipt bytes are decoded into Alloy RPC types. Bounded on-demand number/range queries are implemented for configured sources that declare every required capability complete. ## 1. Product promise Leani can replace a paid RPC for the common methods its configured sources can answer exactly. It is not a full execution/archive node. Three answer paths exist: * **Recent:** serve from the rolling raw block/receipt cache. * **On-demand:** fetch the relevant historical chunk from Xatu/EraE/another backend, answer slowly, and discard it. * **Materialized:** serve from an operator-enabled durable index. The response should expose a non-standard diagnostic header or companion method indicating source, finality, and whether an answer was recent, on-demand, or materialized. ## 2. Method matrix The currently shipped `leani-evm-free-v1` profile implements `web3_clientVersion`, `net_version`, `eth_chainId`, `eth_blockNumber`, `eth_syncing`, `eth_config`, and `leani_getCapabilities` over HTTP, including JSON-RPC batches and notifications. The WebSocket endpoint also accepts regular calls and implements `newHeads` and filtered `logs` subscriptions from post-commit live events. Methods needing raw blocks, receipts, or logs return the explicit server error `-32004` until that material is present; state/EVM methods remain unsupported. The capability response is the machine-readable authority for a running node. `eth_config` follows final EIP-7910: it accepts no parameters and returns `current`, `next`, and `last` fork configurations with JSON-number blob parameters, chain ID, fork hash, precompiles, and system contracts. Its checked Mainnet schedule uses exact activation timestamps; it does not call an upstream configuration RPC. `current` is the fork active at the timestamp of the newest retained raw canonical block. This timestamp can be unavailable even when `eth_blockNumber` knows a newer canonical anchor. Forks activate by timestamp, so a node that retains no head frame, such as an empty one or one that has only backfilled, fails with `-32004` (`eth_config_head_timestamp_unavailable`) instead of choosing a fork by block number or taking the newest one. A head older than the first scheduled fork fails with `eth_config_head_predates_schedule`, and a node whose chain the schedule is not for fails with `eth_config_chain_schedule_unavailable`. Blob transaction receipts report `blobGasPrice` as the protocol blob base fee of the fork active at the block’s timestamp, from the same schedule and fee function as the `blobs-money` processor. Receipts are built for a whole block, so when that price cannot be computed, every receipt of a block with a blob transaction fails with `-32004`, the receipts of its other transactions included, from `eth_getBlockReceipts` and `eth_getTransactionReceipt` alike, rather than carrying a guessed price. `error.data.reason` says why: * `blob_gas_price_schedule_unavailable`: no scheduled fork covers the block’s timestamp, or the schedule is for another chain. A node on a chain other than Mainnet without a blobs processor uses the checked Mainnet schedule, so the receipts of every block with a blob transaction fail this way. * `header_excess_blob_gas_missing`: the block header has no excess blob gas. * `quantity_exceeds_u128`: the price does not fit in 128 bits. Only an excess blob gas that no valid chain reaches, as a forged header may claim, gets there. `eth_getLogs` can use Xatu’s predicate-complete projection under the trusted-dataset policy. Every requested block and required log field must be present; a partial projection, a narrower address/topic scope, or a broken parent link fails explicitly. Complete-cryptographic policy still requires complete material. Retained raw-log projections can serve the same requests without upstream access when their scope and fields cover the query. Decoded processor entities alone are not a lossless raw-log store: enable on-demand history or retain suitable raw material for historical RPC. `eth_blockNumber`, `latest`, and `eth_syncing.currentBlock` report the canonical tip: the newest delivered block at any level, `unattested` included, as an Ethereum RPC’s `latest` (see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes)). They report it independently of raw-frame pruning, or the highest block the progress processor has finalized when that is higher or no tip is known, as on a node without live following. A node that knows no block reports `0x0`, as a fresh Ethereum client does; `eth_blockNumber` never fails for want of a head. A reported head can lag the network, while disconnected or on a historical-only node; consult `eth_syncing`, which reports `liveReady: false` then, before treating it as the network tip. For a number/range request outside the recent window, the router selects the lowest-priority viable `HistorySource`, enforces finality/trust and hard byte/frame/concurrency/timeout limits, validates ordered parent-linked frames, and fails over only at the whole-request boundary. Frames are discarded after serialization. A range crossing into the recent window must have an exact parent-hash handoff. Hash-only transaction/receipt history still requires a durable locator and therefore remains recent-only by default. | Method | EVM-free support | Default behavior | Caveat | | ------------------------------- | ----------------------------------- | ----------------------------------- | ---------------------------------------------------------------------- | | `web3_clientVersion` | Exact | Local | — | | `net_version` | Exact | Local | — | | `eth_chainId` | Exact | Local | — | | `eth_blockNumber` | Exact for the known canonical chain | Canonical tip or finalized coverage | `0x0` before any block is known; never an unfinalized processor cursor | | `eth_syncing` | Exact for this node | Canonical head and live readiness | A disconnected or historical-only node does not claim live readiness | | `eth_getBlockByNumber` | Exact with body source | Recent/on-demand | Historical latency depends on backend | | `eth_getBlockByHash` | Exact with locator | Recent; optional index | Hash-to-chunk lookup otherwise expensive | | `eth_getBlockTransactionCount*` | Exact with body | Recent/on-demand | — | | `eth_getTransactionByBlock*` | Exact with body | Recent/on-demand | — | | `eth_getTransactionByHash` | Exact with locator | Recent; optional index | Global tx locator consumes storage | | `eth_getTransactionReceipt` | Exact with receipt locator | Recent; optional index | Same lookup issue | | `eth_getBlockReceipts` | Exact with receipts | Recent/on-demand by block | Excellent fit for P2P/EraE | | `eth_getLogs` | Exact with complete logs | Recent/on-demand/materialized | Filtered datasets are trusted for completeness | | `eth_feeHistory` | Not shipped | Explicit unavailable error | Requires a separately tested client-compatible policy | | `eth_gasPrice` | Planned | Unsupported | Client policy semantics differ | | `eth_maxPriorityFeePerGas` | Planned | Unsupported | Client policy semantics differ | | `eth_subscribe(newHeads)` | Exact | Live | Replacement heads are emitted after branch commit | | `eth_subscribe(logs)` | Exact | Live | Reorged logs emit `removed: true` before replacements | | filter polling methods | Feasible | Later | Stateful server-side filters | | `eth_sendRawTransaction` | Feasible transport | Optional | P2P broadcast or fallback, not indexing core | | `eth_getBalance` | Not general | Unsupported | Requires executed state, including internal effects | | `eth_getTransactionCount` | Not general | Unsupported | Contract nonces/internal creation require execution | | `eth_getCode` | No | Unsupported | Requires state | | `eth_getStorageAt` | No | Unsupported | Requires state | | `eth_call` | No | Unsupported/optional forward | Requires historical code/storage and EVM | | `eth_estimateGas` | No | Unsupported/optional forward | Requires EVM | | debug/trace methods | No from receipts | Unsupported | May be served only by a trace-capable dataset/backend | “On-demand” cannot make every hash-only lookup cheap. Block-number/range queries map naturally to chunked archives. Transaction-hash queries need a locator service or a compact local `tx_hash -> block_number` index. ### Requests, block tags, and missing blocks A call whose `id` is `null` gets a response with a `null` ID; only a call without an `id` member is a notification. An `id` that is an object, an array, or a boolean gets `-32600` with a `null` ID. A call must be a JSON object: an array in its place, such as `["2.0", 1, "eth_chainId"]`, gets `-32600` with a `null` ID too. A notification runs like any other call, side effects included, and gets no response, as JSON-RPC 2.0 requires: an `eth_unsubscribe` notification ends its subscription, and an `eth_subscribe` notification opens one whose ID the client never learns. A batch answers only its calls with IDs, and a batch of only notifications gets no response at all (HTTP 204). Block numbers and transaction indexes are QUANTITY values: `0x` and one or more hexadecimal digits, without leading zeros except in `0x0`, at most 64 bits. Anything else, such as `0x+1`, `0x01`, or `0x`, gets `-32602`. Wherever a block number is accepted, including `fromBlock` and `toBlock` of `eth_getLogs`, these tags are too: | Tag | Block | | ----------- | ------------------------------------------------------------------------------ | | `latest` | the head `eth_blockNumber` reports, at any level | | `finalized` | the newest canonical block the node has verified as finalized | | `safe` | the same block as `finalized` | | `earliest` | block 0 | | `pending` | none: Leani builds no pending block, so `-32004` (`pending_block_unavailable`) | Until the node has verified a finalized block, `finalized` and `safe` fail with `-32004` (`finalized_block_unavailable`). Leani does not track the justified checkpoint that execution clients call safe, so `safe` names the finalized head, which is never newer. A client that waits for `safe` blocks to avoid reorgs therefore stays safe, but it sees an older block than other clients report. Block lookups return `null` only for a number above every block the node knows exists: its canonical tip and the progress processor’s indexed height. Ethereum clients read `null` as “no such block”, and a reorg checker concludes that a block which turned `null` was reorged out. So a block at or below the head that neither the recent window nor on-demand history serves fails with `-32004` instead: `block_not_retained`, or the history source’s reason when history was asked. A block hash that the recent window does not hold fails with `block_not_retained` unless on-demand history offers a block-hash lookup, in which case `null` means that lookup does not know the hash. This applies to `eth_getBlockByNumber`, `eth_getBlockByHash`, `eth_getBlockReceipts`, `eth_getBlockTransactionCountBy*`, and `eth_getTransactionByBlock*AndIndex`, which still returns `null` for an index past the last transaction of a block it serves. `latest` resolves the canonical tip and may fetch its raw material on demand; without a known tip it fails with `block_not_retained`. `earliest` names block 0, so it requires retained or historical material for block 0. `eth_getLogs` with `blockHash` returns only logs of the block with that hash. If a reorg replaces that block between resolving the hash and reading the block, the call fails with `-32004` rather than return the replacement’s logs, as it does once the reorg has happened (`block_hash_not_retained`). An empty alternatives array at a topic position, `[]`, matches any topic there, as `null` does. ### Transport rules and limits HTTP calls need `content-type: application/json`, with any parameters, and get HTTP 415 otherwise. A browser `Origin` other than a loopback one (`http(s)://localhost`, `127.0.0.1`, or `[::1]` on any port) or one listed in `rpc.allowed_origins` gets HTTP 403, on HTTP calls and WebSocket upgrades alike; requests without an `Origin` header pass. A cross-site page therefore cannot post a `text/plain` batch or open a WebSocket to a loopback node. A request body that is not valid UTF-8 JSON gets HTTP 200 with a `-32700` parse error and a `null` ID. A body over 1 MiB gets HTTP 413 with a JSON-RPC error, not plain text: `-32005` with a `null` ID, and `error.data` holding `reason` `request_size_limit_exceeded` and the `limit` in bytes. | Limit | Default | Setting | When exceeded | | ---------------------------------------------------------------------------------------------- | ------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Request body or WebSocket message | 1 MiB | — | HTTP 413 with a `-32005` error; a longer WebSocket message ends the connection | | Calls per batch | 100 | `rpc.max_batch_requests` | one `-32600` error object for the whole batch | | Response bytes | 16 MiB | `rpc.max_response_bytes` | `-32005` for the call that passes the limit, or whose `eth_getLogs` result would, and for every later call in the batch, which do not run; later notifications still run | | `eth_getLogs` results | 10,000 | `rpc.max_log_results` | `-32005` “query returned more than 10000 results. Try with this block range \[0x…, 0x…].” | | Filter addresses | 1,000 | `rpc.max_log_addresses` | `-32602` | | Alternatives per topic position | 1,000 | `rpc.max_log_topic_alternatives` | `-32602` | | Subscriptions per WebSocket connection | 128 | `rpc.max_subscriptions_per_connection` | `-32005` from `eth_subscribe` | | WebSocket connections | 256 | `rpc.max_websocket_connections` | HTTP 503 on the upgrade | | Subscription notifications of one chain event, per connection | 16 MiB | `rpc.max_subscription_event_bytes` | the connection is closed with code `1008` | | Unsent subscription notifications, per connection | 16 MiB | `rpc.max_subscription_event_bytes` | the connection is closed with code `1013` | | Queued or sending notifications, all WebSocket connections | 64 MiB | `rpc.max_outbound_bytes` | the connection holding the most is closed with code `1013` | | Notifications kept queued past a quarter of `rpc.max_subscription_event_bytes`, per connection | 30 s | — | the connection is closed with code `1013` | The `eth_getLogs` hint names the range from `fromBlock` up to the block before the one where the results passed the limit, and `error.data` carries it as `from`, `to`, and `limit`. The hint is missing when the first block alone has more matching logs than the limit; read that block with `eth_getBlockReceipts` or raise the limit. The filter limits also apply to `eth_subscribe("logs")` filters. Closing a WebSocket connection frees its slot and ends its subscriptions. The subscriptions of one WebSocket connection share an output budget per chain event: the notifications that one block, or one reorg with its removed and replacement logs, produces for the connection may take at most `rpc.max_subscription_event_bytes` encoded bytes. The node serializes a log or head once, however many of the connection’s subscriptions it matches, but each notification counts in full. A connection whose subscriptions pass the budget is closed with code `1008` (policy violation), without that event’s notifications, and with a close reason that names the setting. Its subscriptions would pass the budget again on a like block, so reconnect with fewer or narrower subscriptions, or spread them over several connections. The node also holds at most `rpc.max_subscription_event_bytes` of a connection’s notifications unsent. A client that reads more slowly than its subscriptions produce notifications is closed with code `1013` (try again later), as it is when it falls behind the node’s bounded event channel, and may reconnect. Notifications are queued one at a time, so while earlier ones are still unsent, a connection can get `1013` before one event’s notifications pass the budget that would close it with `1008`. Notifications still unsent when a connection is closed are dropped, and a connection that cannot take its close frame within a second is dropped without one. Other connections are unaffected. A call’s response is sent before the connection’s next message is read, so a client that stops reading stops being answered too. ## 3. RPC profiles ### Minimal * chain metadata; * current head; * recent blocks, transactions, receipts, logs; * recent fee history; * new-head and log subscriptions; * custom aggregate queries/subscriptions. Storage is aggregate output plus a small recent window. ### Indexer Minimal plus: * historical `eth_getLogs`; * block-number-based historical blocks/receipts; * configurable log materialization; * longer recent window. Historical answers may be slow unless their data is materialized. ### RPC-heavy Indexer plus: * transaction locator; * transaction/receipt index; * larger raw window; * response cache; * optional forwarding for state methods. This profile intentionally uses more storage. ## 4. ERC-20 balance semantics ### What can be derived For a token where every balance change produces a standard: ```solidity Transfer(address indexed from, address indexed to, uint256 value) ``` the node can maintain: ```text balance[token, from] -= value balance[token, to] += value ``` with zero-address handling for mint and burn. Logs from reverted transactions do not appear in the canonical receipt and therefore do not change the derived balance. ### What can break the model * rebasing or reflection changes balances without per-holder `Transfer` events; * deliberately non-standard tokens; * unusual proxy upgrades that change balance semantics; * tokens whose historical starting state predates the selected start block; * metadata changes or contracts that revert on metadata calls; * migrations that create balances outside normal transfer events. The API should call this an `event-derived token balance`, include the processed block, and expose its assumptions. “Declared” means that the processor manifest and query result declare: * the chain, token/address scope, and starting block; * the derivation method, such as `erc20_transfer_ledger`; * the highest processed and finalized blocks; * whether historical backfill and the live handoff are complete; * the token behavior assumptions under which the value is exact. For a conventional ERC-20 indexed from before its initial mint, this value should be exact. The declaration exists to distinguish that result from generic EVM state for rebasing or non-standard tokens. The node should not transparently intercept arbitrary `eth_call(balanceOf)` and claim universal equivalence. Illustrative response metadata: ```json { "token": "0x...", "address": "0x...", "balance": "123000000", "asOfBlock": 25620123, "finalizedThrough": 25620064, "method": "erc20_transfer_ledger", "coverageFrom": 1234567, "complete": true } ``` A custom `idx_getTokenBalances` method should be the canonical API. An optional `alchemy_getTokenBalances` compatibility adapter could be added for migrations, but it must return only the declared indexed coverage and completeness metadata. ### Storage modes | Mode | Retained state | New arbitrary address | | --------------------- | -------------------------------- | ------------------------------- | | Address watchlist | Only watched address/token pairs | Requires targeted backfill | | Token allowlist | All holders for selected tokens | Immediate within allowed tokens | | Global current ledger | All non-zero token/address pairs | Immediate, larger database | | Full balance history | Every change/checkpoint | Largest; opt-in | For minimal deployments, watchlist mode is the default. Xatu’s predecoded ERC-20 transfer Parquet table makes a targeted historical backfill possible without querying a paid RPC. The live source still sees all receipt logs but retains only relevant deltas. Adding a watch address should be a first-class job: 1. atomically register address and start cursor; 2. capture new live deltas for it immediately; 3. backfill historical incoming/outgoing transfers; 4. reconcile at an overlapping block; 5. mark the derived balance complete. This is the same cold/hot handoff problem in miniature. ### Historical balance retention Current balance requires one row per retained `(token, address)`. Historical `balance at block N` requires more: * every transfer delta; * periodic snapshots plus later deltas; * or re-running a public-data query on demand. The default should keep current balances and a small undo window. Historical balance queries can be slower and replay public transfer data unless explicitly materialized. ## 5. Native ETH balances Native balances are not equivalent to top-level transaction value transfers. They also change through: * gas charges and priority fees; * internal calls; * contract creation; * withdrawals; * protocol rewards in older history; * other execution-level balance changes. Xatu currently publishes `canonical_execution_balance_diffs` and native transfer tables, so it can act as a trusted historical executed-state source. That does not solve low-latency live native balances through receipt-only P2P: generating complete live balance diffs requires EVM execution or another executed-data provider. Therefore: * `idx_getNativeBalance` may be offered when a `STATE_DIFFS` backend is configured and sufficiently current; * standard `eth_getBalance` remains unsupported until the node has an exact state source; * responses must report backend lag and trust. ## 6. NFT balances * ERC-721 ownership is generally derivable from `Transfer`. * ERC-1155 requires both `TransferSingle` and `TransferBatch`. * The same non-standard/rebase caveats and retention modes apply. These should follow ERC-20 support rather than complicate the first implementation. ## 7. Aggregate subscriptions Native aggregate subscriptions should use durable cursors rather than transient WebSocket connection state: ```text processor_id processor_version block_number block_hash change_sequence finality ``` Event kinds: * `apply` * `undo` * `finalized` * `reset_required` * `processor_complete` Clients resume from a cursor. If the node has pruned the required change history, it returns `reset_required` with the latest queryable checkpoint. The v1 native transport is HTTP query plus SSE, with a TypeScript SDK that handles reconnect and opaque cursors. Standard `eth_subscribe` remains WebSocket JSON-RPC and follows Ethereum’s subscription semantics. See [native API and delivery protocol](https://leani.dev/docs/reference/native-api-and-delivery/). # Native HTTP API > Generated inventory of the versioned native HTTP contract from the tracked OpenAPI document. This inventory is generated from `api/openapi.yaml`. Operation IDs, methods, paths, summaries, authentication policy, and response codes change here in the same pull request as the contract. Native API, not Ethereum JSON-RPC The native API exposes coverage, processors, durable delivery, and typed query extensions. Ethereum JSON-RPC is a separate listener with an intentionally bounded compatibility surface. ## Administration | Method | Path | Purpose | Auth | | -------- | --------------------------------------------------- | --------------------------------------- | --------------- | | `GET` | `/admin/v1/backfill-subscriptions` | List backfill subscriptions | optional bearer | | `POST` | `/admin/v1/backfill-subscriptions` | Create a backfill subscription | optional bearer | | `GET` | `/admin/v1/backfill-subscriptions/{id}` | Inspect a backfill subscription | optional bearer | | `DELETE` | `/admin/v1/backfill-subscriptions/{id}` | Delete a terminal backfill subscription | optional bearer | | `POST` | `/admin/v1/backfill-subscriptions/{id}/cancel` | Cancel a backfill subscription | optional bearer | | `POST` | `/admin/v1/backfill-subscriptions/{id}/retry` | Retry a failed backfill subscription | optional bearer | | `GET` | `/admin/v1/materialization-jobs` | List materialization jobs | optional bearer | | `POST` | `/admin/v1/materialization-jobs` | Create a materialization job | optional bearer | | `GET` | `/admin/v1/materialization-jobs/{id}` | Inspect a materialization job | optional bearer | | `DELETE` | `/admin/v1/materialization-jobs/{id}` | Delete a terminal materialization job | optional bearer | | `POST` | `/admin/v1/materialization-jobs/{id}/cancel` | Cancel a materialization job | optional bearer | | `DELETE` | `/admin/v1/processors/{processor}/artifacts` | Delete retained processor artifacts | optional bearer | | `POST` | `/admin/v1/processors/{processor}/artifacts/replay` | Replay retained processor artifacts | optional bearer | | `POST` | `/admin/v1/processors/{processor}/lanes/live/reset` | Reset a processor live lane for replay | optional bearer | | `GET` | `/admin/v1/raw-history-jobs` | List raw-history jobs | optional bearer | | `POST` | `/admin/v1/raw-history-jobs` | Create a raw-history job | optional bearer | | `GET` | `/admin/v1/raw-history-jobs/{id}` | Inspect a raw-history job | optional bearer | | `DELETE` | `/admin/v1/raw-history-jobs/{id}` | Delete a terminal raw-history job | optional bearer | | `POST` | `/admin/v1/raw-history-jobs/{id}/cancel` | Cancel a raw-history job | optional bearer | ## Other | Method | Path | Purpose | Auth | | ------ | ---------------- | -------------------------------------------- | ------ | | `GET` | `/debug/network` | Render the dependency-free network dashboard | public | ## Health and observability | Method | Path | Purpose | Auth | | ------ | --------------- | ----------------------- | ------ | | `GET` | `/health/live` | Check process liveness | public | | `GET` | `/health/ready` | Check node readiness | public | | `GET` | `/metrics` | Read Prometheus metrics | public | ## Node | Method | Path | Purpose | Auth | | -------- | ------------------------------------------------------------------------ | --------------------------------------------- | --------------- | | `POST` | `/v1/backfill-subscriptions/{subscription}/consumers/{consumer}/ack` | Acknowledge a backfill subscription batch | optional bearer | | `GET` | `/v1/backfill-subscriptions/{subscription}/consumers/{consumer}/changes` | Poll changes from one backfill subscription | optional bearer | | `POST` | `/v1/backfill-subscriptions/{subscription}/consumers/{consumer}/lease` | Renew a backfill consumer lease | optional bearer | | `DELETE` | `/v1/backfill-subscriptions/{subscription}/consumers/{consumer}/lease` | Release a backfill consumer lease | optional bearer | | `GET` | `/v1/backfill-subscriptions/{subscription}/consumers/{consumer}/stream` | Stream changes from one backfill subscription | optional bearer | | `GET` | `/v1/capabilities` | Discover node capabilities | optional bearer | | `GET` | `/v1/network/status` | Read execution network manager status | public | | `GET` | `/v1/processors` | List configured processors | optional bearer | | `GET` | `/v1/status` | Read node and processor status | optional bearer | ## Processor data and delivery | Method | Path | Purpose | Auth | | -------- | ---------------------------------------------------------------------- | ----------------------------------------------------------- | --------------- | | `GET` | `/v1/processors/{processor}/artifacts/{block}` | Read one retained processor artifact | optional bearer | | `GET` | `/v1/processors/{processor}/artifacts/export` | Export a retained processor artifact range | optional bearer | | `GET` | `/v1/processors/{processor}/artifacts/status` | Read retained processor artifact coverage | optional bearer | | `GET` | `/v1/processors/{processor}/changes` | List processor changes | optional bearer | | `GET` | `/v1/processors/{processor}/changes/head` | Read the processor change head | optional bearer | | `GET` | `/v1/processors/{processor}/checkpoints` | List processor recovery checkpoints | optional bearer | | `POST` | `/v1/processors/{processor}/checkpoints/{checkpoint}/restore` | Restore a processor recovery checkpoint | optional bearer | | `GET` | `/v1/processors/{processor}/collections` | List processor collections | optional bearer | | `GET` | `/v1/processors/{processor}/collections/{collection}/entities` | Query processor entities | optional bearer | | `POST` | `/v1/processors/{processor}/collections/{collection}/entities` | Snapshot processor entities | optional bearer | | `GET` | `/v1/processors/{processor}/collections/{collection}/entities/{key}` | Read a processor entity | optional bearer | | `POST` | `/v1/processors/{processor}/collections/{collection}/query-and-follow` | Query a collection and open a race-free follow boundary | optional bearer | | `GET` | `/v1/processors/{processor}/consumers` | List durable consumers | optional bearer | | `POST` | `/v1/processors/{processor}/consumers` | Create a durable consumer | optional bearer | | `GET` | `/v1/processors/{processor}/consumers/{consumer}` | Read a durable consumer | optional bearer | | `DELETE` | `/v1/processors/{processor}/consumers/{consumer}` | Revoke a durable consumer | optional bearer | | `POST` | `/v1/processors/{processor}/consumers/{consumer}/ack` | Acknowledge durable consumer changes | optional bearer | | `GET` | `/v1/processors/{processor}/consumers/{consumer}/changes` | List durable consumer changes | optional bearer | | `POST` | `/v1/processors/{processor}/consumers/{consumer}/lease` | Renew a durable consumer lease | optional bearer | | `DELETE` | `/v1/processors/{processor}/query-snapshots/{snapshot}` | Release a processor query snapshot | optional bearer | | `GET` | `/v1/processors/{processor}/query/blocks/{number}` | Read an indexed Ethereum block summary by block number | optional bearer | | `GET` | `/v1/processors/{processor}/query/latest` | Read the latest indexed Ethereum block summary | optional bearer | | `GET` | `/v1/processors/{processor}/query/pools` | List pools configured for a Uniswap observation processor | optional bearer | | `GET` | `/v1/processors/{processor}/query/pools/{address}/latest` | Read the latest observation for one configured Uniswap pool | optional bearer | | `GET` | `/v1/processors/{processor}/savepoints` | List processor savepoints | optional bearer | | `POST` | `/v1/processors/{processor}/savepoints` | Create a processor savepoint | optional bearer | | `GET` | `/v1/processors/{processor}/savepoints/{savepoint}` | Export a processor savepoint | optional bearer | | `DELETE` | `/v1/processors/{processor}/savepoints/{savepoint}` | Delete a processor savepoint | optional bearer | | `GET` | `/v1/processors/{processor}/schema` | Read a processor schema | optional bearer | | `GET` | `/v1/processors/{processor}/status` | Read processor status | optional bearer | | `GET` | `/v1/processors/{processor}/stream` | Stream processor changes | optional bearer | | `POST` | `/v1/processors/{processor}/streams/live/consumers/{consumer}/ack` | Acknowledge a live-lane consumer batch | optional bearer | | `POST` | `/v1/processors/{processor}/streams/live/consumers/{consumer}/lease` | Renew a live-lane consumer lease | optional bearer | | `DELETE` | `/v1/processors/{processor}/streams/live/consumers/{consumer}/lease` | Release a live-lane consumer lease | optional bearer | | `GET` | `/v1/processors/{processor}/streams/live/consumers/{consumer}/stream` | Stream the dedicated live lane for a durable consumer | optional bearer | ## Typed query extensions | Method | Path | Purpose | Auth | | ------ | ---------------------------------------- | ---------------------------------- | --------------- | | `GET` | `/v1/q/blobs/blocks` | List indexed blob blocks | optional bearer | | `GET` | `/v1/q/blobs/blocks/{number}` | Read an indexed blob block | optional bearer | | `GET` | `/v1/q/blobs/schedule` | Read the active blob schedule | optional bearer | | `GET` | `/v1/q/blobs/snapshot` | Read an atomic blob block snapshot | optional bearer | | `GET` | `/v1/q/blobs/transactions` | List indexed blob transactions | optional bearer | | `GET` | `/v1/q/blobs/transactions/{hash}` | Read an indexed blob transaction | optional bearer | | `GET` | `/v1/q/erc20/balances/{token}/{address}` | Read an indexed ERC-20 balance | optional bearer | | `GET` | `/v1/q/uniswap/pools/{address}` | Read an indexed Uniswap pool | optional bearer | # Native API and delivery protocol > Understand snapshots, retained queries, durable consumers, change envelopes, errors, and capability discovery. Status: implemented v1 contract baseline, 2026-08-01. Leani exposes two related interfaces: 1. Ethereum JSON-RPC for applications expecting an execution-client-shaped endpoint. 2. A native aggregate API for processor status, queries, and durable change streams. The TypeScript SDK targets the native API. Existing Ethereum libraries such as ethers or viem can use the node’s JSON-RPC URL directly. ## 1. Why HTTP plus SSE first blobs.money runs TypeScript on Bun and already uses `fetch`, ethers, and WebSockets. The first aggregate transport should therefore be: * ordinary versioned HTTP/JSON for queries; * server-sent events over streaming `fetch` for a one-way durable change feed; * OpenAPI for machine-readable query/request/response types. SSE fits the required direction, works through common proxies, has a simple sequence/resume model, and does not require the application to implement a bidirectional protocol. The SDK uses streaming `fetch` rather than the browser `EventSource` API so it can send authorization headers and work consistently in Bun, Node, and modern browsers. Standard `eth_subscribe` remains WebSocket JSON-RPC. gRPC can be added for high-throughput non-JavaScript consumers without changing change semantics. ## 2. Interface layout Default listeners: ```text http://127.0.0.1:8080/v1/... aggregate API http://127.0.0.1:8545 Ethereum JSON-RPC ws://127.0.0.1:8546 Ethereum subscriptions ``` The current binary serves all three listeners. The WebSocket endpoint accepts ordinary JSON-RPC calls plus `eth_subscribe`/`eth_unsubscribe` for `newHeads` and filtered `logs`. Subscription events are published only after the live branch commits; reorged logs are sent with `removed: true` before replacement logs. A client that falls behind the bounded event channel, or leaves more than `rpc.max_subscription_event_bytes` of notifications unread, is closed with code 1013 and must reconnect. `eth_subscribe` limits apply per connection: at most `rpc.max_subscriptions_per_connection` subscriptions, whose notifications for one block or reorg may take at most `rpc.max_subscription_event_bytes`. A connection past that is closed with code 1008; reconnect with fewer or narrower subscriptions, or spread them over several connections. When the node shuts down, every connection is closed with code 1001. See [Ethereum JSON-RPC limits](https://leani.dev/docs/reference/ethereum-json-rpc/#transport-rules-and-limits). Aggregate routes: ```text GET /health/live GET /health/ready GET /debug/network GET /metrics GET /v1/status GET /v1/network/status GET /v1/capabilities GET /v1/processors GET /v1/processors/{processor}/status GET /v1/processors/{processor}/schema GET /v1/processors/{processor}/changes/head GET /v1/processors/{processor}/changes GET /v1/processors/{processor}/stream GET /v1/processors/{processor}/collections GET /v1/processors/{processor}/collections/{collection}/entities POST /v1/processors/{processor}/collections/{collection}/entities GET /v1/processors/{processor}/collections/{collection}/entities/{key} POST /v1/processors/{processor}/collections/{collection}/query-and-follow DELETE /v1/processors/{processor}/query-snapshots/{snapshot} GET /v1/processors/{processor}/checkpoints POST /v1/processors/{processor}/checkpoints/{checkpoint}/restore GET /v1/processors/{processor}/savepoints POST /v1/processors/{processor}/savepoints GET /v1/processors/{processor}/savepoints/{savepoint} DELETE /v1/processors/{processor}/savepoints/{savepoint} GET /v1/processors/{processor}/consumers POST /v1/processors/{processor}/consumers GET /v1/processors/{processor}/consumers/{consumer} DELETE /v1/processors/{processor}/consumers/{consumer} POST /v1/processors/{processor}/consumers/{consumer}/lease GET /v1/processors/{processor}/consumers/{consumer}/changes POST /v1/processors/{processor}/consumers/{consumer}/ack GET /v1/processors/{processor}/query/* GET /v1/q/blobs/blocks GET /v1/q/blobs/blocks/{number} GET /v1/q/blobs/snapshot GET /v1/q/blobs/transactions GET /v1/q/blobs/transactions/{hash} GET /v1/q/blobs/schedule GET /v1/q/erc20/balances/{token}/{address} GET /v1/q/uniswap/pools/{address} ``` Health distinguishes process/store liveness from required live/finality readiness and processor range coverage. The operational routes, `/health/*`, `/debug/network`, `/metrics`, and `/v1/network/status`, are not behind the optional API bearer token so a protected local supervisor can probe them. When a bearer token is configured, every other route above, reads and writes alike, needs it, as the `/admin/v1` routes below do: `/v1` names the API version, not a public tier. Neither health route scans store tables: liveness runs a trivial SQLite query, and readiness adds in-memory actor state. `/metrics` and `/v1/network/status` reuse store row and byte counts for up to 10 seconds. ERC-20 and Uniswap streams share the generic processor cursor contract; their typed queries and changes encode large quantities as lossless decimal strings. The `/query/*` subtree is supplied by the selected processor’s optional native query extension. It is always scoped to that exact processor instance. The short `/v1/q/blobs`, `/v1/q/erc20`, and `/v1/q/uniswap` forms are aliases installed only when a single configured extension claims the name; canonical paths are reported by `/v1/capabilities`. Local administration, intentionally separate from the public SDK: ```text POST /admin/v1/materialization-jobs GET /admin/v1/materialization-jobs GET /admin/v1/materialization-jobs/{id} POST /admin/v1/materialization-jobs/{id}/cancel DELETE /admin/v1/materialization-jobs/{id} POST /admin/v1/backfill-subscriptions GET /admin/v1/backfill-subscriptions GET /admin/v1/backfill-subscriptions/{id} POST /admin/v1/backfill-subscriptions/{id}/cancel POST /admin/v1/backfill-subscriptions/{id}/retry DELETE /admin/v1/backfill-subscriptions/{id} POST /admin/v1/raw-history-jobs GET /admin/v1/raw-history-jobs GET /admin/v1/raw-history-jobs/{id} POST /admin/v1/raw-history-jobs/{id}/cancel DELETE /admin/v1/raw-history-jobs/{id} POST /admin/v1/processors/{processor}/lanes/live/reset DELETE /admin/v1/processors/{processor}/artifacts POST /admin/v1/processors/{processor}/artifacts/replay ``` There is no route that rebuilds a processor instance in place. A processor whose coverage contradicts the canonical chain is rebuilt as a replacement instance; its live-lane reset answers `409 live_lane_requires_rebuild` (see the operations runbook). The first administration client can be the Rust CLI over a Unix socket or localhost-only HTTP. Starting or cancelling a backfill changes durable state and should not be mixed with public queries. Processor historical scheduling is independent from RPC history. With `history_mode = "on_demand"`, the canonical live lane and normal JSON-RPC/head subscriptions still start immediately, while that processor waits for an explicit historical job instead of automatically filling from its semantic `start_block`. Create an idempotent node-owned range-to-SQLite job: ```http POST /admin/v1/materialization-jobs content-type: application/json { "processor": "blobs-production", "fromBlock": 25600000, "toBlock": 25610000, "mode": "fill_missing", "idempotencyKey": "local-analysis-25600000-25610000-v1" } ``` `history_control = "node_owned"` plus `history_mode = "on_demand"` permits these jobs. They process missing finalized coverage, retain output according to the processor’s SQLite output policy, and require no consumer or acknowledgement. The request range is not artificially capped; bounded source chunks and runtime budgets keep memory independent from total range length. Application database repair instead uses `POST /admin/v1/backfill-subscriptions` on an `application_subscriptions + on_demand` processor. That operation creates its required consumer and independent history delivery stream atomically. Its `fill_missing` and `recompute` publication semantics are consumer-facing and do not grant the application scheduling authority over a node-owned processor. All control routes share protected API authorization and do not alter JSON-RPC or WebSocket availability. A `fill_missing` subscription owns the creation-time gaps in its requested ranges. If another overlapping job covers one of those blocks first, the subscription still republishes that verified block into its independent stream; shared processor coverage cannot silently satisfy another subscription’s delivery obligation. Completed, failed, and cancelled jobs are terminal and are never retried implicitly; re-submitting a request with the same idempotency key returns the terminal job’s status. A failed subscription is retried explicitly with `POST /admin/v1/backfill-subscriptions/{id}/retry`, for example after a transient outage or after upgrading past the fault that failed it. It resumes from its committed progress with the same history stream, consumer, credential, and acknowledgement, and its error is cleared. Retrying a subscription that is still active returns its status unchanged; a completed or cancelled subscription gets `409 backfill_conflict`. A failed subscription whose processor instance is no longer configured gets `400 invalid_request` and stays as it was. A failed materialization job is not retried: delete it and create it again, which is what `leani backfill` does with a fresh idempotency key. A failed or cancelled subscription’s stream still delivers every record its job committed, then holds its session, with heartbeats, until the consumer has acknowledged what it was delivered, and only then ends with `backfill_failed` or `backfill_cancelled`. Deleting such a subscription is refused with `409 backfill_conflict` while its required consumer has unacknowledged records: read the stream to that end and acknowledge, or delete with `?discardUnacknowledged=true` when the application does not need them. The parameter never applies to a subscription that is still running. Deleting a job right after cancelling it waits up to 10 seconds for its task to stop, and answers `503 backfill_unavailable`, which is retryable, if it has not; a retry waits the same way for a failed job’s task. A subscription whose job has finished is `draining` until its required consumer acknowledges the `backfill_complete` record, which makes it `complete_reclaimable`; its stream ends once that acknowledgement lands. A stream opened after that sends the same `backfill_complete` record again, with the same `cursor`, and ends at once instead of sending heartbeats indefinitely. Acknowledging that cursor again with the stream’s session token succeeds even after the stream has ended, until another stream replaces that session. Should the record no longer be readable, the stream ends with `backfill_completed` instead. A draining subscription can be cancelled as well, for example when its application will never read it to the end. It becomes `cancelled` and keeps its job’s report. Its stream still serves the records committed before the cancellation, `backfill_complete` among them, but acknowledging them leaves the subscription `cancelled`, and a stream opened after that ends with `backfill_cancelled`. Delete it once they are acknowledged, or with `?discardUnacknowledged=true` to abandon them. Cancelling it and deleting it with `?discardUnacknowledged=true` work even after its processor instance was removed from the configuration, when its stream and acknowledgement routes refuse (normally `404 not_found`). A subscription’s status names its required consumer in `consumer`, so an application finds its own subscriptions in `GET /admin/v1/backfill-subscriptions` by that field instead of parsing the opaque `id`; a materialization job’s `consumer` is `null`. `processorConfigured` says whether the node configures the processor instance named in `processor` now: exactly that instance, not another instance of the same kind. While it is `false`, the stream and acknowledgement routes refuse the subscription. That can be temporary, as when a restart’s configuration leaves the instance out by mistake. If the instance will not return, cancel the subscription and delete it with `?discardUnacknowledged=true`, which discards its unacknowledged records; [Processor rebuild or rollback](/docs/operations/runbook/#processor-rebuild-or-rollback) gives the steps around replacing an instance. ## 3. Common response model Every processor query includes coverage: ```ts export interface ProcessorCoverage { chainId: number; chainFinalizedHead: { number: number; hash: `0x${string}`; } | null; requested?: { fromBlock: number; toBlock: number; } | null; available: Array<{ fromBlock: number; toBlock: number; finality: "unattested" | "attested" | "finalized"; }>; configuredStartBlock: number; processedThrough: number | null; finalizedThrough: number | null; complete: boolean; state: | "starting" | "backfilling" | "catching_up" | "live" | "degraded" | "paused" | "failed"; } ``` `paused` and `failed` report the processor’s live lane; `/v1/network/status` gives its `pauseReason` and first unapplied block (`liveGap`). Otherwise `starting` means nothing is applied yet and `backfilling` that the processor’s range has gaps. `catching_up` means it has none but the processor has not reached the chain’s finalized head (`chainFinalizedHead`), or the node is not ready yet. Only then is it `live`. Every `available` interval is inclusive and has one exact finality for every block it contains. Intervals split at both coverage gaps and finality boundaries, so consumers selecting finalized material must filter these ranges rather than treating `finalizedThrough` as a contiguous watermark. `finalizedThrough` is only the highest covered block recorded as finalized; lower gaps or non-finalized coverage can still exist. For a block-local processor, `available` can contain several intervals while a backfill and head follower operate concurrently. `processedThrough` is the last block of the highest interval, even when earlier ones leave gaps; `complete` says whether the requested range, or the range from the configured start to the processor’s cursor, has none. All list responses use opaque pagination cursors: ```ts export interface Page { data: T[]; nextCursor: string | null; coverage: ProcessorCoverage; } ``` Clients must not parse a cursor or construct one from a block number. Materialized-view consumers bootstrap without racing the live stream through the generic query-and-follow operation: 1. `POST /v1/processors/{processor}/collections/{collection}/query-and-follow`; 2. import every page, reading the next one with `GET .../entities?cursor=` and the page’s opaque `nextCursor`; and 3. follow `/v1/processors/{processor}/changes` or `/stream` after the returned `boundaryCursor`. Snapshot rows and the change boundary are captured in one SQLite transaction. Concurrent commits therefore fall strictly after the boundary. Pages remain stable for the advertised TTL even while new blocks commit. The response reports actual `rowCount`/`valueBytes`, retained bounds, coverage, and an explicit recovery action. The typed `/v1/q/blobs/snapshot` route remains a built-in convenience wrapper. ### Durable consumer registration Required or best-effort consumers have a server-side cumulative delivered and acknowledged watermark. A `required` consumer also holds back pruning until it acknowledges, so only a stream that retains until acknowledged accepts one: a live stream with `until_acknowledged` delivery, or a backfill stream. A `window` or `best_effort` stream prunes by its own limits and rejects a new `required` consumer with `400`. Re-creating a consumer that already exists returns `409 consumer_exists` whatever its role. Creation always names a start position: ```http POST /v1/processors/usdc-weth-latest/consumers content-type: application/json { "id": "prices-api", "role": "best_effort", "start": { "position": "earliest_retained" }, "leaseTtlSeconds": 300, "credential": "" } ``` `position` is `earliest_retained`, `current_head`, or `cursor` with a `cursor` field. An old, wrong-store, wrong-instance, or incompatible cursor returns `reset_required`; the node never substitutes another position. `leaseTtlSeconds` is 1 to 3,153,600,000 (100 years), for a backfill subscription’s consumer too; another value gets `400`. A consumer created with a credential needs it on every consumer-scoped call, in addition to any bearer token: the lease, change, and acknowledgement routes below, and its live and backfill streams with their lease and acknowledgement routes: ```http x-leani-consumer-credential: ``` A missing or wrong credential gets `403`, whatever the bearer token. The node compares credentials in constant time and stores each as a BLAKE3 hash keyed with a random per-store secret. A backfill subscription’s credential also enters the subscription’s idempotency identity, an unkeyed BLAKE3 hash of the whole request that the store keeps, so use a random credential. A new credential holds 32 to 512 printable ASCII characters without spaces, such as the output of `openssl rand -hex 32`. A consumer or backfill subscription created before that rule keeps its credential: re-creating the consumer still gets `409 consumer_exists`, and re-submitting the subscription’s request unchanged its status. A consumer declared in the node configuration has no credential and takes none. Credentials isolate applications on these consumer-scoped delivery routes only. Listing consumers, inspecting one with `GET .../consumers/{consumer}`, and revoking one with `DELETE` on that path are administrative operations that only the bearer token protects, whether or not the consumer has a credential. The list already shows what an inspection does, an operator must be able to revoke an abandoned consumer whose credential is lost, and without a bearer token every client that reaches the API is trusted with the administrative routes, such as live-lane resets, anyway. Read a batch: ```http GET /v1/processors/usdc-weth-latest/consumers/prices-api/changes?limit=500 x-leani-consumer-credential: ... ``` After committing the destination transaction and its cursor atomically: ```http POST /v1/processors/usdc-weth-latest/consumers/prices-api/ack x-leani-consumer-credential: ... content-type: application/json { "cursor": "" } ``` Acknowledgement is monotonic and idempotent: repeating the acknowledged cursor succeeds and moves no progress. The session token of a stream that has ended may still repeat it, without renewing the lease, until another stream replaces that session. A page read or a stream batch records the last sequence it hands out as the consumer’s delivered sequence (`deliveredSequence`), and a cursor beyond it gets `409 acknowledgement_beyond_delivered`, so a buggy client cannot let retention prune changes it never received. A live stream batch holds one block’s whole commit, and a live acknowledgement must end one, as each batch’s `acknowledgeableCursor` does (`409 acknowledgement_mid_block` otherwise). A backfill acknowledgement must be a batch’s `acknowledgeableCursor` or the completion’s `cursor`. A processor with a split live stream (`block_versioned_idempotent` delivery ordering, such as `blobs-money`) fences renewal and acknowledgement by streaming session. Its consumers renew and acknowledge through `/streams/live/consumers/{consumer}/lease` and `/ack` with the `x-leani-consumer-session` token from the stream’s `hello`, and the generic lease and acknowledgement routes answer `409 consumer_session_required`. Once another stream replaces a session, its token gets `409 consumer_session_lost`; a session that has only ended may still repeat its acknowledged cursor. A session token carries a MAC keyed with the per-store secret; one the node cannot verify, such as a token from before an upgrade, gets `409 consumer_session_lost`, and the client opens a new stream. The live stream’s `hello` also carries the processor’s `coverage` as the session starts, so an application can diff its own copy on every connection without polling the status route. `commit_then_ack` in the Rust API helper does not construct or execute the acknowledgement future until the destination commit succeeds. The TypeScript SDK exposes the equivalent `commitThenAcknowledge(() => destinationCommit(), cursor => client.processors.consumers.acknowledge(...))`; its acknowledgement callback is likewise never invoked after a failed destination commit. ### Generic retained queries Entity queries accept `fromBlock`, `toBlock`, `fromTimestamp`, `toTimestamp`, and `limit`. Entity keys are URL-safe hex. Every row contains the processor schema, decoded JSON data, and block/timestamp/finality metadata. There are two ways to page through a collection: * `GET .../entities` with the query as URL parameters reads current retained output in entity-key order and creates nothing. Its page holds `data`, `nextCursor`, `retainedBounds`, and `coverage`; pass `nextCursor` as `cursor` to read on. Later pages see later commits: each row appears as it was when its page was read, and a row added behind the read position is not returned. * `POST .../entities` with the query as a JSON body copies the matching rows into a stable snapshot that counts against the node-wide snapshot limits, and returns its first page with the snapshot ID, change boundary, row and byte counts, and expiry. `GET .../entities?cursor=` with its `nextCursor` reads the following pages, which stay stable while new blocks commit. ```http POST /v1/processors/{processor}/collections/{collection}/entities content-type: application/json { "fromBlock": 17000000, "toBlock": 17000999, "limit": 500 } ``` Explicit block bounds require complete processor coverage. Missing blocks return HTTP 409 `range_incomplete`, including the requested interval and coverage. The resolved block range is kept with the cursor, or the snapshot, across pagination and node restart, and each GET page checks coverage again. A block-count window uses its retention policy boundary, rather than the first matching entity, so covered blocks without events remain queryable. `retainedBounds` describes the entities present, not the limits of processor coverage. Timestamp filters select retained rows; include block bounds when you need a completeness check for a specific interval. `output = "none"` or a range outside the retained window returns `output_not_retained` with a rebuild/source-scan action. It never returns a successful empty page for unavailable local history. The TypeScript SDK’s `client.processors.queryAndFollow(processor, collection, query)` returns this snapshot and its `boundaryCursor` in one operation. Continue with `client.processors.subscribe(processor, { after: page.boundaryCursor })`. With delivery disabled, ordinary entity queries still work, and snapshots return `boundaryCursor: null` and `recovery.follow: null`. Query-and-follow, change-head, change-page, and stream requests return non-retryable HTTP 409 `delivery_disabled`; enable a delivery policy before subscribing. The snapshots of a processor that publishes from `attested` or `finalized` also return `boundaryCursor: null` and `recovery.follow: null`, and query-and-follow returns HTTP 409 `deferred_publication_snapshot`: its retained state already holds blocks whose changes are published only once they reach that level, and the stream never undoes a block it never published, so a copy followed from a boundary could keep rows of a reorged block. Release a no-longer-needed snapshot early: ```http DELETE /v1/processors/{processor}/query-snapshots/{snapshotId} x-leani-request: 1 ``` ### Compact processor artifacts When `[processors.artifacts]` is `window` or `full`, the node retains the finalized checksummed map result independently from queryable entities and delivery. Inspect its bounds and logical footprint with: ```http GET /v1/processors/{processor}/artifacts/status GET /v1/processors/{processor}/artifacts/{blockNumber} ``` Portable export is paged by block range: ```http GET /v1/processors/{processor}/artifacts/export?fromBlock=1&toBlock=1000&limit=1000 ``` The response uses `application/vnd.leani.processor-artifacts`. Its versioned durable envelope and `x-leani-artifact-digest` cover the immutable map identity, delta schema, canonical ordering, payload checksums, and requested/exported ranges. `x-leani-artifact-complete` is false when another page is needed or the retained range has a gap. Replay reduces artifacts into a distinct configured processor instance with the same immutable map/delta contract: ```http POST /admin/v1/processors/{source}/artifacts/replay content-type: application/json { "targetProcessor": "analysis-view-v1", "fromBlock": 1, "toBlock": 1000, "limit": 1000 } ``` Artifact deletion is explicit and owner-scoped. Releasing one owner preserves rows protected by another processor job or operator pin: ```http DELETE /admin/v1/processors/{processor}/artifacts?fromBlock=1&toBlock=1000 ``` The default releases the processor-instance owner. `ownerKind` and `ownerId` select `processor_job` or `operator_pin` ownership. Delivery acknowledgement, output pruning, raw-history deletion, and checkpoint pruning never delete artifacts. `[budgets.artifacts]` caps retained finalized bytes and pending unfinalized-block candidates across all processors. Admission is atomic: exceeding either limit rolls back artifacts, processor mutations, coverage, and cursor advancement. The metrics endpoint reports current and maximum artifact bytes separately; window pruning, explicit owner release, or a larger configured budget lets storage-backpressured work resume. ### Checkpoints and portable savepoints Automatic recovery checkpoints are read-only through `GET /v1/processors/{processor}/checkpoints`, are taken at most once per 1,000 finalized blocks and when a historical job completes, and obey configured count retention. `POST .../checkpoints/{checkpoint}/restore` atomically repairs private processor state only when the processor’s cursor sits exactly on the checkpoint; finality reaching that block afterwards does not matter. Between checkpoints no checkpoint can be restored: restore a full-store backup or rebuild the processor. Older checkpoints are rejected because rewinding state without also rewinding independent output, delivery, coverage, and undo storage would be unsafe; use a deterministic rebuild or a full-store backup for that case. Portable savepoints are operator-created exports, which the node cannot restore: ```http POST /v1/processors/{processor}/savepoints content-type: application/json { "id": "before-v2-upgrade" } ``` Each processor instance keeps at most 16 savepoints; creating another returns `409 savepoint_limit` until one is deleted. A savepoint is also admitted against the physical store budget and gets `507 physical_storage_limit` when it does not fit. `GET .../savepoints/{id}` exports a checksummed, versioned `application/vnd.leani.savepoint` archive. Deletion is explicit with `DELETE`; acknowledgement or checkpoint pruning never deletes a portable savepoint. Checkpoints and savepoints are bound to the processor contract without its lifecycle policies, so raising a limit does not invalidate them; a different processor version does. ## 4. Change protocol ### 4.1 Envelope ```ts export interface ChangeEnvelope { apiVersion: "1"; sequence: string; cursor: string; operation: "apply" | "undo" | "attested" | "finalized" | "reset_required"; originKind: "live" | "live_recovery" | "historical_backfill" | "recompute"; originId: string; publicationRevision: string; chainId: number; block: { number: number; hash: `0x${string}`; parentHash: `0x${string}`; timestamp: number; } | null; finality: "unattested" | "attested" | "finalized"; kind: string; schema: string; key: string | null; /** When the payload carries its own finality field, it equals the envelope finality. */ data: T | null; /** Present only on reset_required, as the current coverage hint. */ coverage?: ProcessorCoverage; emittedAt: string; } ``` Envelopes carry change-scoped facts, including the processing lane’s origin and publication revision. `originKind: "live_recovery"` marks a finalized block that a live lane applied late, from retained frames or finalized history, as in a gap replay or a live gap fill (section 4.2). Connection-scoped processor identity and node coverage are served by the stream’s `hello` frame (section 4.3) and by `/v1/status`; poll responses continue to carry one top-level `coverage` per page. `sequence` is a decimal string because a long-running stream can exceed JavaScript’s safe integer range. It is monotonically increasing in one store epoch, including across reorgs. `cursor` is an opaque base64url token that includes at least: ```text cursor format version store epoch chain ID processor instance change sequence block number/hash when applicable checksum ``` The checksum detects accidental corruption, not malicious tampering. A cursor from another store, chain, processor version, or expired epoch is rejected. ### 4.2 Operations `apply` * introduces or updates the keyed domain value; * is emitted only after the state transaction commits; * may be `unattested` or `attested` but not yet finalized (see [finality levels](/docs/concepts/coverage-finality-and-reorgs/#finality-levels-and-publish-modes)). `attested` * marks every block through `throughBlock` attested, as kind `system.finality.put` with `data: { throughBlock }`: the highest block the sync committee has attested; * is emitted only by processors that publish `unattested` blocks; * never repeats an entity. `undo` * reverses a previously committed `apply`; * contains the inverse value/change required by downstream stores; * has its own newer sequence/cursor. `finalized` * advances finality for a processor/chain, as kind `system.finality.put` with `data: { throughBlock }`; * need not repeat every entity; * permits a consumer to mark earlier changes durable. A `finalized` change of kind `system.live_gap.put` instead announces blocks the live lane skipped and did not fill, which the processor never processed, as after a restart at a later finalized anchor: `data: { fromBlock, toBlock, reason: "not_filled" }`. Only block-local processors with `history_mode = "on_demand"` announce them, before the block that follows the range, live or replayed, and in a batch of their own. After a crash between the two, the next block past the cursor announces again, the same range or a longer one; a retry the stream’s last notice already covers adds none. A processor that fills live gaps (below) may instead fill the range then, after a crash or a restart of its live lane, so `live_recovery` batches can follow a notice for the same blocks. Deduplicate batches by cursor: the notice and the following block share a block number. A processor with [`[processors.live_gap_fill]`](/docs/operations/configuration-guide/#live-gap-fill) first fills a range of at most `max_blocks` blocks from history. The live stream delivers those blocks before the following block, one batch each, in block order, as `apply` changes with `originKind: "live_recovery"` and `finality: "finalized"`, and announces nothing for them. It announces a longer range, a range history cannot serve yet, and whatever a fill leaves when it fails or runs out of time, from the first block it did not fill. Otherwise Leani does not fill the range: request it as history, such as with a `fill_missing` backfill subscription (see [Keep your application in sync](/docs/guides/keep-your-application-in-sync/)). `reset_required` * means the requested cursor is no longer resumable or belongs to an incompatible store/processor epoch; * is the only operation whose envelope carries `coverage`, as the current coverage/snapshot hint; * causes the stream to close after the event. The protocol is at-least-once across network reconnects. Consumers deduplicate by cursor/sequence and apply changes transactionally. It is exactly-once in the node’s own state, not magically exactly-once across an HTTP connection. ### 4.3 Poll and stream Polling: ```http GET /v1/processors/blobs-money/changes?after=&limit=500 ``` Streaming: ```http GET /v1/processors/blobs-money/stream?after= Accept: text/event-stream ``` Every change includes the processor descriptor’s `schema`. Built-in and custom native processors may render typed JSON in `data`; processors without a public renderer expose schema-labelled `{ "encoding": "hex", "value": "..." }`. Consumers inspect `finality` on each event rather than filtering it through a subscription query parameter. Every stream connection — initial and reconnect — opens with one `hello` frame before any change events: ```text event: hello data: {"apiVersion":"1","chainId":1,"processor":{...},"coverage":{...}} ``` The hello is synthesized at accept time from live state; it is never stored, carries **no `id:` line** (it has no position in the change log, so it can never become a resume cursor), and is never acknowledged. The SDK surfaces it through the optional `onHello` subscribe callback. Change events follow as before: ```text id: event: apply data: {"apiVersion":"1",...} ``` A browser `EventSource` reconnects on its own and sends the last event’s `id:`, its cursor, as `Last-Event-ID`; without `after`, the stream resumes after that cursor. An explicit `after` takes precedence. The server sends comments as heartbeats. When the node shuts down, it ends the stream after a whole event, with no error event, and the consumer delivery streams after a whole record. An `error` event carries the error body below and ends the stream; its `retryable` says whether reconnecting can help. A stored change the node cannot render is not retryable. The SDK reconnects with the last fully yielded cursor, uses exponential backoff with jitter, and treats these differently: * clean server close, such as the node’s shutdown: reconnect; * transient network/5xx: reconnect; * no bytes, keep-alive comments included, for `idleTimeoutMs`: reconnect; * `error` event with `retryable: true`: reconnect, and after five in a row without a new change, surface the last one; * `error` event with `retryable: false`: surface error; * authentication/validation 4xx, or a redirect: surface error; * `reset_required`: stop and require a snapshot/resubscription decision; * abort signal: stop without error. Slow consumers are disconnected before they exhaust server memory. Durable resume comes from the change log, not an unbounded per-connection buffer. ## 5. Errors HTTP errors use a stable machine-readable body: ```ts export interface LeaniErrorBody { error: { code: | "invalid_request" | "unsupported_capability" | "range_incomplete" | "source_unavailable" | "query_too_expensive" | "cursor_invalid" | "cursor_expired" | "processor_rebuilding" | "finality_unavailable" | "internal"; message: string; retryable: boolean; details?: Record; requestId: string; }; } ``` Relevant status mappings: | Status | Meaning | | ------------- | ---------------------------------------------------------------------- | | `400` | malformed parameters or invalid cursor | | `401` / `403` | authentication/authorization | | `404` | entity or processor does not exist | | `409` | requested processor version/coverage is rebuilding or incomplete | | `410` | stream cursor expired; snapshot reset required | | `413` | requested range/result exceeds configured limit | | `422` | source exists but cannot provide the required capability | | `429` | concurrency/query budget exceeded | | `503` | source/finality temporarily unavailable | | `507` | the physical store budget refuses the write (`physical_storage_limit`) | Standard JSON-RPC uses standard errors where applicable and a documented custom server-error range for unsupported capability, incomplete coverage, and resource limits. Unsupported state methods must not return `null` or zero. ## 6. Capability discovery `GET /v1/capabilities` reports actual configured capability, not everything the software could theoretically implement: ```json { "chainId": 1, "ethereumRpc": { "methods": { "eth_call": { "supported": false, "reason": "evm_state_unavailable" } } }, "processors": [ { "id": "blobs-money", "instance": "blobs-production", "version": "1.6.0", "codeHash": "0x...", "configHash": "0x...", "genericApi": "processor-v1", "changeSchema": "blobs-money.change.v1", "queryExtensions": [ { "id": "blobs-v1", "basePath": "/v1/processors/blobs-production/query", "aliasPath": "/v1/q/blobs" } ], "subscriptions": true, "artifactRetention": "none", "deliveryOrdering": "block_versioned_idempotent" } ], "queryExtensions": [ { "processor": "blobs-production", "id": "blobs-v1", "basePath": "/v1/processors/blobs-production/query", "aliasPath": "/v1/q/blobs" } ] } ``` `genericApi` identifies the common status/query/change contract and `changeSchema` identifies stream payloads. `queryExtensions` separately describes optional typed read wrappers. Applications should persist or select the processor instance and use `basePath`; aliases are conveniences and may be absent when several instances use the same extension. `ethereumRpc` names only `eth_call`, which the node never serves. The JSON-RPC method `leani_getCapabilities` reports the support and coverage of each RPC method on the RPC listener itself, so an Ethereum-library integration can inspect the endpoint without knowing the aggregate API port. ## 7. Blobs API ### 7.1 Types The native API should expose chain-derived values using lossless strings for wei and other 256-bit quantities: ```ts export interface BlobsBlock { network: string; blockNumber: number; blockHash: `0x${string}`; parentHash: `0x${string}`; timestamp: number; size: string | null; blobCount: number; blobGasUsed: string; excessBlobGas: string; blobBaseFee: string; executionBaseFee: string | null; gasUsed: string | null; gasLimit: string | null; executionEthBurnedWei: string | null; blobEthBurnedWei: string; reserveFeeWei: string | null; transactionCount: number; targetBlobsPerBlock: number; maxBlobsPerBlock: number; transformVersion: number; finality: "unattested" | "attested" | "finalized"; } export interface BlobTransaction { network: string; blockNumber: number; blockHash: `0x${string}`; txHash: `0x${string}`; senderAddress: `0x${string}`; toAddress: `0x${string}`; blobVersionedHashes: `0x${string}`[]; blobCount: number; totalBurnedWei: string; executionBurnedWei: string; blobBurnedWei: string; } export interface BlobSchedule { name: string; activationBlock: number; activationTimestamp: number; forkId: `0x${string}`; targetBlobsPerBlock: number; maxBlobsPerBlock: number; baseFeeUpdateFraction: string; eip7918: boolean; source: "chain_spec" | "eth_config" | "configured"; } ``` `blobBaseFee` is the protocol blob base fee in wei per blob gas: the EIP-4844 fee for the block’s excess blob gas under the fork active at its timestamp. `blobEthBurnedWei` and each transaction’s `blobBurnedWei` multiply it by the blob gas used. `reserveFeeWei` is the EIP-7918 reserve price from Fusaka on (`null` before); it only changes how excess blob gas evolves and is never applied to `blobBaseFee`. A schedule’s parameters apply from its `activationTimestamp`; `activationBlock` is the block-number label `atBlock` lookups use, and for Dencun it is 19,426,589, the first block the processor covers. The API should not use decimal floating point for ETH values. Presentation as ETH belongs in blobs.money. ### 7.2 Query examples ```http GET /v1/q/blobs/blocks/21000000 GET /v1/q/blobs/blocks?fromBlock=21000000&toBlock=21001000&limit=500 GET /v1/q/blobs/transactions?blockNumber=21000000 GET /v1/q/blobs/transactions/0x... GET /v1/q/blobs/schedule?atBlock=... ``` Range bounds are inclusive. Results sort by block number and, for transactions, canonical transaction index. Pagination cursors preserve this order. Querying a block-local range with a gap returns: * `409 range_incomplete` by default; * partial data plus explicit coverage only when `allowPartial=true`. The default prevents a dashboard from silently treating a backfill gap as a zero-blob period. ### 7.3 Change kinds The blobs stream emits: ```text blobs.block.put blobs.block.delete blobs.schedule.put ``` One Ethereum block produces one logical `blobs.block.put` containing its blob transactions, so a consumer can update its database atomically. Blocks and transactions remain independently queryable through their retained entity collections when processor output retention is enabled. Example domain payload: ```ts export interface BlobsBlockChange { block: BlobsBlock; transactions: BlobTransaction[]; } ``` On a reorg, `undo` contains the previous block plus transactions so a consumer can delete or restore them by canonical keys. The replacement branch then arrives as ordinary `apply` changes. A reorg of several blocks, like the revert at a restart after a long outage, undoes them newest first, and the live stream delivers each reverted block’s `undo` as a batch of its own. [Undo runs](/docs/concepts/coverage-finality-and-reorgs/#undo-runs) shows how to rebuild derived aggregates once per such run instead of once per block. ## 8. TypeScript SDK Package: ```text @smart-byte/leani-sdk ``` The SDK release workflow builds the JavaScript and type declarations and runs packed-package smoke tests in Node and Bun before publication. Prerelease versions use npm’s `latest` tag until the first stable release, and `next` after it. Runtime support: * Bun, first-class and tested in the blobs.money repository; * current Node LTS with native `fetch`; * modern browsers for public read-only endpoints; * no Node-only APIs in the core client. Public shape: ```ts import { createLeaniClient } from "@smart-byte/leani-sdk"; const node = createLeaniClient({ baseUrl: "http://127.0.0.1:8080", token: process.env.LEANI_TOKEN, timeoutMs: 15_000, }); const status = await node.status(); const capabilities = await node.capabilities(); const page = await node.blobs.listBlocks({ fromBlock: 21_000_000, toBlock: 21_001_000, }); for await (const event of node.blobs.subscribe({ after: savedCursor, signal: abortController.signal, })) { await database.transaction(async (tx) => { await applyBlobChange(tx, event); await saveCursor(tx, event.cursor); }); } ``` The change stream currently publishes every retained finality transition. Consumers inspect or filter `event.finality`; `SubscribeOptions` has no finality filter, only `after`, `signal`, `onHello`, and `idleTimeoutMs`. Core client modules: ```ts client.status() client.capabilities() client.request(path, params, options) client.processors.list() client.processors.status(id) client.processors.changeHead(id) client.processors.changes(id, options) client.processors.subscribe(id, options) client.blobs.getBlock(number) client.blobs.listBlocks(query) client.blobs.listSnapshot(query) client.blobs.getTransaction(hash) client.blobs.listTransactions(query) client.blobs.getSchedule(query) client.blobs.subscribe(options) ``` Typed helpers default to the `/v1/q/*` aliases. In a multi-instance setup, select the desired canonical `basePath` from `client.capabilities()` and pass it through `queryBasePaths` when constructing the client. This keeps typed responses while making the processor instance explicit. `request` is the safe primitive for custom query extensions. Every path resolves under the base URL’s path: `v1/processors//query/summary` and a capabilities `basePath` such as `/v1/q/my-protocol/summary` both keep a prefixed base URL’s prefix. Schemes, protocol-relative paths, segments that decode to `.` or `..`, backslashes, encoded slashes, backslashes, and percent signs, and control characters are rejected before the bearer token is attached, and redirects are refused rather than followed. A custom extension whose path parameter can hold an encoded `/` or `%` must take that value in the query string, through `request`’s `params`, instead. Built-in query namespaces use this same primitive. SDK responsibilities: * URL and query encoding; * abortable deadlines; * typed error mapping; * JSON validation at trust boundaries where practical; * SSE parsing across arbitrary byte chunk boundaries; * heartbeat handling; * reconnect/backoff/jitter; * cursor persistence hooks; * duplicate cursor suppression; * exposure of coverage on every query/stream event. SDK non-responsibilities: * automatically discarding `undo`; * silently resetting an expired cursor; * storing application checkpoints; * converting wei to floating point; * wrapping the complete Ethereum JSON-RPC API; * hiding incomplete coverage. The SDK exports pure helpers: ```ts import { applyEntityChange, compareSequences, createInMemoryCursorStore, // Tests and development only. isResetRequired, } from "@smart-byte/leani-sdk"; ``` Application database integration remains explicit because domain changes must commit before the application advances Leani’s durable acknowledgement. A crash between those steps can replay a change, so destination writes must be idempotent; Leani remains the resume authority. ## 9. OpenAPI and versioning `api/openapi.yaml` is normative for HTTP queries and error bodies. The workflow: 1. update the OpenAPI document; 2. validate it in CI; 3. generate TypeScript wire types; 4. compile the Rust route response types against schema snapshots/tests; 5. handwrite the ergonomic SDK around generated wire types; 6. run SDK contract tests against the real Rust server. Do not generate the whole SDK as a thin collection of awkward endpoint functions. Generated wire types prevent drift; the handwritten client owns streaming, retries, errors, and domain ergonomics. Compatibility rules within `/v1`: * fields are added only as optional unless a new media/API version is created; * enum expansion must be handled as an unknown value at SDK trust boundaries; * field meaning cannot change; * processor output schemas have their own semantic version; * cursors are always opaque and may change representation; * a processor rebuild that changes semantics receives a new instance/version. ## 10. Authentication and limits Localhost is the default trust boundary. The current optional API bearer token protects all native query, stream, consumer, and `/admin/v1` routes on one listener, and a bind beyond loopback requires it unless `api.allow_unauthenticated_remote` is set. A consumer’s credential, when it has one, is required as well on that consumer’s delivery routes, its lease, change, and acknowledgement routes and its streams, and fences that consumer’s lease and cursor; it does not replace the listener bearer when one is configured. Listing, inspecting, and revoking consumers need only the bearer. `/health/live`, `/health/ready`, `/metrics`, `/v1/network/status`, and `/debug/network` remain unauthenticated operational routes. The JSON-RPC listeners have no built-in authentication. Every route, operational ones included, applies these checks, in this order, to keep web pages from reading or changing a loopback node: | Request | Answer | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `Host` name other than `localhost` or one in `api.allowed_hosts`; IP addresses pass | 421 `host_not_allowed` | | browser `Origin` other than a loopback one or one in `api.allowed_origins` | 403 `origin_not_allowed` | | GET or HEAD without an `Origin`, with `Sec-Fetch-Site: cross-site` or `same-site` and a `Sec-Fetch-Mode` other than `navigate`, such as another page’s image load | 403 `cross_site_request` | | request to a consumer’s live or backfill `stream` route without an allowed `Origin`, with `Sec-Fetch-Site: cross-site` or `same-site` in any mode, navigations included | 403 `cross_site_request` | | POST or DELETE without `content-type: application/json` or `x-leani-request: 1` | 415 `unsupported_media_type` | The SDK, curl, and other non-browser clients send no `Origin` or `Sec-Fetch-*` headers, and their `Host` is the name or address in the URL they use, so their requests pass the first four checks whenever that is an IP address, `localhost`, or a name in `api.allowed_hosts`. So do navigations to other routes, including from another site’s page or into a frame, and requests from browsers that send no Fetch Metadata. A page cannot read those responses, and they create no query snapshot. Opening a consumer’s stream takes its session lease, so the stream routes refuse navigations from other sites too; a browser without Fetch Metadata can still open one, which the bearer token and the consumer’s credential prevent. The SDK sends `x-leani-request: 1` on its requests without a JSON body; other clients must add it, for example `curl -X DELETE -H 'x-leani-request: 1' ...`. When remotely exposed: * use TLS, authentication, and request/connection limits at a reverse proxy; * restrict operational routes to the monitoring network; * keep the API and RPC listeners off the public Internet unless an edge policy explicitly protects them; * use the built-in range, response-byte, concurrent-scan, stream, and storage limits as resource bounds, not as per-tenant authorization; * do not assume separate admin credentials or listeners until that isolation is implemented. The SDK never logs tokens or complete URLs containing secrets. ## 11. blobs.money integration blobs.money now carries the native consumer and application-owned reconciliation described below. The remaining work is production validation, shared-SDK adoption, cutover, and rollback evidence. ### Path A: JSON-RPC compatibility Point the existing ethers/fetch clients at the node and support: ```text eth_blockNumber eth_getBlockByNumber(number, true) eth_getBlockReceipts eth_getTransactionReceipt eth_subscribe(newHeads) eth_config, from the node's checked/versioned chain schedule ``` This minimizes initial application changes and proves RPC parity. It still causes blobs.money to fetch and transform raw blocks, so it is a bridge rather than the final design. ### Path B: native SDK Add a small ingestion adapter in blobs.money: ```text SDK blobs stream -> one database transaction upsert/delete blocks upsert/delete blob_transactions -> acknowledge the Leani-owned durable cursor -> existing application analytics ``` The implemented integration derives required historical ranges from PostgreSQL gaps, creates bounded backfill subscriptions, consumes atomic history batches and live changes, commits application rows, then acknowledges the Leani-owned durable cursor. A crash between those steps causes an idempotent replay rather than data loss; PostgreSQL does not maintain a second resume checkpoint. `full_blocks` remains a legacy/fallback concern rather than a required native-consumer input. The node owns: * Ethereum input acquisition; * fork/reorg/finality handling; * blob schedule and fee formulas; * chain-derived block and blob-transaction projections. blobs.money continues to own: * rollup address attribution unless deliberately moved into a processor; * ETH/USD presentation and staleness policy, while a slim Leani Uniswap processor supplies the indexed WETH/USDC pool observation; * API caching and presentation; * product-specific daily/weekly aggregations; * its public website schema. ### Cutover sequence 1. Replace the application-local client with the published `@smart-byte/leani-sdk` package. 2. Start Leani from application-requested ranges derived from PostgreSQL state. 3. Compare Leani output with existing `blocks` and `blob_transactions`. 4. Run the native consumer into shadow PostgreSQL tables. 5. Verify reorg, restart, expired-cursor, and backfill-gap behavior. 6. Make the SDK consumer primary while retaining Alchemy as a disabled-by- default emergency switch. 7. Observe CU usage and parity. 8. remove normal Alchemy block, receipt, and head traffic. 9. Remove `full_blocks` only after its remaining application readers are gone and a recoverable migration is approved. The application can also keep the local JSON-RPC configured for development. That removes the tendency for ordinary developer startup to consume Alchemy CUs. ## 12. API acceptance criteria * Every response validates against the checked-in OpenAPI v1 schema. * Every processor query exposes coverage and finality. * A resumed stream produces no unreported gap. * Reorg apply/undo events drive a test PostgreSQL projection to the same final state as a clean replay. * A cursor committed with an application transaction resumes safely after a process kill. * An expired cursor produces `reset_required`, never a silently truncated stream. * SDK tests pass on Bun and Node. * blobs.money can ingest a golden range using only the SDK. * JSON-RPC capability discovery accurately reports configured indexes and unsupported state methods. # Processor contracts > Reference the configuration, state, output, change, and query contracts of Leani's built-in processors. These processors ship in the standard binary. Downstream projects can assemble a custom binary and register independent native processors through the public factory/registry API; see [Build a custom processor](https://leani.dev/docs/guides/custom-processor/). ## blobs-money 1.6.0 Block-local Dencun-and-later blob economics and type-3 transaction output. It accepts the filtered Xatu projection or complete normalized execution frames. It declares the `Header`, `Body`, `Senders`, `BlobFields`, `BlockSize` and `ReceiptOutcome` field groups, filtered to type-3 transactions. All wei quantities are lossless integers. `history_mode = "on_demand"` disables only automatic historical processor coverage. It does not disable canonical live processing, normal JSON-RPC, or WebSocket head subscriptions, and it still allows `[processors.live_gap_fill]`, which fills short ranges the live lane skips (see [Live gap fill](/docs/operations/configuration-guide/#live-gap-fill)). The processor emits one atomic block change containing the block and all of its blob transactions. Materialized output stores only the block entity, transaction entities, and transaction-by-block index; snapshot queries reconstruct that same bundle without retaining a second permanent copy. The processor retains the exact post-Fusaka BPO activation schedule and keeps consensus-block size distinct from the exact execution-block RLP size. Each block uses the parameters of the fork active at its timestamp. The blob base fee and every blob burn use the protocol fee, EIP-4844’s `fake_exponential(1, excess_blob_gas, update_fraction)`, under every fork. From Fusaka on, the EIP-7918 reserve price (execution base fee × 2^13 / 2^17) is reported separately as `reserveFeeWei` and is not applied to the fee. Version 1.5 fixes the fee and burn that 1.4 overstated wherever the reserve bound; blobs.money’s own formula still clamps the fee to the reserve. Each transaction reports its `toAddress` besides its `senderAddress`. EIP-4844 forbids contract creation, so every blob transaction has one: usually a rollup’s inbox, which stays put while its batchers rotate. Version 1.6 adds it and reads the `Body` group for it. ## evm-events 1.1.0 A declarative event decoder. Configuration supplies contract addresses, Solidity event ABI fragments, and output mappings. The factory parses and validates the ABI before a source is opened, derives exact topic-zero signatures, pushes address/topic predicates into capable sources, and decodes without EVM execution. It declares the `Logs` field group. An event’s `abi` is a human-readable fragment, one JSON ABI event object as a block explorer shows it, or `log0` (below). Fragments may use every ABI type Solidity events use except `function` and fixed-point: `address`, `bool`, `bytes1..bytes32`, `uint`/`int` widths from 8 through 256, `bytes`, `string`, fixed-size and dynamic arrays, and tuples, nested at most six levels deep. One element of a dynamic array holds at most 64 values, counting each nested array and tuple as one, because the decoder copies that layout for every element. A tuple is written with its component names, as `(uint256 amount, bool exactOutput) params` or `tuple(…) params`. Every parameter and tuple component needs a name (`type name` or `type indexed name`), because decoded values are keyed by name. Unnamed parameters, anonymous events, duplicate signatures, malformed keys, and more than three indexed fields fail deterministically during `doctor`. The order of the configured events is part of the configuration hash, so reordering them creates a new processor instance. Decoded `values` are exact strings: integers in decimal; `address`, `bytesN`, and `bytes` as lowercase `0x` hex; `bool` as `true` or `false`; and `string` as text, with invalid UTF-8 replaced by U+FFFD. A tuple that is not indexed flattens to one key per component, such as `params.amount`, and nested tuples extend the path. An array, also one inside a tuple, is one JSON string whose elements render as above, with tuple elements as objects in ABI order: `["1","2"]` or `[{"maker":"0x…","data":"0x01"}]`. Ethereum stores an indexed `string`, `bytes`, array, or tuple only as the keccak hash of its value, so its value is that 32-byte topic as `0x` hex. `key_fields` may name any value key, including a flattened one. Any contract can emit a log with a configured event’s topic zero. A log whose topics or data do not decode as that event, such as an ERC-721 `Transfer` (four topics, no data) against the ERC-20 `Transfer` ABI or a `bool` word other than 0 or 1, is not that event: it is skipped rather than failing the block, and the block’s mapped delta counts the skipped logs. Only the canonical encoding decodes, the one Solidity’s encoder produces: offsets without gaps or shared tails, zero padding, and no trailing bytes. A log that breaks these rules, as a few old compilers’ event data did, is skipped and counted rather than guessed at. The layout is checked before anything decodes, so decoding takes memory in proportion to the log’s size. `abi = "log0"` stores every topicless log of the configured contracts raw: the entity’s `event` and `signature` are `log0`, and its `values` hold one key, `data`, with the log data as `0x` hex. Contracts such as Ekubo Core emit LOG0 with a packed layout that no ABI describes, so the application decodes those bytes. A `log0` entry requires `addresses`, a processor may have one, and its `key_fields` may name `data`. A topicless log cannot match a topic-zero filter, so a processor with a `log0` entry reads every log of its contracts and selects the configured events itself. An anonymous event with indexed parameters has topics but no signature, and is not decoded. Without `key_fields`, an event’s key is its transaction hash and log index, which no other event shares, so the processor is block-local: its history and live blocks apply independently. A `key_fields` key can repeat across blocks, and its entity holds the key’s newest event, so any keyed output makes the processor ordered, like `erc20-balances` (`ordered_state` reduction and `canonical` delivery ordering). It reduces blocks strictly in chain order: a block emits one change per key, for its last event, and undoing a block restores the key’s event from before it. Its live lane therefore waits for its cold backfill, holding live blocks as pending deltas within `[budgets] pending_delta_bytes` until history reaches them. Consumers read a keyed instance’s stream through the generic consumer routes. Application subscriptions and `POST /admin/v1/materialization-jobs` require a block-local processor, so they refuse a keyed instance. With `history_mode = "on_demand"`, its history comes from the `backfill` command instead, which applies an ordered processor’s history contiguously from `start_block` upward: each range must start at the block after the last one applied, and the command refuses any other start. Finalized-coverage compaction is block-local only, so a keyed instance keeps one coverage row per block it applies, without bound, as other ordered processors do. `publish = "finalized"` holds unfinalized changes in the store until verified finality promotes them. The stream then contains only finalized `apply` records and `system.finality` markers; no `undo` is ever published. Held bytes count against `[processors.delivery] max_bytes` from admission. ```toml [[processors]] id = "evm-events" instance = "weth-transfers" version = "1.1.0" start_block = 12965000 publish = "finalized" [processors.state] mode = "durable" [processors.artifacts] mode = "none" [processors.output] mode = "window" [processors.output.window] max_blocks = 216000 [processors.delivery] mode = "none" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "none" safety_blocks = 0 [processors.settings] addresses = ["0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"] [[processors.settings.events]] abi = "event Transfer(address indexed from, address indexed to, uint256 value)" [processors.settings.events.output] collection = "weth.transfers" kind = "weth.transfer" key_fields = [] ``` Every entity/change uses `evm.event.entity.v1`/`evm.event.change.v1`, includes block and transaction identity as camelCase `0x`-hex fields, and renders decoded values as exact decimal or hex strings. `bucket_seconds` plus `_bucket_start` in `key_fields` supports deterministic wall-clock buckets from canonical block timestamps. ## transaction-stats 1.0.0 An ordered A-to-B aggregate that retains constant-sized private working state: transaction count, exact total value, cursor, checkpoints, and the bounded unfinalized contribution journal. It declares the `Body` and `Senders` field groups and sender/recipient predicates, and no receipt group, so it cannot tell whether a transaction reverted: `count` and `totalValueWei` cover every submitted transaction from the sender to the recipient. `totalValueWei` is the submitted value, not the value the recipient received. ```toml [[processors]] id = "transaction-stats" instance = "treasury-flow-total" version = "1.0.0" start_block = 15537394 publish = "finalized" [processors.state] mode = "checkpointed" [processors.artifacts] mode = "none" [processors.output] mode = "latest" [processors.delivery] mode = "none" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "none" safety_blocks = 0 [processors.settings] from = "0x1111111111111111111111111111111111111111" to = "0x2222222222222222222222222222222222222222" complete_from_start = true ``` Use `output = "latest"` for a continuing aggregate or `[processors.delivery] emit = "job_result"` for a bounded backfill. Job-result jobs commit state and coverage for every block but spool only the final aggregate; after acknowledgement the job is complete/reclaimable and deletion remains an explicit operator action. ## rollup-txs 1.0.0 A block-local processor for rollups’ L1 transactions other than blobs: calldata batches, state root and output proposals, and proof submissions. It declares the `Header`, `Body`, `Senders`, `Calldata` and `ReceiptOutcome` field groups, filtered to the rules’ recipients. A transaction matches a rule when it is sent to `to`, from `from` when set, calls `selector` when set, carries `chain_id_arg` as its first ABI word when set (shared settlement contracts serving several chains), in a block whose timestamp lies in `[since, until)`. The first matching rule per recipient wins, so list the more specific rules first. A rule also names the `rollup` its transactions belong to and their `purpose` (`data`, `state`, or `proof`), which the reported rows carry. Type-3 transactions never match: `blobs-money` reports them. Only top-level calls match; a call made through a multisig or another contract is not seen. The processor supports Ethereum mainnet only, and only from London (block 12,965,000) on, because `executionBurnedWei` and `tipWei` need the base fee that earlier blocks lack. Startup refuses another chain and a pre-London `start_block`. `since` and `until` are UNIX timestamps in seconds, below 10^10 (the year 2286), not block numbers or milliseconds: a block matches from `since` (inclusive) up to `until` (exclusive). Startup refuses a `since` or `until` below mainnet’s genesis timestamp (1438269973), as any block number is, and one of 10000000000 or more, as any timestamp in milliseconds is: a millisecond `until` would never close its window, and a millisecond `since` would never match. It also refuses a window whose `since` is not below its `until`. A rule with `chain_id_arg` also needs a `selector`: the chain id is the argument after it, and without one the rows would render `selector: null`, so calls to different functions could not be told apart. Every block emits one `rollups.block` change (`rollups.block-bundle.v1`), also when nothing matched, so a consumer can tell a block without rollup activity from one it never received. Materialized output stores each block’s bundle by block number in the `rollups.blocks` collection (`rollups.entity.v1`) and renders it as the same JSON. The bundle is camelCase JSON: * `block`: `network`, `blockNumber`, `blockHash`, `parentHash`, and `timestamp`. The bundle carries no finality of its own: the block’s finality is the `finality` of the change envelope, or of the retained entity, that carries the bundle. * `transactions`, in block order: `network`, `blockNumber`, and `blockHash`, repeated from the block so that each transaction stands alone as a row; then `txHash`, `transactionIndex`, `rollupId` and `purpose` (`data`, `state`, or `proof`), which are the matched rule’s `rollup` and `purpose`, `success`, `senderAddress`, `toAddress`, `selector`, `gasUsed`, `executionBurnedWei` = base fee × gas used, and `tipWei` = (effective price − base fee) × gas used. A reverted transaction is reported like the others, with `success: false`, because its gas was paid. `selector` is the matched rule’s selector, or `null` when that rule has none: the first bytes of a calldata batch are batch data, not a function selector. Hashes, addresses, and selectors are `0x` followed by lowercase hex digits: 64 for a hash, 40 for an address, and 8 for a selector. `gasUsed`, `executionBurnedWei`, and `tipWei` are decimal strings; `blockNumber`, `timestamp`, and `transactionIndex` are JSON numbers. The rules and the `start_block` are the processor’s identity: changing either needs a new `instance`. ```toml [[processors]] id = "rollup-txs" instance = "rollup-txs-1" version = "1.0.0" history_control = "application_subscriptions" history_mode = "on_demand" start_block = 26000000 [processors.state] mode = "durable" [processors.output] mode = "none" [processors.delivery] mode = "until_acknowledged" [[processors.delivery.consumers]] id = "app" required = true lease_ttl = "5m" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "unfinalized" safety_blocks = 256 [[processors.settings.rules]] rollup = "base" purpose = "state" to = "0x43edb88c4b80fdd2adff2412a7bebf9df42cb40e" selector = "0x82ecf2f6" ``` Xatu serves this processor’s history from its transaction projection, for the rule recipients (see [data sources](/docs/concepts/data-sources/#3-xatu)). ## erc20-balances 1.1.0 An ordered watchlist ledger derived from canonical ERC-20 `Transfer` logs. Configuration declares watched addresses, an optional token allowlist, a start block, and whether the opening coverage is complete. It declares the `Logs` field group. The stored method is explicitly `erc20_transfer_ledger`. Rebasing, reflection, and non-standard tokens are not represented as general EVM balances. Any contract can emit a `Transfer` log, so a log with that signature that is not a canonical ERC-20 transfer (three topics, zero-padded addresses, and a 32-byte value), such as an ERC-721 transfer, is skipped rather than failing the block; the block’s mapped delta counts the skipped logs. A transfer the ledger cannot apply proves it incomplete for that token and holder: an outgoing transfer above the derived balance means transfers before the start block or outside the ERC-20 model, such as a rebasing or forged token, and an incoming one can overflow it. The pair is then marked incomplete: `incompleteFrom` holds that block, `complete` is `false`, `balance` keeps its last derived value, and later transfers for the pair are ignored. Undoing that block restores the derived pair. Only a token in the `tokens` allowlist with `complete_from_start = true` fails the processor instead, because the configuration declares that ledger complete. ```toml [[processors]] id = "erc20-balances" instance = "erc20-watchlist" version = "1.1.0" start_block = 1234567 [processors.state] mode = "durable" [processors.output] mode = "latest" [processors.delivery] mode = "window" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "unfinalized" safety_blocks = 256 [processors.coverage] verification_segment_blocks = 8192 [processors.settings] addresses = ["0x1111111111111111111111111111111111111111"] tokens = ["0x2222222222222222222222222222222222222222"] complete_from_start = true ``` ## block-summary 1.2.0 A receipt-free, block-local view of every Ethereum execution block. It declares the `Header` and `Body` field groups, then publishes timestamp, parent/hash identity, gas limit and usage, decimal base fee, blob gas fields, transaction count, exact execution-block size when available, and finality. Blocks come from verified material or, under the trusted-dataset policy, from Xatu’s dataset-declared header and transaction projections, whose body indices and payload transaction count must agree. Dataset rows carry no execution-block size, so `sizeBytes` is `null` for them. A block a dataset summarized first matches the same block from verified material, as when the live lane overlaps a Xatu backfill, and keeps `null`. The reverse, remapping a block that verified material summarized from a dataset, as a recompute may, is refused as a conflicting apply. Summaries are retained by block hash with a canonical block-number index; the query extension exposes `/latest` and `/blocks/{number}`. It never executes the EVM and does not request receipts. The compact `[blocks]` preset supplies the standard checkpointed lifecycle, full query output, 64 MiB/24 hour delivery window, and undo records kept until 256 blocks past finality. Advanced configurations can override those policies explicitly. ## uniswap-observations 2.1.0 / uniswap-latest 2.0.0 Two configured-pool contracts share the same exact event decoder: * Uniswap V2 `Sync(uint112,uint112)` reserve observations; * Uniswap V3 `Initialize(uint160,int24)` and `Swap(address,address,int256,int256,uint160,uint128,int24)` square-root-price observations. They store exact reserves and `sqrtPriceX96`, never floating-point prices. V3 swap observations also retain `amount0` and `amount1` as signed decimal int256 values, preserving the exact pool deltas needed to derive volume. Applications combine these values with verified token metadata. `uniswap-observations` is block-local and identifies immutable observations by pool, block hash, and log index; it supports independent history/live delivery and full or windowed SQLite retention. Version 2.1 also maintains a query-only latest-observation pointer per pool so new clients can bootstrap immediately before following the immutable stream. Transaction hashes are deliberately not part of this processor’s output, allowing archive and P2P sources to omit transaction bodies when acquiring filtered pool logs. Both processors declare the `Logs` field group. `uniswap-latest` is ordered/canonical and retains only the latest observation per pool for slim price queries. ```toml [[processors]] id = "uniswap-observations" # or "uniswap-latest" instance = "usdc-weth-observations" version = "2.1.0" # use 2.0.0 with uniswap-latest start_block = 12369621 history_control = "node_owned" history_mode = "on_demand" [processors.state] mode = "checkpointed" [processors.artifacts] mode = "none" [processors.output] mode = "full" [processors.delivery] mode = "none" [processors.checkpoint] mode = "automatic" keep = 3 [processors.undo] mode = "unfinalized" safety_blocks = 256 [processors.settings] [[processors.settings.pools]] address = "0x3333333333333333333333333333333333333333" kind = "v3" ``` Pool factory discovery is intentionally separate from event reduction. A catalog adapter can discover pools and render these immutable processor configs; each processor identity then commits to the exact pool set. # Built-in processors > Generated catalog of every processor factory registered by the standard Leani binary. This catalog comes from `ProcessorRegistry::standard()`. Adding, removing, or renaming a built-in factory changes the tracked fixture in the same Rust commit. Runtime `/v1/capabilities` remains authoritative for one configured node. ## `blobs-money` Mainnet blob blocks, transactions, fees, burn, capacity, and fork-aware schedule ## `block-summary` Ethereum block summaries with transaction counts, from verified or dataset-declared blocks ## `erc20-balances` Watchlist ERC-20 transfer ledger with current balances ## `evm-events` ABI-decoded EVM logs with configurable entity keys and time buckets ## `rollup-txs` Rollups' non-blob L1 transactions (calldata batches, state updates, proofs) matched by configured rules, with execution burn and tips ## `transaction-stats` Directional transaction counts and submitted value totals for one address pair ## `uniswap-latest` Latest materialized Uniswap V2/V3 pool state with typed query routes ## `uniswap-observations` Append-only Uniswap V2/V3 pool observations # TypeScript SDK > Typed queries, snapshot lifecycle, resumable streams, and generated public SDK contracts. The public package is `@smart-byte/leani-sdk`, for Node 20.3+ and Bun 1.2+. In a source checkout, run `bun install --cwd packages/sdk` and `bun run --cwd packages/sdk build`. The [query/follow guide](/docs/guides/query-then-follow/) includes an executable client, a matching node config, and pagination/reset handling. For a node started with `leani init blocks` and `leani serve`: ```ts import { createLeaniClient } from "@smart-byte/leani-sdk"; const leani = createLeaniClient({ baseUrl: "http://127.0.0.1:18080" }); const stop = new AbortController(); for await (const change of leani.blocks.subscribe({ signal: stop.signal })) { if (change.operation === "apply" && change.data) { console.log(change.data.blockNumber); } } ``` Pass an `AbortSignal` to cancel an idle subscription; breaking out of the loop also cancels its response body. Transport disconnects reconnect from the last yielded cursor. Protocol errors and expired history require application action. An incomplete final event is discarded before reconnecting. Retryable HTTP errors, including rate limiting, use the configured reconnect backoff, and so do `error` events marked `retryable`; after five of them in a row without a new change, the subscription throws the last one. A stream that sends nothing, not even the node’s 15-second keep-alive comments, for `idleTimeoutMs` (default 60,000) while the loop waits reconnects too. An event over 8 MiB fails the subscription with a non-retryable `invalid_response` error. This print-only example does not maintain state; stateful consumers must apply undo mutations too. A request that fails below HTTP rejects with `TransportError`, a `LeaniError` with `status` 0, `code: "transport"`, `retryable: true`, and the runtime’s error as `cause`: a refused or reset connection, a response body that breaks off, a passed deadline, or a silent stream. Aborting your own signal rejects with its reason instead, and the SDK never retries it. The SDK does not follow redirects: a 3xx answer rejects with the non-retryable `redirect_refused`, with the target in `details.location`, so configure the final URL as `baseUrl`. Every request path resolves under the base URL’s path, one starting with `/` included. A path segment that decodes to `.` or `..`, or holds a backslash, an encoded slash or backslash (`%2F`, `%5C`), or an encoded percent sign (`%25`), and a control character anywhere in the path, are refused before any request. `request()` applies the same rule to a custom query extension’s path, so a path parameter whose value can hold `/` or `%` must move to the query string, `request()`’s `params`. Typed subscriptions accept `processor` to select a configured instance, for example `leani.blobs.subscribe({ processor: "blobs-production" })`. The blobs apply payload is `{ block, transactions }`. Narrow `operation` before reading processor data: `finalized` carries `{ throughBlock }` instead, or, when `isLiveGapChange()` holds, the `{ fromBlock, toBlock }` the live lane skipped. `liveGapBackfill()` in `@smart-byte/leani-sdk/backfill` turns that range into an idempotent `fill_missing` subscription request; see [Keep your application in sync](/docs/guides/keep-your-application-in-sync/). Discover canonical query paths with `capabilities()` and pass `queryBasePaths` when aliases are ambiguous. `queryAndFollow()` returns the **first page** of an immutable snapshot and its stream boundary. Follow every `nextCursor` with `queryEntities()`, release the snapshot in `finally`, then subscribe from `boundaryCursor`. SSE resets throw `ResetRequiredError`; HTTP expiry throws `LeaniError` with `code: "cursor_expired"` or `"query_snapshot_expired"`. Rebuild the seed for either case. `queryEntities()` also works when delivery is disabled; its `boundaryCursor` and `recovery.follow` are then `null`. `queryAndFollow()` requires delivery and returns a `FollowSnapshotPage` with a non-null boundary, or HTTP 409 `delivery_disabled`. The stream and change endpoints return the same error when delivery is disabled. Durable live/history sessions live in `@smart-byte/leani-sdk/backfill`. Construct them with `createBackfillSubscriptionClient({ baseUrl, token, fetch, timeoutMs })`; both entry points use the same `LeaniError` and `TransportError` classes and HTTP error fields (`status`, `code`, `retryable`, `requestId`). The deadline applies to JSON requests, lease renewals, and acknowledgements, and to a stream until its hello arrives. An established stream that sends nothing, heartbeats included, for three `heartbeatIntervalMs` while its iterator waits fails with a `TransportError`; time spent handling a record does not count. A lease renewal that fails with a retryable error is retried with backoff while the lease holds; one that fails otherwise, or keeps failing until the lease would lapse, ends the session at once, and the iterator throws its error. The lanes of `subscribe()` reconnect after either, and after any other retryable error. A stream record over 65 MiB fails the session with a non-retryable `invalid_response` error. A backfill stream can request smaller batches with `batching.maximumEncodedBytes`; `streamLive()` takes no batching option, so a live stream’s batches follow the node’s `api.delivery.live_batches.maximum_encoded_bytes`. Always close delivery sessions in `finally`. Commit idempotent destination writes before acknowledging a batch. Acknowledge a batch’s boundary, never a change’s own cursor: the node refuses one inside a live block or a history progress unit. A backfill session’s `deliveryBatches()` yields every boundary as `ackCursor` with the `events` it covers, including batches without changes and the completion; `events()` flattens the changes without boundaries. A consumer with a credential passes it as `credential` on every call and stream. `retry(subscription)` re-queues a failed subscription to resume from its committed progress, and `delete(subscription, { discardUnacknowledged: true })` deletes a failed or cancelled one whose consumer has not acknowledged every record. See [durable delivery](/docs/concepts/durable-delivery/) for lease and reset semantics. ## Run a history subscription in one call `runBackfill(client, request, options)` in `@smart-byte/leani-sdk/backfill` runs an application-owned subscription to its end; `request` must name its `consumer`. It creates the subscription, applies each history batch with `options.onBatch` and acknowledges it once that resolves, acknowledges the completion, closes the session, and deletes the subscription. It resolves with the `outcome` (`completed`, `cancelled`, or `failed`), the `subscription` as it ended, the `completion` record it received, if it streamed one, and the `deletion`. Creation is idempotent by `idempotencyKey`: running the identical request again, as after a crash, resumes the same subscription after its last acknowledged batch, and the node refuses a different request under the same key. A batch applied but not acknowledged is delivered again, so `onBatch` must be idempotent. A subscription already complete when created is deleted without opening its stream. After a lost connection or consumer session, or while another connection still holds the consumer’s session, it waits a backoff of 1 second, doubling up to 30 seconds, and reads the subscription’s state before it opens the stream again: a subscription that ended meanwhile, as when the node applied the completion’s acknowledgement but its answer was lost, ends the run instead. A failed subscription follows `options.onFailure`. `"retry"`, the default, re-queues it with `retry()`, up to `maxRetries` times (3 by default), then throws a `BackfillStreamError` with `code: "backfill_failed"` and leaves it failed. `"discard"` deletes it with `discardUnacknowledged: true` and resolves with the outcome `failed`, and `"throw"` throws at once. A cancelled subscription is deleted with `discardUnacknowledged: true`. An error from `onBatch`, a `reset_required` stream, any other failure, and an abort of `options.signal` or `request.signal` close the session and reject without deleting the subscription, so the identical request can run again once the cause is fixed. The stream presents `options.credential`, by default the request’s `consumer.credential`. ## Inventory The inventory below is generated from both public entry points and shared error types. The `backfill.*` members belong to the backfill client. ## `baseUrl` | Member | Contract | | --------- | ----------------------- | | `baseUrl` | `readonly baseUrl: URL` | ## `request` | Member | Contract | | --------- | ------------------------------------------------------------------------------------------------------------------ | | `request` | `request( path: string, params?: Record, options?: { signal?: AbortSignal }, ): Promise` | ## `health` | Member | Contract | | -------------- | ------------------------------------------------------------ | | `health.live` | `live(options?: { signal?: AbortSignal }): Promise` | | `health.ready` | `ready(options?: { signal?: AbortSignal }): Promise` | ## `status` | Member | Contract | | -------- | ----------------------------------------------------------------- | | `status` | `status(options?: { signal?: AbortSignal }): Promise` | ## `capabilities` | Member | Contract | | -------------- | ------------------------------------------------------------------------- | | `capabilities` | `capabilities(options?: { signal?: AbortSignal }): Promise` | ## `processors` | Member | Contract | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `processors.list` | `list(options?: { signal?: AbortSignal }): Promise` | | `processors.status` | `status( id: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.changes` | `changes( id: string, options?: ChangesOptions, ): Promise>>` | | `processors.changeHead` | `changeHead( id: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.resetLiveLane` | `resetLiveLane( id: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.listCollections` | `listCollections( id: string, options?: { signal?: AbortSignal }, ): Promise<{ processor: ProcessorSummary; outputMode: "none" \| "latest" \| "window" \| "full"; retained: boolean; data: string[]; }>` | | `processors.queryEntities` | `queryEntities( id: string, collection: string, options?: OutputQueryOptions, ): Promise>` | | `processors.queryAndFollow` | `queryAndFollow( id: string, collection: string, options?: Omit, ): Promise>` | | `processors.getEntity` | `getEntity( id: string, collection: string, key: Hex, options?: { signal?: AbortSignal }, ): Promise>` | | `processors.releaseSnapshot` | `releaseSnapshot( id: string, snapshotId: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.consumers.list` | `list( id: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.consumers.create` | `create( id: string, options: CreateConsumerOptions, ): Promise` | | `processors.consumers.inspect` | `inspect( id: string, consumer: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.consumers.revoke` | `revoke( id: string, consumer: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.consumers.renew` | `renew( id: string, consumer: string, credential: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.consumers.changes` | `changes( id: string, consumer: string, credential: string, options?: { limit?: number; signal?: AbortSignal }, ): Promise>>` | | `processors.consumers.acknowledge` | `acknowledge( id: string, consumer: string, credential: string, cursor: string, options?: { signal?: AbortSignal }, ): Promise` | | `processors.subscribe` | `subscribe( id: string, options?: SubscribeOptions, ): AsyncGenerator>` | ## `blobs` | Member | Contract | | ------------------------ | ------------------------------------------------------------------------------------------------------- | | `blobs.getBlock` | `getBlock( number: number, options?: { signal?: AbortSignal }, ): Promise` | | `blobs.listBlocks` | `listBlocks(options: ListBlocksOptions): Promise>` | | `blobs.listSnapshot` | `listSnapshot( options: ListBlocksOptions, ): Promise>` | | `blobs.getTransaction` | `getTransaction( hash: Hex, options?: { signal?: AbortSignal }, ): Promise` | | `blobs.listTransactions` | `listTransactions( options: ListTransactionsOptions, ): Promise>` | | `blobs.getSchedule` | `getSchedule(options?: { atBlock?: number; signal?: AbortSignal; }): Promise` | | `blobs.subscribe` | `subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>` | ## `erc20` | Member | Contract | | ------------------ | ---------------------------------------------------------------------------------------------------- | | `erc20.getBalance` | `getBalance( token: Hex, address: Hex, options?: { signal?: AbortSignal }, ): Promise` | | `erc20.subscribe` | `subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>` | ## `uniswap` | Member | Contract | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `uniswap.getPool` | `getPool( address: Hex, options?: { signal?: AbortSignal }, ): Promise` | | `uniswap.getLatestObservation` | `getLatestObservation( address: Hex, options?: { processor?: string; signal?: AbortSignal }, ): Promise` | | `uniswap.listConfiguredPools` | `listConfiguredPools(options?: { processor?: string; signal?: AbortSignal; }): Promise` | | `uniswap.subscribe` | `subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>` | ## `blocks` | Member | Contract | | -------------------- | --------------------------------------------------------------------------------------------------------------------- | | `blocks.getLatest` | `getLatest(options?: { processor?: string; signal?: AbortSignal; }): Promise` | | `blocks.getByNumber` | `getByNumber( number: number, options?: { processor?: string; signal?: AbortSignal }, ): Promise` | | `blocks.subscribe` | `subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>` | ## `backfill` | Member | Contract | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `backfill.baseUrl` | `readonly baseUrl: URL` | | `backfill.create` | `create(request: CreateBackfillSubscription): Promise` | | `backfill.list` | `list(options?: { signal?: AbortSignal }): Promise` | | `backfill.inspect` | `inspect( subscription: string, options?: { signal?: AbortSignal }, ): Promise` | | `backfill.cancel` | `cancel( subscription: string, options?: { signal?: AbortSignal }, ): Promise` | | `backfill.retry` | `retry( subscription: string, options?: { signal?: AbortSignal }, ): Promise` | | `backfill.delete` | `delete( subscription: string, options?: { signal?: AbortSignal; discardUnacknowledged?: boolean }, ): Promise` | | `backfill.changes` | `changes( subscription: string, consumer: string, options?: { credential?: string; limit?: number; signal?: AbortSignal; }, ): Promise<{ data: ChangeEnvelope[]; nextCursor: string \| null; coverage: unknown; }>` | | `backfill.stream` | `stream( subscription: string, consumer: string, options?: { credential?: string; batching?: BackfillBatchingRequest; signal?: AbortSignal; }, ): Promise>` | | `backfill.streamLive` | `streamLive( processor: string, consumer: string, options?: { credential?: string; signal?: AbortSignal }, ): Promise>` | | `backfill.subscribe` | `subscribe( request: MultiLaneSubscriptionRequest, ): Promise>` | ## Exported types and helpers `Hex`@smart-byte/leani-sdk ``` export type Hex = `0x${string}`; ``` `Finality`@smart-byte/leani-sdk ``` export type Finality = "unattested" | "attested" | "finalized"; ``` `ChangeOperation`@smart-byte/leani-sdk ``` export type ChangeOperation = | "apply" | "undo" | "attested" | "finalized" | "reset_required"; ``` `LeaniClientOptions`@smart-byte/leani-sdk ``` export interface LeaniClientOptions { baseUrl: string | URL; token?: string; timeoutMs?: number; fetch?: FetchLike; /** * Override typed query-extension mounts with canonical `basePath` values * returned by `capabilities()` when an ergonomic alias is unavailable. */ queryBasePaths?: { blobs?: string; erc20?: string; uniswap?: string; }; reconnect?: { initialDelayMs?: number; maxDelayMs?: number; jitter?: number; }; } ``` `FetchLike`@smart-byte/leani-sdk ``` export type FetchLike = ( input: RequestInfo | URL, init?: RequestInit, ) => Promise; ``` `ProcessorSummary`@smart-byte/leani-sdk ``` export interface ProcessorSummary { id: string; instance: string; version: string; codeHash: Hex; configHash: Hex; genericApi: "processor-v1"; changeSchema: string; queryExtensions: QueryExtensionSummary[]; subscriptions: boolean; artifactRetention: "none" | "window" | "full"; deliveryOrdering: "canonical" | "block_versioned_idempotent"; } ``` `QueryExtensionSummary`@smart-byte/leani-sdk ``` export interface QueryExtensionSummary { id: string; basePath: string; aliasPath: string | null; } ``` `QueryExtensionCapability`@smart-byte/leani-sdk ``` export interface QueryExtensionCapability extends QueryExtensionSummary { processor: string; } ``` `Capabilities`@smart-byte/leani-sdk ``` export interface Capabilities { chainId: number; ethereumRpc: Record; processors: ProcessorSummary[]; queryExtensions: QueryExtensionCapability[]; } ``` `NodeStatus`@smart-byte/leani-sdk ``` export interface NodeStatus { apiVersion: "1"; project: string; version: string; chainId: number; uptimeSeconds: number; databaseBytes: number; processor: ProcessorSummary; processors: ProcessorSummary[]; readiness: Readiness; } ``` `Readiness`@smart-byte/leani-sdk ``` export interface Readiness { liveRequired: boolean; liveReady: boolean; finalityRequired: boolean; finalityReady: boolean; ready: boolean; } ``` `Health`@smart-byte/leani-sdk ``` export interface Health { status: "live" | "ready" | "not_ready" | "failed"; uptimeSeconds: number; database: "available" | "unavailable"; readiness: Readiness; reasons: string[]; } ``` `CoverageInterval`@smart-byte/leani-sdk ``` export interface CoverageInterval { fromBlock: number; toBlock: number; finality: Finality; } ``` `ProcessorCoverage`@smart-byte/leani-sdk ``` export interface ProcessorCoverage { chainId: number; chainFinalizedHead: { number: number; hash: string; } | null; requested?: { fromBlock: number; toBlock: number; } | null; available: CoverageInterval[]; configuredStartBlock: number; processedThrough: number | null; finalizedThrough: number | null; complete: boolean; state: | "starting" | "backfilling" | "catching_up" | "live" | "degraded" | "paused" | "failed"; } ``` `Page`@smart-byte/leani-sdk ``` export interface Page { data: T[]; nextCursor: string | null; coverage: ProcessorCoverage; } ``` `BlobsBlock`@smart-byte/leani-sdk ``` export interface BlobsBlock { network: string; blockNumber: number; blockHash: Hex; parentHash: Hex; timestamp: number; size: string; blobCount: number; blobGasUsed: string; excessBlobGas: string; /** * Protocol blob base fee in wei per blob gas: the EIP-4844 fee for * `excessBlobGas` under the fork active at `timestamp`. EIP-7918 does not * change it. */ blobBaseFee: string; executionBaseFee: string; gasUsed: string; gasLimit: string; executionEthBurnedWei: string; /** `blobBaseFee` times `blobGasUsed`. */ blobEthBurnedWei: string; /** * EIP-7918 reserve price in wei per blob gas from Fusaka on, `null` before. * It only changes how excess blob gas evolves and is not applied to * `blobBaseFee`. */ reserveFeeWei: string | null; transactionCount: number; targetBlobsPerBlock: number; maxBlobsPerBlock: number; transformVersion: number; finality: Finality; } ``` `BlobTransaction`@smart-byte/leani-sdk ``` export interface BlobTransaction { network: string; blockNumber: number; blockHash: Hex; txHash: Hex; transactionIndex: number; senderAddress: Hex; /** * The transaction's to address. Every blob transaction has one: a rollup's * inbox, or a contract that routes the call, such as Multicall3. */ toAddress: Hex; blobVersionedHashes: Hex[]; blobCount: number; totalBurnedWei: string; executionBurnedWei: string; /** The block's `blobBaseFee` times this transaction's blob gas. */ blobBurnedWei: string; } ``` `BlobsSnapshotEntry`@smart-byte/leani-sdk ``` export interface BlobsSnapshotEntry { block: BlobsBlock; transactions: BlobTransaction[]; } ``` `ChangeHead`@smart-byte/leani-sdk ``` export interface ChangeHead { earliestSequence: string | null; latestSequence: string | null; cursor: string | null; } ``` `GenericOutputEntity`@smart-byte/leani-sdk ``` export interface GenericOutputEntity { ordinal: string; key: Hex; schema: string; data: T; blockNumber: number; blockTimestamp: number; finality: Finality; } ``` `OutputBounds`@smart-byte/leani-sdk ``` export interface OutputBounds { earliestBlock: number; latestBlock: number; earliestTimestamp: number; latestTimestamp: number; } ``` `GenericSnapshotPage`@smart-byte/leani-sdk ``` export interface GenericSnapshotPage { data: GenericOutputEntity[]; nextCursor: string | null; snapshotId: string; boundaryCursor: string | null; rowCount: string; valueBytes: string; expiresAtUnixMs: string; retainedBounds: OutputBounds | null; coverage: ProcessorCoverage; recovery: { follow: string | null; outsideRetention: "create_processor_instance_or_source_scan"; }; } ``` `FollowSnapshotPage`@smart-byte/leani-sdk ``` export interface FollowSnapshotPage extends GenericSnapshotPage { boundaryCursor: string; recovery: GenericSnapshotPage["recovery"] & { follow: string }; } ``` `OutputQueryOptions`@smart-byte/leani-sdk ``` export interface OutputQueryOptions { fromBlock?: number; toBlock?: number; fromTimestamp?: number; toTimestamp?: number; limit?: number; cursor?: string; signal?: AbortSignal; } ``` `ConsumerRole`@smart-byte/leani-sdk ``` export type ConsumerRole = "required" | "best_effort"; ``` `ConsumerState`@smart-byte/leani-sdk ``` export type ConsumerState = "active" | "reset_required" | "revoked"; ``` `ConsumerStart`@smart-byte/leani-sdk ``` export type ConsumerStart = | { position: "earliest_retained" } | { position: "current_head" } | { position: "cursor"; cursor: string }; ``` `CreateConsumerOptions`@smart-byte/leani-sdk ``` export interface CreateConsumerOptions { id: string; role: ConsumerRole; start: ConsumerStart; leaseTtlSeconds: number; credential: string; signal?: AbortSignal; } ``` `DurableConsumer`@smart-byte/leani-sdk ``` export interface DurableConsumer { id: string; processorInstance: string; /** The delivery stream this consumer reads. */ streamId: string; role: ConsumerRole; state: ConsumerState; acknowledgedSequence: string; deliveredSequence: string; acknowledgedCursor: string; deliveredCursor: string; /** Stream sessions that have taken the lease; `"0"` before the first. */ leaseGeneration: string; leaseTtlMs: string; leaseExpiresAtUnixMs: string; leaseActive: boolean; lagChanges: string; lagBlocks: string; lagBytes: string; lagAgeMs: string; createdAtUnixMs: string; updatedAtUnixMs: string; } ``` `BlobSchedule`@smart-byte/leani-sdk ``` export interface BlobSchedule { name: string; /** * Block-number label for `atBlock` lookups. For Dencun it is 19426589, the * first block the blobs processor covers. */ activationBlock: number; /** The fork's parameters apply to blocks from this timestamp on. */ activationTimestamp: number; forkId: Hex; targetBlobsPerBlock: number; maxBlobsPerBlock: number; baseFeeUpdateFraction: string; eip7918: boolean; source: "chain_spec" | "eth_config" | "configured"; } ``` `Erc20Balance`@smart-byte/leani-sdk ``` export interface Erc20Balance { token: Hex; address: Hex; balance: string; asOfBlock: number; blockHash: Hex; finality: Finality; coverageFrom: number; complete: boolean; /** * Block whose transfer proved the ledger incomplete for this token and * holder. From that block the pair is no longer derived and `balance` keeps * its last derived value; `null` while the pair is derived. */ incompleteFrom: number | null; method: "erc20_transfer_ledger"; } ``` `EvmEvent`@smart-byte/leani-sdk ``` export interface EvmEvent { schemaVersion: number; contract: Hex; event: string; signature: string; blockNumber: number; blockHash: Hex; blockTimestamp: number; bucketStartTimestamp?: number; transactionHash: Hex; transactionIndex: number; logIndex: number; finality: Finality; values: Record; } ``` `TransactionStats`@smart-byte/leani-sdk ``` export interface TransactionStats { from: Hex; to: Hex; /** Submitted transactions from `from` to `to`, reverted ones included. */ count: number; /** Submitted value in wei, including the value of reverted transactions. */ totalValueWei: string; asOfBlock: number; blockHash: Hex; finality: Finality; coverageFrom: number; complete: boolean; } ``` `UniswapPoolPrice`@smart-byte/leani-sdk ``` export interface UniswapPoolPrice { pool: Hex; kind: "v2" | "v3"; reserve0: string | null; reserve1: string | null; /** Signed token0 pool delta for a V3 swap, in raw token units. */ amount0: string | null; /** Signed token1 pool delta for a V3 swap, in raw token units. */ amount1: string | null; sqrtPriceX96: string | null; blockNumber: number; blockHash: Hex; logIndex: number; finality: Finality; } ``` `LatestUniswapObservation`@smart-byte/leani-sdk ``` export interface LatestUniswapObservation { data: UniswapPoolPrice; timestamp: number; } ``` `ConfiguredUniswapPool`@smart-byte/leani-sdk ``` export interface ConfiguredUniswapPool { address: Hex; kind: "v2" | "v3"; } ``` `ConfiguredUniswapPools`@smart-byte/leani-sdk ``` export interface ConfiguredUniswapPools { data: ConfiguredUniswapPool[]; } ``` `BlockSummary`@smart-byte/leani-sdk ``` export interface BlockSummary { chainId: number; blockNumber: number; blockHash: Hex; parentHash: Hex; timestamp: number; gasLimit: number | null; gasUsed: number | null; baseFeePerGas: string | null; blobGasUsed: number | null; excessBlobGas: number | null; transactionCount: number | null; sizeBytes: number | null; finality: Finality; } ``` `LatestBlockSummary`@smart-byte/leani-sdk ``` export interface LatestBlockSummary { data: BlockSummary; } ``` `ChangeMetadata`@smart-byte/leani-sdk ``` export interface ChangeMetadata { apiVersion: "1"; sequence: string; cursor: string; originKind: | "live" | "live_recovery" | "historical_backfill" | "recompute"; originId: string; publicationRevision: string; chainId: number; block: { number: number; hash: Hex; parentHash: Hex; timestamp: number; } | null; finality: Finality; kind: string; schema: string; key: string | null; /** Present only on `reset_required` events, as the current coverage hint. */ coverage?: ProcessorCoverage; /** RFC 3339 UTC emission time with millisecond precision. */ emittedAt: string; } ``` `LiveGapChangeData`@smart-byte/leani-sdk ``` export interface LiveGapChangeData { fromBlock: number; toBlock: number; /** * Leani did not fill the range: the processor's history is on demand, and a * `live_gap_fill` limit, if it has one, did not cover or complete it. */ reason: "not_filled"; } ``` `ChangeEnvelope`@smart-byte/leani-sdk ``` export type ChangeEnvelope = ChangeMetadata & ( | { operation: "apply" | "undo"; data: T | null } | { operation: "attested"; data: { throughBlock: number } } | { operation: "finalized"; data: { throughBlock: number } } | { operation: "finalized"; data: LiveGapChangeData } | { operation: "reset_required"; data: { earliestAvailableSequence: string; latestAvailableSequence: string; action: "query_snapshot_then_resume"; } } ); ``` `isLiveGapChange`@smart-byte/leani-sdk ``` export function isLiveGapChange( change: ChangeEnvelope, ): change is Extract, { data: LiveGapChangeData }>; ``` `LiveLaneReset`@smart-byte/leani-sdk ``` export interface LiveLaneReset { processor: string; state: "paused"; reason: "operator_reset_pending_replay"; firstUnappliedBlock: number; firstUnappliedHash: Hex; requiredDeliveryBytes: string; } ``` `StreamHello`@smart-byte/leani-sdk ``` export interface StreamHello { apiVersion: "1"; chainId: number; processor: ProcessorSummary; coverage: ProcessorCoverage; } ``` `ListBlocksOptions`@smart-byte/leani-sdk ``` export interface ListBlocksOptions { fromBlock: number; toBlock: number; limit?: number; cursor?: string; allowPartial?: boolean; signal?: AbortSignal; } ``` `ListTransactionsOptions`@smart-byte/leani-sdk ``` export interface ListTransactionsOptions { blockNumber: number; limit?: number; cursor?: string; signal?: AbortSignal; } ``` `ChangesOptions`@smart-byte/leani-sdk ``` export interface ChangesOptions { after?: string; limit?: number; signal?: AbortSignal; } ``` `SubscribeOptions`@smart-byte/leani-sdk ``` export interface SubscribeOptions { after?: string; signal?: AbortSignal; /** Receives the connection handshake frame; called once per (re)connect. */ onHello?: (hello: StreamHello) => void; /** * Reconnect when the stream sends nothing, not even the node's 15-second * keep-alive comments, for this long while the loop waits for a change. * Defaults to 60,000 ms. */ idleTimeoutMs?: number; } ``` `ProcessorSubscribeOptions`@smart-byte/leani-sdk ``` export interface ProcessorSubscribeOptions extends SubscribeOptions { processor?: string; } ``` `LeaniClient`@smart-byte/leani-sdk ``` export interface LeaniClient { readonly baseUrl: URL; request( path: string, params?: Record, options?: { signal?: AbortSignal }, ): Promise; health: { live(options?: { signal?: AbortSignal }): Promise; ready(options?: { signal?: AbortSignal }): Promise; }; status(options?: { signal?: AbortSignal }): Promise; capabilities(options?: { signal?: AbortSignal }): Promise; processors: { list(options?: { signal?: AbortSignal }): Promise; status( id: string, options?: { signal?: AbortSignal }, ): Promise; changes( id: string, options?: ChangesOptions, ): Promise>>; changeHead( id: string, options?: { signal?: AbortSignal }, ): Promise; resetLiveLane( id: string, options?: { signal?: AbortSignal }, ): Promise; listCollections( id: string, options?: { signal?: AbortSignal }, ): Promise<{ processor: ProcessorSummary; outputMode: "none" | "latest" | "window" | "full"; retained: boolean; data: string[]; }>; queryEntities( id: string, collection: string, options?: OutputQueryOptions, ): Promise>; /** First snapshot page plus an atomic stream boundary. Read all nextCursor pages and releaseSnapshot before following. */ queryAndFollow( id: string, collection: string, options?: Omit, ): Promise>; getEntity( id: string, collection: string, key: Hex, options?: { signal?: AbortSignal }, ): Promise>; releaseSnapshot( id: string, snapshotId: string, options?: { signal?: AbortSignal }, ): Promise; consumers: { list( id: string, options?: { signal?: AbortSignal }, ): Promise; create( id: string, options: CreateConsumerOptions, ): Promise; inspect( id: string, consumer: string, options?: { signal?: AbortSignal }, ): Promise; revoke( id: string, consumer: string, options?: { signal?: AbortSignal }, ): Promise; renew( id: string, consumer: string, credential: string, options?: { signal?: AbortSignal }, ): Promise; changes( id: string, consumer: string, credential: string, options?: { limit?: number; signal?: AbortSignal }, ): Promise>>; acknowledge( id: string, consumer: string, credential: string, cursor: string, options?: { signal?: AbortSignal }, ): Promise; }; subscribe( id: string, options?: SubscribeOptions, ): AsyncGenerator>; }; blobs: { getBlock( number: number, options?: { signal?: AbortSignal }, ): Promise; listBlocks(options: ListBlocksOptions): Promise>; listSnapshot( options: ListBlocksOptions, ): Promise>; getTransaction( hash: Hex, options?: { signal?: AbortSignal }, ): Promise; listTransactions( options: ListTransactionsOptions, ): Promise>; getSchedule(options?: { atBlock?: number; signal?: AbortSignal; }): Promise; subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>; }; erc20: { getBalance( token: Hex, address: Hex, options?: { signal?: AbortSignal }, ): Promise; subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>; }; uniswap: { getPool( address: Hex, options?: { signal?: AbortSignal }, ): Promise; getLatestObservation( address: Hex, options?: { processor?: string; signal?: AbortSignal }, ): Promise; listConfiguredPools(options?: { processor?: string; signal?: AbortSignal; }): Promise; subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>; }; blocks: { getLatest(options?: { processor?: string; signal?: AbortSignal; }): Promise; getByNumber( number: number, options?: { processor?: string; signal?: AbortSignal }, ): Promise; subscribe( options?: ProcessorSubscribeOptions, ): AsyncGenerator>; }; } ``` `createLeaniClient`@smart-byte/leani-sdk ``` export function createLeaniClient( options: LeaniClientOptions, ): LeaniClient; ``` `QueryValue`@smart-byte/leani-sdk ``` export type QueryValue = string | number | boolean | undefined; ``` `parseSse`@smart-byte/leani-sdk ``` export async function* parseSse( body: ReadableStream, signal?: AbortSignal, ): AsyncGenerator; ``` `compareSequences`@smart-byte/leani-sdk ``` export function compareSequences(left: string, right: string): -1 | 0 | 1; ``` `isResetRequired`@smart-byte/leani-sdk ``` export function isResetRequired( change: ChangeEnvelope, ): change is ChangeEnvelope & { operation: "reset_required" }; ``` `EntityChangeTarget`@smart-byte/leani-sdk ``` export interface EntityChangeTarget { put(key: string, value: T): void | Promise; delete(key: string): void | Promise; } ``` `applyEntityChange`@smart-byte/leani-sdk ``` export async function applyEntityChange( target: EntityChangeTarget, change: ChangeEnvelope, ): Promise; ``` `commitThenAcknowledge`@smart-byte/leani-sdk ``` export async function commitThenAcknowledge( destinationCommit: () => Promise, acknowledge: (committed: T) => Promise, ): Promise; ``` `CursorStore`@smart-byte/leani-sdk ``` export interface CursorStore { get(): string | undefined; set(cursor: string): void; clear(): void; } ``` `createInMemoryCursorStore`@smart-byte/leani-sdk ``` export function createInMemoryCursorStore(initial?: string): CursorStore; ``` `BackfillExecutionMode`@smart-byte/leani-sdk/backfill ``` export type BackfillExecutionMode = "fill_missing" | "recompute"; ``` `BackfillState`@smart-byte/leani-sdk/backfill ``` export type BackfillState = | "waiting_for_consumer" | "queued" | "running" | "backpressured" | "storage_backpressured" | "draining" | "complete_reclaimable" | "completed" | "failed" | "cancelled"; ``` `BackfillRange`@smart-byte/leani-sdk/backfill ``` export interface BackfillRange { fromBlock: number; toBlock: number; } ``` `RequestedBackfillRange`@smart-byte/leani-sdk/backfill ``` export interface RequestedBackfillRange { fromBlock: number; toBlock: number | "finalized"; } ``` `BackfillStatus`@smart-byte/leani-sdk/backfill ``` export interface BackfillStatus { id: string; owner: "subscription"; processor: string; /** * Whether the node configures the processor instance `processor` now. * Without it, the stream and acknowledgement routes refuse the * subscription. That can be temporary; if the instance will not return, * cancel the subscription and delete it with `discardUnacknowledged: true`, * which discards its unacknowledged records. */ processorConfigured: boolean; /** The required consumer; tell your subscriptions apart by it, not by `id`. */ consumer: string | null; deliveryStreamId?: string; publicationRevision?: string; fromBlock: number; toBlock: number; ranges: BackfillRange[]; requestedBlocks: number; processedBlocks: number; remainingBlocks: number; capturedFinalizedTarget?: number; mode: BackfillExecutionMode; batching?: EffectiveBackfillBatching; state: BackfillState; attempts: number; updatedAtUnixMs: number; report: unknown | null; lastError: string | null; } ``` `HistoricalWorkDeletion`@smart-byte/leani-sdk/backfill ``` export interface HistoricalWorkDeletion { id: string; owner: "subscription"; removedJobs: number; removedSubscriptionRanges: number; removedConsumers: number; removedDeliveryRecords: number; removedDeliveryStreams: number; removedCoverageIntervals: number; removedCoverageSegments: number; removedExactCoverage: number; removedAppliedBlocks: number; removedFinalizedUndo: number; retainedProcessorOutput: boolean; retainedLiveStream: boolean; } ``` `BackfillBatchingRequest`@smart-byte/leani-sdk/backfill ``` export type BackfillBatchingRequest = Partial< Omit< EffectiveBackfillBatching, "maximumBufferedBatches" | "maximumBufferedBytes" > >; ``` `BackfillBatchingProfile`@smart-byte/leani-sdk/backfill ``` export type BackfillBatchingProfile = "latency" | "balanced" | "throughput"; ``` `backfillBatchingProfile`@smart-byte/leani-sdk/backfill ``` export function backfillBatchingProfile( profile: BackfillBatchingProfile, ): Required; ``` `EffectiveBackfillBatching`@smart-byte/leani-sdk/backfill ``` export interface EffectiveBackfillBatching { targetEncodedBytes: number; maximumEncodedBytes: number; maximumEvents: number; maximumProcessedBlocks: number; maximumDelayMs: number; maximumBufferedBatches: number; maximumBufferedBytes: number; compression: "none" | "gzip"; } ``` `CreateBackfillSubscription`@smart-byte/leani-sdk/backfill ``` export type CreateBackfillSubscription = CreateBackfillSubscriptionBase & ( | { fromBlock: number; toBlock: number | "finalized"; ranges?: never; } | { ranges: RequestedBackfillRange[]; fromBlock?: never; toBlock?: never; } ); ``` `liveGapBackfill`@smart-byte/leani-sdk/backfill ``` export function liveGapBackfill( range: BackfillRange, request: Omit & { consumer: NonNullable; }, ): CreateBackfillSubscription; ``` `BackfillStreamHello`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamHello { type: "hello"; apiVersion: "1"; chainId: number; processor: ProcessorSummary; subscriptionId: string; streamId: string; streamKind: "backfill"; publicationRevision: string; ranges: BackfillRange[]; acknowledgedCursor: string; storeEpoch: string; heartbeatIntervalMs: string; sessionExpiresAtUnixMs: string; leaseTtlMs: string; } ``` `BackfillStreamBatch`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamBatch { type: "batch"; streamId: string; originKind: "historical_backfill" | "recompute"; originId: string; publicationRevision: string; fromBlock: number; throughBlock: number; processedBlockCount: string; domainChangeCount: string; progressUnitCount: string; rawPayloadBytes: string; uncompressedEncodedBytes: string; transmittedBytes: string; buildDelayMs: string; firstCursor: string | null; lastCursor: string | null; acknowledgeableCursor: string; changes: ChangeEnvelope[]; } ``` `BackfillStreamCompletion`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamCompletion { type: "backfill_complete"; subscriptionId: string; streamId: string; ranges: BackfillRange[]; throughBlock: number; cursor: string; mode: BackfillExecutionMode; disposition: | "published_all" | "published_missing_only" | "already_covered_noop"; requestedBlockCount: string; coveredBeforeRequestBlockCount: string; coveredBeforeRequestRanges: BackfillRange[]; newlyProcessedBlockCount: string; republishedBlockCount: string; domainChangeCount: string; preexistingCoverageSkipped: boolean; repairHint?: "recompute"; } ``` `BackfillStreamHeartbeat`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamHeartbeat { type: "heartbeat"; emittedAt: string; } ``` `BackfillStreamReset`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamReset { type: "reset_required"; code: "reset_required"; message: string; earliestAvailableSequence: string | null; latestAvailableSequence: string | null; } ``` `BackfillStreamFailure`@smart-byte/leani-sdk/backfill ``` export interface BackfillStreamFailure { type: "error"; code: string; message: string; earliestAvailableSequence: string | null; latestAvailableSequence: string | null; } ``` `BackfillStreamRecord`@smart-byte/leani-sdk/backfill ``` export type BackfillStreamRecord = | BackfillStreamBatch | BackfillStreamCompletion | BackfillStreamHeartbeat; ``` `LiveStreamHello`@smart-byte/leani-sdk/backfill ``` export interface LiveStreamHello { type: "hello"; apiVersion: "1"; chainId: number; processor: ProcessorSummary; streamId: string; streamKind: "live"; acknowledgedCursor: string; storeEpoch: string; heartbeatIntervalMs: string; sessionExpiresAtUnixMs: string; leaseTtlMs: string; /** What the processor covers as the session starts: diff it against the * destination to find blocks to request as history. */ coverage: ProcessorCoverage; } ``` `LiveStreamBatch`@smart-byte/leani-sdk/backfill ``` export interface LiveStreamBatch { type: "batch"; streamId: string; originKind: "live" | "live_recovery"; originId: string; publicationRevision: "0"; fromBlock: number; throughBlock: number; processedBlockCount: "1"; domainChangeCount: string; progressUnitCount: "1"; rawPayloadBytes: string; uncompressedEncodedBytes: string; transmittedBytes: string; buildDelayMs: string; firstCursor: string; lastCursor: string; acknowledgeableCursor: string; changes: ChangeEnvelope[]; } ``` `LiveStreamRecord`@smart-byte/leani-sdk/backfill ``` export type LiveStreamRecord = | LiveStreamBatch | BackfillStreamHeartbeat; ``` `DeliverySession`@smart-byte/leani-sdk/backfill ``` export interface DeliverySession extends AsyncIterable { readonly hello: THello; acknowledge( cursor: string, options?: { signal?: AbortSignal }, ): Promise; close(): Promise; } ``` `BackfillDeliverySession`@smart-byte/leani-sdk/backfill ``` export interface BackfillDeliverySession extends DeliverySession> { batches(): AsyncIterable>; /** * Every acknowledgement boundary with the changes it covers: each batch, * including one without changes, then the completion. Commit `events`, * then acknowledge `ackCursor`. */ deliveryBatches(): AsyncIterable>; /** * The batches' changes, flattened. They carry no acknowledgement * boundary; acknowledge from `deliveryBatches()` or `batches()`. */ events(): AsyncIterable>; } ``` `LiveDeliverySession`@smart-byte/leani-sdk/backfill ``` export type LiveDeliverySession = DeliverySession< LiveStreamHello, LiveStreamRecord >; ``` `MultiLaneSubscriptionRequest`@smart-byte/leani-sdk/backfill ``` export interface MultiLaneSubscriptionRequest { processor: string; consumer: string; credential?: string; lanes: { live?: boolean; history?: Array<{ subscriptionId: string }>; }; view?: "live_priority" | "lane_specific"; batching?: BackfillBatchingRequest; signal?: AbortSignal; } ``` `UnifiedDeliveryBatch`@smart-byte/leani-sdk/backfill ``` export type UnifiedDeliveryBatch = | { laneKind: "live"; laneId: "live"; streamId: string; ackCursor: string; events: ChangeEnvelope[]; completion?: never; record: LiveStreamBatch; } | { laneKind: "history"; laneId: string; streamId: string; ackCursor: string; events: ChangeEnvelope[]; completion?: BackfillStreamCompletion; record: BackfillStreamBatch | BackfillStreamCompletion; }; ``` `HistoryDeliveryBatch`@smart-byte/leani-sdk/backfill ``` export type HistoryDeliveryBatch = Extract< UnifiedDeliveryBatch, { laneKind: "history" } >; ``` `MultiLaneDelivery`@smart-byte/leani-sdk/backfill ``` export interface MultiLaneDelivery { batches(): AsyncIterable>; liveBatches(): AsyncIterable>; historyBatches( subscriptionId: string, ): AsyncIterable>; acknowledge( batch: UnifiedDeliveryBatch, options?: { signal?: AbortSignal }, ): Promise; close(): Promise; } ``` `BackfillSubscriptionClientOptions`@smart-byte/leani-sdk/backfill ``` export interface BackfillSubscriptionClientOptions { baseUrl: string | URL; token?: string; fetch?: FetchLike; /** Deadline for JSON requests; established delivery streams stay open. */ timeoutMs?: number; } ``` `BackfillSubscriptionClient`@smart-byte/leani-sdk/backfill ``` export interface BackfillSubscriptionClient { readonly baseUrl: URL; create(request: CreateBackfillSubscription): Promise; list(options?: { signal?: AbortSignal }): Promise; inspect( subscription: string, options?: { signal?: AbortSignal }, ): Promise; cancel( subscription: string, options?: { signal?: AbortSignal }, ): Promise; /** Re-queue a failed subscription to resume from its committed progress. */ retry( subscription: string, options?: { signal?: AbortSignal }, ): Promise; /** * Delete a terminal subscription. A failed or cancelled one is refused * while its required consumer has unacknowledged records, unless * `discardUnacknowledged` is set. */ delete( subscription: string, options?: { signal?: AbortSignal; discardUnacknowledged?: boolean }, ): Promise; changes( subscription: string, consumer: string, options?: { credential?: string; limit?: number; signal?: AbortSignal; }, ): Promise<{ data: ChangeEnvelope[]; nextCursor: string | null; coverage: unknown; }>; stream( subscription: string, consumer: string, options?: { credential?: string; batching?: BackfillBatchingRequest; signal?: AbortSignal; }, ): Promise>; streamLive( processor: string, consumer: string, options?: { credential?: string; signal?: AbortSignal }, ): Promise>; subscribe( request: MultiLaneSubscriptionRequest, ): Promise>; } ``` `createBackfillSubscriptionClient`@smart-byte/leani-sdk/backfill ``` export function createBackfillSubscriptionClient( options: BackfillSubscriptionClientOptions, ): BackfillSubscriptionClient; ``` `BackfillFailurePolicy`@smart-byte/leani-sdk/backfill ``` export type BackfillFailurePolicy = "retry" | "discard" | "throw"; ``` `RunBackfillOptions`@smart-byte/leani-sdk/backfill ``` export interface RunBackfillOptions { /** Apply one history batch. It is acknowledged after this resolves. */ onBatch(batch: HistoryDeliveryBatch): void | Promise; /** What a failed subscription does: retry it (default), delete it, or throw. */ onFailure?: BackfillFailurePolicy; /** Retries before "retry" gives up and throws. Default 3. */ maxRetries?: number; /** The consumer's credential; `request.consumer.credential` by default. */ credential?: string; batching?: BackfillBatchingRequest; /** * Stops the run, as `request.signal` does too. The run rejects with the * abort reason and keeps the subscription. */ signal?: AbortSignal; } ``` `RunBackfillResult`@smart-byte/leani-sdk/backfill ``` export interface RunBackfillResult { outcome: "completed" | "cancelled" | "failed"; /** The subscription as it ended, before its deletion. */ subscription: BackfillStatus; /** The completion record this run received, if it streamed one. */ completion?: BackfillStreamCompletion; deletion: HistoricalWorkDeletion; } ``` `runBackfill`@smart-byte/leani-sdk/backfill ``` export async function runBackfill( client: BackfillSubscriptionClient, request: CreateBackfillSubscription, options: RunBackfillOptions, ): Promise; ``` `parseNdjson`@smart-byte/leani-sdk/backfill ``` export async function* parseNdjson( body: ReadableStream, signal?: AbortSignal, ): AsyncGenerator; ``` `LeaniErrorBody`@smart-byte/leani-sdk ``` export interface LeaniErrorBody { error: { code: string; message: string; retryable: boolean; details?: Record | null; requestId: string; }; } ``` # Architecture > Learn Leani's invariants, component boundaries, normalized data model, planning, lanes, storage classes, and reorg model. ## 1. Terminology **Protocol state** is Ethereum account/storage/code state required to execute the EVM. The core node does not retain it. **Index state** is application-defined state such as balances for watched addresses, the latest Uniswap pool price, or blobs.money aggregates. This is the storage the operator intentionally chooses. **Raw history** is block headers, bodies, transactions, receipts, logs, traces, and state diffs before projection. **Delta** is the compact deterministic result of mapping one canonical block for one processor. **Checkpoint** records a processor version, canonical block hash, state cursor, and source/verification metadata. ## 2. Architectural invariants 1. Processors consume a source-neutral representation. 2. Missing capabilities cause an error, never silently empty fields. 3. Raw historical material is transient unless retention is explicitly enabled. 4. State transitions are applied in canonical block order. 5. Every unfinalized state transition is reversible until finalized. 6. Historical and live lanes overlap and verify before handoff. 7. Outputs carry provenance, processor version, and a canonical cursor. 8. Source trust and cryptographic verification are different dimensions. 9. Resource budgets are enforced in bytes, not only in block counts. 10. An RPC response is served only when its completeness semantics are known. ## 3. Component view  ## 4. Source contracts A source must declare rather than imply what it can provide. ```rust struct SourceDescriptor { id: String, kind: SourceKind, range: BlockRange, capabilities: CapabilitySet, trust: TrustModel, finality: FinalityModel, partitioning: Partitioning, expected_lag: Duration, } trait HistorySource { fn describe(&self) -> SourceDescriptor; async fn plan(&self, request: RangeRequest) -> Vec; async fn stream(&self, chunk: SourceChunk) -> Stream; } trait LiveSource { fn describe(&self) -> SourceDescriptor; async fn subscribe(&self, from: Cursor) -> Stream; } ``` Initial capabilities: ```text HEADER BODY TRANSACTIONS CALLDATA RECEIPTS LOGS WITHDRAWALS BLOB_SIDECARS TRACES STATE_DIFFS CONSENSUS_FINALITY MEMPOOL ``` Some fields can be derived from stronger capabilities. For example, `LOGS` can be decoded from full `RECEIPTS`. Derived capabilities retain the provenance of the source material. ## 5. Normalized data The normalized envelope should be field-oriented and cheaply projectable. It should not force every source to create a trace-rich Ethereum Firehose block. ```text BlockEnvelope chain_id block_ref number hash parent_hash timestamp finality unattested | attested | finalized header? body? transactions? receipts? logs? withdrawals? blob_sidecars? traces? state_diffs? capabilities provenance[] verification ``` `verification` should be structured: ```text header_hash verified | failed | not_checked parent_continuity verified | failed | not_checked transactions_root verified | failed | unavailable receipts_root verified | failed | unavailable consensus_anchor finalized | attested | unavailable dataset_checksum verified | failed | unavailable ``` Filtered Xatu logs may be canonical and operationally trustworthy but cannot be independently checked for completeness without all receipts. That is represented as `trusted_dataset`, not `receipts_root=verified`. ## 6. Planning and pushdown Processor manifests are compiled into a `DataRequest`: ```text block range required capabilities addresses topic filters transaction types fields finality policy ``` Backends push down whatever they can: * Xatu reads only required Parquet tables/columns and filters address/topics. * A predecoded transfer table can bypass general log decoding. * EraE reads complete bodies and receipts but only emits selected normalized fields. * P2P retrieves complete bodies and/or receipts when the requirement needs them. Predicate-complete log-only requests use verified header blooms to skip definite negatives, fetch whole-block receipts for positives, and exact-filter logs. Bodies are omitted when the processor’s declared log identity does not include transaction hash. The planner may combine sources within one processor. Example: * beacon Parquet for blob header/transaction fields; * canonical execution transaction Parquet for gas used/status; * P2P for the current head. At source boundaries it requests an overlap and compares block hashes and, where possible, roots. ### Architecture payoff: work follows processor requirements The processor descriptor controls both acquisition and output. A narrowly filtered processor can request less source material, perform less mapping, and publish fewer bytes without changing the source, coordinator, delivery, or ACK contracts. The blobs transform emits a block delta for every block and validates and joins projected transactions and receipts while calculating blob fee/burn fields. The Uniswap transform scans only matching pool logs and emits nothing for blocks without a matching observation. Equal block ranges therefore do not imply equal input, compute, or output work. The built-in benchmark command can compare these shapes across acquisition, processing, durable delivery, and SDK/PostgreSQL destinations. Its generated reports are local artifacts and must retain machine, source, commit, configuration, and cold/warm-run context when results are shared. ## 7. Cold and hot lanes ### Cold lane The cold lane: 1. snapshots a target block near the current finalized head; 2. plans historical chunks; 3. downloads/streams chunks with bounded concurrency; 4. maps blocks in parallel; 5. writes compact deltas tagged by block hash; 6. lets the reducer apply them in order; 7. discards the source chunk after verification and map completion. ### Hot lane The hot lane starts immediately: 1. observes head/fork/finality events; 2. retrieves block bodies and receipts; 3. validates and maps them; 4. retains recent raw envelopes for RPC/reorg; 5. retains compact deltas until the cold reducer catches up; 6. optionally publishes block-local provisional results. ### Joining The lanes must overlap by at least the configured reorg window or a smaller finalized overlap:  The scheduler deduplicates by `(chain_id, block_number, block_hash, processor_version)`. A block number alone is never an identity. ### Stateful outputs while backfilling There are three valid modes: 1. **Block-local:** publish hot outputs immediately because history is irrelevant. 2. **Checkpoint-seeded:** load a verified aggregate checkpoint and continue at the head. 3. **Catch-up:** capture hot deltas, mark the aggregate incomplete, and publish complete state only when the ordered reducer reaches the head. The runtime must not present an empty-seeded token balance as a complete current balance. ## 8. Processor model The first processor API should be deliberately small: ```rust trait Processor { fn descriptor(&self) -> ProcessorDescriptor; fn map_block(&self, block: &BlockView) -> Result; } trait Reducer { fn apply( &self, tx: &mut ReducerTransaction, cursor: Cursor, delta: BlockDelta, ) -> Result; } ``` `ReducerTransaction` is a core-owned namespaced state/index API. It captures preimages and constructs undo automatically, then atomically commits state, changes, cursor/coverage, provenance, and sink outbox rows. A native processor does not receive a raw SQLite transaction. Separating map from reduce enables: * parallel historical decode/filter work; * canonical ordered state; * small live catch-up buffers; * reuse of map output across multiple sinks; * deterministic source equivalence tests. Processor descriptors include: * semantic version and code hash; * start block; * required capabilities; * address/topic filters; * output schema; * current-only, bucketed, or full-history retention; * publication policy: unattested, attested or finalized; * whether deltas are commutative or strictly ordered. The concrete first-release modes are: * `block_local`: block N cannot depend on prior output, so hot and historical coverage can be filled concurrently; * `ordered_state`: mapping is parallel but one reducer applies blocks in order. The blobs.money processor is block-local. ERC-20 balances and Uniswap pool state are ordered-state processors. ## 9. Storage classes ### Mandatory * processor state chosen by the application; * committed cursor and processor version; * source/checksum/provenance metadata; * unfinalized undo records; * bounded hot deltas while cold reduction catches up. ### Bounded * recent raw block/receipt window; * one or a few Parquet/EraE chunks; * mapper work queue; * response cache for on-demand RPC; * checkpoints. ### Optional and potentially large * complete log history; * all token-holder balances; * per-transfer balance history; * transaction hash to block locator; * raw block/receipt history; * Firehose merged blocks; * Substreams module caches. Default pruning rules: 1. discard mapped historical input immediately; 2. prune raw live envelopes after finality plus the configured RPC window; 3. prune undo records only after finality; 4. keep checkpoints according to processor policy; 5. expose byte use by storage class and processor. ## 10. Reorg and finality model Post-Merge canonicality is anchored to consensus safe/finalized execution payloads. A future pre-Merge/genesis profile must additionally validate canonical total difficulty or an appropriate history accumulator/checkpoint; parent and trie-root validity alone is insufficient to select the canonical proof-of-work branch. Every unfinalized block application is one atomic transaction: ```text validate parent persist delta apply output changes persist undo advance unfinalized cursor commit publish apply event ``` On reorg, Leani identifies the common ancestor, undoes the old branch in reverse order and publishes undo events, then applies the new branch in forward order and publishes apply events:  Each such transaction covers one processor. The shared live lane retains a block’s canonical frame, and switches the canonical chain for a reorg, before any processor applies or undoes it, and a delta can be persisted as pending before its apply; each of those steps commits separately. Startup reconciliation repairs an interruption between them before the live lanes open: it undoes unfinalized processor blocks that are not canonical, deletes pending deltas no processor can apply, and replays retained canonical frames to lanes that trail them. When consensus finality advances: * mark matching outputs finalized; * publish finality events; * remove undo/raw data older than policy; * detect and halt on any finalized-chain contradiction. An operator may choose `publish = "finalized"` for simpler downstream semantics at the cost of roughly two epochs of latency. ## 11. Lean processing runtime and ecosystem adapters Firehose provides a useful cursor/block transport and ecosystem format. Its normal architecture also persists one-block and merged 100-block files, while Substreams caches module outputs. Permanent raw storage conflicts with this project’s default. The project can implement the parts needed for its use cases directly: 1. canonical, fork-aware historical/live block coordination; 2. capability-aware source planning and filter pushdown; 3. deterministic per-block mapping; 4. canonical ordered state reduction; 5. transactional apply/undo and finality; 6. checkpoints and resumable cursors; 7. output queries and subscriptions; 8. pluggable database sinks. This is much smaller than recreating distributed Firehose and Substreams. The core does not need a permanent raw-block object store, tier-1/tier-2 workers, shared module caches, or the ability to execute arbitrary third-party packages. Compatibility should be layered rather than built into the internal model: 1. **Output compatibility:** optionally emit standard database/entity change protobufs. This is the cheapest and may enable existing sinks. 2. **Firehose reader compatibility:** adapt a complete enough `BlockEnvelope` into `FIRE INIT`/`FIRE BLOCK`, allowing an operator to run upstream Firecore and accept its storage model. 3. **Firehose gRPC compatibility:** implement the downstream historical/live service. This is more work because cursor, fork, and retention behavior must match. 4. **Substreams package compatibility:** execute event-only `.spkg` modules. This is the most expensive layer because it requires the Wasm host API, module DAG/store semantics, protobuf versions, parameters, caches, and package validation. Proposed compatibility path: 1. keep the internal envelope minimal and capability-aware; 2. use versioned protobufs and deterministic cursor/apply/undo semantics internally; 3. keep processors independent of storage and transport; 4. implement native Rust processors first; 5. experiment with standard output-change protobufs and a Firehose reader adapter; 6. populate an Ethereum `sf.ethereum.type.v2.Block` only with fields genuinely available; 7. test known event-only Substreams packages; 8. reject packages that require calls, traces, or state changes; 9. adopt a Substreams package runtime only if real reusable packages justify its complexity. Full compatibility is not an MVP dependency. The architecture should make adapters possible, but it should not make `sf.ethereum.type.v2.Block`, Firehose cursor internals, or the Substreams Wasm ABI the project’s foundational representation. Ethereum packages can reasonably assume richer executed-block data than receipts provide. ## 12. Example configuration A node reads strict, versioned TOML, whose machine-readable schema is `config/schema-v1.json`. `config/example.toml` is a complete advanced document, `config/modes/` holds the checked operating profiles, and `config/defaults/ethereum-mainnet.toml` is what a compact document expands to. Tests load every shipped configuration and hold the schema to the configuration types, so these files, not an illustration, show the current shape. The [configuration guide](/docs/operations/configuration-guide/) explains each section. ## 13. Execution networking Leani consumes isolated networking crates from Reth 2.5.2 rather than forking SHiNode or running a complete execution client. One manager persists for the process lifetime, uses a stable key in the data directory, shares a bounded peer cache across live/history work, enables Discv4, Discv5, and DNS discovery, and hedges warm startup across cached and newly discovered peers. Only the most recent capability-proven, currently viable body servers are dialed immediately; the broader cache is admitted in refill-paced batches so stale records cannot monopolize the first dial window. Cache refreshes merge the current Reth view into that bounded discovery set, so a short failed run cannot erase the next run’s alternatives. Leani multiplexes at most four bounded material requests per connected peer under one global, live-priority admission gate. Body and receipt requests start at an evidence-backed eight-block ceiling, shrink independently after a partial/failed response, and grow again only after sustained success. Finalized P2P history is a protocol-verified, operator-bounded fallback after retained and archive sources: Leani proves ancestry to a beacon-derived execution anchor before emitting commitment-checked material. Filtered log-only processors use verified header blooms to avoid negative-block receipts and can omit bodies when transaction hash is not part of their declared output identity; generic RPC and broad processors retain the full-material path. Durable processor coverage and retained parent-linked canonical material remain the restart authorities. # Automated verification > Run Leani's network-free correctness fixtures and keep performance, Mainnet, fault-injection, and soak evidence on controlled hosts. The project has two separate end-to-end gates. They answer different questions and neither substitutes for the other. ## Network-free binary fixture For a clean-host/native/container smoke test: ```bash leani e2e fixture \ --data-dir /tmp/leani-fixture \ --blocks 128 \ --report /tmp/leani-fixture/report.json ``` The command rejects an existing database, creates a deterministic finalized chain, runs the real historical runtime and SQLite reducer, retains changes and bounded checkpoints, verifies continuous coverage, runs SQLite integrity checks, and emits a versioned JSON report. It proves packaging and local durability without claiming current upstream interoperability. ## Deterministic 10,000-block gate Run: ```bash cargo test -p leani-runtime \ ten_thousand_block_hot_cold_restart_converges_and_serves_coverage \ --locked -- --nocapture ``` This test uses real runtime, SQLite, reducer, handoff, and native API code with deterministic source adapters. It: 1. creates a 10,000-block historical chain; 2. starts a 64-block live overlap and follows additional head blocks; 3. runs both block-local and ordered-state processors; 4. interrupts both historical lanes after a committed prefix; 5. closes and reopens the SQLite database; 6. replays the live overlap and resumes historical coverage; 7. compares every live/historical overlap hash; 8. persists a `verified` handoff verdict; 9. drains ordered live deltas without waiting for another head; 10. proves exact continuous coverage with no pending deltas or duplicate applied blocks; 11. verifies that only the overlap and live head remain in recent raw storage; 12. runs SQLite integrity verification; and 13. queries processor coverage through the native API router. It is part of the default workspace test suite and therefore runs in CI on Linux and macOS. ## Deterministic fault matrix The default workspace suite injects these failure classes at source, runtime, store, and API boundaries: | Failure | Evidence test | | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | disconnect/source outage and failover | `historical_run_fails_over_at_a_committed_chunk_boundary`, `only_the_failed_chunk_moves_to_the_next_source`, `only_a_named_hole_moves_to_the_next_source`, `a_hole_names_every_block_from_its_first_to_its_last`, `a_blob_hole_names_every_block_from_its_first_to_its_last`, `a_later_detour_reuses_the_proven_header_chain`, `a_source_failing_again_before_committing_hands_over_the_rest_of_the_gap`, `blocks_a_source_has_not_published_hand_over_the_rest_of_the_gap` | | source corruption and input budget overrun | `history_stream_rejects_corruption_and_budget_overrun` | | reorg apply/undo and removed logs | `shared_live_runtime_fans_out_and_reorgs_one_source_subscription`, `websocket_reorg_marks_old_logs_removed_before_replacements` | | process interruption/restart, and startup reconciliation of what it left | `historical_run_resumes_from_durable_coverage_after_process_reopen`, the 10,000-block gate, `startup_reconciliation_deletes_pending_deltas_no_lane_can_apply`, `startup_reconciliation_replays_retained_frames_to_lanes_that_applied_none`, `startup_reconciliation_replays_retained_frames_above_a_hole`, `startup_reconciliation_leaves_a_consistent_store_untouched`, `a_hole_below_the_retained_window_is_left_to_the_cold_backfill`, `a_lane_the_store_paused_at_the_tip_resumes_after_a_restart_without_a_hole`, `a_lane_the_store_failed_at_the_tip_can_be_reset_after_a_restart`, `a_retained_tip_the_finalized_anchor_proves_is_kept_not_undone`, `a_proof_from_above_a_finalized_head_keeps_and_finalizes_the_retained_tail`, `a_branch_reorged_away_during_downtime_is_undone_only_above_the_fork`, `an_anchored_overlap_redelivers_proven_blocks_without_applying_them_twice`, `a_failed_or_stalled_retained_proof_falls_back_to_an_empty_ancestry`, `a_cancelled_retained_proof_seeds_nothing` | | crash injected into a live commit, reorg, or backfill commit, then startup reconciliation | `startup_reconciliation_replays_a_block_a_crash_kept_from_some_lanes`, `startup_reconciliation_applies_a_delta_a_crash_left_pending`, `startup_reconciliation_undoes_a_branch_a_crashed_reorg_left_applied`, `startup_reconciliation_records_where_a_crash_paused_a_lane_at_its_delivery_limit`, `startup_reconciliation_records_where_a_crash_failed_a_lane_at_its_delivery_limit`, `startup_reconciliation_records_where_a_crash_stopped_a_ledger_at_its_delivery_limit`, `startup_reconciliation_deletes_a_delta_a_crashed_backfill_left_pending` | | checkpoint cadence and retention, restore of corrupted state at an exact checkpoint cursor, and portable savepoint integrity | `automatic_checkpoints_are_bounded_and_savepoints_are_explicit`, `portable_savepoints_are_bounded_per_processor` | | restorable checkpoints at job completion and after finality promotion | `completed_jobs_leave_a_restorable_checkpoint`, `checkpoint_restore_accepts_later_finality_promotion` | | undo retention past finality | `ordered_undo_rows_stay_within_the_safety_depth`, `finalized_applies_write_no_undo_rows` | | compacted coverage re-apply and handoff, interrupted handoffs | `compacted_coverage_counts_as_already_applied`, `hot_cold_handoff_verifies_across_compacted_coverage`, `running_handoffs_hold_back_only_their_overlap`, `a_new_handoff_supersedes_an_unfinished_one` | | one processor’s reducer, delta, finality, or cold-backfill failure | `reducer_failure_parks_only_its_processor_while_shared_ingestion_continues`, `a_canonical_block_local_reducer_failure_parks_only_its_lane`, `a_lane_parked_by_its_reducer_does_not_fail_the_live_lane_after_restart`, `a_conflicting_pending_delta_fails_only_its_lane`, `a_delta_conflict_lane_resumes_after_a_restart_and_a_reset`, `one_processor_finality_failure_does_not_stop_the_others_or_pruning`, `a_finality_contradiction_is_recorded_for_a_lane_failed_for_another_reason`, `one_failed_cold_backfill_parks_only_its_processor`, `a_lane_parked_below_a_seeded_anchor_walks_past_it_once_history_passes`, `a_lane_failed_outside_the_live_lane_keeps_its_failure_and_is_not_remapped`, `a_failed_canonical_delivery_lane_resets_through_the_api_and_resumes`, `a_failed_lane_keeps_its_first_failure_unless_finality_contradicts_it`, `a_park_racing_a_finality_failure_leaves_the_failed_lane_alone`, `pausing_a_lane_that_another_component_failed_is_benign`, `resetting_a_contradicted_live_lane_answers_that_it_needs_a_rebuild` | | parked lanes across subscribe, reorg, finality, and concurrent drains | `a_lane_paused_at_subscribe_receives_material_it_requires_after_resuming`, `a_reorg_leaves_a_paused_lane_paused`, `an_ordered_gapped_lane_resumes_after_a_reorg_below_its_marker`, `a_block_local_gapped_lane_applies_a_shortening_reorg_without_a_hole`, `unfinalized_gap_replay_pauses_until_finality_then_recovers_from_history`, `concurrent_drains_of_one_live_gap_do_not_fail_the_lane`, `rewind_probe_block_local_gap_above_lowest_reverted`, `rewind_probe_ordered_gap_above_lowest_reverted`, `rewind_probe_block_local_gap_at_lowest_reverted`, `gapped_lanes_rewound_without_a_replacement_branch_stay_on_the_chain`, `a_rewind_without_a_replacement_branch_settles_a_paused_lane_at_once`, `a_gapped_lane_whose_whole_history_is_reverted_resumes_at_its_start`, `a_gap_marker_off_the_canonical_chain_is_re_pointed_instead_of_completed`, `a_rewind_onto_a_seeded_anchor_a_paused_lane_never_applied_completes_its_gap`, `an_ordered_replay_onto_a_seeded_anchor_checks_its_parent_link` | | blocks an on-demand live lane skips after downtime, announced or filled before the next block | `an_on_demand_block_local_lane_announces_the_blocks_it_skips_before_applying`, `a_lane_that_replays_past_skipped_blocks_at_startup_announces_them`, `a_live_consumer_sees_blocks_the_live_lane_skipped`, `an_on_demand_lane_that_fills_live_gaps_delivers_the_skipped_blocks_before_the_next_one`, `a_live_gap_larger_than_the_fill_limit_is_announced_without_fetching_history`, `a_live_gap_fill_whose_history_fails_announces_the_whole_range`, `a_live_gap_fill_announces_the_rest_at_a_frame_that_does_not_descend_from_the_last_one`, `a_live_gap_fill_never_applies_a_last_block_the_following_block_does_not_name`, `a_live_gap_fill_waits_for_delivery_capacity_without_announcing_or_refetching`, `a_replayed_live_gap_fill_refused_above_the_resume_mark_waits_without_refetching`, `a_live_gap_fill_that_runs_out_of_time_announces_the_rest`, `a_live_gap_fill_abandons_a_history_request_that_outlasts_its_time_limit`, `a_startup_replay_past_a_hole_fills_it_when_configured`, `a_live_gap_above_the_finalized_head_is_announced_not_filled`, `a_storage_paused_fill_resumes_once_the_store_has_room`, `a_live_gap_fill_held_at_a_node_wide_delivery_budget_resumes_once_the_budget_has_room`, `a_live_gap_fill_refused_at_a_node_wide_delivery_budget_waits_for_room_without_refetching`, `live_delivery_refusal_decides_as_an_apply_without_writing` | | non-contiguous live blocks and reorgs | `shared_live_rejects_a_block_that_does_not_extend_the_canonical_tip`, `shared_live_rejects_a_reorg_that_does_not_revert_the_tip` | | recent storage at its hard limit | `live_ingestion_waits_at_the_recent_hard_limit_until_finality_prunes`, `live_ingestion_fails_after_waiting_too_long_at_the_recent_hard_limit` | | slow required consumer | `required_consumer_remains_protective_after_lease_lapse` | | disk/delivery pressure | `delivery_limit_pauses_and_auto_resumes_below_low_water`, `a_known_unavailable_lane_the_store_paused_without_a_marker_is_mapped_again`, `a_failed_ledger_behind_its_history_gets_no_marker_ahead_of_its_cursor` | | acknowledgement/pruning/upgrade replay faults | `acknowledgement_cannot_exceed_the_consumers_delivered_head`, `delivery_pruner_applies_finality_age_and_bounded_batch_limits`, `unacknowledged_changes_survive_a_v6_to_v7_upgrade` | | stable query during concurrent commits | `query_snapshot_keeps_values_and_stream_boundary_stable`, `generic_query_and_follow_snapshot_is_stable_across_new_commits` | | job-result publication | `terminal_historical_job_spools_only_its_final_result` | | backup/restore | `backup_reopens_as_a_complete_verified_restore` | These are deterministic correctness gates. Filesystem exhaustion, process kill timing, clean-host packaging, and multi-day storage behavior remain staging evidence and must be recorded separately. ## Public Mainnet staging gate This gate uses the production history, execution-P2P, and finality adapters. It does not delete or replace an existing database. Public datasets may lag the verified finalized anchor. The network supervisor therefore tries the configured history backends first and may fill only the remaining finalized suffix through the consensus-anchored execution-P2P history fallback. By default every gap at or after the processor’s configured start block is eligible. Setting `sources.live.history_fallback_blocks` applies a hard maximum suffix ending at the finalized anchor. The fallback fetches bounded header batches concurrently, retains only one 32-byte hash per proven block, and validates the assembled chain through the finalized anchor before yielding any body or receipt. It reuses the process-wide persistent peer pool and fails if an uncovered gap falls outside an explicitly configured bound. Request failure penalizes or rotates the responsible Reth peer without restarting discovery or disconnecting unrelated peers. Body and receipt material starts as one-block probes and grows adaptively to four-block batches after success. Effective concurrency is bounded by connected peers, the global request gate, and the source budget so ordinary Mainnet responses remain below the ETH protocol’s 2 MiB soft response limit. Each successful four-block-or-smaller history unit is emitted before the next unit, successful bodies are retained while receipts retry, and a queued one-block head request takes the next available local request slot before normal catch-up work. On process or lane restart, the node validates the finalized anchor against the retained canonical SQLite suffix. If that suffix is complete and parent-linked, live following resumes after its durable tip without redownloading the handoff overlap. Before any lane opens, startup reconciliation brings each processor back onto that suffix: it undoes blocks an interrupted reorg left applied and replays retained blocks an interrupted commit skipped. When an archive later covers the maximum possible P2P bridge suffix, the reconciliation lane remaps sequential finalized batches and compares their processor-delta checksums with the checksums committed live. This audit does not require retaining the original raw frames. First obtain a recent weak-subjectivity checkpoint root and its finalized beacon slot independently. Put them in a copy of `config/example.toml`. The command refuses the checked-in placeholder. Choose a recent start block. For a 10,000-block test, subtract 9,999 from the finalized execution block reported by: ```bash cargo run --locked --release -- \ --config config/mainnet-e2e.toml \ source probe finality \ --report evidence/finality.json ``` Then run: ```bash cargo run --locked --release -- \ --config config/mainnet-e2e.toml \ e2e mainnet \ --processor blobs-money \ --from-block \ --data-dir ./data/mainnet-e2e \ --minimum-follow-blocks 16 \ --stable-seconds 300 \ --max-head-age-seconds 60 \ --timeout-seconds 7200 \ --report evidence/mainnet-e2e.json ``` Pass `--resume` after an intentional interruption. Without it, an existing `leani.sqlite` is rejected. The command passes only after all of these remain true for the stable window: * the historical/live overlap verdict is durably `verified` for every participating processor; * processor coverage is one continuous range from `--from-block` through its current cursor; * the cursor is at least `--minimum-follow-blocks` beyond the initial finalized handoff anchor; * no ordered live deltas remain pending; * the latest cursor has retained canonical material; * its execution timestamp is at most `--max-head-age-seconds` old; and * the network lane has not exited. Peer availability remains part of the evidence. Lowering `minimum_peers` for a diagnostic run must be recorded and does not satisfy a stricter production profile. On completion or a handled failure, timeout, or interrupt, it cancels the network lane, verifies SQLite, and writes the JSON report. The report contains the exact binary commit, bounds, durable handoff, cursor, coverage, latest-block age, logical store attribution, elapsed time, and failure details. ## Evidence policy The deterministic gate proves scheduling, restart, reconciliation, storage, and API invariants. The Mainnet gate proves current external interoperability. A production claim requires both, followed by a fault-injected staging run and an independent soak whose duration and acceptance criteria are recorded by the release operator. GitHub Actions runs only deterministic correctness and contract checks. It does not run synthetic or real-source benchmarks, Mainnet evidence gates, fault-injected delivery measurements, or soaks, and it does not upload their reports. Performance evidence must be collected explicitly on a controlled host with `scripts/run-controlled-benchmarks.sh`; its default output is under the ignored `.private/evidence/performance` tree. Real-source and Mainnet runs remain manual because source availability, trust inputs, host state, and network conditions are part of the result. Do not commit a weak-subjectivity checkpoint as an evergreen trusted value. Record the checkpoint source and acquisition time alongside each evidence report. # Benchmarking > Measure acquisition, processing, and durable delivery against deterministic corpora, run the controlled suites, and sweep candidates on a quiet host. Performance evidence is collected on controlled hosts, never in CI. The network-free correctness gates are described in [automated verification](/docs/contributing/automated-verification/); this page covers the `benchmark` command, the controlled suites, and candidate sweeps. ## Measure one stage or the complete path Measure acquisition, isolated materialization, durable delivery, or their combined path against deterministic streaming corpora. `blobs-like` runs the production blobs transform and `uniswap-like` runs the block-local WETH/USDC observation transform; zero/sparse/dense remain control and limit shapes. Reports are immutable and versioned. Report v20 records the stage and product profile independently, declares destination/query row semantics, verifies canonical logical output bytes, attributes logical storage layers, and reports final node/destination physical bytes plus `storageAmplificationMilli` (the physical/logical ratio multiplied by 1,000). It also includes the current 128-block/50-ms commit baseline and samples SQLite/WAL/segment physical storage throughout the run. Delivery measurements distinguish producer completion, consumer completion, and post-producer drain time, with producer/consumer block rates plus domain-event, logical-payload, and transmitted-byte rates. Raw-retaining profiles additionally separate raw acquisition from retained-local reads, identify whether that read verifies frames, replays a processor, or feeds delivery, report raw segment/catalog bytes, and fail if the local phase performs any further external source reads: ```bash cargo run -- benchmark \ --mode end-to-end \ --profile externalized \ --corpus blobs-like \ --blocks 10000 \ --chunk-blocks 2048 \ --warmups 1 \ --runs 3 \ --samples-report benchmark-samples.ndjson \ --report benchmark.json ``` `--mode` selects only the measured stage. Pair `acquire` with `--profile acquire-discard` or `raw-only`; `process` with `--profile materialized`, `compact-artifact`, `compact-artifact-segment`, `compact-artifact-tiered`, or `raw-artifact`, `raw-materialized`, or `raw-artifact-materialized`; and `deliver`/`end-to-end` with `--profile externalized` or `raw-externalized`. The compact-artifact profiles retain one immutable, replayable map result per finalized block. Every `raw-*` combination first retains complete processor-reuse frames and then performs processing or verification from the local raw store. Product profiles own storage and delivery policy; selecting a stage never rewrites that policy. `--consumer-delay-ms` adds deterministic destination latency. SDK/PostgreSQL runs can use `--consumer-reconnect-every-batches N` to close and independently fence a new stream session after each N acknowledged batches, and `--consumer-drop-ack-response-once` deterministically exercises recovery when the destination commit and node ACK succeed but the response is lost. The same seeded generator supports one-million-block controlled runs without retaining the corpus in memory. Add `--concurrent-live-blocks N` to an end-to-end SDK/PostgreSQL run to commit a finalized live tail while history is still acquired and delivered; `--live-block-interval-ms` controls its cadence. The report verifies independent history/live digests and durable cursors and records live commit-latency percentiles. Summary reports stay compact; high-frequency resource samples are written only when `--samples-report` is supplied, as immutable NDJSON with a content hash recorded in the summary. The summary reports the minimum, median, and maximum of the measured runs’ elapsed time and block rate. Below 10 runs, which cannot support a tail estimate, `statistics` is `min_median_max` and the p95 fields are null. From 10 runs, `statistics` is `min_median_max_p95`, and the p95 fields describe the slow tail, interpolated between runs: `p95ElapsedMilliseconds` is the time 95% of runs stayed within, and `p95BlocksPerSecondMilli` the rate 95% of runs reached, the 5th percentile of per-run rates. `peakRssMeasurement` says how `peakRssBytes` was measured. `linux_vm_hwm` is the kernel’s resident high-water mark, `VmHWM`, reset as each run’s sampling starts, so it misses no spike. `sampled_rss`, on macOS or where Linux cannot reset `VmHWM`, is the largest resident-set sample taken every `--sample-interval-ms`, so a spike between samples escapes it; a summary over runs measured both ways reports `sampled_rss`. For consumer-neutral delivery capacity, use `--mode end-to-end --profile externalized --destination rust-http`. With the default zero consumer delay, this measures source acquisition, production processor execution, durable delivery publication, real HTTP streaming, payload validation, immediate ACK, and pruning without adding an application database. Add a fixed `--consumer-delay-ms` to the same command to measure backpressure from a slower consumer. In contrast, `--mode deliver --destination rust-http` pre-fills the node before starting its timer and is the delivery-only transport ceiling; `--destination sdk-postgres` measures the concrete SDK-to-PostgreSQL path. Generated benchmark reports and samples are intentionally ignored by Git. Synthetic runs measure the local source/coordinator path, not Xatu, EraE, or P2P transport. Keep machine, source, commit, configuration, and cold/warm-run context with any result shared outside the local checkout. ## Real-source backfills The fixed mainnet profile supports `block-summary`, `blobs-money`, and `uniswap-observations`. The real-source command measures acquisition through durable delivery, acknowledgement, and pruning, using fresh node storage: ```bash cargo build --release --locked -p leani target/release/leani --config config/benchmarks/real-source.toml \ benchmark real-source \ --processor block-summary --source-policy erae-only \ --from-block 19426589 --to-block 19430684 \ --data-dir /tmp/leani-erae-blocks-run1 \ --report /tmp/leani-erae-blocks-run1.json ``` Compare the same processor, source policy, range, configuration, and release build profile. Pass the baseline report’s `outputDigest` as `--expected-output-digest` on candidate runs. A mismatched delivered-output digest fails the benchmark. Reports include source reads, fetched and normalized bytes, time to first frame, consumer completion, and sampled peak RSS. Use unique data directories and report paths for every run. Repeat public-mirror measurements on a quiet host: network bandwidth, archive-cache state, and other work affect results. The [alternating comparison harness](https://github.com/smart-byte/leani/blob/main/benchmarks/erae-backfill/README.md) supports the same policies. Its shell entry point defaults to EraE; set `LEANI_BENCHMARK_SOURCE_POLICY=xatu-only`, `p2p-only`, or `history-portfolio` to compare another policy. The manifest pins that choice, and resumed reports must match it. Each policy needs a compatible source configuration. To isolate EraE acquisition and frame construction from the node and delivery, the source crate includes a streaming example: ```bash cargo run --release --locked -p leani-source-erae --example throughput -- \ file:///absolute/path/to/erae-mirror/ 19426589 19428636 full 4 4 ``` The last three arguments select `headers`, `bodies`, `logs`, or `full`, active chunks, and buffered frames per stream. HTTPS and loopback HTTP mirrors also work. A local mirror must contain the checksum catalog and named archive objects. The example verifies block counts and parent continuity, fingerprints the requested frames, and reports physical reads and bytes. The `full` mode also requests transaction hashes on logs. Compare full-frame fingerprints when checking complete-data equivalence; partial projections can legitimately omit optional fields that an older reader populated. Keep feature sets equal when comparing CPU measurements, particularly the signature-recovery backend. Node delivery measurements and source-only measurements are different workloads and should be reported separately. ## Controlled suites Performance evidence is never collected or uploaded by GitHub Actions. CI tests benchmark logic as ordinary Rust code, but it does not execute benchmark commands. Run the checked suites explicitly on a quiet, controlled host; the runner builds a release binary, records host/tool/commit context, executes cases sequentially to avoid cross-case contention, and writes only beneath an ignored local directory: ```bash # Fast resumability and report-contract check. scripts/run-controlled-benchmarks.sh smoke # The complete synthetic matrix formerly represented by CI configuration. scripts/run-controlled-benchmarks.sh synthetic-million # Select a smaller controlled comparison or an external evidence mount. LEANI_BENCHMARK_CASES="materialized end-to-end-fast-consumer" \ LEANI_BENCHMARK_CORPORA="uniswap-like blobs-like" \ scripts/run-controlled-benchmarks.sh synthetic-million /mnt/leani-evidence/run-01 ``` The default destination is a timestamped directory under `.private/evidence/performance`. Passing the same explicit output directory resumes completed cases and refuses ambiguous partial output. The `all` and `delivery-faults` suites include SDK/PostgreSQL cases and therefore require a dedicated disposable database: ```bash # Compose resolves the node's LEANI_API_TOKEN for every service; the # database ignores it. LEANI_API_TOKEN=unused docker compose --profile benchmark up -d benchmark-postgres export LEANI_BENCHMARK_POSTGRES_URL=postgres://leani_benchmark:leani_benchmark@127.0.0.1:55432/leani_benchmark scripts/run-controlled-benchmarks.sh delivery-faults ``` ## Candidate sweeps Run or resume one selected-stage candidate sweep with: ```bash cargo run -- benchmark sweep \ --manifest config/benchmark-sweep.example.json \ --output-directory benchmark-sweep-results ``` The manifest holds common benchmark arguments and explicit candidate arguments. Each successful candidate produces content-addressed immutable summary/sample files; rerunning skips complete candidates and rebuilds `sweep-summary.json` with correctness/resource-gate decisions and the throughput/RSS/storage Pareto frontier. Concurrent-live sweeps can also set `maximumLiveP95CommitLatencyUs`; live p95 then becomes a required gate and a Pareto dimension, preventing a history-throughput win from hiding live writer starvation. Keep a tuned argument out of `baseArguments` and set it once per candidate. ## SDK and PostgreSQL destinations Delivery defaults to the real HTTP path with a Rust discard sink. Use `--destination rust-direct` for the durable-store protocol ceiling. The full SDK/PostgreSQL fixture uses a dedicated disposable database because it truncates its benchmark tables: ```bash LEANI_API_TOKEN=unused docker compose --profile benchmark up -d benchmark-postgres LEANI_BENCHMARK_POSTGRES_URL=postgres://leani_benchmark:leani_benchmark@127.0.0.1:55432/leani_benchmark \ cargo run -- benchmark --mode end-to-end --profile externalized --destination sdk-postgres \ --corpus blobs-like --blocks 10000 --warmups 1 --runs 3 ``` That default writes the stable one-event-per-row protocol fixture. Add `--postgres-schema blobs-application` with `--corpus blobs-like` to write and measure application-shaped block and blob-transaction tables instead; the report records the selected schema, its independent row semantics, table/index bytes, WAL bytes, and canonical output digest. # Manual verification > Rehearse catch-up, live following, subscriptions, JSON-RPC behavior, restart, and recovery against explicit acceptance checks. This guide exercises Leani the way an application such as blobs.money uses its native integration: 1. start from a fresh database 10,000 blocks behind the finalized head; 2. backfill application-specific output while following the head; 3. consume the durable `blobs-money` transformation stream; 4. query the materialized native API and Ethereum JSON-RPC facade; and 5. interrupt, restart, and resume from a saved change cursor. For repeatable automated gates and their evidence requirements, see [Automated verification](https://leani.dev/docs/contributing/automated-verification/). ## Test modes | Mode | History | Follows head | Serves APIs | Primary purpose | | -------------------------- | --------------------------------------- | ----------------------- | ---------------------- | --------------------------------------------------------- | | Deterministic runtime test | Synthetic | Synthetic | In-process API | Fast scheduling, crash, handoff, and storage verification | | Historical-only `backfill` | Public datasets | No | After starting `serve` | Isolate dataset ingestion and transformations | | Continuous `serve` | Public datasets plus bounded P2P bridge | Yes | Yes | Application integration rehearsal | | `e2e mainnet` | Production adapters | Yes, for a bounded gate | No | Produce explicit pass/fail Mainnet evidence | Use a separate `data_dir` for each mode. Do not point a manual test at a production or otherwise valuable database. ## Build the binary ```bash cd leani cargo build --locked --release mkdir -p data/manual-mainnet evidence ``` The commands below use `./target/release/leani` so the release binary is built only once. ## Prepare a recent finality anchor Copy the operational template: ```bash cp config/example.toml config/manual-mainnet.toml ``` Edit at least these fields. A relative `data_dir` is relative to the configuration file, so from `config/` this names the `data/manual-mainnet` directory created above: ```toml data_dir = "../data/manual-mainnet" [finality] kind = "consensus_p2p" checkpoint = "" checkpoint_slot = 0 # Replace 0 with ITS_FINALIZED_BEACON_SLOT. minimum_peers = 2 discovery_port = 0 ``` Obtain the finalized root and slot independently. For example, query two independently operated Beacon APIs and require the responses to agree: ```bash curl -s "$BEACON_API/eth/v1/beacon/headers/finalized" \ | jq '{root: .data.root, slot: .data.header.message.slot}' ``` Do not commit this checkpoint as an evergreen value. Record its sources and acquisition time with the test evidence. The checkpoint is the bootstrap trust anchor; `consensus_p2p` verifies subsequent light-client updates locally over P2P. Validate the configuration without opening the database: ```bash ./target/release/leani \ --config config/manual-mainnet.toml \ doctor ``` Then exercise the configured finality transport: ```bash ./target/release/leani \ --config config/manual-mainnet.toml \ source probe finality \ --report evidence/manual-finality.json ``` Derive an inclusive 10,000-block range from the verified finalized execution anchor: ```bash jq -e '.accepted == true' evidence/manual-finality.json TO=$(jq -r '.selected.execution_block_number' evidence/manual-finality.json) FROM=$((TO - 9999)) echo "Testing blocks $FROM through $TO" ``` In the copied `blobs-money` entry, set the processor start to the numeric value printed as `FROM`, and keep the entry’s other fields and policy tables: ```toml [[processors]] id = "blobs-money" instance = "blobs-production" version = "1.6.0" start_block = 0 # Replace 0 with the numeric FROM value printed above. ``` `serve` uses this configured value. The finite `e2e mainnet` command overrides it with its `--from-block` argument. ## Recommended rehearsal: catch up and follow head Start the node in terminal 1: ```bash ./target/release/leani \ --config config/manual-mainnet.toml \ --log-filter 'info,leani_source_p2p=debug' \ serve ``` This launches the following work concurrently: * verify the consensus checkpoint and seed the finalized execution anchor; * start the execution-P2P head follower; * backfill every configured processor through the finalized anchor; * use the bounded P2P history bridge for the archive-to-head gap; * compare the hot/cold overlap and persist a verified handoff; and * expose the native API, HTTP JSON-RPC, and WebSocket JSON-RPC listeners. The example configuration permits a 1,048,576-block P2P history fallback, so the complete 10,000-block range is bridgeable even when the public archives lag. The normal path still prefers capable retained/archive sources in configured priority order; execution P2P remains the probabilistic last resort. In terminal 2, inspect process and processor readiness: ```bash curl -s http://127.0.0.1:8080/health/ready | jq ``` ```bash curl -s http://127.0.0.1:8080/v1/network/status | jq ``` Open `http://127.0.0.1:8080/debug/network` for the same information as a live-updating dashboard. It distinguishes the long-lived `live` session from short-lived `history` sessions used to bridge dataset gaps. A zero-peer history row does not mean the head follower is disconnected. ```bash curl -s \ http://127.0.0.1:8080/v1/processors/blobs-money/status \ | jq ``` `/health/ready` reports the required network actors. It does not by itself prove that the requested historical processor range is complete. The processor status should eventually show: * `configuredStartBlock` equal to `FROM`; * one continuous `available` interval; * `complete: true`; * no missing range between `FROM` and `processedThrough`; and * a `processedThrough` cursor that continues advancing with the head. `complete: true` by itself is only the no-internal-gaps assertion. During catch-up, `/v1/network/status` can therefore report `storedRangeContiguous: true` and a non-zero `blocksRemaining` while `readiness.liveReady` remains false. The dashboard renders this as “contiguous so far” plus the current target instead of calling the node complete. If the required network lane exits, inspect `network.supervisor.lastError` and `retryInSeconds`; the dashboard displays the same failure and retry countdown above the processor coverage. The API and RPC listeners are: ```text native HTTP/SSE: http://127.0.0.1:8080 Ethereum JSON-RPC: http://127.0.0.1:8545 Ethereum WebSocket: ws://127.0.0.1:8546 ``` ## Consume blobs.money transformations Open the stream while catch-up is still running: ```bash curl -N --no-buffer \ -H 'Accept: text/event-stream' \ http://127.0.0.1:8080/v1/processors/blobs-money/stream ``` Without `after`, the native stream begins with the first retained durable change, replays the backfill, and then remains attached to live changes. A typical envelope contains: ```json { "sequence": "12345", "cursor": "", "operation": "apply", "finality": "unattested", "kind": "blobs.block.put", "block": { "number": 123, "hash": "0x..." }, "data": {} } ``` The blobs processor emits one `blobs.block.put` bundle per block. Its data contains the block and all blob transactions for an atomic destination update. The envelope operation communicates `apply`, `undo`, `finalized`, or `reset_required`; consumers must not infer reorg semantics only from the domain-change suffix. To resume from the cursor of the last atomically committed event: ```bash CURSOR='' curl -N --no-buffer \ -H 'Accept: text/event-stream' \ --get \ --data-urlencode "after=$CURSOR" \ http://127.0.0.1:8080/v1/processors/blobs-money/stream ``` The stream is at-least-once. A production consumer must apply the domain change and save its opaque cursor in the same database transaction. The current API and SDK do not accept a `finality` subscription query option. Inspect or filter `event.finality` in the consumer. Do not append `?finality=attested`; unknown change-stream query fields are rejected. ## Exercise the TypeScript SDK The SDK can be imported directly from this workspace for a manual Bun consumer: ```ts import { createLeaniClient, } from "./packages/sdk/src/index.ts"; const node = createLeaniClient({ baseUrl: "http://127.0.0.1:8080", timeoutMs: 15_000, }); let savedCursor: string | undefined; for await (const event of node.blobs.subscribe({ after: savedCursor, })) { console.log({ sequence: event.sequence, operation: event.operation, finality: event.finality, kind: event.kind, block: event.block?.number, }); // The real application boundary should be: // // await database.transaction(async (transaction) => { // await applyBlobChange(transaction, event); // await saveCursor(transaction, event.cursor); // }); savedCursor = event.cursor; } ``` Run a saved script from the repository root with: ```bash bun run ./manual-consumer.ts ``` The generic equivalent is `node.processors.subscribe("blobs-money", { after })`. ## Query transformed output Fetch a page of materialized blob blocks: ```bash curl -s \ "http://127.0.0.1:8080/v1/q/blobs/blocks?fromBlock=${FROM}&toBlock=${TO}&limit=3" \ | jq ``` Inspect the first durable transformation records: ```bash curl -s \ "http://127.0.0.1:8080/v1/processors/blobs-money/changes?limit=5" \ | jq '.data[] | { sequence, operation, finality, kind, block: .block.number, data }' ``` Range queries reject incomplete coverage by default. This prevents an application from silently treating an archive or catch-up gap as an empty result. ## Exercise Ethereum JSON-RPC Query the current block: ```bash curl -s http://127.0.0.1:8545 \ -H 'content-type: application/json' \ --data '{ "jsonrpc":"2.0", "id":1, "method":"eth_blockNumber", "params":[] }' \ | jq ``` Fetch the current full block: ```bash curl -s http://127.0.0.1:8545 \ -H 'content-type: application/json' \ --data '{ "jsonrpc":"2.0", "id":1, "method":"eth_getBlockByNumber", "params":["latest",true] }' \ | jq ``` WebSocket JSON-RPC supports raw `newHeads` and filtered `logs`. Application aggregations such as blobs output use the native durable SSE/SDK stream rather than `eth_subscribe`. ## Historical-only mode This mode isolates free public datasets from P2P and head following. Copy `deploy/container.toml` to `config/manual-history.toml`, then configure local bind addresses, a fresh data directory, relative to the configuration file, and the processor start; keep the other settings and the processor’s policy tables as copied: ```toml data_dir = "../data/manual-history-10k" [sources.live] kind = "disabled" minimum_peers = 0 handoff_overlap_blocks = 32 [finality] kind = "disabled" checkpoint = "" endpoints = [] minimum_agreement = 1 [[processors]] id = "blobs-money" instance = "blobs-container-1-6" version = "1.6.0" history_mode = "on_demand" start_block = 0 # Replace 0 with the numeric FROM value printed above. [rpc] http_bind = "127.0.0.1:8545" ws_bind = "127.0.0.1:8546" historical_mode = "on_demand" minimum_recent_blocks = 128 [api] bind = "127.0.0.1:8080" ``` Backfill the bounded range and verify the database: ```bash ./target/release/leani \ --config config/manual-history.toml \ backfill \ --processor blobs-money \ --from "$FROM" \ --to "$TO" ./target/release/leani \ --config config/manual-history.toml \ db verify ./target/release/leani \ --config config/manual-history.toml \ db inspect ``` Start `serve` against the same database to query the result and replay its durable native change stream: ```bash ./target/release/leani \ --config config/manual-history.toml \ serve ``` The native processor stream can replay committed historical transformations. JSON-RPC WebSocket head subscriptions remain quiet because this profile has no live source. A historical-only backfill may fail near the current head when no configured public dataset has published the complete requested range. That result isolates archive availability; the continuous profile is responsible for bridging the bounded archive-to-head gap over P2P. ## Controlled synthetic benchmarks Synthetic performance evidence is deliberately excluded from GitHub Actions. Run the checked suites on a quiet controlled host; cases execute sequentially and record the exact commit, dirty state, host, binary, Rust, Cargo, and Bun versions beside their reports: ```bash scripts/run-controlled-benchmarks.sh smoke scripts/run-controlled-benchmarks.sh synthetic-million ``` Outputs default to an ignored timestamped directory under `.private/evidence/performance`. Set `LEANI_BENCHMARK_CASES` and `LEANI_BENCHMARK_CORPORA` to select a comparison, or pass an explicit second argument when evidence belongs on a dedicated local volume. The runner keeps complete results when resuming and stops on partial output instead of silently overwriting an interrupted measurement. The `delivery-faults` and `all` suites require `LEANI_BENCHMARK_POSTGRES_URL` to reference a dedicated disposable database. When running an installed binary outside the repository root, set `LEANI_SOURCE` to the Leani checkout containing the SDK fixture. Start the checked local service with: ```bash # Compose resolves the node's LEANI_API_TOKEN for every service; the # database ignores it. LEANI_API_TOKEN=unused docker compose --profile benchmark up -d benchmark-postgres export LEANI_BENCHMARK_POSTGRES_URL=postgres://leani_benchmark:leani_benchmark@127.0.0.1:55432/leani_benchmark scripts/run-controlled-benchmarks.sh delivery-faults ``` ## Real-source delivery benchmark `benchmark real-source` measures the production fixed-range path end to end: real archive/dataset acquisition, processor mapping and durable publication, gzip HTTP delivery, canonical event decoding/hashing, acknowledgement, and pruning. Its built-in consumer intentionally does no application-database work, so the result answers “how fast can the node index and deliver this processor output to an immediately acknowledging consumer?” Run benchmarks with the release binary, a fresh data directory, and an immutable report path. Like a node, the benchmark holds its data directory, execution P2P peer store and identity included, so it refuses a directory that a running node or another benchmark holds: ```bash ./target/release/leani \ --config config/benchmarks/real-source.toml \ benchmark real-source \ --processor blobs-money \ --source-policy xatu-only \ --from-block 19426589 \ --to-block 20426588 \ --data-dir ./data/benchmark-xatu-blobs-1m \ --timeout-seconds 7200 \ --report evidence/benchmark-xatu-blobs-1m.json ``` The inclusive example contains exactly one million post-Dencun blocks. The checked-in benchmark config also defines the single Uniswap V3 USDC/WETH 0.05% pool processor and enables EraE. Select `erae-only`, use `p2p-only` to measure the finalized execution-P2P fallback, or use `history-portfolio` to exercise normal priority and failover. By default a P2P range may extend back to the processor start block. If `sources.live.history_fallback_blocks` is configured, the range must fit that explicit cost bound. P2P results are highly sensitive to public-peer availability and block payload size. Record cold and persisted-peer-store runs separately, preserve `execution-p2p-secret` plus `execution-network.sqlite` for the latter, and never compare a recent receipt-complete run directly with a header-only or state-snapshot sync headline. `material_request_blocks` is intentionally tunable: the evidence-backed default is eight, and larger values must earn promotion through repeated real-network runs. For an immediate versus slow-consumer comparison, repeat the exact same range and build with a fresh directory/report and add, for example, `--consumer-delay-ms 100`. To make cross-source equivalence machine-checked, pass the first successful report’s `outputDigest` as `--expected-output-digest` on the second run. The report separates: * source operation time, attempts/failures, assigned chunk ranges, frames, and normalized in-memory bytes; * exact EraE successful response bytes/read attempts; * Xatu source-object size, projected compressed column bytes, logical Parquet byte-range requests, and bytes returned to the reader (coalescing, caching, retries, HTTP headers, and failed partial responses remain below that adapter and are not mislabeled as physical HTTP requests); * P2P aggregate and anchored-proof header requests, body/receipt requests, sparse bloom avoidance, retries, elapsed request time, and RLP response payload bytes before request-id framing and Snappy compression; * sampled time to first source frame, first processor commit, and source completion, plus exact first HTTP batch, first acknowledgement, producer completion, and consumer completion timings; * canonical payload bytes, HTTP encoded/transmitted/observed bytes, batches, acknowledgements, retained-delivery high-water, pruning, SQLite/WAL size, and peak RSS; and * processor output digest, coverage, pending-output state, and optional digest agreement with another source run. The wall-clock result includes real public-source network latency and local processing in parallel. It is intentionally separate from deterministic synthetic benchmarks, which establish stable processor and delivery ceilings. ## Finite Mainnet evidence gate The Mainnet E2E command runs production ingestion and verification actors but does not open the API listeners: ```bash ./target/release/leani \ --config config/manual-mainnet.toml \ e2e mainnet \ --processor blobs-money \ --from-block "$FROM" \ --data-dir ./data/mainnet-e2e \ --minimum-follow-blocks 16 \ --stable-seconds 300 \ --max-head-age-seconds 60 \ --timeout-seconds 7200 \ --report evidence/mainnet-e2e.json ``` After an intentional interruption, reuse that isolated state explicitly: ```bash ./target/release/leani \ --config config/manual-mainnet.toml \ e2e mainnet \ --processor blobs-money \ --from-block "$FROM" \ --data-dir ./data/mainnet-e2e \ --resume \ --minimum-follow-blocks 16 \ --stable-seconds 300 \ --max-head-age-seconds 60 \ --timeout-seconds 7200 \ --report evidence/mainnet-e2e.json ``` For a quick diagnostic, two follow blocks and a 30-second stable window shorten the run. They do not satisfy the stricter production evidence profile. ## Restart and cursor-resume exercise For the most representative manual failure test: 1. start `serve` against a fresh 10,000-block configuration; 2. attach the blobs stream and save an event cursor; 3. stop the node with Ctrl-C during catch-up or live following; 4. restart `serve` with the same configuration and `data_dir`; 5. resume the stream with `after=`; 6. confirm sequences remain monotonic and coverage remains continuous; 7. stop the node and run `db verify`; and 8. inspect storage attribution with `db inspect`. The SDK handles URL encoding, reconnect backoff, duplicate cursor suppression, and SSE framing. The application remains responsible for atomically applying events and saving its cursor. ## Deterministic 10,000-block gate Before spending time on public network behavior, run: ```bash cargo test -p leani-runtime \ ten_thousand_block_hot_cold_restart_converges_and_serves_coverage \ --locked -- --nocapture ``` This uses the real runtime, SQLite store, reducers, handoff logic, restart path, and native API with deterministic source adapters. It proves the internal 10,000-block invariants without depending on public peers or datasets. ## Diagnosing a stalled public run Execution-peer availability is part of the test, not an implementation detail. If no blocks arrive, check the logs for the connected execution-peer count before investigating processors or the API. The production example requires three execution peers. Lowering `sources.live.minimum_peers` to two can be useful for a diagnostic run, but record that change and do not treat it as evidence for a three-peer profile. The finality P2P source has its own independent `finality.minimum_peers`. Permit outbound TCP and UDP. For the configured fixed execution P2P port, publish both TCP and UDP through Docker and the host firewall. Inbound reachability improves discovery and peer diversity but is not a trust requirement. Restricted NAT, firewalls, or peers that do not serve receipts can delay the history bridge even when headers are available. Use `/debug/network` to compare: * connected peers and the manager age; * successful, timed-out, and failed material requests; * average local request-gate wait; and * the current requested range and phase. High queue wait means local work is competing for the available peer slots. Low queue wait plus repeated timeouts points to remote serving quality. The manager and its node identity should remain stable across these failures. ## Manual acceptance checklist A successful application rehearsal demonstrates: * verified finality bootstrap and a live execution-P2P lane; * continuous processor coverage from `FROM` through the current cursor; * a durable verified hot/cold handoff; * transformed historical changes followed by new live changes on one stream; * successful native blob queries for the tested range; * supported recent JSON-RPC methods returning canonical material; * restart without losing committed coverage; * cursor resume without a gap; * no pending ordered deltas after handoff; * bounded recent raw storage; and * a successful SQLite integrity check. ## Public quickstart proof To compare Xatu transport strategies without changing processor state, run: ```bash cargo test -p leani-source-xatu compare_public_xatu_range_strategies --locked -- \ --ignored --nocapture ``` This manual diagnostic compares the production coalescing reader with a 256 KiB split strategy on identical Parquet columns, reverses their order for the second pair, and checks that completed reads return identical bytes. Each attempt has a 20-second deadline and prints its outcome; inspect those outcomes even if the test completes successfully. Repeated reads do not establish a cold CDN baseline. Record the host, time, and all attempts when comparing performance. After building the current binary, run `scripts/verify-mainnet-quickstart.sh`. This manual check requests the documented Xatu range `17000000..17000999`, reopens the node, checks finalized coverage and decoded non-zero Sync reserves through the API, releases its query snapshot, and verifies SQLite integrity. It uses an isolated temporary directory and local ports 18180, 18545, and 18546. `LEANI_BIN` selects a built binary; `LEANI_VERIFY_DIR` reuses a prior verification directory to exercise restart/idempotence. The directory contains the binary SHA-256, dated report, responses, and logs. Backfill has a ten-minute deadline; `LEANI_VERIFY_TIMEOUT_SECONDS` overrides it for diagnostics. A timeout or unavailable source is not a passing result. Installation and release artifact checks follow this code/API/example hardening phase. # Processor extension contract > Understand the native processor factory, settings, map/reduce, query extension, generic API, and future Wasm boundary. The node is an embeddable Rust library as well as a standard binary. Downstream projects can register application-specific native processors and compile their own binary without forking or modifying node assembly. This is the first extension model. It is also the host boundary for a future sandboxed Wasm package runtime: ```text ProcessorRegistry +-- native ProcessorFactory +-- future WasmProcessorFactory ``` The source planner, historical/live runtimes, SQLite reducer, undo journal, finality handling, generic API, and durable subscriptions operate on the `Processor` trait and do not branch on how a processor was constructed. ## Current extension contract An external binary supplies a `ProcessorRegistry`: ```rust use leani::{ProcessorRegistry, run_with_registry}; let mut processors = ProcessorRegistry::standard(); processors.register(MyProcessorFactory)?; run_with_registry(processors).await?; ``` `ProcessorRegistry::standard()` includes the checked-in blobs.money, ERC-20, and Uniswap factories. `ProcessorRegistry::new()` starts empty, allowing a product-specific binary with no built-in application processor. Package loaders can register discovered `Arc` values through `register_shared`. A factory constructs an immutable processor from the common instance configuration and its processor-owned settings table: ```rust pub trait ProcessorFactory: Send + Sync { fn id(&self) -> &str; fn description(&self) -> &str { self.id() } fn create( &self, configured: &ProcessorConfig, context: ProcessorFactoryContext, ) -> Result; } ``` `ProcessorComponents` always contains the processor and may contain one processor-owned `QueryExtension`. Returning them together prevents query routes from being registered for a different processor instance. Construction is deterministic and performs no file or network I/O. The registry rejects: * unknown or duplicate processor IDs; * invalid processor-owned settings; * a returned descriptor with another ID or version; * descriptor/configuration disagreement about start block, publication, or retention; and * invalid descriptor requirements or schemas. `doctor`, `serve`, `backfill`, `benchmark`, conformance, database inspection, and the Mainnet E2E command all use the same supplied registry. ## Processor-owned settings Common configuration remains strict: ```toml [[processors]] id = "my-protocol" version = "1.0.0" start_block = 18000000 retention = "full_output_history" [processors.settings] contracts = ["0x..."] bucket_seconds = 3600 ``` The node preserves `[processors.settings]` as a `toml::Table`. The selected factory calls `ProcessorConfig::decode_settings()` using a processor-specific Serde type, normally with `deny_unknown_fields`. The processor hashes the canonical meaning of its immutable settings into its descriptor `config_hash`. Changing output-affecting settings therefore creates a distinct durable processor instance. Code works the same way: `version` and `code_hash` name what `map` and `reduce` compute. Bump both whenever a change alters a delta, a reduced value, or an emitted change, and run the result as a new instance, as in a [processor rebuild](/docs/operations/runbook/#processor-rebuild-or-rollback). The store cannot see code. A binary that maps blocks differently under an unchanged identity fails only when it maps a block again that the old binary already holds, such as a pending live delta at the next start’s hot/cold overlap, possibly hours later: the backfill then fails with `already has a different pending delta`. ## Map, reduce, and changes Every processor implements: ```rust #[async_trait] pub trait Processor: Send + Sync { fn as_any(&self) -> &dyn Any; fn descriptor(&self) -> &ProcessorDescriptor; async fn map(&self, block: &BlockFrame) -> Result; fn finality_variant_checksums( &self, delta: &EncodedDelta, ) -> Result, ProcessorError>; async fn reduce( &self, transaction: &mut dyn ReducerTransaction, cursor: &ProcessorCursor, delta: &EncodedDelta, ) -> Result; fn change_json( &self, change: &DomainChange, ) -> Result, ProcessorError>; } ``` `map` is deterministic and may execute in parallel. It converts requested source material into a compact, versioned delta. `finality_variant_checksums` defaults to the exact delta checksum. A processor whose mapped payload embeds included/finalized state must override it by decoding the validated delta and returning only the checksums produced by changing that finality field. This lets restart recovery prove archive/live equivalence even after the raw frame has been pruned; it must not treat any other payload difference as equivalent. `reduce` applies that delta through the core-owned transaction. Processors can read and mutate namespaced collections/indexes and emit domain changes, but cannot access SQLite directly. The core captures preimages and commits state, coverage, cursor, undo, and changes atomically. The core derives each change’s undo from its key’s mutation, so emitted changes must describe each key’s net mutation in the block: at most one change per key, an upsert of the key’s final value or a delete of a removed key. A key the block created may carry a different change payload, because its undo is a delete. A reducer that writes a key more than once in a block emits one change, for the last write; the store rejects an unfinalized block whose changes do not match its mutations. A mapper must not fail a block because of a log another contract emitted. Any contract can emit any topic zero, so a log that does not decode as the expected event is skipped, and preferably counted in the delta, rather than treated as invalid input. `change_json` is an optional public rendering hook. Durable output remains the processor’s compact bytes. A processor with a stable JSON representation can render those bytes for HTTP/SSE clients; otherwise the API publishes schema-labelled hexadecimal data. Every change envelope includes the descriptor’s `change_schema`. ## Query extensions Processors that need a domain-specific read API can return a native query extension from their factory: ```rust use std::sync::Arc; use axum::{Router, routing::get}; use leani::{ProcessorComponents, QueryContext, QueryExtension}; #[derive(Debug)] struct MyQueryExtension; impl QueryExtension for MyQueryExtension { fn id(&self) -> &str { "my-protocol-v1" } fn alias(&self) -> Option<&str> { Some("my-protocol") } fn routes(&self) -> Router { Router::new().route("/summary", get(summary)) } } let processor = Arc::new(MyProcessor::new(configured)?); let components = ProcessorComponents::new(processor) .with_query_extension(Arc::new(MyQueryExtension)); ``` The canonical mount is always instance-scoped: ```text GET /v1/processors/{instance}/query/summary ``` The optional `GET /v1/q/my-protocol/summary` alias is mounted only when exactly one configured extension requests that alias. With multiple instances, every canonical route remains available and the ambiguous alias is omitted. `GET /v1/capabilities` reports the canonical and active alias paths. `QueryContext` gives the extension bounded, read-only access to its owning processor’s entities and indexes, coverage, committed cursor, page limits, and extension-scoped opaque cursors. It cannot read another processor’s namespace. Extension routes inherit the aggregate API’s bearer authentication and standard error envelope. Entity scans observe latest committed state and are suitable for ordinary queries. A consumer that needs a stable multi-page bootstrap followed by an exact change boundary must use the generic `query-and-follow` operation. ## Generic API All configured processors automatically receive: ```text GET /v1/processors/{instance}/status GET /v1/processors/{instance}/schema GET /v1/processors/{instance}/changes GET /v1/processors/{instance}/stream ``` Typed aliases are conditional. For example, `/v1/q/blobs/*` exists only when a single configured blobs extension can claim that alias. A custom-only binary remains a valid node. JSON-RPC progress is independent of query-extension registration. The SDK consumes custom changes through: ```ts for await (const event of node.processors.subscribe( "my-protocol", { after: savedCursor }, )) { await database.transaction(async (transaction) => { await applyChange(transaction, event); await saveCursor(transaction, event.cursor); }); } ``` The delivery contract remains at-least-once. Applying the change and saving the cursor in one consumer transaction is required. ## Complete example The complete [`examples/custom-node` walkthrough](https://leani.dev/docs/guides/custom-processor/) is a separate downstream binary inside the workspace. It: * defines its own strict settings type; * implements a block-summary processor; * registers a native factory without changing `crates/node`; * configures only that custom processor, with no blobs processor; * exposes typed JSON changes through the generic stream; and * exposes `GET /{number}` through a processor-owned query extension; and * reuses the standard CLI and every node command. Validate it: ```bash cargo run -p leani-custom-example -- \ --config examples/custom-node/node.toml \ doctor --json ``` Run a bounded backfill: ```bash cargo run -p leani-custom-example --release -- \ --config examples/custom-node/node.toml \ backfill \ --processor example-block-summary \ --from-block 19426589 \ --to-block 19426688 ``` Start the query service: ```bash cargo run -p leani-custom-example --release -- \ --config examples/custom-node/node.toml \ serve ``` Consume its changes: ```bash curl -N --no-buffer \ http://127.0.0.1:9080/v1/processors/example-block-summary/stream ``` The processor instance is discoverable through `GET /v1/processors`. Query a stored summary through its canonical route: ```bash curl http://127.0.0.1:9080/v1/processors//query/19426589 ``` Because the example has one instance, its shorter `/v1/q/block-summaries/19426589` alias is also available. Running the standard binary against this configuration fails `doctor` with an explicit unregistered-processor diagnostic. This proves that processor availability belongs to the assembled binary, not a global hard-coded list. ## Security model Native processors are trusted code linked into the operator’s binary. They provide maximum performance and the simplest debugging experience, but they are not a multi-tenant sandbox. Do not load native `.so`, `.dylib`, or `.dll` processors dynamically. Rust has no stable native ABI for this contract, such libraries are platform-sensitive and unsandboxed, and loading them would undermine the workspace’s `unsafe_code = "forbid"` policy. The supported choices are: 1. statically linked native Rust processors now; and 2. versioned, metered, sandboxed Wasm packages later. ## Wasm product path A hosted product can implement `WasmProcessorFactory` over this registry. A package will need: * a signed/versioned manifest and content-addressed artifact; * declared capabilities, filters, start point, schemas, publication, and retention; * a stable component ABI for map/reduce/state/change operations; * deterministic execution without ambient network, filesystem, clock, or randomness; * memory, instruction/fuel, mutation, delta, and emitted-byte limits; * schema-aware JSON/Protobuf rendering; * package/version migrations and reproducible rebuilds; and * per-tenant resource accounting and observability. The first Wasm release should support one map/reduce processor per package. Module DAG composition and restricted event-only Substreams package import can follow after the package ABI and storage behavior are proven. General Substreams compatibility is not implied. Packages requiring calls, traces, state diffs, arbitrary host functions, or retained Firehose history must fail capability negotiation when the configured node sources cannot supply those inputs. # Release process > How Leani builds, protects, and publishes its binaries, crates, SDK, container, and release-pinned documentation from one reviewed commit. Every Leani release publishes native binaries, four Rust authoring libraries, the TypeScript SDK, a multiarchitecture container, a Homebrew formula, and release-pinned documentation from one reviewed commit. This page explains how releases work and what contributors must keep intact. Users check a release with [Verify a release](/docs/operations/verify-releases/). Release maintainers follow the [maintainer runbook](https://github.com/smart-byte/leani/blob/main/docs/maintainers/releasing.md) for repository settings and publication steps. ## Release flow The manually dispatched **Coordinate release** workflow (`release-coordinate.yml`) runs from `main` in two steps: 1. **prepare** opens a pull request that updates workspace and SDK versions, exact internal crate requirements, lockfile versions, license notices, current install pins, and the changelog heading. It is reviewed and merged like any other change. 2. **release** runs after one maintainer approval. It reserves annotated `v` and `sdk-v` tags on the merged commit, which are never moved, builds and tests every artifact once, and records the runs and checksums in a release plan. It then dispatches the publishers in their protected environments, checks each publication against the plan, promotes the documentation site, and waits until leani.dev serves the release. Once a stable release exists, a prerelease leaves the site on it. Publication is resumable rather than atomic: a retry reuses matching publications, and different bytes stop it. ## Release-facing changes Update `CHANGELOG.md`, OpenAPI/SDK compatibility tables, durable schema versions, and processor version declarations with the change that needs them. When the SDK version in `packages/sdk/package.json` changes, pin every documented SDK install, such as the `bun add` commands in `README.md` and the install guide, to that exact version in the same commit. CI’s `scripts/check-sdk-pins.rb` fails on a documented install that names another version, a range, a tag, or none. ## Verify publication history Install the local commit and push guards as described in `CONTRIBUTING.md`. Run these checks on the exact commit being released: ```bash scripts/check-publication.rb HEAD scripts/check-secrets.sh HEAD ``` The first check examines every reachable commit, including files deleted in later commits. The second uses a pinned credential scanner with redacted output. Private plans, agent instructions, design explorations, and raw reports must stay in ignored local storage. Do not push recovery or archive refs. ## Protected publication Every publishing workflow separates building from publishing. Its build and test jobs run with a read-only token, without secrets or an OIDC token, and upload the artifact they tested. One publish job per workflow runs in a protected GitHub environment, downloads that artifact, and publishes it without building or installing project code: | Workflow | Environment | Publish job permissions | Dispatch from | | --------------------------------------- | ----------------- | ---------------------------------------------------------------------------- | ------------- | | `release.yml` | `release-github` | `contents: write`, `actions: read`, `id-token: write`, `attestations: write` | `v*` tag | | `container.yml` | `release-ghcr` | `packages: write`, `actions: read`, `id-token: write`, `attestations: write` | `v*` tag | | `crates-release.yml` | `release-crates` | `contents: read`, `actions: read`, `id-token: write` | `v*` tag | | `sdk-release.yml` | `release-npm` | `actions: read`, `id-token: write` | `sdk-v*` tag | | `site-promote.yml` | `site-production` | `contents: write` | `v*` tag | | `release-coordinate.yml` preparation PR | `release-control` | `contents: read` | `main` | | `release-coordinate.yml` coordination | `release-control` | `contents: read`, `actions: read` | `main` | No other job may request a write permission, `id-token: write` included, read a secret, or keep its checkout credentials. The crates.io publish job checks out the repository because Cargo cannot upload a prepared archive. The coordinator checks out reviewed control scripts, without building or installing project dependencies, and writes only with its `RELEASE_CONTROL_TOKEN` secret. npm publishes a local archive file named right after `publish`. No workflow may run on `pull_request_target` or `workflow_run`: both run with the base repository’s token, secrets, and cache on behalf of events a pull request controls. `ruby scripts/check-release-ci.rb --workflows` enforces this, and CI runs it through `scripts/test-release-ci.rb`. Change its policy table only together with this page. ## Provenance and dependency updates Release workflows attest the build provenance of every archive and container image. CycloneDX SBOMs ship with the GitHub release for archives, whose publication stops if the candidate lacks the node’s SBOM, `leani.cdx.json`. Container SBOMs are bound to each image digest instead, so they verify against the exact image and the container job needs no `contents: write`. [Verify a release](/docs/operations/verify-releases/) shows how to check them. The Dockerfile pins its base images to multiarchitecture digests. Dependabot proposes weekly updates for GitHub Actions, Cargo, Bun (`packages/sdk` and `site`), and those image digests. CI requires the Dockerfile’s Rust image to match `rust-toolchain.toml`, so merge a Rust image update only together with the same toolchain update. The weekly `Dependency audit` workflow runs the cargo-deny advisory check and `bun audit` against both Bun lockfiles. Releases do not wait for it, so review its failures when they arrive. Linux archives are built on Ubuntu 24.04, and their build fails if a binary needs a newer glibc than the documented floor, 2.39. Raising the floor changes the [supported distributions](/docs/getting-started/install/#published-releases). ## Rust libraries The crates.io release surface is `leani-primitives`, `leani-processor-api`, `leani-source-api`, and `leani-testkit`. Other workspace crates have publication disabled. Keep these libraries on the workspace release version; their internal prerelease dependencies use exact version requirements. Use the repository’s pinned Cargo version to rehearse all four packages together: ```bash cargo publish --dry-run --locked \ -p leani-primitives \ -p leani-processor-api \ -p leani-source-api \ -p leani-testkit ``` Cargo verifies the packaged sources against a temporary registry containing the selected local packages. This allows verification before the first upload. Inspect each archive and its normalized manifest, license, README, lockfile, and source contents. Run the publication/privacy checks and credential scanner on the exact release commit and scan the extracted archives too. Validate a standalone consumer against the packages, then against crates.io after upload. The full `leani` node is distributed through native archives, Homebrew, and containers. Its Git-only networking and consensus dependencies prevent publication on crates.io. Publishing the authoring libraries does not enable dynamic processor loading; custom native processors are compiled into a node. ## Rollback and migrations Rollback means deploying the previous binary with a store format it supports or restoring the pre-migration backup. Every irreversible migration must be preceded by a tested export/rebuild path. Store migrations are forward only, and a start the store refuses for a processor identity leaves an older store at its schema.