Backfill while following live

Leani does not make a large historical scan block head following. The live lane starts independently, commits canonical head work, and can recover a short live gap. Historical jobs fill missing committed intervals for each processor.

1. Run doctor and inspect which historical sources satisfy the processor’s declared material shape.
2. Start serve with live execution and finality configured. Live readiness is independent of older processor coverage.
3. For history_control = "node_owned", let the node schedule missing ranges. For application_subscriptions, create bounded backfill subscriptions from the application and consume their independent history lanes. Request the ranges the live stream announces in system.live_gap.put changes too, those the live lane does not fill itself; see Keep your application in sync.
4. Observe exact coverage intervals, not only processedThrough.
5. Accept the handoff only when parent hash, canonical ancestry, and processor coverage meet at the boundary.

The default gap policy

If sources.live.history_fallback_blocks is omitted, P2P fallback is eligible to fill every history gap at or after the processor start block. Retained or archive sources are selected first when capable. Configure a numeric limit only when you intentionally want to restrict P2P recovery to a finalized suffix.

A failed job is terminal until acted upon

Terminal failed backfill jobs are not retried forever. Inspect the recorded error and correct its cause. Then retry a failed subscription with POST /admin/v1/backfill-subscriptions/{id}/retry, which resumes from its committed progress, or delete the job and create it again. Completion is published only after all overlapping ownership and coverage invariants are satisfied.

For application-owned work, each history lane has its own lease and acknowledgement. A live acknowledgement never advances a history lane, and one history lane reconnects without resetting its siblings.

Run a history subscription in one call

runBackfill() from @smart-byte/leani-sdk/backfill takes one application-owned subscription through its lifecycle: it creates the subscription, applies and acknowledges each history batch, acknowledges the completion, and deletes the subscription. It resolves once the subscription has ended and been deleted, so run it beside the live lane rather than before it, and catch its rejection: an abort or a failure keeps the subscription, so the same request can run again once the cause is fixed.

import {
  createBackfillSubscriptionClient,
  runBackfill,
} from "@smart-byte/leani-sdk/backfill";

const leani = createBackfillSubscriptionClient({ baseUrl: "http://127.0.0.1:18080" });
const result = await runBackfill(
  leani,
  {
    processor: "blobs-production",
    fromBlock: 19_426_589,
    toBlock: 19_436_588,
    idempotencyKey: "blobs-api:19426589-19436588",
    consumer: { id: "blobs-api", role: "required", leaseTtlSeconds: 300 },
  },
  {
    // Your destination's write. It must be idempotent: a batch applied but
    // not acknowledged is delivered again.
    onBatch: (batch) => saveChanges(batch.events),
  },
);
console.log(`history ${result.outcome}`);

The idempotency key makes a rerun after a crash resume the same subscription after its last acknowledged batch. The request must be identical, consumer lease and credential included: the node refuses a different request under the same key. After a lost connection, it backs off and opens the stream again, unless the subscription ended meanwhile. A failed subscription is retried from its committed progress up to maxRetries times, then its error is thrown, leaving it failed; pass onFailure: "discard" to delete it instead, or "throw" to keep it failed and handle the error yourself.