Skip to content

Storage products and lifecycles

“Retention” is not one switch. Each product answers a different recovery or query need and has its own budget.

Product What it is for Typical policy
State Mutable reducer state needed for future blocks durable or checkpointed
Output Entities served by generic or processor-owned queries latest, window, full, or none
Artifacts Compact immutable results reusable for replay/export bounded retention or none
Delivery Ordered change history for external consumers best effort, until acknowledged, or none
Checkpoints Recovery points for processor state automatic, at most once per 1,000 finalized blocks and when a historical job completes, with a finite keep count
Undo Unfinalized branch replacement every unfinalized block, plus a safety depth past finality
Raw history Reusable normalized source material explicitly enabled, segmented, and bounded

The policies are orthogonal. An externalized processor can retain durable state and an acknowledgement-fenced change log while keeping no query output. A latest-value processor can retain one materialized entity with no delivery log.

What each output policy keeps

Four common output policies, from least to most retained:

PolicyRetained by leaniBoundBest fitExample
externalizedState, coverage, and delivery only until the required consumer ACKsByte/age budgets with pause-on-limit backpressurePostgreSQL, ClickHouse, object storage, or another application-owned sink1M real blocks: 9.5 MB blobs / 11.4 MB Uniswap peak node files
latest / aggregateCurrent entities or constant-sized ordered stateOutput cardinality plus checkpoints and configured undoBalances, pool state, counters, totals, dashboardsTransaction totals remain one current aggregate
windowOnly the selected block, age, row, or byte windowExplicit max_blocks / max_age / max_rows / max_bytesRecent APIs, rolling analytics, replay windows30-day event profile ships as a checked example
full outputEvery materialized processor entity and required indexesGrows with output cardinality and indexed historyReusable local historical query surfaces1M dev-native blobs blocks: 1.58 GiB compacted DB
  1. Measured figures are workload-specific evidence, not host-size or cost guarantees. The real-source runs used gzip HTTP delivery, immediate acknowledgement, four source workers, and did not include an application database.
  2. The 1.58 GiB full-output figure retained 2.55 million entities and 550,000 index rows. Externalized, latest, or windowed policy is the intended choice when that reusable query surface is unnecessary.
  3. Trusted Xatu input and commitment-checked EraE input have different trust and performance profiles; retention policy does not change source trust.

Hard ceilings and backpressure

Process memory, temporary material, mapped deltas, SQLite size, artifacts, and delivery history have operator ceilings. A limit action is part of the contract: pause work, expire a best-effort consumer with reset required, or reject growth. Leani must not silently discard required delivery history to remain “healthy.”