Storage products and lifecycles
“Retention” is not one switch. Each product answers a different recovery or query need and has its own budget.
| Product | What it is for | Typical policy |
|---|---|---|
| State | Mutable reducer state needed for future blocks | durable or checkpointed |
| Output | Entities served by generic or processor-owned queries | latest, window, full, or none |
| Artifacts | Compact immutable results reusable for replay/export | bounded retention or none |
| Delivery | Ordered change history for external consumers | best effort, until acknowledged, or none |
| Checkpoints | Recovery points for processor state | automatic, at most once per 1,000 finalized blocks and when a historical job completes, with a finite keep count |
| Undo | Unfinalized branch replacement | every unfinalized block, plus a safety depth past finality |
| Raw history | Reusable normalized source material | explicitly enabled, segmented, and bounded |
The policies are orthogonal. An externalized processor can retain durable state and an acknowledgement-fenced change log while keeping no query output. A latest-value processor can retain one materialized entity with no delivery log.
What each output policy keeps
Four common output policies, from least to most retained:
| Policy | Retained by leani | Bound | Best fit | Example |
|---|---|---|---|---|
| externalized | State, coverage, and delivery only until the required consumer ACKs | Byte/age budgets with pause-on-limit backpressure | PostgreSQL, ClickHouse, object storage, or another application-owned sink | 1M real blocks: 9.5 MB blobs / 11.4 MB Uniswap peak node files |
| latest / aggregate | Current entities or constant-sized ordered state | Output cardinality plus checkpoints and configured undo | Balances, pool state, counters, totals, dashboards | Transaction totals remain one current aggregate |
| window | Only the selected block, age, row, or byte window | Explicit max_blocks / max_age / max_rows / max_bytes | Recent APIs, rolling analytics, replay windows | 30-day event profile ships as a checked example |
| full output | Every materialized processor entity and required indexes | Grows with output cardinality and indexed history | Reusable local historical query surfaces | 1M dev-native blobs blocks: 1.58 GiB compacted DB |
- Measured figures are workload-specific evidence, not host-size or cost guarantees. The real-source runs used gzip HTTP delivery, immediate acknowledgement, four source workers, and did not include an application database.
- The 1.58 GiB full-output figure retained 2.55 million entities and 550,000 index rows. Externalized, latest, or windowed policy is the intended choice when that reusable query surface is unnecessary.
- Trusted Xatu input and commitment-checked EraE input have different trust and performance profiles; retention policy does not change source trust.
Hard ceilings and backpressure
Process memory, temporary material, mapped deltas, SQLite size, artifacts, and delivery history have operator ceilings. A limit action is part of the contract: pause work, expire a best-effort consumer with reset required, or reject growth. Leani must not silently discard required delivery history to remain “healthy.”
Built by Smart-Byte · Legal notice (Impressum) · Privacy