Run your own processor as Wasm
A Wasm processor is a module you build with the guest SDK, in Rust with the
leani-wasm-guest crate or in TypeScript. The
standard leani binary loads it at runtime:
leani processor runruns it once over finalized history and prints its changes;- the node’s playground and
POST /admin/v1/wasm/runsrun it on recent and live blocks without persisting anything; - a
wasmprocessor entry in the configuration keeps it running, with the same durable state, coverage, finality, reorg handling, generic API, and SDK subscriptions as a built-in processor.
A Wasm processor implements the same Processor contract as a native one, so
both kinds run side by side in one node and share its sources. Choose by the
trade-off:
| Wasm module | Native processor | |
|---|---|---|
| Ships with | any stock leani binary |
your own node binary |
| Change it | rebuild the module, restart or rerun | rebuild the node |
| Runs | sandboxed, metered, deterministic interpreter | trusted native code |
| Speed | map ~3.5–5×, reduce ~8× native; see Performance | fastest |
| Outputs | JSON entities and changes | any encoding, typed query extensions |
Start in Wasm; move a processor to native when it is proven and its compute matters. The descriptor contract is the same, so the move is a rebuild under a new instance.
Try a template
The binary embeds four templates: block-gas, the gas, base fee, and burned
base fee of every block, from headers alone; erc20-transfers, transfers of chosen tokens at or
above a minimum value; token-volume, a running transfer count and volume per
token, in ordered state; and event-decoder, any contract’s events, decoded
from Solidity signatures given as parameters. Run one over twelve Mainnet
blocks with a configuration whose history sources reach them, such as the one
leani init blocks writes:
leani processor run --template erc20-transfers \ --from-block 19426589 --to-block 19426600 \ --params '{"tokens":["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]}'Every change prints as one JSON line; progress and the backfill report go to stderr:
{"block":19426589,"blockHash":"0xa291…3791","direction":"apply","key":"0x1c02…0038","kind":"erc20.transfer","operation":"upsert","value":{"blockNumber":19426589,"from":"0xfe0d…43d2","logIndex":56,"timestamp":1710338159,"to":"0x0b8a…66dc","token":"0xa0b8…eb48","transactionHash":"0x1c02…10a3","value":"58295353165"}}The event decoder answers “what did this contract emit” for any finalized range, with no code to write:
leani processor run --template event-decoder \ --from-block 19426589 --to-block 19426600 \ --params '{"addresses":["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"],"events":["Transfer(address indexed from, address indexed to, uint256 value)"]}'leani processor run is an ordinary on-demand backfill of a temporary wasm
instance into a throwaway data directory: it uses the configuration’s sources,
trust, and finality, ignores its processors, and never touches its data
directory. Give it a module file instead of --template to run your own, and
--param key=value to set single parameters.
Write a processor
leani processor create writes a processor crate for you. Describe the onchain
data and what to keep, and it generates the code:
cat > spec.json <<'EOF'{ "data": { "kind": "events", "contracts": ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"], "events": ["Transfer(address indexed from, address indexed to, uint256 value)"] }, "keep": { "kind": "totals", "by": "to", "sum": ["value"] }}EOFleani processor create usdc-volume --spec spec.jsonleani processor run usdc-volume --from-block 19426589 --to-block 19426600data.kind is events (contracts, events), transactions (to, from,
and functions, whose calls it decodes), or blocks (header fields); each
takes fields to keep, all by default. keep.kind is rows, one entity per
match, or totals, a running count and sums of unsigned fields per value of
by, or of everything. --template event-decoder, or another template,
starts from that template’s source instead, and without either
leani processor create writes a USDC transfer decoder to start from. The
crate carries the leani-wasm-guest SDK of the installed binary, its build
settings, and a lockfile, so it builds against exactly the ABI that binary
runs. Building needs the Wasm target once:
rustup target add wasm32-unknown-unknown. Given the project’s directory,
leani processor run builds it as
cargo build --release --target wasm32-unknown-unknown does, into
target/wasm32-unknown-unknown/release/usdc_volume.wasm, and runs that
module; the compiler’s errors print as it writes them, and a failed build runs
nothing.
Edit src/lib.rs where a spec falls short. A processor declares what it
needs, maps one block to a delta, and reduces the delta into state and
changes:
use leani_wasm_guest::{ Block, BlockRef, Capability, Error, Manifest, Processor, Store, export_processor,};use serde::{Deserialize, Serialize};
#[derive(Deserialize)]#[serde(deny_unknown_fields, rename_all = "camelCase")]struct Params { #[serde(default)] min_gas_used: u64,}
#[derive(Deserialize, Serialize)]struct Busy { gas_used: u64, gas_limit: u64,}
struct BusyBlocks;
impl Processor for BusyBlocks { type Params = Params; type Delta = Option<Busy>;
fn manifest(_: &Params) -> Result<Manifest, Error> { Ok(Manifest::new("busy-blocks").require(Capability::Header)) }
fn map(params: &Params, block: &Block) -> Result<Option<Busy>, Error> { let header = block.header.as_ref().ok_or("the header is required")?; let (Some(gas_used), Some(gas_limit)) = (header.gas_used, header.gas_limit) else { return Ok(None); }; Ok((gas_used >= params.min_gas_used).then_some(Busy { gas_used, gas_limit })) }
fn reduce(_: &Params, block: &BlockRef, busy: Option<Busy>, store: &mut Store) -> Result<(), Error> { match busy { Some(busy) => store.upsert("busy", "busy.block", &block.number.to_be_bytes(), &busy), None => Ok(()), } }}
export_processor!(BusyBlocks);The SDK’s solidity module parses Solidity signatures and decodes event and
call arguments, and U256 holds 256-bit amounts with checked addition,
subtraction, and multiplication. The templates’ sources in
examples/wasm-processors show log filters, ERC-20 decoding, and ordered
state.
Write it in TypeScript
The same processors can be written in TypeScript, as
AssemblyScript compiles it: TypeScript’s
syntax with Wasm’s types, such as u64 and u32, and without any, union
types other than | null, or closures that capture variables. With the spec
above:
leani processor create usdc-volume-ts --language typescript --spec spec.jsonleani processor run usdc-volume-ts --from-block 19426589 --to-block 19426600A spec, a template (--template event-decoder --language typescript), and
the starter generate the same processor as in Rust: the same manifest, rows,
and totals. The package carries the TypeScript SDK of the installed binary in
assembly/leani/ and a lockfile pinning the AssemblyScript compiler; the run
installs that compiler with npm and builds the package with npm run build
into build/usdc-volume-ts.wasm, which needs Node.js 20 or later. Every
template has a TypeScript twin beside its Rust source,
examples/wasm-processors/<name>/assembly/index.ts.
Where Rust and TypeScript differ:
| Rust | TypeScript |
|---|---|
impl Processor for X and export_processor!(X) |
class X extends Processor, register(new X()), and export * from "./leani/exports" |
type Params, decoded by serde |
params: Json: get, field, asString, asU64; only refuses unknown keys |
type Delta, encoded by serde |
map writes the delta with a Writer; reduce reads it back in the same order with a Reader |
Manifest filters by transaction type and names its schemas |
Manifest does neither |
Option<u64> |
Some<u64> | null; hashes, addresses, and U256 are null when missing |
[u8; 20], [u8; 32], Vec<u8> |
Uint8Array |
#[derive(Serialize)] entities; put takes JSON bytes |
Json.object().set(key, value): bytes as 0x hex, U256 as a decimal string; put takes the Json |
return Err(...) |
throw new Error(...) fails the call with its message |
solidity::Signature |
import { Signature } from "./leani/solidity" |
A TypeScript module is about a fifth the size of its Rust twin. Its map runs
about as fast, and its reduce 2–3× slower, so a whole block with the SQLite
store takes about 1.1× the Rust module’s time.
Try it on live blocks
GET /playground on a running node serves a playground. Pick a template, your
module, or build your own: pick events, transactions, or blocks, the
fields, and rows or totals, and the node writes the code. Choose how many
recent blocks to run over and whether to follow new ones, and watch each
block’s changes arrive. Each edit to the parameters, each other template, and
each new build of your module reruns the same blocks, and the playground marks
the changes the new version adds, alters, or drops before it follows new
blocks with it. In a Chromium browser, watch .wasm reloads your module
whenever a build rewrites it. The playground also writes the configuration
entry, leani processor commands, SDK subscription, and curl call for what
you run, and its node configuration tab builds a node
configuration
from what you need.
From TypeScript, @smart-byte/leani-sdk starts the same runs: leani.wasm.run
yields a run’s events until done, and leani.wasm.swap gives a running run
another module or parameters:
for await (const event of leani.wasm.run({ template: "block-gas", blocks: 4 })) { if (event.event === "block") { console.log(event.data.number, event.data.changes); }}With api.allow_wasm_builds = true, the playground’s editor compiles the
code on the node and swaps the new module into the run; compiler errors point
at their lines. Rust builds need cargo and the Wasm target on the node’s host,
TypeScript builds Node.js 20 or later.
The playground calls POST /admin/v1/wasm/runs, which you can call directly
with a template name or the module as hex in module:
curl -N -X POST http://127.0.0.1:8080/admin/v1/wasm/runs \ -H 'content-type: application/json' \ -d '{"template":"erc20-transfers","params":{"minValue":"1000000000"},"blocks":32,"follow":true}'The response is a stream of server-sent events: ready with the run’s id,
then one block per applied block with its changes, and undo for each block
a reorg reverts. A run covers the newest blocks retained blocks, 16 by
default and at most 1,024, then follows new ones for maxSeconds, 600 by
default and at most 3,600. It reduces into a throwaway store, so its changes
are what a configured instance would publish, but nothing is kept. While it
runs, PUT /admin/v1/wasm/runs/{run}/processor gives it another module or
other parameters: it reruns the blocks it holds and follows on. A node runs
two runs at a time, of modules up to 4 MiB, with a quarter of the default
fuel per call. api/openapi.yaml describes every event and refusal.
A run first uses the node’s retained blocks. When those lack material the module requires, such as logs of contracts no configured processor reads, the run fetches its own from the node’s P2P peers: up to 64 recent blocks, then each new one. That feed fetches every log or receipt of the kinds the module requires, so a processor swapped in with another filter reruns the blocks without downloading them again, and the run and the live lane download a block they both need once.
Keep it running
Add a wasm processor to the configuration. settings.module is relative to
the configuration file, and settings.params is passed to the module as JSON:
[[processors]]id = "wasm"instance = "usdc-transfers"version = "1.0.0"history_mode = "on_demand"start_block = 19426589publish = "unattested"
[processors.state]mode = "durable"
[processors.output]mode = "full"
[processors.delivery]mode = "window"
[processors.checkpoint]mode = "automatic"keep = 3
[processors.undo]mode = "unfinalized"safety_blocks = 256
[processors.settings]module = "processors/erc20_transfers.wasm"# Refuse any other module bytes: `b3sum` prints the hash, as does the# `moduleHash` of a run's `ready` event.# blake3 = "0x<64 hex digits>"
[processors.settings.params]tokens = ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"]minValue = "1000000000"
# Optional overrides of the defaults below.[processors.settings.limits]fuel = 2000000000leani doctor loads and describes the module, so a missing file, a stale pin,
invalid parameters, or a bad manifest fail before serve. The instance then
behaves like any other: leani backfill --processor usdc-transfers --from-block N --to-block M, the generic /v1/processors/usdc-transfers/…
routes, entity queries by collection, and SDK subscriptions with
leani.processors.subscribe("usdc-transfers").
The processor’s code hash is the module’s BLAKE3 hash, and its configuration hash covers the parameters. The store refuses a changed module or changed parameters under an existing instance, so a new module version runs as a new instance, as a processor rebuild does.
Add it to a running node
With api.allow_processor_changes = true under [api], a running node takes a
processor through its API, without a restart. The request carries the entry
above as JSON, which needs only instance and start_block: the rest takes
the values above, id = "wasm" and version 1.0.0 included. Your own module
goes in module, its bytes as hex (xxd -p processor.wasm | tr -d '\n'), or
name a bundled template:
curl -s http://127.0.0.1:8080/admin/v1/processors \ -H 'content-type: application/json' \ -d '{"processor": {"instance": "usdc-transfers", "start_block": 19426589, "settings": {"params": {"tokens": ["0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48"], "minValue": "1000000000"}}}, "template": "erc20-transfers"}'The node checks the entry as leani doctor checks the configuration, keeps it
in its data directory’s processors.toml and the module in modules/, named
by its hash, and restarts its network or JSON-RPC lanes in-process to run it.
Live processing pauses for every processor while the lanes restart and the
added processor runs the unfinalized blocks the node keeps: about two seconds
on Ethereum, longer on rollups. Every start loads processors.toml after the
configuration file; an instance configured in both runs as the configuration
file says, with a warning. DELETE /admin/v1/processors/usdc-transfers
removes an added processor and keeps its stored data. The node refuses a
second processor of one kind, version, and parameters, which its live lane
could not tell apart. With api.bearer_token_env, send the token as for any
/admin/v1 route. From TypeScript, leani.processors.add sends the same
request and leani.processors.remove removes the processor.
The contract
Manifest. manifest(params) returns the processor’s name, its
reduction, and its requirements: capabilities (header, body,
transactions, calldata, senders, blob_fields, block_size,
receipts, receipt_outcome, logs, withdrawals), an optional filter
(addresses, topics, senders, recipients, transaction_types), and
whether logs need their transaction hashes. Sources fetch only that.
What map sees. Exactly the required materials and field groups, reduced
to the filter, whatever the source delivered: a filtered archive and a full
P2P block map alike. A log is kept when its transaction matches the
transaction filters and the log matches the address and topic filters. The
block’s finality is never visible, so a delta never depends on it.
Each field arrives only with its capability; without it the field is None
(null in TypeScript) or empty, whatever the source holds. The names are the
Rust SDK’s; TypeScript spells them in camel case.
| Capability | Fields |
|---|---|
header |
the header’s gas_limit, gas_used, base_fee_per_gas, blob_gas_used, excess_blob_gas, and fee_recipient |
block_size |
the header’s size_bytes and transaction_count |
body |
a transaction’s to, nonce, gas_limit, and value |
calldata |
a transaction’s input |
senders |
a transaction’s from |
blob_fields |
a type-3 transaction’s max_fee_per_blob_gas and blob_versioned_hashes, and its receipt’s blob_gas_used |
receipt_outcome |
a receipt’s success, gas_used, and effective_gas_price |
receipts |
a receipt’s logs, with receipt_outcome |
logs |
the block’s logs; a log’s transaction_hash also needs logs_with_transaction_hash |
withdrawals |
the block’s withdrawals |
transactions stands for body, calldata, senders, blob_fields, and
block_size. A transaction’s hash, index, and type, and a receipt’s, come
with any group of theirs.
Not every source fills every combination. Xatu’s public tables fill
blob_fields and block_size only for type-3 transactions, without
calldata, logs, withdrawals, or a sender or recipient filter, and
logs only without transaction or receipt groups. Such a manifest takes its
history from eraE or execution P2P instead: see
Xatu.
Reduction. block_local processors reduce each block alone: they cannot
read state, and their keys must not repeat across blocks. ordered
processors reduce in chain order and may read what earlier blocks wrote; their
live lane waits for their history.
State. Public collections hold JSON entities served by the API:
get/put/delete/scan_prefix, plus index_put/index_delete. Private
state namespaces hold any bytes and are never exposed. Reads see the block’s
own earlier writes. The node captures undo and commits state, coverage,
cursor, and changes atomically.
Changes. Publish with emit_upsert/emit_delete, or upsert/remove to
write and publish together. A block publishes at most one change per key; the
host keeps the last one emitted. An upsert of an existing key must carry the
key’s final value.
Failures. A guest error or panic fails the block with its message; the node retries and halts the lane rather than skip the block. Skip input that does not decode, such as an ERC-721 log under the ERC-20 signature, instead of failing.
Sandbox. A module has no clock, randomness, network, filesystem, or WASI:
the node refuses any import outside the leani ABI. Every call runs in a
fresh instance of the wasmi
interpreter, so nothing leaks from one block to the next, and floats follow
Wasm’s deterministic profile. map is a pure function of the module, its
parameters, and the block.
Limits
Defaults, overridable per instance in [processors.settings.limits]:
| Limit | Default | Exceeded when |
|---|---|---|
module_bytes |
16 MiB | the module file is larger |
memory_bytes |
64 MiB | an instance grows its memory past it |
fuel |
1,000,000,000 | one describe, map, or reduce call runs about that many instructions, or copies that many bytes through the host |
value_bytes |
8 MiB | a delta, entity, state value, change payload, or hash input is larger |
changes |
100,000 | one block emits more changes |
host_calls |
1,000,000 | one block’s reduce makes more state accesses and emits |
Fixed for every module: names of collections, namespaces, indexes, and change kinds up to 128 bytes; keys up to 64 KiB; at most 10,000 entries per scan; 16 KiB of log output and a 4 KiB failure message per call; 64 MiB of changes per call; and a function table of at most 10,000 elements.
Performance
Measured in release on an Apple Silicon laptop with the same ERC-20 decoding
done by the native evm-events processor and by the erc20-transfers
template, per block:
| Transfers per block | Native map | Wasm map | Native reduce | Wasm reduce | Map + apply with the SQLite store, native / Wasm |
|---|---|---|---|---|---|
| 0 | 1.5 µs | 19 µs | 1.6 µs | 8 µs | 0.38 / 0.41 ms |
| 10 | 12 µs | 54 µs | 14 µs | 119 µs | 1.3 / 1.4 ms |
| 100 | 91 µs | 342 µs | 130 µs | 1.1 ms | 8.3 / 12 ms |
| 1,000 | 0.9 ms | 3.1 ms | 1.5 ms | 11 ms | 114 / 121 ms |
Each call pays about 10–20 µs to start a fresh instance; beyond that, map
runs 3.5–5× and reduce, where building JSON dominates, about 8× slower than
native. Maps run on the blocking thread pool, so concurrent maps use several
cores, and with storage and sources in the loop the difference shrinks: source
throughput usually bounds a backfill before either does. Filter in the
manifest rather than in map, and prefer typed #[derive(Serialize)]
entities to serde_json::Value.
ABI
A module is a core Wasm module (wasm32), ABI version 1.
crates/wasm-guest/src/ffi.rs documents its exports and its imports from
module leani, and abi.rs beside it the inputs and outputs with their
postcard and JSON encodings. Another language implements the ABI by decoding
postcard and exporting the same functions, as the TypeScript SDK does.
Built by Smart-Byte · Legal notice (Impressum) · Privacy