A project to collect LN gossip traffic and compute useful metrics to inform a set-reconciliation based protocol (Erlay-like) to replace message flooding.
This project is inspired by earlier work on measuring properties of the LN gossip network, and more recent work on adapting Minisketch for LN gossip messages.
Architecture
-
The collector is a daemon that runs a forked version of ldk-node which does not forward gossip (it should still reply to gossip queries). It collects to a random set of nodes, and then forwards gossip messages to a NATS server. It runs for a fixed time period, set via config file.
-
The archiver applies some basic config to an already-running NATS server, creates a DuckDB database for storing gossip data from the collector, and batch wrties said data to the DB. The archiver MUST be started before the collector.
-
Tailscale is used as a P2P VPN here due to convinience; Nebula or vanilla Wireguard would also work.
Once data is collected, the DuckDB built-in UI can be used to run queries:
# This will grow the DB file by ~5x, but should speed up queries.
./db_build_indices.sh $DB_FILENAME
# The DuckDB UI can be resource-hungry; add limits to not affect other processes.
systemd-run --scope -p MemoryMax=16G -p CPUQuota=50% --user duckdb -ui