Kaspa Explorer Infrastructure

Proposal to Eliminate the Explorer Bottleneck

March 2026 • Prepared for Kaspa Core & Ecosystem Teams

The Problem

Kaspa's consensus layer handles 10 BPS (2000+ TPS) without issues. However, wallets and exchanges that depend on explorer infrastructure experience stale data, delayed confirmations, and API timeouts during sustained high throughput. The bottleneck is entirely in the off-chain indexing pipeline.

The current architecture funnels all reads and writes through a single PostgreSQL instance that cannot sustain the required throughput:

rusty-kaspa node ──wRPC──► simply-kaspa-indexer ──SQL INSERT──► PostgreSQL (SINGLE INSTANCE) ◄──SQL SELECT── kaspa-rest-server ──HTTP──► Wallets / Exchanges ▲ Write contention blocks reads. Reads slow down under write pressure. ▲ Single point of failure. No horizontal scaling path. ▲ PostgreSQL not designed for sustained 1000s of inserts/sec + concurrent analytical reads.

Proposed Fixes

PostgreSQL Read Replicas Phase 1 — Now
Redis Caching Layer Phase 1 — Now
Exchange wRPC Integration Guide Phase 1 — Now
REST API Rewrite (Rust) Phase 2 — Medium
Table Partitioning & Materialized Views Phase 2 — Medium
WebSocket Push Notifications Phase 2 — Medium
Write-Optimized Storage Engine Phase 3 — Architecture
Node-Level Transaction Index Phase 3 — Architecture

Phase 1 — Deploy Now

1. PostgreSQL Read Replicas

Estimated impact: 40-60% reduction in API latency under load • Low effort

The Problem

The indexer's heavy INSERT/UPDATE workload and the REST API's SELECT queries compete for the same PostgreSQL instance. Under sustained high TPS, write-ahead log (WAL) flushing, index maintenance, and row locking cause read queries to queue behind writes.

The Fix

Deploy PostgreSQL streaming replication with one or more read replicas. Route all REST API traffic to replicas. The indexer writes exclusively to the primary.

Indexer ──INSERT──► PostgreSQL Primary (writes only) │ streaming replication (async, <100ms lag) │ ┌─────────┼─────────┐ ▼ ▼ ▼ Replica 1 Replica 2 Replica 3 (reads only) ▲ ▲ ▲ └─────────┼─────────┘ │ REST API (load balanced across replicas)

Implementation

Configure wal_level = replica and max_wal_senders = 5 on the primary
Set up streaming replication to 1-3 read replicas using pg_basebackup
Use pgBouncer or application-level routing to direct REST API connections to replicas
Set hot_standby = on on replicas to allow read queries during replication
Monitor replication lag — at sub-100ms async replication, explorer data is effectively real-time

Zero write contention on readsREST API queries never compete with indexer inserts

Horizontal read scalingAdd more replicas as exchange/wallet traffic grows

No code changes to indexerCompletely transparent — indexer writes to primary as before

Built-in PostgreSQL featureMature, battle-tested, no new dependencies

Note: Async replication introduces a small lag (typically <100ms). This is acceptable for explorer queries but should be monitored. If an exchange requires absolute consistency, they can query the primary for critical balance checks.

Phase 1 — Deploy Now

2. Redis Caching Layer

Estimated impact: 70-90% reduction in database read queries • Low effort

The Problem

Exchanges poll the same address endpoints repeatedly (every few seconds) to check for deposit confirmations. The REST API hits PostgreSQL for every request, even when the data hasn't changed since the last query.

The Fix

Add Redis as a caching layer between the REST API and PostgreSQL. Cache hot data with short TTLs appropriate to each query type.

Exchange/Wallet Request │ ▼ REST API ── cache hit? ──► Redis ── yes ──► return cached response (sub-ms) │ │ │ no │ │ │ ▼ └────────── cache miss ──────► PostgreSQL Replica │ store in Redis with TTL │ ▼ return response

Recommended TTLs by Query Type

Endpoint	TTL	Rationale
`/addresses/{addr}/balance`	3-5 seconds	Balances change with blocks (~1 per 100ms at 10 BPS)
`/addresses/{addr}/transactions`	5-10 seconds	Tx history is append-only, slight staleness acceptable
`/transactions/{txid}`	60 seconds	Once confirmed, tx data is immutable
`/blocks/{hash}`	300 seconds	Block data is immutable once indexed
`/info/blockdag`	1-2 seconds	DAG info changes frequently but is lightweight
`/info/health`	5 seconds	Health status, low frequency is fine

Implementation

Deploy Redis alongside the REST API server (single instance is sufficient to start)
Add cache middleware to kaspa-rest-server — FastAPI supports this via fastapi-cache2 with Redis backend
Key format: kaspa:{endpoint}:{params_hash} with per-endpoint TTL configuration
Add Cache-Control and X-Cache-Status headers so clients know if they got a cached response
Implement cache invalidation on new block notifications (optional, improves freshness)

Massive DB load reductionMost exchange polling hits cache, not PostgreSQL

Sub-millisecond responsesCached responses are 100-1000x faster than DB queries

Protects DB during spikesTraffic surges absorbed by cache, DB stays stable

Simple to addFastAPI cache middleware — minimal code changes

Phase 1 — Deploy Now

3. Exchange wRPC Integration Guide

Estimated impact: Removes explorer dependency for real-time operations • Low effort

The Problem

Exchanges use the REST API for everything — including real-time balance monitoring and deposit detection — when the node's wRPC interface can handle these natively with zero bottleneck.

The Fix

Publish integration documentation and reference code showing exchanges how to use a hybrid approach: wRPC for real-time operations, REST API only for historical queries.

═══════════════════════════════════════════════════════ REAL-TIME (via wRPC — no bottleneck, no explorer needed) ═══════════════════════════════════════════════════════ Exchange ──wRPC──► rusty-kaspa node │ ├── SubscribeUtxosChanged(addresses[]) │ Push notification when deposit arrives │ ├── GetBalanceByAddress(addr) │ Current confirmed balance, real-time │ └── GetUtxosByAddresses(addresses[]) Full UTXO set for withdrawal construction ═══════════════════════════════════════════════════════ HISTORICAL ONLY (via REST API — lighter load) ═══════════════════════════════════════════════════════ Exchange ──HTTP──► kaspa-rest-server │ ├── Transaction history for user display ├── Block confirmation depth verification └── Address activity reports / auditing

What This Achieves

Deposit detection becomes instant — wRPC push notifications vs polling REST every few seconds
Balance queries bypass the explorer entirely — node handles these natively at any TPS
REST API load drops dramatically — historical queries are infrequent compared to real-time polling
No single point of failure — exchanges can run their own node, independent of api.kaspa.org uptime

Implementation

Write reference integration code in JavaScript/TypeScript (most exchange backends), Rust, Python, and Go
Document the wRPC subscription lifecycle: connect, subscribe, handle events, reconnect
Provide a Docker-based "exchange integration kit" with a pre-configured rusty-kaspa node + --utxoindex
Coordinate with major exchanges to pilot the hybrid approach

Prerequisite: This requires exchanges to run (or connect to) a rusty-kaspa node with --utxoindex enabled. This adds ~20% memory overhead to the node but is manageable. Alternatively, ecosystem-run wRPC endpoints could be provided.

Phase 2 — Medium Term

4. REST API Rewrite in Rust

Estimated impact: 5-10x throughput improvement for API layer • Medium effort

The Problem

The current REST API is Python/FastAPI running behind gunicorn with Uvicorn workers. Python's GIL limits true concurrency, gunicorn workers have been observed deadlocking under load (hashrate population was removed for this reason), and the per-request overhead is high compared to native alternatives.

The Fix

Rewrite kaspa-rest-server in Rust using axum or actix-web. This aligns with the rest of the Kaspa stack (rusty-kaspa, simply-kaspa-indexer) and provides:

True async concurrency — tokio runtime handles thousands of concurrent connections without GIL contention
Connection pooling — sqlx or deadpool-postgres with efficient prepared statements
Memory efficiency — no Python interpreter overhead, ~10x lower memory per connection
Integrated Redis — native async Redis client (fred or redis-rs) for cache layer
Type-safe query building — compile-time SQL validation prevents runtime query errors

// Example: Rust axum handler with caching and read replica routing

async fn get_address_balance(
    State(ctx): State<AppState>,
    Path(address): Path<String>,
) -> Result<Json<BalanceResponse>, ApiError> {
    // Check cache first
    let cache_key = format!("balance:{address}");
    if let Some(cached) = ctx.redis.get(&cache_key).await? {
        return Ok(Json(cached));
    }

    // Query read replica (not primary)
    let balance = sqlx::query_as!(BalanceResponse,
        "SELECT balance FROM address_balances WHERE address = $1",
        &address
    )
    .fetch_one(&ctx.read_replica_pool)
    .await?;

    // Cache for 5 seconds
    ctx.redis.set_ex(&cache_key, &balance, 5).await?;

    Ok(Json(balance))
}

Migration Path

Implement the most-queried endpoints first (balance, tx lookup, address txs)
Run Rust API alongside Python API behind a load balancer, gradually shifting traffic
Maintain API compatibility — same URL structure and response format
Deprecate Python API once Rust version covers all endpoints

Phase 2 — Medium Term

5. Table Partitioning & Materialized Views

Estimated impact: 30-50% write throughput improvement + faster queries • Medium effort

The Problem

As the blockchain grows, the transactions table becomes massive. Index maintenance on large tables is expensive — every INSERT requires updating B-tree indexes across the entire table. Queries that only need recent data still scan large index structures.

The Fix

A. Range Partitioning by DAA Score

Partition the transactions table into time-based chunks. The active partition (current DAA score range) is small and fast. Historical partitions are rarely written to and can be optimized for reads.

-- Partition transactions by DAA score ranges (each ~1 day of blocks)

CREATE TABLE transactions (
    transaction_id TEXT NOT NULL,
    block_daa_score BIGINT NOT NULL,
    -- ... other columns
) PARTITION BY RANGE (block_daa_score);

-- Recent partition (active writes + most reads)
CREATE TABLE transactions_current
    PARTITION OF transactions
    FOR VALUES FROM (86400000) TO (86500000)
    TABLESPACE fast_nvme;

-- Historical partitions (rarely written, optimized for reads)
CREATE TABLE transactions_2026_q1
    PARTITION OF transactions
    FOR VALUES FROM (85000000) TO (86400000)
    TABLESPACE standard_ssd;

-- Auto-create new partitions via pg_partman
SELECT partman.create_parent(
    p_parent_table := 'public.transactions',
    p_control := 'block_daa_score',
    p_interval := '100000',
    p_type := 'range'
);

B. Materialized Views for Common Queries

Pre-compute expensive aggregations that exchanges query frequently, refreshed incrementally by a background worker.

-- Pre-computed address balances (refreshed every block)
CREATE MATERIALIZED VIEW address_balances AS
SELECT
    script_public_key AS address,
    SUM(amount) AS balance,
    COUNT(*) AS utxo_count,
    MAX(block_daa_score) AS last_active_daa
FROM current_utxos
GROUP BY script_public_key;

CREATE UNIQUE INDEX ON address_balances (address);

-- Refresh incrementally (triggered by indexer after each batch)
REFRESH MATERIALIZED VIEW CONCURRENTLY address_balances;

Smaller active indexWrites only update the current partition's indexes, not the entire history

Faster recent queriesMost queries target recent data — scans a small partition instead of the full table

Independent maintenanceVACUUM, REINDEX, and backups per-partition without locking the whole table

Storage tieringMove old partitions to cheaper storage, keep active partition on NVMe

Phase 2 — Medium Term

6. WebSocket Push Notifications

Estimated impact: Eliminates polling, reduces API requests by 80%+ • Medium effort

The Problem

The REST API is entirely poll-based. Exchanges hit endpoints every 2-5 seconds asking "has anything changed?" The answer is usually "no" — but the database still has to execute the query. This creates massive unnecessary load.

The Fix

Add a WebSocket endpoint to the REST API (or as a separate service) that pushes events to subscribers. Clients subscribe to address events and receive notifications when new transactions are indexed.

CURRENT (poll-based): Exchange ──GET /balance──► API ──SELECT──► DB (every 3 sec) Exchange ──GET /balance──► API ──SELECT──► DB (every 3 sec) Exchange ──GET /balance──► API ──SELECT──► DB (every 3 sec) ... 28,800 queries/day per address being monitored PROPOSED (push-based): Exchange ──WebSocket──► Notification Service ◄──LISTEN──► PostgreSQL │ │ { "event": "tx_indexed", "address": "kaspa:qr...", "txid": "abc..." } │ ▼ Exchange receives push only when something changes ... ~50-200 events/day per address (only actual transactions)

Implementation

-- PostgreSQL: Notify on new indexed transaction
CREATE OR REPLACE FUNCTION notify_tx_indexed()
RETURNS TRIGGER AS $$
BEGIN
    PERFORM pg_notify('tx_indexed', json_build_object(
        'address', NEW.script_public_key,
        'txid', NEW.transaction_id,
        'amount', NEW.amount,
        'daa_score', NEW.block_daa_score
    )::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER tx_indexed_trigger
    AFTER INSERT ON transactions
    FOR EACH ROW EXECUTE FUNCTION notify_tx_indexed();

// WebSocket server (Rust/axum) — listens to PostgreSQL NOTIFY, pushes to subscribers

async fn ws_handler(ws: WebSocketUpgrade, State(ctx): State<AppState>) -> Response {
    ws.on_upgrade(|socket| async move {
        let (mut sender, mut receiver) = socket.split();

        // Client sends subscription: { "subscribe": ["kaspa:qr..."] }
        while let Some(msg) = receiver.next().await {
            let sub: Subscription = serde_json::from_str(&msg.to_text()?)?;
            ctx.subscriptions.add(sub.addresses, sender.clone());
        }
    })
}

// Background task: listen to PostgreSQL NOTIFY channel
async fn pg_listener(pool: PgPool, subs: Arc<SubscriptionManager>) {
    let mut listener = PgListener::connect_with(&pool).await?;
    listener.listen("tx_indexed").await?;

    while let Ok(notification) = listener.recv().await {
        let event: TxEvent = serde_json::from_str(notification.payload())?;
        subs.notify(&event.address, &event).await;
    }
}

Phase 3 — Architectural

7. Write-Optimized Storage Engine

Estimated impact: 10-100x write throughput, scales to 32+ BPS • High effort

The Problem

PostgreSQL uses a B-tree index structure optimized for balanced read/write workloads. Kaspa's indexer workload is 95%+ writes with bursty reads — a pattern PostgreSQL was not designed for. As BPS increases beyond 10, PostgreSQL will not scale, regardless of hardware.

The Fix

Adopt a tiered storage architecture with a write-optimized engine for ingestion and a read-optimized layer for queries.

Database Comparison for Kaspa's Workload

Database	Write Throughput	Read Pattern	Fit for Kaspa
PostgreSQL (current)	~5-10K rows/sec with indexes	Excellent for complex joins	Struggles at sustained high TPS. Index maintenance is the killer.
ClickHouse	~500K-1M+ rows/sec	Excellent for analytical/aggregate queries. Not ideal for point lookups.	Best fit for tx history, block data, analytics. Append-only model matches blockchain data perfectly.
ScyllaDB	~100K-500K rows/sec per node	Excellent for point lookups by key. No joins.	Best fit for tx-by-ID lookups, address balance lookups. Linear horizontal scaling.
TiKV / TiDB	~50-100K rows/sec	MySQL-compatible SQL with distributed storage	Good compromise — SQL compatibility + horizontal scaling. Higher operational complexity.
QuestDB / TimescaleDB	~100K-500K rows/sec	Time-series optimized queries	Good for DAA-score-ordered tx data. TimescaleDB runs on PostgreSQL (easier migration).

Recommended Architecture: ClickHouse + ScyllaDB Hybrid

rusty-kaspa node │ │ wRPC ▼ Indexer ──────────┬──────────────────────────┐ │ │ ▼ ▼ ClickHouse ScyllaDB (analytical store) (key-value store) │ │ tx history by address tx by ID lookup block analytics address balances aggregate queries UTXO set snapshots │ │ └────────────┬─────────────┘ │ ▼ REST API (Rust) routes queries to appropriate store │ ▼ Wallets / Exchanges

Why This Works

ClickHouse uses an LSM-tree/MergeTree engine — inserts go to memory, batch-merged to disk. No index maintenance on write. Perfectly suited for append-only blockchain data.
ScyllaDB provides single-digit millisecond point lookups by primary key. "Get transaction by ID" and "get balance by address" are O(1) operations.
Both scale horizontally — add nodes to handle higher BPS. PostgreSQL cannot do this for writes.
Separation of concerns — each database handles the query pattern it's optimized for, instead of PostgreSQL trying to do everything.

Migration consideration: This is a significant architectural change. A pragmatic first step is TimescaleDB — it runs as a PostgreSQL extension, so the indexer requires minimal changes, while gaining hypertable partitioning and compression that can 5-10x write throughput. This buys time before a full database migration.

Phase 3 — Architectural

8. Node-Level Transaction Index

Estimated impact: Eliminates explorer dependency for tx lookups • High effort

The Problem

The rusty-kaspa node deliberately does not index transactions by ID or maintain address-to-transaction mappings. This keeps the node lean, but forces every service that needs this data through the explorer bottleneck. GitHub issue #610 requested transaction lookup by ID — it remains unimplemented.

The Fix

Add an optional --txindex flag to rusty-kaspa (similar to the existing --utxoindex) that maintains:

Transaction-by-ID lookup — maps txid → block hash + position (compact, low overhead)
Address-to-transaction index — maps script_public_key → list of txids (optional, higher overhead)

This would be stored in the node's existing RocksDB instance (which is already LSM-tree based and write-optimized) and exposed via additional wRPC methods:

// New wRPC methods (proposed)

GetTransactionById(txid: Hash) -> TransactionInfo
    Returns: transaction data + containing block + acceptance status

GetTransactionsByAddress(address: Address, limit: u32, offset: Hash?) -> Vec<TransactionInfo>
    Returns: paginated transaction history for an address

SubscribeTransactionsByAddress(addresses: Vec<Address>) -> Stream<TransactionEvent>
    Returns: push notifications for new transactions involving watched addresses

Impact on the Ecosystem

Exchanges could operate entirely on Track A — no explorer dependency whatsoever
The explorer becomes optional — useful for analytics and browsing, not critical infrastructure
Node operators choose their overhead — --txindex is opt-in, like --utxoindex
RocksDB handles the write pattern natively — LSM-tree, no B-tree index maintenance penalty

This requires a KIP. Adding transaction indexing to the node is a significant scope change. It should be proposed as a formal KIP with performance benchmarks, storage overhead analysis, and consensus from the core team. The storage cost is estimated at ~2-4x the UTXO index depending on address-level granularity.

Phase 1 — Now

Deploy within days. No architectural changes.

PostgreSQL read replicas
Redis caching layer
Exchange wRPC integration guide

Phase 2 — Medium Term

Weeks to months. Targeted improvements.

Rust REST API rewrite
Table partitioning + materialized views
WebSocket push notifications

Phase 3 — Architecture

Months. Fundamental redesign for 32+ BPS.

Write-optimized storage engine
Node-level transaction index (KIP)

Target Architecture (All Phases Complete)

rusty-kaspa node (with --utxoindex --txindex) │ │ │ wRPC │ wRPC │ │ │ ▼ │ Exchanges / Wallets (direct, no explorer needed) │ ├── SubscribeUtxosChanged (deposits) │ ├── GetTransactionById (confirmation) │ └── GetBalanceByAddress (real-time) │ │ wRPC ▼ Indexer (Rust, batch-optimized) │ │ │ │ ▼ ▼ ClickHouse ScyllaDB Redis (analytics) (point lookups) (hot cache) │ │ │ └──────────────┼───────────────────┘ │ ▼ REST API (Rust/axum) ├── HTTP REST (backwards compat) └── WebSocket (push notifications) │ ▼ Explorer UI / Analytics / Auditing (nice-to-have, not critical infrastructure)

In this architecture, the explorer is useful but not critical. Exchanges and wallets operate directly against the node for real-time operations. The explorer stack serves historical data and analytics with write-optimized storage that scales horizontally. No single PostgreSQL instance bottleneck.