Sync Protocol

The sync protocol ensures nodes converge on the same ledger state by exchanging missing blocks. It uses a frontier-based approach: each node tells its peer what it already has, and the peer sends back what it is missing.

Protocol ID: /xe/sync/1.0.0

Workflow

  Client                                  Server
  ──────                                  ──────
     │                                       │
     │  SyncRequest (frontiers, page_size)   │
     │ ─────────────────────────────────────▶│
     │                                       │
     │                          Compare frontiers
     │                          Walk chains for
     │                          missing blocks
     │                                       │
     │  SyncResponse (blocks[], has_more)    │
     │◀───────────────────────────────────── │
     │                                       │
     │  SyncResponse (blocks[], has_more)    │
     │◀───────────────────────────────────── │
     │                                       │
     │  SyncResponse (blocks[], has_more=f)  │
     │◀───────────────────────────────────── │
     │                                       │
     │  Validate and add blocks to ledger    │
     │                                       │

1. Client sends frontiers -- A map of account -> latest block hash representing its current view of the ledger, plus a requested page size.
2. Server compares frontiers -- For each account, the server determines which blocks the client is missing.
3. Server streams pages -- Missing blocks are sent in paginated responses. Each page contains up to pageSize blocks and a HasMore flag.
4. Client receives and validates -- The client collects all blocks, then adds them to the ledger via AddSyncedBlock() with retry logic for cross-account dependencies. AddSyncedBlock skips the timestamp window check (which would reject historical blocks) while keeping all other validation (signatures, PoW, balances, chain integrity).
5. Stream closes -- After the final page (HasMore=false), the stream ends.

Wire Types

SyncRequest

type SyncRequest struct {
    Frontiers map[string]string `json:"frontiers"` // account → frontier block hash
    PageSize  int               `json:"page_size"`
}

SyncResponse

type SyncResponse struct {
    Blocks   []*core.Block `json:"blocks"`
    HasMore  bool          `json:"has_more"`
    Cursor   string        `json:"cursor"` // hash of last block sent
    Rejected []string      `json:"rejected,omitempty"`
}

Rejected is populated on the terminal page only (HasMore = false) and lists accounts whose advertised frontier the server could not locate in its own chain. See unrecognized frontiers.

Constants

Triggers

Sync is triggered in three ways:

1. On Peer Connection

When a new peer connects, the node immediately initiates a sync:

h.Network().Notify(&network.NotifyBundle{
    ConnectedF: func(n network.Network, conn network.Conn) {
        gate.spawn(func() {
            pid := conn.RemotePeer()
            if !outboundRL.allow(pid) {
                return
            }
            requestSync(h, pid, ledger, quarantine, tracker)
        })
    },
})

2. Periodic Re-Sync (with frontier tracking)

A background goroutine checks peers every 10 seconds, but only syncs when something has changed. A SyncTracker records the frontiers last sent to each peer and a dirty flag that is set when a block is added locally (via gossip, API, or sync):

• Dirty flag set: sync all peers on the next tick, then clear the flag
• Dirty flag clear: skip all peers (frontiers haven't changed)
• Full resync: forced every 60 seconds as a safety net regardless of dirty state

This eliminates the constant stream-open/close chatter when the network is idle. With 5 peers and a clean ledger, the node produces zero sync traffic between the 60-second safety ticks.

ticker := time.NewTicker(periodicSyncInterval) // 10s
for range ticker.C {
    for _, pid := range h.Network().Peers() {
        if !outboundRL.allow(pid) || !peerMaySupportSync(h, pid) {
            continue
        }
        if tracker.shouldSync(pid, currentFrontiers) {
            go requestSync(h, pid, ledger, quarantine, tracker)
        }
    }
}

The loop stops when the node's context is cancelled, and shutdown waits for every in-flight sync goroutine (periodic, per-peer and connect-triggered) before the store is closed.

3. Incoming Sync Requests

The node also serves sync requests from other peers via the stream handler registered at /xe/sync/1.0.0.

Serving a request can trigger the server's own catch-up. The stream is pull-only — blocks flow server→client — so a node that fell behind on some accounts during a partition could otherwise serve its peers forever while never catching up itself. Before responding, the server checks for accounts it holds where the requester advertised a tip the server disagrees with and is not ahead of; if any exist, it marks itself dirty so its next periodic round re-pulls them.

Accounts the server does not hold at all are deliberately ignored here — chasing a phantom account is exactly the amplification vector the anti-amplification guard exists to prevent, so a forged hash for a non-existent account never trips it.

Skipping Sync-Incapable Peers

Not every peer speaks /xe/sync/1.0.0. Before opening a stream, the node checks the peer's libp2p-advertised protocol set; if the peerstore positively knows the set and the sync protocol is absent, the peer is skipped. Every "unknown" case (identify not yet complete, empty protocol list, peerstore error) attempts sync anyway, so a real full node is never starved.

A peer that fails protocol negotiation is added to a skip list for syncReprobeInterval (10 minutes), then retried once in case it has since upgraded. Without this, such peers would be re-selected every tick forever — wasted work, log spam, and a minor amplification vector where a peer advertises partial protocols to burn sync attempts indefinitely.

Rate Limiting

Both inbound and outbound sync are rate-limited per peer with separate syncRateLimiter instances:

The rate limiter tracks the last sync timestamp per peer ID. Expired entries are evicted on each check to prevent unbounded memory growth.

func (rl *syncRateLimiter) allow(pid peer.ID) bool

[!INFO] Why Rate Limit? Without rate limiting, the periodic 10-second re-sync combined with peer connection events could cause excessive sync traffic, especially in large networks. The 5-second cooldown ensures at most one sync per peer per direction every 5 seconds.

Server-Side Logic

The server handles an incoming sync stream as follows:

1. Decode request -- Read and JSON-decode the SyncRequest from a size-limited reader (maxSyncRequestBytes)
2. Cap frontiers -- If the request contains more than maxFrontiers entries, excess entries are silently trimmed
3. Clamp page size -- Page size is clamped to [1, maxPageSize]
4. Walk chains -- For each account in the server's ledger:
   • If the client has no frontier for the account, send the entire chain
   • If the client's frontier matches the server's, skip (already in sync)
   • If the client's frontier is behind, send blocks after the frontier
   • If the client's frontier is unrecognized, skip the account and add it to the Rejected list (prevents amplification attacks)
5. Send pages -- Blocks are grouped into pages of pageSize and streamed as JSON-encoded SyncResponse objects. Exactly one terminal page carries HasMore = false plus the full Rejected list, so the protocol is unambiguous for the client. When there is genuinely nothing to communicate (an idle sync between two in-sync peers) the channel simply closes and the client reads EOF.

Unrecognized Frontiers and Recovery

[!WARNING] Why the server refuses If a peer claims a frontier hash that doesn't exist in the server's chain for that account, the account is skipped entirely. This prevents a bandwidth amplification attack where a malicious peer sends fake frontier hashes for every account to receive the full ledger.

The refusal is correct but, on its own, it wedges an honest node whose local frontier has genuinely diverged from the canonical chain: it advertises a hash the server doesn't recognise, gets nothing, and stays diverged forever.

The Rejected list closes that hole. The client counts consecutive rejections per (peer, account). Once an account has been rejected unrecognizedDropThreshold (3) rounds in a row, the client omits it from its outgoing frontier map — which puts the server on the "client has no frontier for this account" branch, so it dumps the full chain and the node recovers. A single non-rejection resets the counter, and once recovery lands the next round sees no rejection and steady state resumes.

With a 10-second tick and a 60-second forced full re-sync, worst-case recovery is on the order of a few minutes after a wedge first becomes observable.

Client-Side Logic

The client side of a sync:

1. Open stream -- Create a new stream to the target peer with a 60-second timeout
2. Send frontiers -- JSON-encode SyncRequest with the ledger's current frontiers
3. Close write -- Signal to the server that the request is complete
4. Read pages -- Decode SyncResponse objects until HasMore=false or EOF, accumulating blocks up to maxTotalBlocks
5. Add blocks with retry -- Blocks are added to the ledger via AddSyncedBlock() in up to 10 passes, to handle cross-account dependency chains. AddSyncedBlock bypasses the timestamp window check — synced blocks are historical data that was already validated when originally published. All other validation (signatures, PoW, balances, chain integrity) still applies.

Cross-Account Dependencies

A receive block references a send block from a different account. If the send block arrives later in the sync stream, the receive block fails to validate. The retry mechanism handles this:

Pass 1: Add all blocks → some receive blocks fail (send not yet in ledger)
Pass 2: Retry failed blocks → most succeed (sends now in ledger)
...       up to 10 passes; stops early once a pass makes no progress

Whether a failure is retried is decided by one shared classifier, core.IsRetryableError — the same one conflict promotion uses, so the two can never drift. It is an allowlist, so a permanently invalid block (bad signature, PoW, structural or value validation) is never retried. Retryable patterns include:

If no progress is made in a retry pass (the same blocks fail again), retrying stops. Leftovers are logged with a summary of why they are stuck, not just how many, so a node that never converges is diagnosable.

Block Quarantine

Blocks that fail with non-retryable errors (invalid signature, wrong hash, missing attestations) are added to an in-memory quarantine set. On subsequent sync rounds, quarantined blocks are skipped entirely — no validation attempt, no log output. This prevents repeated "block rejected" log spam from permanently-invalid blocks (e.g. blocks from a previous network epoch).

The quarantine is bounded at maxQuarantineEntries (8,192) with FIFO eviction: the keys are peer-supplied block hashes, so an unbounded map would grow without limit under a flood of distinct rejected hashes. The only cost of evicting a still-bad hash is re-validating and re-rejecting it on a later round.

The quarantine resets on node restart, which is correct — new state after restart may make previously invalid blocks valid.

Security Considerations

[!DANGER] Frontier Privacy The sync protocol reveals the full frontier set to the responding peer. A malicious peer can learn which accounts exist and their current block heights. Future improvement: use a bloom filter or frontier hash instead of sending the full frontier map.

Mitigations in place:

• Request size limit (1 MiB) -- Prevents OOM from huge frontier maps
• Response size limit (10 MiB) -- Prevents OOM from oversized responses
• Frontier cap (10,000) -- Limits frontier entries per request
• Block caps (10,000 client + server) -- Prevents CPU/memory exhaustion from chain walking
• Rate limiting (5s cooldown) -- Prevents sync flooding
• Stream deadline (60s) -- Prevents hung connections
• Amplification resistance -- Unrecognized frontiers are skipped, not replied with full chains
• Bounded quarantine (8,192, FIFO) -- Peer-supplied hashes can't grow the quarantine map without limit
• Drained producers -- If encoding a page fails (e.g. the peer disconnected), the server keeps draining its producer to completion, so an abandoned stream can't park a goroutine holding block slices

Relationship to Gossip

Sync and gossip are complementary:

Gossip handles the fast path -- new blocks propagate in near-real-time. Sync handles the slow path -- catching up after downtime, missed messages, or network partitions.

[!IMPORTANT] Sync does not carry finalization state The sync protocol transfers block bodies only. A node that catches up on blocks may still hold its own non-final losing fork at a position the network has already finalized, because the votes that would resolve it were cleaned up network-wide and are never re-gossiped. Recovering from that needs a targeted vote_request pull, which the stale-conflict sweep issues for exactly this case.