fleet-router
fleet-router listener eos · 0.0.0.0:17000 · v0.6.0
  • route set
  • failover
  • stale
  • offline
  • quarantined
Hyperion's stream is set to SHiP A, whose trace range covers its start block. Atomic took the least-loaded healthy node, SHiP B. SHiP C stopped advancing its chain state, so it is flagged stale and deprioritized, not excluded.

One WebSocket in front of the whole SHiP fleet.

fleet-router is a Rust reverse proxy and load balancer for Antelope State History nodes. Hyperion, Atomic and other indexers open a single connection. The router picks a healthy upstream, and when that node drops it resumes the stream at the next block on another one. No reconnect, no duplicate blocks.

cargo install fleet-router

Also as Docker image and prebuilt binaries. MIT licensed. Source on GitHub · crates.io · Changelog

How a stream survives a node drop

Every client session is a route across block sections. The router sets it, watches it, and re-sets it when the node behind it fails.

  1. Connect

    The client opens one WebSocket to the router. The router selects an upstream and forwards that upstream's ABI, so the client talks to a single SHiP endpoint.

  2. Route

    Selection prefers an upstream whose trace range covers the requested block, then the lowest priority tier, then the fewest connections per unit of weight. A get_blocks_request_v1 with fetch_finality_data only goes to a node whose finality range covers that block.

  3. Fail over

    When the active upstream closes, errors or goes silent, the client socket stays open. A quiet upstream is pinged and abandoned if nothing comes back within 4 s. The router re-sends the in-flight get_blocks from the next block on another suitable node and replays the client frames buffered during the swap.

  4. De-duplicate

    Blocks the client already received are dropped on replay. The client never reconnects and never sees the same block twice.

The bound

A session that keeps dying on an unclean drop at the same next block backs off from 100 ms, doubling to 3.2 s, and is closed with WebSocket code 1011 and the reason block K unavailable on all upstreams after five such drops. An idle timeout or a clean close still fails over and does not count. The bound is always on. Opt-in block_failure_quarantine additionally skips that upstream for K..K+range after repeated strikes, without marking it offline.

What it does

The legend for the panel above, in the order an operator meets it.

Single resilient endpointws://router:17000
Hyperion, Atomic and other indexers share one WebSocket instead of each consumer pinning itself to one SHiP node.
Range-aware least-connectionstrace_begin_block … trace_end_block
Prefers an upstream whose trace range covers the requested block, then the least-loaded healthy node. Finality-data requests go only to a node whose finality range covers that block.
Upstream preferencepriority · weight
Among nodes that can serve a request equally well, the router uses the lowest priority tier, then the fewest connections per unit of weight. A high priority makes a node a last resort, and failover still moves off a failed preferred node to a backup that covers the block.
Transparent failoverresume at next block
On upstream loss the router reconnects to another suitable node, resumes get_blocks at the next block, de-duplicates replayed blocks and replays buffered client frames. That includes a node that goes silent without closing the connection, such as a pulled cable or a frozen VM: a quiet upstream is pinged and abandoned after upstream_liveness_timeout_ms (4 s by default).
Stale-upstream deprioritizationupstream_stale = 1
An upstream that stops advancing its chain state is flagged stale and deprioritized when routing. It is not hard-excluded.
Many networks, one processlisteners[] · /ready/<name>
listeners serves each chain on its own port with its own upstream pool, chain_id, connection cap and /ready/<name> check.
chain_id verificationrequire_chain_id
An upstream whose reported chain_id disagrees with the pin or the rest of its pool stays offline. Nodes that omit chain_id, such as FIO and other EOSIO 2.0 builds, stay usable unless require_chain_id is set.
Bounded failover and quarantineblock_failure_quarantine
Repeated unclean drops at the same next block are bounded: the session closes with code 1011 after five. Opt-in block_failure_quarantine skips a corrupted SHiP log range without marking the node offline.
Live dashboard/ui/ · /status · /api/v1
Opt-in fleet view with block ranges, sessions, throughput, failovers and an event log, plus a no-JavaScript status page and a JSON API with a server-sent events stream. Off by default.
Operator controlsdrain · disable · move
With a control token set, drain an upstream for maintenance, disable or enable it, change its priority and weight at runtime, clear its quarantines, or move a single session. A drain uses the normal failover, so clients stay connected.
Prometheus metrics/metrics · server, endpoint, listener
Per-upstream series carry server and endpoint labels, plus listener in multi-listener mode, with gauges for the start and end of each SHiP range.
Upstream wss://wss://host:port
Optional TLS to SHiP nodes, verified with rustls against the OS trust store. A bare host:port stays ws://. The client listener remains plaintext.
Resource safetymax_connections · idle_timeout_ms
Connection cap, handshake and idle timeouts, and a bounded maximum WebSocket message size.
Runs as a servicesystemd · Docker · launchd
Ships systemd units for one process or one process per network, with a raised open-file limit and basic sandboxing. The operations guide also covers the Docker restart policy, launchd on macOS and a service wrapper on Windows.
Graceful shutdownSIGINT · SIGTERM
Handles SIGINT and SIGTERM with a bounded drain and WebSocket close frames to connected clients. Written in Rust on tokio and tokio-tungstenite; ABI checks use pure-Rust rs_abieos, so libclang is not required.

Install

Three ways to get the binary. All of them are the same single executable.

crates.io

Requires Rust 1.99 or newer. Building wss:// support needs a C compiler and CMake; they are not required at runtime.

cargo install fleet-router

Docker

Published to GitHub Container Registry on tagged releases. The image runs as a non-root user.

docker pull ghcr.io/eosrio/fleet-router

Prebuilt binaries

Each GitHub Release ships archives for these targets.

  • x86_64-unknown-linux-gnuLinux x86_64
  • aarch64-apple-darwinmacOS arm64
  • x86_64-apple-darwinmacOS x86_64
  • x86_64-pc-windows-msvcWindows x86_64

Quick start

  1. Write a sample config
    fleet-router config init ./config.json
  2. Validate it and test upstream connectivity
    fleet-router config test ./config.json
  3. Run the proxy
    fleet-router run --config ./config.json

To keep it running across reboots, install the systemd unit from deploy/. See Run as a service for systemd, Docker, launchd and Windows.

Two chains, one process

A multi-listener config.json. Each listener is its own port, pool and readiness check. Pin chain_id on every listener.

config.json
{
  "listen_address": "0.0.0.0",
  "upstream_reconnect_ms": 3000,
  "upstream_monitoring_ms": 5000,
  "upstream_status_ms": 5000,
  "metrics_port": 9090,
  "listeners": [
    {
      "name": "eos",
      "listen_port": 17000,
      "chain_id": "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906",
      "servers": [
        { "name": "EOS SHiP 1", "endpoint": "10.0.1.10:8080", "enabled": true },
        { "name": "EOS SHiP 2", "endpoint": "10.0.1.11:8080", "enabled": true, "weight": 2 },
        { "name": "EOS SHiP backup", "endpoint": "10.0.1.12:8080", "enabled": true, "priority": 10 }
      ]
    },
    {
      "name": "wax",
      "listen_port": 17001,
      "chain_id": "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4",
      "servers": [
        { "name": "WAX SHiP 1", "endpoint": "10.0.2.10:8080", "enabled": true },
        { "name": "WAX SHiP 2", "endpoint": "10.0.2.11:8080", "enabled": true }
      ]
    }
  ]
}
listeners[]
One ingress port per network, each with an isolated upstream pool and its own /ready/<name>.
chain_id
Upstreams reporting a different chain_id are kept offline. Older nodes that omit it stay usable unless require_chain_id is set.
endpoint
A bare host:port is dialed as ws://. Use wss://host:port when the node serves TLS.
priority · weight
Optional per server. Lower priority is preferred (default 0); weight (default 1) shares load within a tier. Here SHiP 2 takes twice the sessions of SHiP 1, and the backup serves only when neither can.
*_ms intervals
Reconnect, monitoring and status polling. Lower values detect failures faster at the cost of more polling traffic.
metrics_port
Enables the HTTP endpoint for /health, /ready and /metrics, and for the dashboard when it is enabled. Omit it to run without one.
Single-port mode
Replace listeners with top-level listen_port and servers for one chain. The two modes are mutually exclusive.

Health and metrics

Set metrics_port and the five states the panel shows, plus operator-disabled, become HTTP answers and Prometheus series.

HTTP endpoints
EndpointAnswer
/health200 while the process is running.
/ready200 when at least one upstream is online and not disabled by an operator. In multi-listener mode, when every listener has one. Otherwise 503.
/ready/<name>200 or 503 for that listener, or 404 if the name is unknown.
/metricsPrometheus text exposition of router and upstream state.
/ui/
/status
/api/v1
The dashboard, status page and JSON API, only with dashboard.enabled.

Exported series

Each per-upstream series carries server and endpoint labels, plus listener in multi-listener mode. fleet_router_up is a process gauge with no labels.

fleet_router_up
fleet_router_upstream_up{endpoint,server}
fleet_router_upstream_stale{endpoint,server}
fleet_router_active_connections{endpoint,server}
fleet_router_upstream_trace_begin_block{endpoint,server}
fleet_router_upstream_chain_state_begin_block{endpoint,server}
fleet_router_upstream_chain_state_end_block{endpoint,server}
fleet_router_upstream_finality_data_begin_block{endpoint,server}
fleet_router_upstream_finality_data_end_block{endpoint,server}
fleet_router_upstream_block_failures_total{endpoint,server}
fleet_router_upstream_quarantined{endpoint,server}
fleet_router_upstream_quarantine_block{endpoint,server}
fleet_router_upstream_operator_disabled{endpoint,server}
fleet_router_upstream_priority{endpoint,server}
fleet_router_upstream_weight{endpoint,server}
route set
upstream_up is 1 and the stream is on this node.
failover
An unclean mid-stream drop increments block_failures_total; the session moves to another node.
stale
upstream_stale is 1: chain state stopped advancing. Deprioritized, still online.
offline
upstream_up is 0: unreachable, a missed status deadline, or a chain_id mismatch.
quarantined
upstream_quarantined is 1 and quarantine_block names the skipped block. Online for every other range.
disabled
upstream_operator_disabled is 1: drained or disabled through the control API. Takes no new sessions until enabled.

Logging is tracing, controlled with RUST_LOG (default info).

Live dashboard and controls

Opt-in. Set dashboard.enabled and the metrics port also serves a live fleet view, a plain status page and a JSON API. Add a control token to manage the fleet from the same page.

The fleet-router dashboard listing four upstreams with their block ranges, priority and weight, sessions, throughput, head and failovers. The fleet-router dashboard listing four upstreams with their block ranges, priority and weight, sessions, throughput, head and failovers.
Four upstreams on one listener. Each row shows the node's trace range up to the head, its priority and weight, sessions, throughput and failovers. The Sessions and Events tabs list every client stream and the router's event log.
config.json
"metrics_address": "127.0.0.1",
"metrics_port": 9090,
"dashboard": {
  "enabled": true,
  "control_token_file": "/etc/fleet-router/control.token"
}
Drain
Moves live sessions off an upstream with the normal failover, so clients stay connected. A session moves only to a node whose range covers its next block, so draining the only archive node leaves its history readers in place.
Disable, enable, tune
Take an upstream out of rotation, bring it back, change its priority and weight, or clear its quarantines. Runtime changes are not written back to the config.
Move or close a session
Send one client stream to another upstream at its next block, or close it.
Access
Read views need no token. Controls need the bearer token from control_token_file or FLEET_ROUTER_CONTROL_TOKEN. Dashboard routes answer only to an IP address, localhost or a name in allowed_hosts, which blocks DNS rebinding.

The full contract is in the API reference.