fleet-router

Documentation / Configuration · v0.8.1

Configuration reference

fleet-router reads one JSON file (./config.json by default; pass another with --config). Run fleet-router config init <path> to write a sample and fleet-router config test <path> to validate it and test upstream connectivity.

A config uses one of two modes:

Each listener presents its upstreams in one of two topologies: pool (identical full nodes, the default) or stitched (live nodes plus frozen range servers, presented to clients as one SHiP node). See Topology and stitching.

All *_ms fields are in milliseconds.

View source on GitHub

On this page

Reloading configuration

With the dashboard and a control token enabled, open Configuration and unlock controls:

Both routes validate the whole configuration and reserve new listening ports before applying it. A validation error or occupied port leaves the running configuration unchanged; a failed save also leaves the file unchanged. Stale browser drafts and intervening file edits return a conflict instead of overwriting a newer revision. The configuration source is limited to 1 MiB. File edits are applied explicitly; there is no automatic file watcher.

Listeners and upstreams can be added, removed or replaced, including chain API and dashboard ports. New connections use the new configuration. Established SHiP streams retain their previous upstream pool, failover policy and limits until they disconnect; established chain API connections likewise retain their previous policy and pool. Removing a listener stops new connections and releases its port. Its monitors remain while established connections need them, then stop. Configuration and Sessions show these connections as draining. There is no forced drain deadline, so long-lived streams can retain an old pool indefinitely. Keep removed upstreams available if those streams still depend on them.

Unchanged listeners keep their running state. Connection caps on a listener include its draining generations; lowering a cap rejects new connections until enough old ones finish. For a retained SHiP endpoint with the same listener and chain policy, upstream connection counts and cumulative counters survive replacement, including removal and re-addition while old streams remain. A renamed listener is a new identity. Other request limits remain specific to each connection's generation.

The editor reads the file selected at startup by --config (resolved to an absolute path). Saving uses an atomic replacement in the same directory; the service account needs permission to create and replace files there. Existing file permissions are retained. In container deployments, mount a writable configuration directory rather than a single bind-mounted file if dashboard saving is needed. A read-only file can still be reloaded after an external edit. A conflicting address change, such as replacing a wildcard bind with a specific address on the same occupied port, is refused: use a free port or restart for that change.

Inline dashboard.control_token values appear as __FLEET_ROUTER_KEEP_SECRET__; leave this placeholder unchanged to retain the saved token. Token files and environment tokens are resolved again on apply. If you change the dashboard address or token, reconnect or unlock with the new settings. Disabling the dashboard or removing its metrics endpoint also removes the reload API; re-enable it through a process restart. Existing event streams close when the dashboard is disabled or their hostname is no longer allowed.

The configuration API exposes the same validation, revision checks and apply path.

Minimal example

json
{
  "listen_address": "0.0.0.0",
  "listen_port": 17000,
  "upstream_reconnect_ms": 3000,
  "upstream_monitoring_ms": 5000,
  "upstream_status_ms": 5000,
  "servers": [
    {
      "name": "SHiP Node 1",
      "endpoint": "127.0.0.1:18080",
      "enabled": true
    },
    {
      "name": "SHiP Node 2",
      "endpoint": "127.0.0.1:28080",
      "enabled": true
    }
  ]
}

Top-level fields

Listener and upstreams

Polling intervals

Lower values detect failures and staleness faster at the cost of more polling traffic to upstreams; raise them to reduce chatter.

Chain verification

A status that omits chain_id stays online unless require_chain_id is set. On get_status_result_v0 that field is an optional extension, and older nodeos builds (FIO mainnet/testnet and other EOSIO 2.0-based nodes) never send it. In that case the router logs one warning per upstream that chain_id cannot be verified. The warning is cleared if a later status reports a chain_id, and logged again on the next omission.

Failure handling

Limits and timeouts

Health and metrics endpoint

Dashboard

The control API (drain, enable/disable, live priority and weight, clear quarantine, move or close a session) is on only when a token is configured. The FLEET_ROUTER_CONTROL_TOKEN environment variable overrides both fields. A token must be at least 16 characters; generate one with openssl rand -hex 32. Without a token the dashboard is read-only.

json
"metrics_address": "127.0.0.1",
"metrics_port": 9090,
"dashboard": {
  "enabled": true,
  "control_token_file": "/etc/fleet-router/control.token"
}

Read endpoints and the dashboard need no token, and they show client IP addresses and upstream addresses. Keep metrics_address on a private interface or behind a reverse proxy with access control. The router logs a warning when the control API is enabled on a non-loopback address. Changes made through the control API are not written back to this file.

Upstream servers

Each entry in servers is an object:

Uniqueness is the dial target: 127.0.0.1:8080 and ws://127.0.0.1:8080 are the same upstream, and so are ws://host and ws://host:80. ws:// and wss:// to the same host are different. Unknown schemes (https://, http://, …) are rejected.

Upstream preference

By default every upstream is equal and new sessions go to the one with the fewest active connections. Use priority and weight to change that.

Preference never overrides whether an upstream can serve the request. For each new session and each failover in a pool listener, the router ranks candidates in this order (a get_blocks stays on the session's upstream when it covers the start block and does not lag, and otherwise moves to the best of these that serves it better):

  1. Enabled, not disabled by an operator, below max_sessions, not quarantined for the requested block (always required), and online (on failover, an offline upstream is a last resort).
  2. Covers the requested block and is advancing, then covers but stale or lagging (see live_lag_blocks), then does not cover but advancing, then neither. Covering means having the block in every log the request asks for (traces, chain state for deltas, finality data) that some online upstream has at that block. A live node covers every block from its first one on, including the next block at the head.
  3. Lowest priority.
  4. Fewest connections per unit of weight, then the higher weight.
  5. A hash of the session and the endpoint, so equal upstreams are picked evenly and the same way for a session.

So a preferred node that is stale or missing the requested block loses to a backup node that has it. When the upstream serving a session fails, failover moves to another upstream that covers the block as well, even if that upstream is in a lower-preference tier.

A stitched listener picks the upstream for each leg of a request (see Range-split fleets) in this order: online; fresh and not lagging; below max_sessions; for a request with fetch_block, not known to lack a block log; the role history_preference prefers; not the upstream the session just failed over from; lowest priority; the longest run of blocks it can serve from the cursor; fewest connections per unit of weight, then the higher weight; then the hash. Priority and weight come after the role preference and capacity, so neither pulls history reads onto a live node or a session onto a full upstream. With the default priority and weight, the longest run decides, which saves handoffs, and weight splits sessions among replicas of one range.

Priority is applied when a session is placed. Sessions already on a backup upstream stay there when a preferred upstream comes back; new sessions go to the preferred one.

json
"servers": [
  { "name": "local-1",  "endpoint": "10.0.0.10:8080",         "enabled": true, "weight": 2 },
  { "name": "local-2",  "endpoint": "10.0.0.11:8080",         "enabled": true },
  { "name": "provider", "endpoint": "wss://ship.example:443", "enabled": true, "priority": 100 }
]

Here local-1 and local-2 share all traffic about 2:1, and provider is used only when neither local node can serve a request.

/metrics reports each upstream's settings as fleet_router_upstream_priority and fleet_router_upstream_weight.

Topology and stitching

topology decides how a listener presents its upstreams to clients:

These fields go on each listener, or at the top level in single-port mode:

Two global fields matter most in a stitched listener, although they apply to pool listeners too: upstream_stall_timeout_ms and live_lag_blocks.

A minimal stitched listener, with two replicas of one archived range and two live nodes:

json
{
  "name": "main",
  "listen_port": 17000,
  "chain_id": "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906",
  "servers": [
    { "name": "archive-a", "endpoint": "10.0.3.10:8080", "enabled": true, "role": "frozen", "range": { "begin": 2, "end": 50000001 }, "max_sessions": 64 },
    { "name": "archive-b", "endpoint": "10.0.3.11:8080", "enabled": true, "role": "frozen", "range": { "begin": 2, "end": 50000001 }, "max_sessions": 64 },
    { "name": "live-1",    "endpoint": "10.0.1.10:8080", "enabled": true },
    { "name": "live-2",    "endpoint": "10.0.1.11:8080", "enabled": true }
  ]
}

example.stitched.config.json is a complete file with two archived ranges and a chain API.

Validation rejects:

These are warnings, logged at startup and printed by config test:

fleet-router config test --strict <path> also probes every server and prints the coverage each stitched listener adds up to, and exits 1 on a hole in the trace or chain-state coverage, a frozen server that did not report its ranges, or (with a chain API) archived blocks that no frozen server with an api_endpoint covers.

Chain API

chain_api opens an HTTP port in front of a listener's servers that answers like a nodeos chain API: block endpoints are routed by block number to the frozen range server or live node that holds the block, and get_info and read-only state calls go to live nodes. It is off when omitted. Each server takes part through its api_endpoint. See Chain API for how requests are routed and answered.

json
"chain_api": { "listen_port": 18888, "listen_address": "10.0.0.5" }

Set it on each listener, or at the top level in single-port mode. An older fleet-router ignores the field without an error.

A chain_api needs at least one enabled server with api_endpoint. Three more settings apply to it: handshake_timeout_ms bounds a client's request head and body (10 s when 0) and closes idle keep-alive connections; live_lag_blocks drops laggards from state calls; and an upstream connect is bounded like a session's upstream connect.

These are warnings, logged at startup and printed by config test:

config test prints each chain API listener's servers with their role and api_endpoint, with a warning for each enabled frozen server that has none. With --strict it exits 1 when archived blocks of the listener's coverage (the trace and chain-state ranges of its enabled frozen servers) have no enabled frozen server with an api_endpoint covering them, so that only live nodes could answer them. A frozen server without a block log cannot serve a chain API at all; it passes --strict as long as a replica with an api_endpoint covers its range.

Upstream TLS (wss://)

wss:// is dialed with rustls (not OpenSSL). The router verifies the upstream certificate against the operating-system trust store. The published container image installs CA certificates, so public CAs work there. For a private CA, install it into that store, or set SSL_CERT_FILE / SSL_CERT_DIR to PEM certificates. When either variable is set, those paths replace the OS store.

Client TLS termination

Client-facing listeners default to plaintext (ws:// for SHiP and http:// for chain_api). Set tls at the top level, on a listener in listeners, or on chain_api to terminate TLS directly in fleet-router (wss:// and https://) using tokio-rustls:

json
"tls": {
  "cert_file": "/etc/fleet-router/tls/fullchain.pem",
  "key_file": "/etc/fleet-router/tls/privkey.pem",
  "sni": [
    {
      "server_names": ["api.wax.example.com", "*.wax.example.com"],
      "cert_file": "/etc/fleet-router/tls/wax-fullchain.pem",
      "key_file": "/etc/fleet-router/tls/wax-privkey.pem"
    }
  ]
}

All configured PEM files are validated at startup (and by fleet-router config test) and monitored every 5 seconds in the background. When a certificate or private key file is renewed on disk, fleet-router atomically reloads it without dropping active connections. Each certificate/key pair reloads independently: if one fails (for example while a renewal script is mid-write), its previous valid certificate stays active and a warning is logged, while valid renewals for other domains still take effect. Failed pairs are retried on the next poll.

Client message limits

Client requests are capped by max_client_message_bytes (default 64 KiB) on both the WebSocket message size and the frame size. Upstream block payloads still use max_message_bytes.

A client frame that is not a valid SHiP request closes the session: a malformed request with WebSocket close code 1007 (invalid request). A request variant the router does not know is forwarded to the upstream in a pool listener, for nodes newer than the router; a stitched listener closes the session with code 1003 (unsupported request variant <n>).

While no upstream link is up (during a failover), acks are added to the client's credit and status requests are queued; neither is buffered. Only those unknown-variant frames are buffered, up to 64 KiB per session. Each buffered frame counts as at least one byte, so empty frames cannot grow the queue without a bound. Exceeding that cap closes the session with WebSocket close code 1008 (policy violation). More than 1024 status requests waiting for an answer also close it with 1008.

Multiple networks

To expose several ingress ports for different chains, each backed by its own redundant SHiP pool, you can either:

  1. Use listeners in one fleet-router process. Each entry binds its own port and keeps an isolated upstream pool, monitor loop, and connection semaphore.
  2. Run one fleet-router instance per network for process or container isolation. See deploy/docker-compose.multi-network.yml and deploy/fleet-router@.service.

Listener fields

Always pin chain_id in multi-network setups. When a listener's chain_id is omitted, fleet-router logs a startup warning and trusts the first currently-online upstream in the pool as the reference chain_id. If a misconfigured wrong-chain node answers first, the remaining nodes are rejected until it goes offline.

Example

json
{
  "listen_address": "0.0.0.0",
  "upstream_reconnect_ms": 3000,
  "upstream_monitoring_ms": 5000,
  "upstream_status_ms": 5000,
  "metrics_port": 9090,
  "listeners": [
    {
      "name": "eos",
      "listen_port": 17000,
      "chain_id": "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906",
      "servers": [
        { "name": "EOS SHiP 1", "endpoint": "10.0.1.10:8080", "enabled": true },
        { "name": "EOS SHiP 2", "endpoint": "10.0.1.11:8080", "enabled": true }
      ]
    },
    {
      "name": "wax",
      "listen_port": 17001,
      "chain_id": "1064487b3cd1a897ce03ae5b6a865651747e2e152090f99c1d19d44e01aea5a4",
      "servers": [
        { "name": "WAX SHiP 1", "endpoint": "10.0.2.10:8080", "enabled": true },
        { "name": "WAX SHiP 2", "endpoint": "10.0.2.11:8080", "enabled": true }
      ]
    }
  ]
}