> For the complete documentation index, see [llms.txt](https://docs.ergo.services/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ergo.services/networking/network-stack.md). # Network Stack Understanding the network stack for distributed communication The network stack makes remote messaging work like local messaging. When you send to a process on another node, the framework discovers where that node is, establishes a connection if needed, encodes the message, sends it over TCP, and delivers it to the recipient's mailbox. From your perspective, it's just `Send(pid, message)` - whether the PID is local or remote. This transparency requires three systems working together: service discovery to find nodes, connection management to establish reliable links, and message encoding to serialize data for transmission. Each system handles a specific problem, and together they create the illusion that remote communication is just local communication. ## The Big Picture When you send a message to a remote process: 1. **Routing decision** - The framework examines the node portion of the PID. Local node? Direct mailbox delivery. Remote node? Continue to step 2. 2. **Connection lookup** - Check if a connection to that node already exists. If yes, use it. If no, continue to step 3. 3. **Discovery** - Query the registrar (or check static routes) to find where the remote node is listening: hostname, port, TLS requirements, protocol versions. 4. **Connection establishment** - Open TCP connections to the remote node, perform mutual authentication via handshake, negotiate capabilities, exchange caching dictionaries, create a connection pool. 5. **Message transmission** - Encode the message into bytes (EDF), optionally compress it, wrap it in a protocol frame (ENP), send it over one of the TCP connections in the pool. 6. **Remote delivery** - The receiving node reads the frame, decompresses if needed, decodes back to Go values, routes to the recipient's mailbox. This entire pipeline is invisible to your code. You call `Send`, and the framework does the rest. ## Service Discovery Before connecting to a remote node, the framework needs to know where that node is. Service discovery translates logical node names (`worker@example.com`) into connection parameters (IP, port, TLS, protocol versions). The embedded registrar provides basic discovery: * One node per host runs a registrar server (whoever started first) * Other nodes connect as clients * Same-host discovery is direct (no network) * Cross-host discovery uses UDP queries * Automatic failover if the server node dies For production clusters, external registrars provide more features: * **etcd** - Centralized discovery, application routing, configuration storage; registration held by a lease, changes delivered by a prefix watch * **Saturn** - Purpose-built for Ergo, immediate event propagation, efficient at scale The embedded registrar works for development and small deployments. For larger clusters or dynamic topologies, use etcd or Saturn. The choice is transparent to your code - you specify the registrar at node startup, and everything else works identically. For details, see [Service Discovery](/networking/service-discovering.md). ## Static Routes Discovery is dynamic - nodes register themselves, and others query to find them. But sometimes you want explicit control. Maybe nodes have fixed addresses. Maybe you're behind a firewall that blocks discovery. Maybe you're connecting to external systems. Static routes let you hardcode connection parameters: ```go route := gen.NetworkRoute{ Route: gen.Route{ Host: "10.0.1.50", Port: 4370, TLS: true, }, } network.AddRoute("prod-db@example.com", route, 100) ``` Now when connecting to `prod-db@example.com`, the framework uses your route directly. No discovery query. No registrar involvement. You've taken control. Static routes support pattern matching (`"prod-.*"`), multiple routes with failover weights, and hybrid approaches (use patterns for selection, resolvers for address lookup). You can configure per-route cookies, certificates, network flags, and atom mappings. The framework checks static routes first, always. A matching static route takes that node out of discovery entirely: every matching route is tried, and when they all fail the attempt ends with `gen.ErrNoRoute` - the registrar is never consulted. Discovery is reached only when **no** static route matched the name. For details, see [Static Routes](/networking/static-routes.md). ## Connection Establishment Once the framework knows where to connect (from discovery or static routes), it establishes a connection pool. ### Handshake The handshake performs mutual authentication using challenge-response. Node A connects to node B: 1. A sends hello with random salt and digest (computed from salt + cookie) 2. B verifies digest - if cookies match, digest is correct 3. B sends its own challenge 4. A verifies B's response 5. Both sides authenticated If TLS is enabled, certificate fingerprints are exchanged and verified too. After authentication, nodes exchange introduction messages: * Node names and version information * Network flags (capabilities: remote spawn? important delivery? fragmentation?) * Caching dictionaries (atoms, types, errors that will be used frequently) The flags negotiation ensures nodes with different feature sets can work together. Features not supported by both sides are disabled for that connection. The caching dictionaries enable efficiency. Instead of encoding `"mynode@localhost"` repeatedly (19 bytes), it gets a cache ID and subsequent uses encode as 2 bytes. ### Connection Pool After handshake, the accepting node tells the dialing node to create a connection pool: * Pool size (default 3 TCP connections) * Acceptor addresses to connect to The dialing node opens additional TCP connections using a shortened join handshake (skips full authentication since the first connection already authenticated). These connections join the pool, forming a single logical connection with multiple physical TCP links. Multiple connections enable parallel message delivery. Each message goes to a connection based on the sender's identity, and the receiving side creates multiple receive queues per TCP connection for concurrent processing. This two-level mechanism (sender-side link selection and receiver-side queue routing) preserves per-sender message ordering while enabling parallelism across different senders. For details on how ordering works, including the `KeepNetworkOrder` flag and when to disable it, see [Message Ordering](/networking/network-transparency.md#message-ordering). ### Software Keepalive *Introduced in v3.3.0.* TCP keepalive operates at the OS level - it detects hard network failures like unplugged cables or crashed hosts. But it can't detect application-level problems: a stuck process that stopped reading from a connection, a flusher that failed silently, a goroutine that never got scheduled. The connection looks alive to TCP while no useful data flows. Software keepalive works at the protocol level. When a connection pool item has nothing to send, its flusher periodically writes a small keepalive packet. The receiving side expects these packets and sets a read deadline based on the sender's advertised period. If nothing arrives - no real messages and no keepalive packets - the deadline fires and the connection is terminated. Each side advertises its keepalive period during handshake. This allows asymmetric configuration: a node in a reliable datacenter might send keepalive every 15 seconds, while a node on an unstable network might send every 5 seconds. The receiver calculates its deadline from the sender's period, not its own. ```go node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ Network: gen.NetworkOptions{ Flags: gen.NetworkFlags{ // ... other flags ... EnableSoftwareKeepAlive: 15, // send keepalive every 15 seconds when idle }, SoftwareKeepAliveMisses: 3, // tolerate 3 missed keepalives before disconnect }, }) ``` The timeout calculation uses the remote node's period, not the local one. If the remote node advertises a 15-second period and you configure 3 misses, the connection is considered dead after 45 seconds of silence. Real messages reset the deadline just like keepalive packets do - on a busy connection, keepalive is never sent because regular traffic keeps the deadline from expiring. When a keepalive timeout fires on any pool item, the entire connection is terminated - not just the affected TCP link. A single unresponsive link is strong evidence that the whole network path to the remote node is down. This triggers the standard cleanup flow: monitors receive `MessageDown`, links receive `MessageExit`, and the connection is removed from the node's connection map. Software keepalive is enabled by default (15-second period, 3 misses, 45-second timeout). Set `EnableSoftwareKeepAlive` to 0 to disable it. Acceptors and routes can override the misses count; zero inherits from `NetworkOptions`. Both sides must have keepalive enabled for the feature to activate. If either side advertises period 0, the connection falls back to TCP-only keepalive with infinite read deadline - neither side sends keepalive packets and neither side sets read deadlines. This means a single node with keepalive disabled in a cluster removes protection for all its connections, not just its own. During a rolling upgrade from older nodes (which don't support the feature) to newer ones, connections between old and new nodes will not have software keepalive until both sides are upgraded. ## Message Encoding and Transmission Once a connection exists, messages flow through encoding and framing. ### EDF (Ergo Data Format) EDF is a binary encoding specifically designed for the framework's communication patterns. It's type-aware - each value is prefixed with a type tag (e.g., `0x95` for int64, `0xaa` for PID, `0x9d` for slice). The decoder reads the tag and knows what follows. Framework types like `gen.PID` and `gen.Ref` have optimized encodings. Structs are encoded field-by-field in declaration order (no field names on the wire). Custom types must be registered on both sides via `node.Network().RegisterType` (typically from an application's `Load` callback). During handshake, nodes exchange their type lists to agree on encoding. Compression is opt-in, per process, and off until you ask for it. There is no node-level compression option: the switch is `ProcessOptions.Compression` at spawn, or `SetCompression` at runtime, and only `Enable: true` makes the wire path consider it. Once enabled, a message larger than the threshold (default 1024 bytes) is compressed with GZIP, ZLIB or LZW; the protocol frame says so, and the receiver decompresses before decoding. A process that never enables it sends everything uncompressed however large the message is. For details on EDF - type tags, struct encoding, registration requirements, compression, caching - see [Network Transparency](/networking/network-transparency.md). ### ENP (Ergo Network Protocol) ENP wraps encoded messages in frames for transmission. Each frame has an 8-byte header with magic byte, protocol version, frame length, order byte, and message type. The frame body contains sender/recipient identifiers and the EDF-encoded payload. The order byte preserves message ordering per sender. Messages from the same sender have the same order value and route to the same receive queue, guaranteeing sequential processing. Messages from different senders have different order values and route to different queues, enabling parallel processing. For details on protocol framing, order bytes, receive queue distribution, and the exact byte layout, see [Network Transparency](/networking/network-transparency.md). ### Message Fragmentation *Introduced in v3.3.0.* When a message exceeds the fragment size threshold (default 65000 bytes), the framework splits it into smaller pieces for transmission and reassembles them on the receiving side. This happens after compression; if a compressed message is still too large, it gets fragmented. From your code's perspective, nothing changes. You send a large message, and it arrives intact. Fragmentation works with all message types: regular sends, important delivery, calls, and events. It composes with compression: a message can be compressed first, then fragmented, and on the receiving side defragmented and then decompressed. When [`KeepNetworkOrder`](/networking/network-transparency.md#message-ordering) is disabled for a process, the framework distributes fragments across all TCP connections in the pool, using the full bandwidth of the connection. This is useful for transferring large payloads where throughput matters more than ordering. When `KeepNetworkOrder` is enabled (the default), all fragments travel through a single TCP connection to preserve message ordering for that sender. Both nodes must have `EnableFragmentation` in their network flags. If either side doesn't support it, large messages are sent as-is (subject to `MaxMessageSize` limits). During handshake, nodes exchange their fragmentation capability, and the feature activates only when both sides agree. `MaxMessageSize` is a logical limit on the EDF-encoded message, checked before compression and fragmentation. On the receiving side, the framework tracks the accumulated size of received fragments and rejects the assembly if it exceeds the limit. ```go node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ Network: gen.NetworkOptions{ Flags: gen.NetworkFlags{ EnableFragmentation: true, // default: true }, FragmentSize: 65000, // bytes per fragment, 0 = default FragmentTimeout: 30, // seconds, assembly timeout, 0 = default MaxFragmentAssemblies: 1000, // max concurrent assemblies, 0 = default }, }) ``` `FragmentSize` controls at what point messages get split. This is a sender-side setting; the receiver reassembles whatever arrives regardless of the sender's fragment size. Two nodes can use different fragment sizes. `FragmentTimeout` sets how long the receiver waits for all fragments before discarding an incomplete assembly. If a sender crashes mid-message or a connection drops, partial assemblies are cleaned up after this timeout. `MaxFragmentAssemblies` limits how many messages can be simultaneously reassembled per connection, protecting against memory exhaustion from many concurrent large messages. ## Network Transparency in Practice Network transparency means remote operations look like local operations. You send to a PID without checking if it's local or remote. You establish links and monitors the same way regardless of location. The framework handles discovery, encoding, and transmission automatically. But transparency has limits: * **Latency** - Remote sends take milliseconds vs microseconds for local * **Bandwidth** - Network links have finite capacity, local operations don't * **Failures** - Networks fail in ways local memory doesn't (packets lost, connections drop, nodes unreachable) * **Partial failures** - Some nodes work while others fail (local systems fail entirely or work entirely) The framework makes distributed programming feel local, but you still need to design for network realities: use timeouts, handle connection failures, prefer async over sync, batch messages, keep payloads small. For deep understanding of how transparency works - EDF encoding, struct serialization, type registration, important delivery, failure semantics - see [Network Transparency](/networking/network-transparency.md). ## Network Configuration Configure the network stack in `gen.NodeOptions.Network`: ```go node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ Network: gen.NetworkOptions{ Mode: gen.NetworkModeEnabled, Cookie: "secret-cluster-cookie", MaxMessageSize: 10 * 1024 * 1024, // 10MB Flags: gen.NetworkFlags{ Enable: true, EnableRemoteSpawn: true, EnableRemoteApplicationStart: true, EnableImportantDelivery: true, EnableFragmentation: true, // default: true EnableSoftwareKeepAlive: 15, // seconds, 0 to disable }, SoftwareKeepAliveMisses: 3, // tolerate 3 missed keepalives FragmentSize: 65000, // 0 = default FragmentTimeout: 30, // seconds, 0 = default Acceptors: []gen.AcceptorOptions{ { Port: 15000, PortRange: 10, BufferSize: 64 * 1024, }, }, }, }) ``` **Mode** - `NetworkModeEnabled` enables full networking with acceptors. `NetworkModeHidden` allows outgoing connections only (no acceptors). `NetworkModeDisabled` disables networking entirely. **Cookie** - Shared secret for authentication. All nodes must use the same cookie to communicate. Set explicitly for distributed deployments. **MaxMessageSize** - Maximum incoming message size. Protects against memory exhaustion. Default unlimited (fine for trusted clusters). **Flags** - Control capabilities. Remote nodes learn your flags during handshake and can only use features you've enabled. `EnableRemoteSpawn` allows spawning (with explicit permission per process). `EnableImportantDelivery` enables delivery confirmation. `EnableFragmentation` enables message fragmentation for large messages (both sides must enable). `EnableSoftwareKeepAlive` sets the keepalive period in seconds (see [Software Keepalive](#software-keepalive)). The defaults are all-or-nothing, and this catches people. `gen.DefaultNetworkFlags` is substituted only while `Flags.Enable` is false - the moment you write `Flags: gen.NetworkFlags{Enable: true, ...}`, your literal stands exactly as written and every field you did not name is `false`. So enabling one flag silently turns off fragmentation, important delivery, tracing, clock skew, proxy accept, simultaneous connect, wrapped errors and the 15-second software keepalive. To change one thing, start from the defaults: ```go flags := gen.DefaultNetworkFlags flags.EnableRemoteSpawn = false options.Network.Flags = flags ``` **Acceptors** - Define listeners for incoming connections. Multiple acceptors on different ports are supported. Each can have its own cookie, TLS, and protocol. ## Custom Network Stacks The framework provides four extension points: **gen.NetworkHandshake** - Control connection establishment and authentication. Implement this to change how nodes authenticate or how connection pools are created. **gen.NetworkProto** - Control message encoding and transmission. The Erlang distribution protocol is implemented as a custom proto, allowing Ergo nodes to join Erlang clusters. **gen.Connection** - The actual connection handling. Implement this for custom framing, routing, or error handling. **gen.TypeRegistry** - Optional capability that proto implementations may declare to expose a wire-format type registry. The default ENP/EDF stack implements it. The Erlang distribution proto does not, since the Erlang external term format is schemaless on the wire. When a node has multiple protos configured, `node.Network().RegisterType` distributes registration to every TypeRegistry-capable proto strictly: any per-proto failure fails the call. Protos that do not implement TypeRegistry are skipped silently. You can register multiple handshakes and protos, allowing one node to support multiple protocol stacks simultaneously: ```go node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ Network: gen.NetworkOptions{ Handshake: customHandshake, Proto: customProto, Acceptors: []gen.AcceptorOptions{ {Port: 15000, Proto: ergoProto}, // Ergo protocol {Port: 16000, Proto: erlangProto}, // Erlang protocol }, }, }) ``` This enables migration scenarios (gradually migrate from Erlang to Ergo) and integration scenarios (connect to systems using different protocols). ## Remote Operations Once connections exist, you can spawn processes and start applications on remote nodes: ```go remote, err := node.Network().GetNode("worker@otherhost") if err != nil { return err } pid, err := remote.Spawn("worker_name", gen.ProcessOptions{}) ``` Remote spawning requires the remote node to explicitly enable it: ```go // On the remote node node.Network().EnableSpawn("worker_name", createWorker) ``` Without explicit permission, remote spawn requests fail. This prevents arbitrary code execution. The same pattern applies to starting applications: ```go remote.ApplicationStart("myapp", gen.ApplicationOptions{}) ``` Requires: ```go node.Network().EnableApplicationStart("myapp") ``` This security model ensures you control exactly what remote nodes can do on your node. ## Where to Go Next This chapter provided an overview of how the network stack operates. For deeper understanding: * [**Service Discovery**](/networking/service-discovering.md) - How nodes find each other, application routing, configuration management, embedded vs external registrars * [**Network Transparency**](/networking/network-transparency.md) - How messages are encoded, EDF details, protocol framing, compression, caching, important delivery * [**Static Routes**](/networking/static-routes.md) - Explicit routing configuration, pattern matching, failover, proxy routes Each of these chapters dives deep into its specific topic, giving you the details needed for production deployments.