1 of 64

Ergo Framework Documentation

Overview

Building reliable concurrent and distributed systems is hard. In Go, you might start with goroutines and channels. As the system grows, you add mutexes to protect shared state. Then you need to coordinate across multiple services, so you introduce message queues or RPC. Before long, you're managing synchronization primitives, handling partial failures, and debugging race conditions that only appear under load.

Ergo Framework offers a different foundation. Think of it as making goroutines addressable and message-passing-only, then extending that model across a cluster. Processes are like goroutines - lightweight, multiplexed onto OS threads - but isolated and communicating only through messages. Each process has an identifier that works whether the process is local or on a remote node. Sending a message looks the same either way.

The actor model isn't new. Erlang proved these patterns work for systems requiring massive concurrency and high reliability. Ergo brings them to Go: no external dependencies, familiar Go idioms, and performance that doesn't sacrifice correctness for speed.

Core Components

The framework consists of a few fundamental pieces that work together.

A node provides the runtime environment. It manages process lifecycles, routes messages, handles network connections, and provides services like logging and scheduled tasks. When you start a node, you get infrastructure. When you spawn a process, the node handles the mechanics.

Processes are lightweight actors. Each has a mailbox where messages queue up, priority-sorted into urgent, system, main, and log queues. The process handles messages one at a time in its own goroutine. When the mailbox empties, the goroutine sleeps. This makes processes efficient - you can have thousands without resource problems. It also makes them safe - sequential message handling means no race conditions within a process.

Supervision trees provide fault tolerance. Supervisors monitor worker processes. When a worker crashes, the supervisor restarts it according to a configured strategy. Supervisors can supervise other supervisors, creating a hierarchy. Failures are isolated to subtrees. The rest of the system continues running while the failed part recovers.

Meta processes solve a specific problem: integrating blocking I/O with the actor model. HTTP servers block waiting for requests. TCP servers block accepting connections. A meta process uses two goroutines - one runs your blocking code (like http.ListenAndServe), the other handles messages from other actors. This bridges synchronous APIs with asynchronous actor communication.

Network Transparency

The framework treats local and remote processes identically. Send a message to a process on the same node or a process on a remote node - the code is the same. The framework handles the difference.

When you send to a remote process, the node extracts the target node from the process identifier, discovers that node's address (through static routes or a registrar), establishes a connection if needed, encodes the message, and sends it. The remote node receives it, decodes it, and delivers it to the target process's mailbox. This happens automatically. Your code just sends a message.

This transparency extends to failure detection. Use the Important delivery flag and you get the same error semantics for remote processes as for local ones. Without it, a message to a missing remote process times out (was it slow or dead?). With it, you get immediate error notification (process doesn't exist), just like local delivery. The network becomes transparent not just for success cases but for failures too.

Nodes discover each other through a registrar. By default, each node runs a minimal registrar. Nodes on the same host find each other through localhost. For remote nodes, the framework queries the registrar on the remote host. For production clusters, configure an external registrar like etcd or Saturn for centralized discovery, cluster configuration, and application deployment tracking.

What This Enables

You write business logic using message passing between processes. The framework handles concurrency (processes run in parallel but each is sequential internally), fault tolerance (supervisors restart failures), and distribution (messages route automatically to remote processes). You're not writing code to manage connections, encode messages, or handle network failures explicitly. Those are solved problems handled by the framework.

Systems built this way have useful properties. They scale by adding nodes and distributing processes across them. The code doesn't change - deployment topology is operational configuration. They handle failures through supervision rather than defensive programming everywhere. They evolve through composition - add new process types, adjust supervision strategies, change message flows - without restructuring the foundation.

The development experience differs from typical microservices. No REST endpoints to define. No service discovery to configure (it's built in). No serialization libraries to manage (the framework handles it). No retry logic scattered throughout (supervision handles recovery). You model your domain as processes exchanging messages, and the framework provides the infrastructure.

Performance

Lock-free queues in process mailboxes avoid contention. Processes sleep when idle, consuming no CPU. Connection pooling uses multiple TCP connections per remote node for parallel delivery. These design choices add up to performance comparable to hand-written concurrent code, but without the complexity.

The real performance benefit is development velocity. You're not debugging race conditions or deadlocks. You're not coordinating distributed transactions. You're not managing connection pools or implementing retry logic. The framework handles those concerns, leaving you to focus on what your system does.

Benchmarks measuring message passing, network communication, and serialization performance are available at .

Zero Dependencies

The framework uses only the Go standard library. No external dependencies means no version conflicts, no supply chain vulnerabilities, no surprise breaking changes from third-party packages. The requirement is just Go 1.20 or higher.

This isn't ideological purity. It's practical stability. The framework's behavior depends only on Go itself. Updates are predictable. Supply chain is simple. The code you write today will compile and run the same way years from now, assuming Go maintains backward compatibility (which it does).

For detailed explanations of these concepts, start with and explore the section. For API documentation, see the godoc comments in the source code.

Basics

Actor Model

The Actor Model and Its Properties

The actor model is a computational approach to building concurrent systems, first proposed in the 1970s. At its core is a simple yet powerful idea: instead of having program components share memory and coordinate through locks, they communicate by sending messages to each other.

The Fundamental Concept

In the actor model, everything is an actor. An actor is an independent entity that has its own private state and processes incoming messages one at a time. Actors never directly access each other's state. Instead, they send messages and wait for responses if needed.

This might seem like a constraint, but it's actually what makes the model powerful. By eliminating shared state, we eliminate entire classes of concurrency bugs that plague traditional multi-threaded programs.

Supervision Tree

Process control and fault tolerance

Building reliable systems means accepting an uncomfortable truth: failures will happen. Hardware fails. Networks partition. Bugs exist in code. The question isn't whether your processes will crash, but what happens when they do.

The supervision tree model provides an answer. Instead of trying to prevent all failures, you structure your system so failures are expected, isolated, and automatically recovered from.

The Supervision Principle

The model divides processes into two distinct roles:

Workers do the actual work. They handle requests, process data, manage state, and inevitably, sometimes crash when things go wrong.

Supervisors watch over workers. Their only job is to start child processes and restart them when they fail. Supervisors don't do application work - they manage lifecycle.

This separation is crucial. If workers handled their own restart logic, a bug in that logic would prevent recovery. By moving restart responsibility to a separate supervisor, you ensure that failures in workers can always be recovered.

How Supervision Works

A supervisor starts its children and monitors them. When a child crashes, the supervisor decides what to do based on its restart strategy. Should it restart just this one child? Restart all children? Restart all children in a specific order?

The strategy depends on the relationships between children. If they're independent, restart just the failed one. If they depend on each other, restart all of them to ensure consistent state. If they have startup dependencies, restart in order.

Supervisors can supervise other supervisors, forming a tree. At the top might be an application supervisor. Below it, supervisors for different subsystems. Below those, the actual workers. When a worker crashes, only its portion of the tree is affected. The rest of the system continues running.

Fault Tolerance Through Isolation

This tree structure creates fault isolation boundaries. A crashed database worker doesn't affect the HTTP handler workers. A failed cache process doesn't take down the authentication processes. Each supervision subtree handles its own failures without cascading them upward.

The Erlang community calls this "let it crash." It sounds reckless, but it's actually disciplined. Instead of defensive programming trying to handle every possible error, you let processes fail and rely on supervisors to restart them in a clean state. Often, a fresh restart clears transient problems that would be difficult to handle explicitly.

Supervision in Ergo Framework

Ergo Framework implements supervision through the act.Supervisor actor. When you create a supervisor, you specify its children and restart strategy. The framework handles the monitoring and restart logic.

Workers are typically act.Actor implementations - regular actors that do application work. Supervisors are act.Supervisor implementations - actors whose behavior is managing children.

Because supervisors are also actors, they can be supervised. This is how you build the tree: supervisors supervising supervisors supervising workers, all the way down.

The tree structure emerges from how you compose supervisors and workers. There's no special tree-building API. You just nest supervisors, and the tree forms naturally.

Building Reliable Systems

The supervision tree model leads to systems with interesting properties.

Self-healing - Failures trigger automatic recovery. Most transient problems resolve themselves through restart.

Graceful degradation - When a subsystem fails, only that part stops working. The rest continues serving requests.

Operational simplicity - Instead of complex error handling throughout your code, you centralize recovery logic in supervisors.

The trade-off is that you need to design processes that can restart cleanly. State that must survive restarts needs to be externalized - in databases, in other processes, or rebuilt from messages. But this discipline leads to more robust designs anyway.

Where to Go From Here

Understanding supervision requires seeing it in practice. The chapter covers the specifics: restart strategies, child specifications, and practical patterns for structuring your application.

The combination of the actor model (isolated processes, message passing) and supervision trees (automatic recovery) gives you the tools to build systems that handle failures gracefully. It's a different approach than traditional error handling, but one that scales well to distributed systems where failures are inevitable.

Generic Types

Data Types and Interfaces Used in Ergo Framework

Ergo Framework uses several specialized types for identifying and addressing processes, nodes, and other entities in the system. Understanding these types is essential for working with the framework.

Identifiers and Names

Node

What is a Node in Ergo Framework?

A node is the runtime environment where your actors live. Think of it as the container that hosts processes, routes messages between them, and handles the complexities of distributed communication.

When you start a node, you're launching a complete system with several subsystems working together: process management, message routing, networking, and logging. Each subsystem has a specific responsibility, and they coordinate to provide the foundation for your application.

What a Node Provides

Process Management - The node tracks every process running on it. When you spawn a process, the node assigns it a unique PID, registers it in the process table, and manages its lifecycle. When a process terminates, the node cleans up its resources and notifies any processes that were linked or monitoring it.

Process

What is a Process in Ergo Framework

A process is an actor - a lightweight entity that handles messages sequentially in its own goroutine. It's the fundamental building block of an Ergo application.

Every process has a mailbox where incoming messages wait to be processed. The mailbox contains four queues with different priorities: Urgent for critical system messages, System for framework control, Main for regular application messages, and Log for logging. When the process wakes up to handle messages, it processes them in priority order, taking from Urgent first, then System, then Main, and finally Log.

The process runs only when it has messages to handle. When the mailbox is empty, the process sleeps, consuming no CPU. When a message arrives, the process wakes, handles the message, and sleeps again if nothing else is waiting. This efficiency is why you can have thousands of processes in a single application.

Links And Monitors

Linking and Monitoring Mechanisms

Building reliable systems from independent processes requires solving a fundamental coordination problem. When a process terminates - whether from a crash, graceful shutdown, or network failure - other processes that depend on it or supervise it need to know. Without this knowledge, a supervisor can't restart failed workers, dependent processes continue attempting to use unavailable services, and the system degrades silently.

The challenge is detecting termination without breaking isolation. Processes can't share memory or directly observe each other's state. The traditional approach in distributed systems uses heartbeats: processes periodically signal they're alive, and silence implies failure. But heartbeats introduce overhead, timing sensitivity, and the fundamental ambiguity of distinguishing "slow" from "dead."

Ergo Framework provides a different mechanism. Processes explicitly declare relationships - links and monitors - and the framework delivers termination notifications through these channels. When a process terminates, the node automatically notifies all processes that established relationships with it. The notification is immediate, deterministic, and part of the normal message flow.

Links and monitors both deliver termination notifications, but they differ in what happens next. A link couples your lifecycle to the target's - when it terminates, you terminate. A monitor simply informs you of termination, leaving the response up to you. The choice depends on whether you need failure propagation or just failure awareness.

Events

Publish/Subscribe Event Mechanism

The actor model excels at point-to-point communication. Process A sends a message to process B. Process C makes a request to process D. Each interaction has a specific sender and receiver.

But some scenarios need one-to-many communication. A price feed updates and dozens of trading strategies need the new price. A user logs in and multiple subsystems need notification. A sensor reading arrives and various monitoring processes need to react. You could send individual messages to each interested process, but then the producer needs to track all consumers. When consumers come and go, the producer's consumer list becomes a maintenance burden.

Events solve this with publish/subscribe semantics. A producer registers an event and publishes values to it. Consumers subscribe to the event without the producer knowing who they are. The framework handles message distribution - when the producer publishes an event, all current subscribers receive it. Subscribers can come and go dynamically, and the producer's code doesn't change.

Registering Events

A process becomes an event producer by calling RegisterEvent with an event name and options. The call returns a token - a unique reference that proves ownership. Only the process holding this token (or a process it delegates to) can publish events under this name.

The Notify option controls whether the producer receives notifications about subscriber changes. When enabled, the producer receives gen.MessageEventStart when the first subscriber appears and gen.MessageEventStop when the last subscriber leaves. This allows the producer to start or stop expensive operations based on demand. If nobody's watching the price feed, why fetch prices?

The Buffer option specifies how many recent events to keep. When a new subscriber joins, it receives the buffered events as a catch-up mechanism. Set this to zero if events are only relevant at the moment they're published. Set it to a reasonable number if new subscribers should see recent history.

Events are identified by name and node. The combination must be unique. Two processes on the same node can't register events with the same name. But processes on different nodes can register events with the same name - they're different events.

Publishing Events

Publishing an event sends it to all current subscribers.

You pass your application data directly. The framework wraps it in gen.MessageEvent automatically, adding the event identifier and timestamp. Subscribers receive the complete gen.MessageEvent structure containing your data.

The producer uses the token obtained during registration. If you try to publish with an incorrect token, the operation fails. This prevents unauthorized processes from publishing events they don't own.

Event publishing is fire-and-forget. The producer doesn't wait for acknowledgment or know how many subscribers received the event. The framework handles distribution asynchronously.

Subscribing to Events

Processes subscribe to events through links or monitors, the same mechanisms used for process lifecycle tracking.

LinkEvent creates a link to an event. You receive event messages as they're published. If the event producer terminates or unregisters the event, you receive an exit signal. The link semantics apply - by default, you'd terminate too.

MonitorEvent creates a monitor on an event. You receive event messages and a down notification if the producer terminates or the event is unregistered, but you don't terminate automatically.

Both methods return buffered events upon successful subscription:

The buffered events let subscribers catch up on what happened before they joined. If the buffer size was 10 and 5 events have been published, new subscribers receive those 5 events immediately.

For local events, you can omit the node name: gen.Event{Name: "price_update"}. The framework fills in the local node name. For remote events, specify the full event identifier including the remote node name.

Event Lifecycle

Events exist from registration until unregistration or producer termination.

When you register an event, it becomes available for subscription. Processes on any node can subscribe if they know the event name and node. The framework tracks all subscribers and distributes published events to them.

When the producer terminates, the event is automatically unregistered. All subscribers receive termination notifications (exit signals for links, down messages for monitors). The event name becomes available for registration again.

The producer can explicitly unregister an event with UnregisterEvent. This triggers the same notifications to subscribers. Use this when you're done publishing events but your process continues running.

If a subscriber terminates or unsubscribes (via UnlinkEvent or DemonitorEvent), the producer doesn't receive notification unless Notify was enabled. With Notify, the producer receives gen.MessageEventStop when the last subscriber leaves.

Network Transparency

Events work across nodes seamlessly. A producer on node A can publish events that subscribers on nodes B, C, and D receive. The framework handles the network distribution.

When you subscribe to a remote event, the framework sends a subscribe request to the remote node. The remote node records your subscription. When the producer publishes an event on the remote node, the remote node sends it to all remote subscribers, including you.

If the network connection fails, subscribers receive termination notifications with reason gen.ErrNoConnection. This is consistent with how links and monitors handle network failures for processes.

The buffered events work across nodes too. When you subscribe to a remote event, the remote node sends you the buffered events as part of the subscription response. This catch-up mechanism works regardless of where the producer and subscribers are located.

Token Delegation

Event tokens can be delegated. The producer can give its token to another process, allowing that process to publish events under the producer's event registration.

This enables patterns where event generation is separated from event registration. A coordinator registers the event and distributes the token to worker processes. Workers publish events as data becomes available. Subscribers don't know or care which process instance published each event - they just receive events on the registered event name.

Token delegation also allows rotating producers. A primary process registers an event and holds the token. A backup process can take over using the same token if the primary fails. Subscribers see a continuous event stream even as the producing process changes.

Event Messages

Event messages have a specific structure:

Each gen.MessageEvent contains:

Event - The event identifier (name and node)
Message - Your application data (any type)
Timestamp - When the event was published (nanoseconds since epoch)

Subscribers receive these wrapped messages and extract the application data. The wrapping provides context: which event this came from, when it was published, allowing subscribers to handle events from multiple sources or correlate timing.

Practical Patterns

Events fit several common scenarios.

Data streaming - A sensor process registers an event and publishes readings. Multiple monitoring processes subscribe. Each reading goes to all monitors. If a monitor crashes and restarts, it subscribes again and receives recent buffered readings to catch up.

State change notification - A user session process registers an event and publishes state changes (login, logout, permission change). Authorization processes subscribe and update their caches. The session process doesn't track who's interested in its state changes.

System telemetry - Processes publish metrics as events. Monitoring processes subscribe and aggregate. If the monitoring process restarts, buffered events provide recent history to rebuild state.

Workflow coordination - An order processing system publishes order state events. Inventory, shipping, and billing processes subscribe. Each subsystem reacts to relevant state changes. The order process doesn't orchestrate the subsystems - they coordinate through events.

For more information on links and monitors as they apply to processes and nodes, see the chapter.

Cron

Schedule tasks on a repetitive basis

Applications often need tasks to run periodically. Generate a daily report at midnight. Clean up expired sessions every hour. Send weekly summary emails. Poll an external API every five minutes.

You could implement this yourself - spawn a process that sleeps, wakes up, performs the task, and sleeps again. But then you're managing wake times, handling timezone changes, accounting for daylight saving time transitions, and ensuring the scheduler itself stays alive. The scheduling logic becomes scattered across your application.

Cron provides scheduled task execution as a framework service. You declare what should run and when using the familiar crontab syntax. The framework handles timing, execution, and all the edge cases around time-based scheduling.

CertManager

TLS Certificate Management

Network communication in production systems needs encryption. TLS provides this, but managing TLS certificates introduces operational challenges. Certificates expire. Security incidents require rotation. Updating certificates traditionally means restarting services, causing downtime.

The naive approach loads certificates at startup from files. When you need to update a certificate, you replace the file and restart the service. For a single service, this works. For distributed systems with dozens of nodes and services, coordinating restarts for certificate updates becomes an operational burden.

Ergo Framework provides gen.CertManager for live certificate updates. Load a certificate at startup, and you can update it later without restarting. All components using that certificate manager - node acceptors, web servers, TCP servers - automatically use the updated certificate for new connections.

Actors

Meta Processes

Networking

Mutual TLS

Mutual TLS authentication between nodes

Standard TLS provides server authentication - the client verifies the server's certificate. Mutual TLS (mTLS) adds client authentication - both sides present and verify certificates. Only clients with certificates signed by a trusted CA can connect.

Configuration

NodeOptions.CertManager is used for:

Default acceptor (created automatically on port 15000)
All outgoing connections

To override per-acceptor, use AcceptorOptions.CertManager.

CertAuthManager

gen.CertAuthManager extends CertManager with CA pool and authentication settings:

Server-side settings:

Setting

Purpose

ClientAuth values:

Value

Behavior

Client-side settings:

Setting

Purpose

Runtime Certificate Rotation

Certificates can be rotated without restart:

New connections use the updated certificate. Existing connections keep their original certificate.

CA pools and ClientAuth are fixed at startup. Restart the node to change these settings.

To use different certificates for specific destinations, see .

Troubleshooting

Connection rejected with certificate error

Verify the client certificate is signed by a CA in the server's ClientCAs pool. Check certificate expiration dates.

Server certificate verification failed

The server's certificate must be signed by a CA in the client's RootCAs pool. For development, disable verification with NetworkOptions.InsecureSkipVerify: true.

SNI mismatch

Set ServerName on the client's CertAuthManager if the certificate's Common Name doesn't match the connection address.

Certificate rotation not taking effect

Updates apply to new connections only. Close existing connections to force reconnection with new certificate.

CA pool changes not taking effect

CA pools are fixed at startup. Restart the node to apply changes.

Testing

Advanced

extra library

Actors

An extra library of actors implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy.

Leader

Distributed leader election actor implementing Raft-inspired consensus algorithm. Provides coordination primitives for building systems that require single leader selection across a cluster.

Use cases: Task schedulers, resource managers, single-writer databases, distributed locks, cluster coordinators.

Prometheus metrics exporter actor that automatically collects and exposes Ergo node and network telemetry via HTTP endpoint. Provides observability for monitoring cluster health and resource usage.

Use cases: Production monitoring, performance analysis, capacity planning, debugging distributed systems.

Applications

The additional application library for Ergo Framework contains packages with a narrow specialization or external dependencies since Ergo Framework adheres to the "zero dependencies" principle.

You can find the source code of these applications in the application library repository at https://github.com/ergo-services/application.

Observer

The Application Observer provides a convenient web interface to view node status, network activity, and running processes in the node built with Ergo Framework. Additionally, it allows you to inspect the internal state of processes or meta-processes. The application is can also be used as a standalone tool Observer. For more details, see the section Inspecting With Observer. You can add the Observer application to your node during startup by including it in the node's startup options:

import (
	"ergo.services/ergo"
	"ergo.services/application/observer"
	"ergo.services/ergo/gen"
)

func main() {
	opt := gen.NodeOptions{
		Applications: []gen.ApplicationBehavior {
			observer.CreateApp(observer.Options{}),
		}
	}
	node, err := ergo.StartNode("example@localhost", opt)
	if err != nil {
		panic(err)
	}
	node.Wait()
}

The function observer.CreateApp takes observer.Options as an argument, allowing you to configure the Observer application. You can set:

Port: The port number for the web server (default: 9911 if not specified).
Host: The interface name (default: localhost).
LogLevel: The logging level for the Observer application (useful for debugging). The default is gen.LogLevelInfo

Meta-Processes

An extra library of meta-process implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy.

Loggers

An extra library of logger implementations that are not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages that have external dependencies, as Ergo Framework adheres to a "zero dependency" policy

Colored

The colored logger provides visual clarity for console output by applying color highlighting to log messages. Instead of monochrome text where errors blend with informational messages, each log level gets a distinct color, and framework types are highlighted automatically. This makes it easier to scan logs during development and debugging.

The logger writes directly to standard output with immediate formatting - no buffering, no delays. When a process logs a message, it appears instantly in your terminal with colors applied. This synchronous approach keeps logs simple and predictable during interactive development.

Visual Organization

Color helps your eyes parse logs quickly. Log levels use consistent colors:

Rotate

The rotate logger writes log messages to files with automatic rotation based on time intervals. Instead of a single growing log file that eventually fills the disk, the logger creates new files periodically and optionally compresses old ones. This keeps disk usage predictable and makes log files manageable for analysis and archival.

The logger operates asynchronously - log messages enter a queue and a background goroutine writes them to the file. This design prevents blocking your processes when disk I/O is slow. Logging happens in the background while your actors continue processing messages without waiting for disk writes to complete.

File Rotation Mechanics

Rotation happens based on time periods. You configure a duration - one minute, one hour, one day - and the logger creates a new file every period. The active file is always named <Prefix>.log. When the period ends, the logger:

Copies the active file to a timestamped filename: <Prefix>.YYYYMMDDHHmi.log
Optionally compresses it with gzip: <Prefix>.YYYYMMDDHHmi.log.gz
Truncates the active file to start fresh for the new period

This approach ensures the active file always has the same name. You can tail it (tail -f <Prefix>.log) and it works across rotations. The timestamped copies accumulate in the log directory, creating a chronological archive.

The timestamp format is YYYYMMDDHHmi - year, month, day, hour, minute. This format sorts lexicographically, so ls -l shows files in chronological order. It's compact but human-readable.

Asynchronous Writing

The logger uses an internal lock-free queue (MPSC - multi-producer single-consumer). When any process logs a message, it pushes to the queue and returns immediately. A single background goroutine pops messages from the queue and writes them to the file.

This design has several advantages:

Non-blocking - Logging never blocks your process. If the disk is slow or the file system stalls, your actors continue running. The queue absorbs bursts of messages.

Ordering - Messages from a single producer maintain order. The queue preserves submission order, so logs reflect the actual sequence of events within each process.

Batching - The background goroutine processes messages continuously. If multiple messages arrive quickly, it writes them in a tight loop, reducing syscall overhead.

Configuration

The logger requires a rotation period and accepts several optional parameters:

Period - The rotation interval. Minimum is time.Minute. Smaller periods create more files with less data each. Larger periods create fewer files with more data each. Choose based on how you analyze logs - if you search specific time ranges, shorter periods help. If you archive logs by day, use 24 * time.Hour.

Path - Directory for log files. Defaults to ./logs relative to the executable. The logger creates the directory if it doesn't exist. Supports ~ for home directory expansion (~/logs becomes /home/user/logs). Use absolute paths in production to avoid ambiguity.

Prefix - Filename prefix. Defaults to the executable name. The active file is <Prefix>.log, rotated files are <Prefix>.YYYYMMDDHHmi.log[.gz]. Use meaningful prefixes if multiple services log to the same directory.

Compress - Enables gzip compression for rotated files. The active file stays uncompressed for fast writing. When rotating, the logger compresses the copy, reducing disk usage by 5-10x for text logs. Compressed files have .log.gz extension. Use compression if disk space matters more than CPU for compression.

Depth - Limits the number of retained log files. When rotating, if the number of files exceeds Depth, the logger deletes the oldest file. Set to 0 (default) for unlimited retention. Set to a specific number (e.g., 24) to keep the last 24 periods. This prevents unbounded disk usage.

TimeFormat - Timestamp format in log messages. Same as colored logger - any format from time package or custom layout. Empty string uses nanosecond timestamps. Choose based on readability vs. precision.

IncludeName - Includes registered process names in log messages. Helps identify which process logged what.

IncludeBehavior - Includes behavior type names in log messages. Useful during development to understand code flow.

ShortLevelName - Uses abbreviated level names ([TRC], [DBG], etc.) instead of full names. Saves space in log files.

Basic Usage

Configure the rotate logger in node options:

This configuration:

Rotates every hour
Stores logs in /var/log/myapp/
Names files myapp.log (active) and myapp.202411191200.log.gz (rotated with compression)

For detailed logger configuration options, see the rotate.Options struct in the package. For understanding how loggers integrate with the framework, see .

Registrars

An extra library of registrars or client implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework follows a "zero dependency" policy.

Available Registrars

A client library for the central registrar. Provides service discovery, configuration management, and real-time cluster event notifications through a centralized registrar service.

Features:

Centralized service discovery
Real-time event notifications
Configuration management
TLS security support

A client library for , a distributed key-value store. Provides decentralized service discovery, hierarchical configuration management with type conversion, and automatic lease management.

Features:

Distributed service discovery
Hierarchical configuration with type conversion from strings ("int:123", "float:3.14")
Automatic lease management and cleanup

Choose Saturn for centralized management with a dedicated registrar service, or etcd for a distributed approach with built-in consensus and reliability guarantees.

Saturn Сlient

This package implements the gen.Registrar interface and serves as a client library for the central registrar, . In addition to the primary Service Discovery function, it automatically notifies all connected nodes about cluster configuration changes.

To create a client, use the Create function from the saturn package. The function requires:

The hostname where the central registrar is running (default port: 4499

Network Protocols

The Ergo Framework allows nodes to run with various network stacks. You can replace the default network stack or add it as an additional stack. For more information, refer to the Network Stack section.

This library contains implementations of network stacks that are not part of the standard Ergo Framework library.

Tools

Saturn - Central Registrar

Ergo Service Registry and Discovery

saturn is a tool designed to simplify the management of clusters of nodes created using the Ergo Framework. It offers the following features:

A unified registry for node registration within a cluster.
The ability to manage multiple clusters simultaneously.

Events

Publish/Subscribe Event Mechanism

The actor model excels at point-to-point communication. Process A sends a message to process B. Process C makes a request to process D. Each interaction has a specific sender and receiver.

Registering Events

Publishing Events

Publishing an event sends it to all current subscribers.

Event publishing is fire-and-forget. The producer doesn't wait for acknowledgment or know how many subscribers received the event. The framework handles distribution asynchronously.

Subscribing to Events

Processes subscribe to events through links or monitors, the same mechanisms used for process lifecycle tracking.

MonitorEvent creates a monitor on an event. You receive event messages and a down notification if the producer terminates or the event is unregistered, but you don't terminate automatically.

Both methods return buffered events upon successful subscription:

The buffered events let subscribers catch up on what happened before they joined. If the buffer size was 10 and 5 events have been published, new subscribers receive those 5 events immediately.

Event Lifecycle

Events exist from registration until unregistration or producer termination.

Network Transparency

Events work across nodes seamlessly. A producer on node A can publish events that subscribers on nodes B, C, and D receive. The framework handles the network distribution.

If the network connection fails, subscribers receive termination notifications with reason gen.ErrNoConnection. This is consistent with how links and monitors handle network failures for processes.

Token Delegation

Event tokens can be delegated. The producer can give its token to another process, allowing that process to publish events under the producer's event registration.

Event Messages

Event messages have a specific structure:

Each gen.MessageEvent contains:

Event - The event identifier (name and node)
Message - Your application data (any type)
Timestamp - When the event was published (nanoseconds since epoch)

Practical Patterns

Events fit several common scenarios.

System telemetry - Processes publish metrics as events. Monitoring processes subscribe and aggregate. If the monitoring process restarts, buffered events provide recent history to rebuild state.

For more information on links and monitors as they apply to processes and nodes, see the chapter.

Application

Grouping and Managing Actors as a Unit

An application groups related actors and manages them as a unit. Instead of starting individual processes and tracking their lifecycles manually, you define an application that specifies which actors to start, in what order, and how the group should behave if individual actors fail.

Think of an application as a recipe. It lists the components (actors and supervisors), describes their startup order, and specifies the rules for what happens when things go wrong. The node follows this recipe when starting the application and monitors the running components according to the specified mode.

The Need for Applications

Starting processes one at a time works for simple systems. But as complexity grows, you face coordination problems. Which processes should start first? What if one fails to start - do you continue or abort? If a critical component terminates, should the service keep running in a degraded state or shut down cleanly?

These aren't implementation details - they're architectural decisions about your service's structure and fault tolerance policy. Applications let you declare these decisions explicitly rather than scattering the logic throughout your code. The specification documents what your service consists of. The mode declares your termination policy. The framework enforces both.

Defining an Application

Applications implement the gen.ApplicationBehavior interface:

The Load callback returns the application specification - what this application consists of and how it should behave. The Start callback runs after all processes start successfully. The Terminate callback runs when the application stops.

A typical application specification:

The Group lists processes to start. Processes start in the order listed. If a process has a Name, it's registered with that name, making it discoverable. Processes without names are anonymous.

Application names and process names exist in separate namespaces. An application named "api" and a process named "api" do not conflict - you can have both registered simultaneously. However, using the same name for both creates confusion when reading code or debugging. Avoid identical names even though the framework allows it.

Application Modes

The mode determines what happens when a process in the application terminates.

Temporary Mode - The application continues running despite individual process terminations. Only when all processes have stopped does the application itself terminate. This mode is for applications where components can fail and restart independently (typically via supervisors) without stopping the whole application.

Transient Mode - The application stops if any process terminates abnormally (crashes, panics, errors). Normal termination doesn't trigger shutdown. When an abnormal termination occurs, all remaining processes receive exit signals and the application shuts down. Use this mode when abnormal failures indicate a systemic problem that requires stopping the entire service.

Permanent Mode - The application stops if any process terminates, regardless of reason. Even normal termination of one process triggers shutdown of all others and the application itself. This mode is for applications where all components must run together - if one stops, the whole application is incomplete.

Loading and Starting

Applications go through two phases: loading and starting.

Loading calls your Load callback, validates the specification, and registers the application with the node. The application is loaded but not running. This separation allows you to load multiple applications and resolve dependencies before starting any of them.

Starting launches the processes in the Group according to their order. If dependencies are specified in ApplicationSpec.Depends, the node ensures those applications are running first. If any process fails to start (including initialization timeout), previously started processes are killed and the application fails to start.

Application processes have a maximum InitTimeout of 15 seconds (3x DefaultRequestTimeout). Setting a higher value in gen.ProcessOptions returns gen.ErrNotAllowed and prevents the application from starting.

Once all processes are running, the Start callback is called and the application enters the running state.

Dependencies

Applications can depend on other applications or network services. If application B depends on application A, the node ensures A is running before starting B. Dependencies are declared in ApplicationSpec.Depends.

This allows you to structure complex systems with clear startup ordering. A database connection pool application starts before the API server application. The API server starts before the web frontend application. The framework handles the ordering automatically.

Stopping Applications

Applications stop in three ways.

You can call ApplicationStop, which sends exit signals to all processes and waits for them to terminate gracefully (5 second timeout by default). Once all processes have stopped, the Terminate callback runs and the application transitions to the loaded state.

You can call ApplicationStopForce, which kills all processes immediately without waiting. Less graceful, but guaranteed to stop quickly.

The application can stop itself based on its mode. In Transient or Permanent mode, process failures trigger automatic shutdown according to the mode's rules.

Environment and Configuration

Applications have environment variables that all their processes inherit. These override node-level variables but are overridden by process-specific variables. This creates a natural layering: node provides defaults, application provides service-specific values, processes can override for their specific needs.

Tags for Instance Selection

Running multiple instances of the same application across a cluster creates a selection problem. Which instance should handle the request? In blue/green deployments, you run two versions and route traffic based on readiness. Canary deployments send a percentage to the new version. Some instances enter maintenance mode while others serve production traffic.

Tags provide metadata for making these decisions. Label each application instance with tags describing its deployment state, version, or role:

Tags are always available through node.ApplicationInfo() or remoteNode.ApplicationInfo(). For clusters using centralized registrars (etcd, Saturn), tags are also published during application route registration. This enables cluster-wide discovery: query the registrar and receive all application instances with their tags.

The embedded in-memory registrar does not support application route registration, so tags in single-node or statically-routed deployments are only accessible via direct ApplicationInfo() calls, not through resolver queries.

In clusters with centralized registrars:

Common tag patterns:

Blue/green deployment: "blue", "green"
Canary rollout: "canary", "stable"
Maintenance state: "maintenance", "active", "draining"

Tags separate deployment strategy from application code. Your application doesn't know it's the "blue" deployment - that's configuration. The routing logic queries tags and makes decisions based on current cluster state.

Process Role Mapping

Applications contain multiple processes with specific responsibilities. An API server handles requests. A connection pool manages database connections. A cache manager stores frequently accessed data. These are logical roles, but the actual process names might be versioned, generated, or environment-specific.

The Map field bridges this gap. Define a mapping from logical role (string) to actual process name (Atom):

To communicate with a process by role, get the application info, look up the role in the map, then use the returned name:

This works for both local and remote applications. When querying a remote application, RemoteNode.ApplicationInfo() retrieves the map from the remote node, letting you discover process names without prior knowledge of the remote application's internal structure.

Why use mapping:

Version changes: Update "api_server_v2" to "api_server_v3" without changing client code
Implementation swaps: Map "db" to different pool implementations based on deployment
Remote discovery: Remote nodes query the map to find process names in foreign applications

The map provides a service contract. External code knows the application has an "api" role and a "db" role. The actual implementations can change as long as the roles remain consistent.

The Application Pattern

Applications provide structure to your actor system. Instead of scattered process creation throughout your code, applications centralize the "what runs in this service" question. The specification documents your system's structure. The mode declares your fault tolerance policy. The dependency mechanism ensures correct startup ordering.

This organization becomes especially valuable in distributed systems where services start on different nodes. An application can be started remotely on another node, bringing all its components with the correct configuration and dependencies.

For more details on application lifecycle and options, refer to the gen.ApplicationBehavior and gen.ApplicationSpec documentation in the code.

Remote Spawn Process

Spawning processes on remote nodes

Remote spawning means starting a process on another node from your code. You call a method, provide a factory name and options, and a process starts on the remote node. From the caller's perspective, it's nearly identical to spawning locally - you get back a gen.PID and can communicate with it immediately.

This capability enables dynamic workload distribution. Your node needs to process a job but doesn't have capacity? Spawn a worker on a remote node with available resources. Your application needs to scale horizontally? Spawn processes across multiple nodes and distribute load. Remote spawning makes the cluster feel like one large computing resource rather than isolated nodes.

But remote spawning isn't automatic. Security matters. You don't want arbitrary nodes spawning arbitrary processes on your infrastructure. The framework requires explicit permission - the remote node must enable each process factory individually and can restrict which nodes are allowed to use it.

Security Model

Remote spawning is disabled by default at the framework level. To enable it, set the EnableRemoteSpawn flag in your node's network configuration:

This flag is a global switch. With it disabled, all remote spawn requests fail immediately with gen.ErrNotAllowed. With it enabled, requests proceed to the next level of security: per-factory permission.

Enabling Process Factories

Even with EnableRemoteSpawn turned on, remote nodes can't spawn anything until you explicitly enable specific process factories:

Now remote nodes can request spawning using the factory name "worker". The factory function createWorker returns a gen.ProcessBehavior, just like local spawning. When a remote spawn request arrives with name "worker", the framework calls createWorker() to instantiate the process.

The factory name is the permission token. Remote nodes must use this exact name when requesting spawns. If they request "worker" and you haven't enabled it, the request fails. If they request "admin_process" without permission, it fails. You control the namespace of what's spawnable.

Access Control Lists

By default, EnableSpawn allows all nodes to use the factory. But you can restrict it to specific nodes:

Now only those two nodes can spawn workers. Requests from other nodes fail with gen.ErrNotAllowed.

You can update the access list dynamically:

Calling EnableSpawn again with the same factory name updates the access list. The factory must be the same (same type) - you can't change which factory is associated with a name after the first EnableSpawn call. Attempting to do so returns an error.

Disabling Access

To remove nodes from the access list:

This removes scheduler@node2 from the allowed list. Other nodes in the list remain allowed.

To completely disable a factory:

Without any node arguments, DisableSpawn removes the factory entirely. All future spawn requests for that name fail.

To re-enable the factory with an open access list (any node can spawn):

This is the explicit "allow all nodes" configuration.

Spawning on Remote Nodes

To spawn a process on a remote node, first get a gen.RemoteNode interface:

GetNode establishes a connection if needed. If a connection already exists, it returns immediately. If discovery or connection fails, you get an error.

With the remote node handle, spawn a process:

The gen.ProcessOptions are the same as local spawning: mailbox size, compression settings, parent process options. The remote node respects these options when creating the process.

Spawn with Arguments

You can pass initialization arguments to the remote process:

These arguments are passed to the factory's Init callback, just like local spawning. The arguments must be serializable via EDF - primitives, registered structs, framework types. Complex arguments require type registration on both sides.

Spawn with Registration

To spawn and register the process with a name:

The first argument is the registration name. The remote process is registered under that name on the remote node, allowing other processes on that node (or other nodes) to find it via gen.ProcessID{Name: "worker-001", Node: "worker@otherhost"}.

Spawning from Processes

The gen.Process interface provides methods for remote spawning from within a process:

This differs from using RemoteNode.Spawn in a subtle but important way: the spawned process inherits properties from the calling process, not from the node.

Inherited properties:

Application name - if the caller is part of an application, the remote process becomes part of that application too
Logging level - the remote process uses the same log level as the caller
Environment variables - if ExposeEnvRemoteSpawn security flag is enabled, the remote process gets a copy of the caller's environment

This inheritance enables application-level distribution. If your application spawns processes remotely using process.RemoteSpawn, those processes belong to your application's supervision tree (conceptually), inherit your configuration, and operate as extensions of your application rather than independent processes.

The same inheritance applies.

Parent Relationship and Inheritance

Remote spawn behavior differs based on whether you spawn from a process or from the node:

From a process (process.RemoteSpawn):

The spawned process inherits attributes from the calling process:

Parent PID: Set to the calling process's PID
Group Leader: Set to the calling process's group leader
Application: Set to the calling process's application name (if caller belongs to an application)

The remote process can send messages to its parent using process.Parent(). If LinkChild: true is set in options, the link is established after spawn. However, the parent is on a different node - if the network connection drops, the remote process receives an exit signal for the lost parent and may terminate if linked.

From the node (RemoteNode.Spawn):

The spawned process receives attributes from the requesting node's core:

Parent PID: Set to the requesting node's core PID
Group Leader: Set to the requesting node's core PID
Application: Not set (empty - process doesn't belong to any application)

This creates independent processes without application affiliation. Use this for standalone remote workers that don't need to be part of an application's logical structure.

Environment Variable Inheritance

By default, remote processes don't inherit environment variables. This is a security decision - you probably don't want to expose your node's configuration to remote processes.

To enable environment inheritance:

Now when you use process.RemoteSpawn, the remote process receives a copy of the calling process's environment. The remote node reads these values and sets them on the spawned process.

Important: Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via edf.RegisterTypeOf. If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote spawn fails entirely with an error like "no encoder for type <type>". The framework doesn't skip problematic variables - any non-serializable value causes the entire spawn request to fail.

Environment inheritance only works with process.RemoteSpawn. Using RemoteNode.Spawn doesn't inherit environment because there's no calling process - it's a node-level operation.

How It Works

When you call remote.Spawn:

Check capabilities - The local node checks if the remote node's EnableRemoteSpawn flag is true (learned during handshake). If false, fail immediately.
Create spawn message - Package the factory name, process options, and arguments into a MessageSpawn protocol message. Include a reference for tracking the response.

If anything fails (factory not found, access denied, remote node terminating, initialization timeout), the error is returned to the caller. The entire operation is synchronous from the caller's perspective - you call Spawn and block until the process is created or an error occurs.

Practical Considerations

Performance - Remote spawning is slower than local spawning. There's network latency, message encoding, and a synchronous request-response roundtrip. If you're spawning hundreds of processes, doing it remotely will be noticeably slower. Consider spawning a pool locally and distributing work via messages rather than spawning on-demand remotely.

Timeouts - Remote spawn has a maximum InitTimeout of 15 seconds (3x DefaultRequestTimeout). If the remote process's ProcessInit takes longer, spawn fails with gen.ErrTimeout. Setting InitTimeout higher than 15 seconds returns gen.ErrNotAllowed immediately without attempting the spawn.

Failure modes - Remote spawn can fail in ways local spawn can't. The network connection can drop mid-request. The remote node can crash before responding. The factory might exist but lack permission. Handle errors explicitly and have fallback strategies (retry, spawn locally, defer the work).

Resource ownership - A process spawned on a remote node runs on that node's resources (CPU, memory). It's part of that node's process table. If the remote node terminates, the process dies. If you're distributing workload, be aware of which node owns which processes.

Linking - Both LinkChild and LinkParent options work for remote spawn. The link is established after the remote process is created. If the network connection drops, linked processes receive exit signals for the lost peer.

Application membership - Processes spawned via RemoteNode.Spawn don't belong to any application. Processes spawned via process.RemoteSpawn inherit the caller's application. This affects supervision, lifecycle, and monitoring.

Registration names - Use SpawnRegister carefully. The name you provide is registered on the remote node. If that name is already taken, spawn fails. Ensure your naming strategy avoids conflicts, especially if multiple nodes are spawning on the same target.

When to Use Remote Spawn

Dynamic scaling - Your application detects high load and spawns additional workers on remote nodes to handle the burst. When load decreases, workers terminate naturally and resources are freed.

Specialized hardware - Some nodes have GPUs, fast storage, or special network access. Spawn processes on those nodes when you need their capabilities, rather than sending data back and forth.

Fault isolation - Spawn risky operations on remote nodes. If they crash or consume excessive resources, they don't affect your local node's stability.

Data locality - If data lives on a specific node (in memory, on local disk), spawn processing near the data rather than transferring it across the network.

Heterogeneous clusters - Different nodes run different process types. Scheduler nodes spawn job processors on worker nodes. API nodes spawn request handlers on computation nodes. Remote spawning enables this separation.

Remote spawning isn't always the right answer. For static topologies where processes have fixed homes, use supervision trees and let supervisors spawn locally. For message-passing workloads where spawning overhead matters, use process pools and distribute work via messages. Remote spawning shines when you need dynamic, on-demand process creation across a cluster.

For understanding the underlying network mechanics, see . For controlling connections to remote nodes, see .

Logging

Logging system and logger implementations

Understanding what happens inside a running system requires logging. But logging in distributed actor systems isn't straightforward. Messages pass between dozens of processes. Processes spawn dynamically, handle requests, and terminate. Network connections form and break. Following a single request's path through the system means tracking its journey across multiple processes, possibly across multiple nodes.

Traditional logging compounds the problem. Each component writes to its own log. Process logs go to one file, network logs to another, node events to a third. When something goes wrong, you're piecing together a timeline from scattered sources, correlating by timestamp and hoping you've found all the relevant entries. It's detective work when you need diagnostic clarity.

Ergo Framework centralizes the logging flow while keeping distribution flexible. Every log call - whether from a process, meta process, or the node itself - flows through a single logging system. That system distributes messages to registered loggers based on configurable filters. One logger might write everything to the console. Another might write only errors to a file. A third might send metrics to a monitoring system. The architecture is simple: centralized input, filtered distribution to multiple outputs.

How Messages Flow

When code calls process.Log().Info("message"), the framework creates a gen.MessageLog structure. This contains the timestamp, severity level, source identifier, message format and arguments, and any attached structured fields. The message enters the node's logging subsystem.

The subsystem maintains loggers organized by severity level. Each logger, when registered, declares which levels it handles - perhaps just errors and panics, perhaps everything from debug upward. When a log message arrives, the subsystem looks up which loggers are registered for that message's level and calls their Log methods.

This is fan-out distribution. A single info-level message goes to every logger registered for info level. The default logger writes it to stdout. A file logger appends it to a file. A metrics logger counts it. Each logger receives the same message and processes it independently.

Hidden Loggers (introduced in v3.2.0) - Prefix a logger name with "." to create a hidden logger that's excluded from fan-out. Hidden loggers only receive logs from processes that explicitly call SetLogger(name). This creates truly isolated logging streams - bidirectional isolation. For example, register ".debug" as a hidden logger, then have a specific process use SetLogger(".debug"). That process's logs go only to the hidden logger (not to other loggers), and the hidden logger receives logs only from that process (not from fan-out). This is useful for separating verbose debugging output or creating per-process log files without mixing logs from other processes.

You can also use SetLogger("filename") to send a process's logs to a specific logger. The process's logs go only to that logger, but the logger still receives fan-out logs from other processes. This routes verbose process logs to a dedicated destination but doesn't create isolation - the logger sees both the process's logs and system-wide fan-out.

Severity Levels

The framework provides six severity levels, ordered from most to least verbose:

gen.LogLevelTrace - Framework internals, message routing, network packets. Extremely verbose, intended only for deep debugging of the framework itself.

gen.LogLevelDebug - Application debugging information. Useful during development but typically disabled in production.

gen.LogLevelInfo - Normal informational messages. This is the default level. Startup events, request handling, normal operations.

gen.LogLevelWarning - Conditions that merit attention but don't prevent operation. Deprecated API usage, approaching resource limits, retry scenarios.

gen.LogLevelError - Errors that prevent specific operations but don't crash the system. Failed requests, unavailable resources, validation failures.

gen.LogLevelPanic - Critical errors requiring immediate attention. Despite the name, logging at this level doesn't trigger a panic - it's just the highest severity marker.

Setting a level creates a threshold. Set a process to gen.LogLevelWarning and it logs warnings, errors, and panics, but suppresses info, debug, and trace. Each level implicitly includes all higher severity levels.

Two special levels control behavior rather than representing severity:

gen.LogLevelDefault - Sentinel meaning "inherit." Nodes with this level become gen.LogLevelInfo. Processes with this level inherit from their parent, leader, or node. This default-then-inherit pattern allows hierarchical log level configuration.

gen.LogLevelDisabled - Stops all logging from the source. The framework doesn't even create log messages. Use this to completely silence a source without removing loggers.

Trace deserves special mention. It's so verbose that enabling it accidentally could flood storage. You can't enable it dynamically via SetLevel. It must be set at startup through gen.NodeOptions.Log.Level or gen.ProcessOptions.LogLevel. This restriction prevents operational mistakes.

The node starts at gen.LogLevelInfo. Processes inherit this unless their spawn options specify otherwise. After startup, you can adjust a process's level dynamically with SetLevel, allowing surgical verbosity changes during debugging.

Identifying Log Sources

The logging subsystem differentiates between four source types: node, process, meta process, and network. Each carries its source information in a typed structure - gen.MessageLogNode, gen.MessageLogProcess, gen.MessageLogMeta, or gen.MessageLogNetwork. This typing allows custom loggers to handle different sources differently, perhaps routing network logs to one destination and process logs to another.

The default logger formats each source type distinctly in its output:

Node logs show the node name as a CRC32 hash:

Process logs show the full PID:

With IncludeName enabled, the registered name appears:

With both IncludeName and IncludeBehavior enabled, the actor type appears:

Meta process logs show the alias:

Network logs show local and remote node hashes:

These visual distinctions make scanning logs easier. At a glance, you can distinguish node events from process activity, meta process operations from network communications. The format itself tells you what layer of the system generated each message.

Adding Context with Fields

Beyond the message text, you can attach structured fields - key-value pairs providing context. Fields enable correlation across log entries and make logs machine-parseable.

Consider a request handler. It receives a request with an ID. Every log entry related to that request should include the ID, allowing you to filter logs to just that request's activity:

With IncludeFields enabled in the logger configuration, output shows:

Fields appear on a separate line below the message, prefixed with "fields" and aligned with the timestamp. Multiple fields are space-separated, each formatted as key:value. In JSON output, fields become separate JSON properties at the message's top level.

Fields only appear in output if the logger is configured to include them. The default logger requires gen.NodeOptions.Log.DefaultLogger.IncludeFields = true. Without this, fields are tracked internally but not displayed - useful if some loggers need fields while others don't.

Fields accumulate. Call AddFields multiple times and you add more fields rather than replacing existing ones. This supports incremental context building. Add session_id when the session starts. Add transaction_id when beginning a transaction. Add payment_id when processing payment. Each subsequent log includes all accumulated fields.

Remove fields with DeleteFields:

This clears the named fields from subsequent logs.

Field Scoping

Field scoping handles nested contexts where you need temporary fields that shouldn't persist beyond a specific operation.

PushFields saves the current field set and starts a new scope. Add temporary fields, perform the operation (with those fields appearing in logs), then PopFields to restore the previous field set:

Output shows:

The operation field exists only within the push/pop scope. After popping, logs include only session_id.

Scopes can nest. Each PushFields returns the stack depth. Each PopFields returns the new depth. This supports complex nested contexts - a request containing a transaction containing multiple operations, each adding its own contextual fields that disappear when the operation completes.

One restriction protects consistency: you can't delete fields while the field stack has active frames. If you've pushed fields, pop back to the base level before deleting. This prevents deleting a field that a pending pop might restore, which would leave the field state inconsistent.

The Default Logger

Every node starts with a default logger writing to os.Stdout. Configure it through gen.NodeOptions.Log.DefaultLogger:

TimeFormat controls timestamp display. Empty means nanoseconds since epoch. Any Go time format works - time.DateTime, time.RFC3339, or custom formats.

IncludeBehavior adds actor type names to process logs, showing which implementation generated each message.

IncludeName adds registered process names to process logs, making output more readable than PIDs alone.

IncludeFields controls whether structured fields appear in output.

EnableJSON switches to JSON format, with each message as a single-line JSON object.

To disable the default logger entirely, set Disable: true. Do this when using only custom loggers.

Adding Custom Loggers

Custom loggers implement gen.LoggerBehavior:

The Log method receives each message. The Terminate method handles cleanup when the logger is removed or the node shuts down.

The filter (final arguments) specifies which levels this logger handles. The logger receives only messages at those levels. Omit the filter to use gen.DefaultLogFilter, which includes all levels from Trace through Panic.

Loggers are stored per-level internally. Registering for Error and Panic stores the logger in both level maps. When an error occurs, the framework looks up the Error map and delivers the message to all loggers in that map.

Logger names must be unique. Reusing a name returns gen.ErrTaken. Remove a logger with LoggerDelete before adding a new one with the same name.

The Log method is called synchronously. If it blocks, it delays the logging path. For expensive operations - compressing logs, sending over network, database writes - make Log queue the work and return immediately, processing asynchronously.

Process-Based Loggers

A process can act as a logger, receiving log messages through its mailbox. This integrates logging with the actor model.

Implement the HandleLog callback in your actor:

Process-based logging queues messages asynchronously. The Log call places the message in the process's Log mailbox and triggers the process. The process handles log messages through HandleLog, processing them sequentially. The code that generated the log continues immediately without waiting.

This queuing prevents blocking. If the logger process is busy or the logging logic is expensive, messages queue and are processed when ready. The logging path stays fast.

One detail matters: when a logger process terminates, it's automatically removed from the logging system. No need to call LoggerDeletePID explicitly.

Using Multiple Loggers

The fan-out architecture supports multiple loggers operating simultaneously with different purposes.

A typical production configuration disables the default logger and adds specialized loggers:

The colored logger handles debug through panic for console display during development. The rotate logger receives everything and writes to rotating files. Trace messages don't appear anywhere because no logger is registered for trace level.

Loggers can be added and removed dynamically. Start with console logging during development. Add file logging in staging. In production, remove console, keep files, add metrics forwarding. The system adapts without code changes.

Controlling Verbosity

Different processes often need different verbosity. Most processes log at Info. Increase a troublesome process to Debug temporarily. Keep infrastructure processes at Warning to reduce noise.

For processes generating high-volume logs, route them to a dedicated logger using a hidden logger. A trading engine logging every order would overwhelm general logs:

This creates isolation - the trading process logs only to .trading, and .trading receives only trading process logs. Other processes and loggers are unaffected. Without the hidden logger (using a regular logger name), the logger would also receive fan-out logs from all other processes.

Process-based loggers enable sophisticated handling. A logger process can aggregate metrics - count errors per minute, track which processes log most frequently. It can detect patterns - the same error repeating indicates a stuck condition. It can forward to external systems - send errors to Slack, metrics to Prometheus. As an actor, it maintains state, can be supervised for reliability, and integrates naturally with the rest of your system.

Logger Implementations

The framework provides two logger implementations in separate packages for common needs:

Colored (ergo.services/logger/colored) - Terminal output with ANSI colors. Highlights Ergo types (PIDs, Atoms, Refs) and colorizes log levels (yellow for warnings, red for errors, etc.). Visual clarity for development, but has performance overhead. Not suitable for high-volume production logging.

Rotate (ergo.services/logger/rotate) - File logging with automatic rotation. Supports size-based and time-based rotation. Compresses old logs with gzip. Configurable retention policies. Production-ready for long-running systems generating substantial logs.

Both integrate with the logging system through node.LoggerAdd. You can combine them - colored for console during development, rotate for persistent storage, both receiving the same filtered log stream.

For implementation details and configuration options, see and in the extra library documentation.

Network Stack

Understanding the network stack for distributed communication

The network stack makes remote messaging work like local messaging. When you send to a process on another node, the framework discovers where that node is, establishes a connection if needed, encodes the message, sends it over TCP, and delivers it to the recipient's mailbox. From your perspective, it's just Send(pid, message) - whether the PID is local or remote.

This transparency requires three systems working together: service discovery to find nodes, connection management to establish reliable links, and message encoding to serialize data for transmission. Each system handles a specific problem, and together they create the illusion that remote communication is just local communication.

The Big Picture

When you send a message to a remote process:

Routing decision - The framework examines the node portion of the PID. Local node? Direct mailbox delivery. Remote node? Continue to step 2.
Connection lookup - Check if a connection to that node already exists. If yes, use it. If no, continue to step 3.
Discovery - Query the registrar (or check static routes) to find where the remote node is listening: hostname, port, TLS requirements, protocol versions.

This entire pipeline is invisible to your code. You call Send, and the framework does the rest.

Service Discovery

Before connecting to a remote node, the framework needs to know where that node is. Service discovery translates logical node names ([email protected]) into connection parameters (IP, port, TLS, protocol versions).

The embedded registrar provides basic discovery:

One node per host runs a registrar server (whoever started first)
Other nodes connect as clients
Same-host discovery is direct (no network)

For production clusters, external registrars provide more features:

etcd - Centralized discovery, application routing, configuration storage, HTTP polling for registration
Saturn - Purpose-built for Ergo, immediate event propagation, efficient at scale

The embedded registrar works for development and small deployments. For larger clusters or dynamic topologies, use etcd or Saturn. The choice is transparent to your code - you specify the registrar at node startup, and everything else works identically.

For details, see .

Static Routes

Discovery is dynamic - nodes register themselves, and others query to find them. But sometimes you want explicit control. Maybe nodes have fixed addresses. Maybe you're behind a firewall that blocks discovery. Maybe you're connecting to external systems.

Static routes let you hardcode connection parameters:

Now when connecting to [email protected], the framework uses your route directly. No discovery query. No registrar involvement. You've taken control.

Static routes support pattern matching ("prod-.*"), multiple routes with failover weights, and hybrid approaches (use patterns for selection, resolvers for address lookup). You can configure per-route cookies, certificates, network flags, and atom mappings.

The framework checks static routes first, always. If a static route exists, discovery is bypassed. If static routes fail or don't exist, the framework falls back to discovery.

For details, see .

Connection Establishment

Once the framework knows where to connect (from discovery or static routes), it establishes a connection pool.

Handshake

The handshake performs mutual authentication using challenge-response. Node A connects to node B:

A sends hello with random salt and digest (computed from salt + cookie)
B verifies digest - if cookies match, digest is correct
B sends its own challenge

If TLS is enabled, certificate fingerprints are exchanged and verified too.

After authentication, nodes exchange introduction messages:

Node names and version information
Network flags (capabilities: remote spawn? important delivery? fragmentation?)
Caching dictionaries (atoms, types, errors that will be used frequently)

The flags negotiation ensures nodes with different feature sets can work together. Features not supported by both sides are disabled for that connection.

The caching dictionaries enable efficiency. Instead of encoding "mynode@localhost" repeatedly (19 bytes), it gets a cache ID and subsequent uses encode as 2 bytes.

Connection Pool

After handshake, the accepting node tells the dialing node to create a connection pool:

Pool size (default 3 TCP connections)
Acceptor addresses to connect to

The dialing node opens additional TCP connections using a shortened join handshake (skips full authentication since the first connection already authenticated). These connections join the pool, forming a single logical connection with multiple physical TCP links.

Multiple connections enable parallel message delivery. Each message goes to a connection based on the sender's identity (derived from sender PID). Messages from the same sender always use the same connection, preserving order. Messages from different senders use different connections, enabling parallelism.

The receiving side creates 4 receive queues per TCP connection. A 3-connection pool has 12 receive queues processing messages concurrently. This parallel processing improves throughput while preserving per-sender message ordering.

Message Encoding and Transmission

Once a connection exists, messages flow through encoding and framing.

EDF (Ergo Data Format)

EDF is a binary encoding specifically designed for the framework's communication patterns. It's type-aware - each value is prefixed with a type tag (e.g., 0x95 for int64, 0xaa for PID, 0x9d for slice). The decoder reads the tag and knows what follows.

Framework types like gen.PID and gen.Ref have optimized encodings. Structs are encoded field-by-field in declaration order (no field names on the wire). Custom types must be registered on both sides - registration happens during init(), and during handshake nodes exchange their type lists to agree on encoding.

Compression is automatic. If a message exceeds the compression threshold (default 1024 bytes), it's compressed using GZIP, ZLIB, or LZW. The protocol frame indicates compression, so the receiver decompresses before decoding.

For details on EDF - type tags, struct encoding, registration requirements, compression, caching - see .

ENP (Ergo Network Protocol)

ENP wraps encoded messages in frames for transmission. Each frame has an 8-byte header with magic byte, protocol version, frame length, order byte, and message type. The frame body contains sender/recipient identifiers and the EDF-encoded payload.

The order byte preserves message ordering per sender. Messages from the same sender have the same order value and route to the same receive queue, guaranteeing sequential processing. Messages from different senders have different order values and route to different queues, enabling parallel processing.

For details on protocol framing, order bytes, receive queue distribution, and the exact byte layout, see .

Network Transparency in Practice

Network transparency means remote operations look like local operations. You send to a PID without checking if it's local or remote. You establish links and monitors the same way regardless of location. The framework handles discovery, encoding, and transmission automatically.

But transparency has limits:

Latency - Remote sends take milliseconds vs microseconds for local
Bandwidth - Network links have finite capacity, local operations don't
Failures - Networks fail in ways local memory doesn't (packets lost, connections drop, nodes unreachable)

The framework makes distributed programming feel local, but you still need to design for network realities: use timeouts, handle connection failures, prefer async over sync, batch messages, keep payloads small.

For deep understanding of how transparency works - EDF encoding, struct serialization, type registration, important delivery, failure semantics - see .

Network Configuration

Configure the network stack in gen.NodeOptions.Network:

Mode - NetworkModeEnabled enables full networking with acceptors. NetworkModeHidden allows outgoing connections only (no acceptors). NetworkModeDisabled disables networking entirely.

Cookie - Shared secret for authentication. All nodes must use the same cookie to communicate. Set explicitly for distributed deployments.

MaxMessageSize - Maximum incoming message size. Protects against memory exhaustion. Default unlimited (fine for trusted clusters).

Flags - Control capabilities. Remote nodes learn your flags during handshake and can only use features you've enabled. EnableRemoteSpawn allows spawning (with explicit permission per process). EnableImportantDelivery enables delivery confirmation.

Acceptors - Define listeners for incoming connections. Multiple acceptors on different ports are supported. Each can have its own cookie, TLS, and protocol.

Custom Network Stacks

The framework provides three extension points:

gen.NetworkHandshake - Control connection establishment and authentication. Implement this to change how nodes authenticate or how connection pools are created.

gen.NetworkProto - Control message encoding and transmission. The Erlang distribution protocol is implemented as a custom proto, allowing Ergo nodes to join Erlang clusters.

gen.Connection - The actual connection handling. Implement this for custom framing, routing, or error handling.

You can register multiple handshakes and protos, allowing one node to support multiple protocol stacks simultaneously:

This enables migration scenarios (gradually migrate from Erlang to Ergo) and integration scenarios (connect to systems using different protocols).

Remote Operations

Once connections exist, you can spawn processes and start applications on remote nodes:

Remote spawning requires the remote node to explicitly enable it:

Without explicit permission, remote spawn requests fail. This prevents arbitrary code execution.

The same pattern applies to starting applications:

Requires:

This security model ensures you control exactly what remote nodes can do on your node.

Where to Go Next

This chapter provided an overview of how the network stack operates. For deeper understanding:

- How nodes find each other, application routing, configuration management, embedded vs external registrars
- How messages are encoded, EDF details, protocol framing, compression, caching, important delivery
- Explicit routing configuration, pattern matching, failover, proxy routes

Each of these chapters dives deep into its specific topic, giving you the details needed for production deployments.

Pool

A single actor processes messages sequentially. This is fundamental to the actor model - it eliminates race conditions and makes reasoning about state straightforward. But it also means one actor can become a bottleneck. If messages arrive faster than the actor can process them, the mailbox grows, latency increases, and eventually the system stalls.

The standard solution is to run multiple workers. Instead of sending requests to one actor, distribute them across several identical actors processing in parallel. This works, but now you need routing logic: pick a worker, check if it's alive, handle mailbox overflow, restart dead workers. This boilerplate appears in every pool implementation.

act.Pool solves this. It's an actor that manages a pool of worker actors and automatically distributes incoming messages and requests across them. You send to the pool's PID, the pool forwards to an available worker. The pool handles worker lifecycle, automatic restarts, and load balancing. From the sender's perspective, it's just one actor. Under the hood, it's N workers processing in parallel.

Creating a Pool

Like act.Actor provides callbacks for regular actors, act.Pool uses the act.PoolBehavior interface:

The key difference from ActorBehavior: Init returns PoolOptions that define the pool configuration. All callbacks are optional except Init.

Embed act.Pool in your struct and implement Init to configure workers:

The pool spawns workers during initialization. Each worker is linked to the pool (via LinkParent: true). If a worker crashes, the pool receives an exit signal and can restart it.

Workers are created using the WorkerFactory. This is the same factory pattern as regular Spawn - it returns a gen.ProcessBehavior instance. The workers can be act.Actor, act.Pool (nested pools), or custom behaviors.

Rate Limiting Through Pool Configuration

The combination of PoolSize and WorkerMailboxSize provides a natural rate limiting mechanism. The pool can buffer at most PoolSize × WorkerMailboxSize messages. If all workers are busy and their mailboxes are full, new messages are rejected:

When a sender tries to send beyond this limit, they receive ErrProcessMailboxFull (if using important delivery) or the message is dropped with a log entry. This backpressure prevents the system from accepting more work than it can handle.

For external APIs (HTTP, gRPC), this translates to returning "503 Service Unavailable" when the pool is saturated. The pool size controls maximum concurrency, and the mailbox size controls burst capacity. Tune both based on your worker processing speed and acceptable latency.

Automatic Message Distribution

When you send a message or make a call to the pool, act.Pool automatically forwards it to an available worker:

Forwarding happens for messages in the Main queue (normal priority). The pool maintains a FIFO queue of worker PIDs. When a message arrives:

Pop a worker from the queue
Forward the message using Forward (preserves original sender and ref)
Check result:

If all workers have full mailboxes, the message is dropped and logged. The pool doesn't have its own buffer beyond the workers' mailboxes. This is intentional - backpressure should propagate to senders.

The pool forwards Regular messages, Requests, and Events. Exit signals and Inspect requests are handled by the pool itself (they're not forwarded to workers).

Workers and the Original Sender

Workers receive the original sender's PID, not the pool's PID. When a worker processes a forwarded message, from points to whoever sent to the pool:

The same applies to Call requests. Workers see the original caller's from and ref. When they return a result or call SendResponse, it goes directly to the original caller, bypassing the pool entirely.

This is why forwarding is transparent. The worker doesn't know it's part of a pool. It processes messages as if they were sent directly to it.

Intercepting Pool Messages

Automatic forwarding applies only to the Main queue (normal priority). Urgent and System queues are handled by the pool itself through HandleMessage and HandleCall callbacks:

The same for synchronous requests:

Important: High-priority requests that return (nil, nil) from HandleCall are not forwarded to workers. They're simply ignored, and the caller times out. Forwarding only happens for Main queue messages. If you want a request to be handled, either:

Send it with normal priority (goes to workers)
Handle it explicitly in pool's HandleCall and return a result

Use high priority only for pool management that should be handled by the pool itself, not for work that should go to workers.

Dynamic Pool Management

Adjust the pool size at runtime with AddWorkers and RemoveWorkers:

AddWorkers spawns new workers with the same factory and options used during initialization. They're added to the FIFO queue and immediately available for work.

RemoveWorkers takes workers from the queue and sends them gen.TerminateReasonNormal via SendExit. The workers terminate gracefully, finishing any in-progress work before shutting down.

Both methods return the new pool size after the operation. They fail if called from outside Running state.

Worker Restarts

Workers are linked to the pool with LinkParent: true. When a worker crashes, the pool receives an exit signal. The forward mechanism detects this (ErrProcessUnknown / ErrProcessTerminated), spawns a replacement with the same factory and arguments, and forwards the message to the new worker.

This is automatic restart, not supervision. The pool doesn't track worker history or apply restart strategies. It just replaces dead workers immediately when detected during forwarding. If you need sophisticated restart strategies, use a Supervisor to manage the pool and its workers.

Pool Statistics

Pools expose internal metrics via Inspect:

Use this for monitoring pool health. High messages_unhandled indicates workers are overwhelmed. High worker_restarts suggests worker stability issues.

When to Use Pools

Use a pool when:

One actor is a bottleneck (mailbox growing, latency increasing)
Work items are independent (no ordering dependencies)
Workers are stateless or can reconstruct state cheaply

Don't use a pool when:

Work items depend on previous items (pools don't guarantee ordering)
Workers maintain critical state that can't be lost on restart
Concurrency isn't the bottleneck (single actor is fast enough)

Pools are for horizontal scaling of stateless work. If workers need state coordination, use multiple independent actors with explicit routing instead.

Patterns and Pitfalls

Set WorkerMailboxSize to limit backpressure propagation. Unbounded mailboxes let workers accumulate huge queues, hiding the overload until memory exhausts. Bounded mailboxes cause forwarding to try next worker, eventually reaching the sender with backpressure.

Don't forward Exit signals intentionally. The pool doesn't forward Exit messages to workers. If you need to broadcast shutdown to all workers, iterate manually and send to each worker PID.

Monitor forwarding metrics. If messages_unhandled increases, your pool is undersized or workers are too slow. Scale up with AddWorkers or optimize worker processing.

Use priority for pool management. Send management commands with MessagePriorityHigh to ensure they go to the pool, not forwarded to workers.

Nested pools are possible but rarely useful. A pool of pools adds latency without much benefit. Prefer one pool with more workers over nested layers.

Erlang

Erlang network stack

This package implements the Erlang network stack, including the DIST protocol, ETF data format, EPMD registrar functionality, and the Handshake mechanism.

It is compatible with OTP-23 to OTP-27. The source code is available on the project's GitHub page at https://github.com/ergo-services/proto in the erlang23 directory.

Note that the source code is distributed under the Business Source License 1.1 and cannot be used for production or commercial purposes without a license, which can be purchased on the project's sponsor page.

EPMD

The epmd package implements the gen.Registrar interface. To create it, use the epmd.Create function with the following options:

Port: Registrar port number (default: 4369).
EnableRouteTLS: Enables TLS for all gen.Route responses on resolve requests. This is necessary if the Erlang cluster uses TLS.
DisableServer: Disables the internal server mode, useful when using the Erlang-provided

To use this package, include ergo.services/proto/erlang23/epmd.

Handshake

The handshake package implements the gen.NetworkHandshake interface. To create a handshake instance, use the handshake.Create function with the following options:

Flags: Defines the supported functionality of the Erlang network stack. The default is set by handshake.DefaultFlags().
UseVersion5: Enables handshake version 5 mode (default is version 6).

To use this package, include ergo.services/proto/erlang23/handshake.

DIST protocol

The ergo.services/proto/erlang/dist package implements the gen.NetworkProto and gen.Connection interfaces. To create it, use the dist.Create function and provide dist.Options as an argument, where you can specify the FragmentationUnit size in bytes. This value is used for fragmenting large messages. The default size is set to 65000 bytes.

To use this package, include ergo.services/proto/erlang/dist.

ETF data format

Erlang uses the ETF (Erlang Term Format) for encoding messages transmitted over the network. Due to differences in data types between Golang and Erlang, decoding received messages involves converting the data to their corresponding Golang types:

number -> int64
float number -> float64

When encoding data in the Erlang ETF format:

map -> map #{}
slice/array -> list []

You can also use the functions etf.TermIntoStruct and etf.TermProplistIntoStruct for decoding data. These functions take into account etf: tags on struct fields, allowing the values to map correctly to the corresponding struct fields when decoding proplist data.

To automatically decode data into a struct, you can register the struct type using etf.RegisterTypeOf. This function takes the object of the type being registered and decoding options etf.RegisterTypeOption. The options include:

Name - The name of the registered type. By default, the type name is taken using the reflect package in the format #/pkg/path/TypeName
Strict - Determines whether the data must strictly match the struct. If disabled, non-matching data will be decoded into any.

To be automatically decoded the data sent from Erlang must be a tuple, with the first element being an atom whose value matches the type name registered in Golang. For example:

The values sent by an Erlang process should be in the following format:

Ergo-node in Erlang-cluster

If you want to use the Erlang network stack by default in your node, you need to specify this in gen.NetworkOptions when starting the node:

In this case, all outgoing and incoming connections will be handled by the Erlang network stack. For a complete example, you can refer to the repository at , specifically the erlang project

If you want to maintain the ability to accept connections from Ergo nodes while using the Erlang network stack as a main one, you need to add an acceptor in the gen.NetworkOptions settings:

Please note that if the list of acceptors is empty when starting the node, it will launch an acceptor with the network stack using Registrar, Handshake, and Proto from gen.NetworkOptions.

If you set the options.Network.Acceptor, you must explicitly define the parameters for all necessary acceptors. In the example, acceptorErlang is created with empty gen.AcceptorOptions (the Erlang stack from gen.NetworkOptions will be used), while for acceptorErgo, the Ergo Framework stack (Registrar, Handshake, and Proto) is explicitly defined.

In this example, you can establish incoming and outgoing connections using the Erlang network stack. However, the Ergo Framework network stack can only be used for incoming connections. To create outgoing network connections using the Ergo Framework stack, you need to configure a static route for a group of nodes by defining a match pattern:

For more detailed information, please refer to the section.

Erlang-node in Ergo-cluster

If your cluster primarily uses the Ergo Framework network stack by default and you want to enable interaction with Erlang nodes, you'll need to add an acceptor using the Erlang network stack. Additionally, you must define a static route for Erlang nodes using a match pattern:

Actor `GenServer`

The erlang23.GenServer actor implements the low-level gen.ProcessBehavior interface, enabling it to handle messages and synchronous requests from processes running on an Erlang node. The following message types are used for communication in Erlang:

regular messages - sent from Erlang using erlang:send or the Pid ! message syntax
cast-messages - sent from Erlang with gen_server:cast
call-requests - from Erlang made with

erlang23.GenServer uses the erlang23.GenServerBehavior interface to interact with your object. This interface defines a set of callback methods for your object, which allow it to handle incoming messages and requests. All methods in this interface are optional, meaning you can choose to implement only the ones relevant to your specific use case:

The callback method HandleInfo is invoked when an asynchronous message is received from an Erlang process using erlang:send or via the Send* methods of the gen.Process interface. The HandleCast callback method is called when a cast message is sent using gen_server:cast from an Erlang process. Synchronous requests sent with gen_server:call or Call* methods are handled by the HandleCall callback method.

If your actor only needs to handle regular messages from Erlang processes, you can use the standard act.Actor and process asynchronous messages in the HandleMessage callback method.

To start a process based on erlang23.GenServer, create an object embedding erlang23.GenServer and implement a factory function for it.

Example:

To send a cast message, use the Cast method of erlnag23.GenServer.

To send regular messages, use the Send* methods of the embedded gen.Process interface. Synchronous requests are made using the Call* methods of the gen.Process interface.

Like act.Actor, an actor based on erlang23.GenServer supports the TrapExit functionality to intercept exit signals. Use the SetTrapExit and TrapExit methods of your object to manage this functionality, allowing your process to handle exit signals rather than terminating immediately when receiving them.

Meta-Process

A meta-process solves a specific problem: how to integrate blocking I/O with the actor model without breaking its guarantees. It runs two goroutines - one executes your blocking I/O code, the other handles actor messages. This separation preserves sequential message processing while allowing continuous external I/O operations.

Meta-processes are owned by their parent process. When the parent terminates, all its meta-processes terminate with it. This dependency is by design - meta-processes extend the parent's capabilities rather than existing as independent entities in the supervision tree.

The Problem

Actors work sequentially. One message arrives, gets processed, completes. Next message. This simplicity eliminates race conditions and makes reasoning straightforward.

Blocking I/O breaks this model. Call net.Listener.Accept() in a message handler and the actor freezes. The goroutine blocks waiting for connections. Other messages pile up unprocessed. The actor becomes unresponsive.

The obvious fix fails. Spawn a goroutine for Accept() and now two goroutines access the actor's state concurrently. You need locks. The sequential guarantee vanishes. The actor model collapses into traditional concurrent programming with all its complexity.

Meta-processes preserve both. One goroutine blocks on I/O. Another goroutine processes messages sequentially. Neither interferes with the other.

Two Goroutines, Two Purposes

When a meta-process starts, the framework launches two goroutines:

External Reader: Runs your Start() method from beginning to end. This goroutine is meant for blocking operations - Accept() loops, ReadFrom() calls, reading from pipes. When external events occur, this goroutine sends messages into the actor system using Send(). It never processes incoming messages.

Actor Handler: Created on-demand when messages arrive in the mailbox. Processes messages sequentially by calling your HandleMessage() and HandleCall() methods. When the mailbox empties, this goroutine terminates. Next time messages arrive, a new actor handler spawns. This goroutine never does I/O directly - it handles requests from actors.

The External Reader runs continuously from spawn until termination. The Actor Handler comes and goes based on message traffic.

Why Regular Processes Cannot Do This

Aspect

Process

Meta-Process

Processes have one goroutine that must handle everything. If it blocks on I/O, message processing stops. If it spawns additional goroutines for I/O, the actor model breaks.

Meta-processes separate concerns. The External Reader handles I/O. The Actor Handler handles messages. Both run independently.

Restrictions Explained

Meta-processes cannot make synchronous calls. Which goroutine should block waiting for the response? The External Reader is blocked on external I/O. The Actor Handler might not be running. Neither can reliably wait for responses.

Meta-processes cannot create links or monitors. When a linked process terminates, it sends an exit signal as a message. The Actor Handler processes messages, but only when running. Signals could be delayed or lost if the Actor Handler is not active. Incoming links and monitors work because other processes send signals that queue in the mailbox. Creating outgoing links requires guarantees that meta-processes cannot provide.

These are not arbitrary limitations. They follow from having two goroutines with distinct responsibilities.

Behavior Implementation

Init() runs once during creation. Initialize state, store the MetaProcess reference, prepare resources. Return an error to prevent spawning.

Start() runs in the External Reader. This is where your blocking I/O lives. Loop forever accepting connections. Block reading datagrams. Read from pipes. When Start() returns, the meta-process terminates.

HandleMessage() processes regular messages sent by actors. Runs in the Actor Handler. Return nil to continue, return an error to terminate.

HandleCall() processes synchronous requests from actors. Return (result, nil) to send the result back. Return (nil, error) to send an error. The framework handles the response automatically.

Terminate() runs during shutdown regardless of how termination occurred. Close resources, flush buffers, clean up. Do not block or panic here.

HandleInspect() returns diagnostic information as string key-value pairs. Used by monitoring tools. Inspect requests are sent to the system queue (high priority) and processed before regular messages. You can inspect meta processes from within a process context using process.InspectMeta(alias) or directly from the node using node.InspectMeta(alias). Both methods only work for local meta processes (same node).

Three States

Sleep: External Reader is running (usually blocked on I/O), Actor Handler does not exist. Mailbox may contain messages waiting to be processed. This is the resting state when no actors are communicating with the meta-process.

Running: Both goroutines active. External Reader continues I/O operations. Actor Handler processes messages from the mailbox. Both work simultaneously without blocking each other.

Terminated: Both goroutines stopped. Start() returned and Actor Handler completed its final message.

Transitions are automatic. Message arrives → Actor Handler spawns → Sleep becomes Running. Mailbox empties → Actor Handler exits → Running becomes Sleep. Start() returns → Terminated regardless of current state.

Data Flow

The External Reader blocks reading while the Actor Handler simultaneously blocks writing. Two blocking operations, two goroutines, neither prevents the other.

Creating Meta-Processes

Define your behavior:

Spawn from a process:

The meta-process lives as long as its parent lives. When Server terminates, the UDP server terminates automatically.

State-Based Operations

Different operations are available in different states:

All states (Sleep, Running, Terminated):

Send(), SendWithPriority() - External Reader sends in Sleep, Actor Handler sends in Running
ID(), Parent() - Identity never changes

Running only:

SendResponse(), SendResponseError() - Only Actor Handler has the gen.Ref from HandleCall()
SetSendPriority(), SetCompression() - Actor Handler controls these

Sleep and Running (not Terminated):

Spawn() - Both goroutines can spawn child meta-processes

The External Reader operates in Sleep state and has minimal capabilities - just sending messages and spawning children. The Actor Handler operates in Running state and has full capabilities for processing requests.

Shared State

Both goroutines access the same struct fields. Use atomic operations for shared counters and flags:

Avoid complex synchronization. If you need mutexes, the design probably belongs in a regular process with meta-processes handling only I/O.

Common Patterns

External events to actors: External Reader reads events, sends them to actors for processing.

Actor-controlled I/O: Actors send commands, Actor Handler executes them against external resources.

Full-duplex communication: External Reader reads, Actor Handler writes, both operate on the same connection.

Server accepting connections: External Reader accepts connections, spawns child meta-processes for each.

When to Use Meta-Processes

Use meta-processes when:

Operating on blocking I/O (TCP accept, UDP read, pipe read, file read)
Bridging external event sources with actors (monitoring filesystems, listening to OS signals)
Wrapping synchronous APIs that cannot be made asynchronous

Do not use meta-processes when:

Implementing business logic
Managing application state
Coordinating between actors
Processing messages that do not involve blocking I/O

Meta-processes sit at the boundary between the external world and the actor system. They translate blocking operations into asynchronous messages and execute actor commands using blocking APIs. Regular processes implement everything else.

For complete examples, see , , , and .

Important Delivery Flag

Guaranteed message delivery with acknowledgment

In the actor model, messages are typically fire-and-forget. You send a message, and it either arrives or it doesn't. For local communication, errors are immediate - if the process doesn't exist or the mailbox is full, Send returns an error. But for remote communication, Send succeeds as soon as the message reaches the network layer. You don't know if it arrived at the remote node, if the target process exists, or if the mailbox had space.

This works fine for many scenarios. Asynchronous messaging doesn't require confirmation. Actors process what arrives and ignore what doesn't. Systems are resilient because actors don't wait for acknowledgments - they keep working.

But some operations need certainty. A payment authorization must definitely be recorded or definitely fail - "maybe it worked" isn't acceptable. A distributed transaction coordinator needs to know that all participants received the commit message before proceeding. Critical state updates can't be silently lost.

Important Delivery provides guaranteed message delivery through acknowledgment. When you send with the important flag, the framework tracks the message, waits for confirmation from the recipient, and reports errors if delivery fails.

The Problem: Network Opacity

Without important delivery, remote communication is opaque:

The remote Send succeeds even if:

The remote process doesn't exist
The remote process's mailbox is full
The remote node received the message but dropped it

You only discover problems through absence - no response arrives, timeouts fire, but you don't know why. Did the request get lost? Did the process crash? Is it just slow?

The Solution: Confirmed Delivery

Important delivery makes remote communication transparent - errors are immediate, just like local:

The framework sends the message, waits for acknowledgment from the remote node, and reports the outcome. Either the message is in the recipient's mailbox (success) or you get an error explaining what went wrong (failure). No ambiguity.

How to Use Important Delivery

There are two ways to enable important delivery:

Method 1: Per-message explicit methods

Use SendImportant and CallImportant instead of Send and Call:

Method 2: Process-level flag

Set the important delivery flag on the process - all outgoing messages use important delivery:

The process-level flag affects all outgoing messages: Send, SendPID, SendProcessID, SendAlias, and Call requests. You don't need to use special methods - regular Send and Call automatically include the important flag.

Use the flag when the process primarily deals with critical messages. Use explicit methods when only specific messages require guarantees.

How Important Delivery Works

Send with Important Delivery

Here's what happens when you send a message with important delivery:

The sender blocks until the acknowledgment arrives. The remote node attempts delivery and sends either success (ACK) or failure (error). The sender's SendImportant unblocks with the result.

For local sends, the behavior is identical to regular Send - immediate error if the process doesn't exist or mailbox is full. The important flag only affects remote sends.

Call with Important Delivery

Call requests already have a response channel (the caller waits for HandleCall to return), so important delivery works differently. The ACK is only sent if there's an error - if delivery succeeds, no ACK is sent, and the caller waits for the actual response:

The key difference from regular Call: with CallImportant, if the remote process doesn't exist or its mailbox is full, you get an immediate error instead of waiting for timeout. If delivery succeeds, you wait for the response just like regular Call.

Without the important flag, ErrProcessUnknown looks like timeout - you can't tell if the process is slow, dead, or never existed. With important delivery, you know immediately.

Combining Call and Response Delivery

Things get interesting when you combine important delivery on requests with important delivery on responses. There are four combinations, each with different guarantees.

Regular Call + Regular Response

Guarantees: None. Request may be lost. Response may be lost. Timeout is ambiguous.

Use case: Fast, non-critical operations where occasional loss is acceptable.

Regular Call + Important Response (RR-2PC)

Guarantees: Response delivery is confirmed. If the handler returns a result, the caller will receive it (or get an error if delivery fails). Request delivery is not confirmed - the handler might never receive the request.

Protocol name: RR-2PC (Response-Reliable Two-Phase Commit)

Use case: The handler's work is critical, the caller must know if it succeeded. Example: committing a transaction. If the transaction commits, the caller must know. But it's okay if the request gets lost (request is idempotent, can be retried).

How it works:

The handler blocks after processing until the caller acknowledges the response. If the caller crashes before sending ACK, the handler's SendResponseImportant returns ErrResponseIgnored or ErrTimeout.

The request has no guarantee - it might be lost, and the caller would timeout. But if the handler processed the request and sends a response, that response is guaranteed to be delivered.

Important Call + Regular Response

Guarantees: Request delivery is confirmed. The handler will receive the request (or caller gets an error immediately). Response delivery is not confirmed - response may be lost.

Use case: The handler must receive the request, but the response is less critical or can be retried. Example: triggering a background job. The job must start, but if the status response is lost, the caller can query status later.

How it works:

The caller gets immediate confirmation that the request arrived, then waits for the response. If the response gets lost, the caller times out - but knows the handler received and processed the request.

Important Call + Important Response (FR-2PC)

Guarantees: Both request and response delivery are confirmed. The handler definitely receives the request, and the caller definitely receives the response. No ambiguity at any point.

Protocol name: FR-2PC (Fully-Reliable Two-Phase Commit)

Use case: Critical operations where both request and response must be guaranteed. Example: distributed transaction commit coordination, financial operations, critical state synchronization.

How it works:

With FR-2PC:

The caller gets immediate error if request can't be delivered (no ambiguous timeout)
If request is delivered, caller waits for response
The handler blocks after sending response until caller confirms receipt

This is the most reliable pattern but also the most expensive. Use it only when guaranteed delivery is essential.

FR-2PC as Foundation for 3PC

FR-2PC provides the messaging reliability needed to implement Three-Phase Commit (3PC) and other distributed transaction protocols at the application level.

Traditional Two-Phase Commit (2PC) has a blocking problem: if the coordinator crashes after participants vote "yes" but before sending commit/abort, participants don't know what to do. They're stuck.

Three-Phase Commit solves this by adding a pre-commit phase:

Prepare: Can you commit?
Pre-commit: Everyone said yes, get ready to commit
Commit: Now commit

If the coordinator crashes after pre-commit, participants know the outcome was "commit" and can proceed independently.

But 3PC only works if messages are reliably delivered. If a pre-commit message gets lost and a participant doesn't receive it, the protocol breaks - some participants think we're committing, others are still waiting.

FR-2PC guarantees that messages are delivered or errors are reported. This lets you implement 3PC confidently:

FR-2PC ensures that:

If CallImportant returns nil, the participant received the message
If CallImportant returns an error, the participant didn't receive the message
No ambiguous timeouts where you don't know if the message arrived

This determinism is essential for 3PC. Without it, you'd need complex timeout-based recovery that can't distinguish "participant is slow" from "participant is dead" from "message was lost."

Performance Considerations

Important delivery adds overhead:

Extra round trip: Sender waits for ACK before proceeding
Sender blocks: Can't process other messages while waiting
Network traffic: Additional ACK messages

For SendImportant, the sender blocks until ACK arrives (success or error) or timeout. For CallImportant, the sender gets immediate error if delivery fails, or waits for response if delivery succeeds (no extra ACK on success).

The blocking is process-local - only the sending actor waits. Other actors on the node continue normally. But the sending actor's mailbox isn't processed during the wait.

Use important delivery selectively:

Use for: Critical state updates, transaction coordination, payment processing, data synchronization
Don't use for: High-frequency updates, informational messages, monitoring events, retryable operations

Most actor communication doesn't need guarantees. The actor model is resilient because actors handle partial failure gracefully. Important delivery is for the cases where partial failure isn't acceptable - where certainty is worth the cost.

Local vs Remote Behavior

Important delivery only affects remote communication. For local sends:

Local mailbox operations are synchronous - pushing to the mailbox either succeeds or fails immediately. The important flag is unnecessary because there's no network uncertainty. The framework silently treats local important sends as regular sends.

This means your code works identically for local and remote processes. You can use SendImportant everywhere without checking if the target is local or remote - the framework optimizes local communication automatically.

Error Types

Important delivery produces specific errors:

ErrProcessUnknown - The remote process doesn't exist. Without important delivery, you'd discover this through timeout. With important delivery, you know immediately.

ErrProcessMailboxFull - The remote process exists but its mailbox is full. Without important delivery, the message would queue in the network layer or be dropped. With important delivery, you get immediate feedback.

ErrTimeout - The remote node received the message but didn't send ACK within the timeout period. This is different from Call timeout - it means the node is unresponsive or overloaded.

ErrResponseIgnored - For important responses, the caller is no longer waiting (timed out or terminated). The response couldn't be delivered. Without important delivery, the handler wouldn't know the response was ignored.

ErrNoConnection - Cannot establish connection to the remote node. This error occurs for both regular and important sends, but important delivery surfaces it immediately instead of silently queueing.

These specific errors let you handle different failure modes appropriately - retry for ErrTimeout, provision more resources for ErrProcessMailboxFull, fail immediately for ErrProcessUnknown.

Summary

Important delivery trades performance for certainty. Messages are guaranteed to be delivered or errors are reported immediately. Use it when:

The operation is critical and must succeed or definitely fail
Ambiguous timeouts are unacceptable
You're implementing distributed protocols that require guaranteed delivery

For most actor communication, fire-and-forget messaging is sufficient. The actor model handles uncertainty through supervision, retries, and eventual consistency. Important delivery is for the cases where uncertainty itself is the problem.

For more on handling synchronous requests, see .

Network Transparency

Making distributed communication feel local

Network transparency means the location of a process - whether it's in the same goroutine, on the same node, or on a remote node halfway across the world - doesn't change how you interact with it. You send messages the same way. You make calls with the same API. You establish links and monitors with the same methods. The framework handles the complexity of discovering nodes, encoding messages, and routing them across the network.

This isn't just convenient. It's fundamental to building distributed systems in the actor model. If remote operations looked different from local operations, you'd be constantly checking location and branching your logic. That locality awareness would spread throughout your code, making it brittle and hard to reason about. Network transparency lets you design systems as collections of communicating actors, and deployment topology becomes an operational concern rather than a code concern.

But transparency has limits. Networks are slower than in-process communication. They fail in ways local operations don't. Messages can be lost. Connections drop. Remote nodes crash or become unreachable. The framework makes remote operations look local, but the network's physical reality still matters.

What Transparency Means in Practice

Consider a simple example. You have a gen.PID and you want to send it a message:

This code is identical whether pid points to a local process or a remote one. You don't check. You don't call different methods. You just send.

Behind the scenes, the framework does different things:

For a local process: The message is placed directly in the recipient's mailbox queue. The framework checks the priority, selects the appropriate queue (Main, System, or Urgent), and pushes the message. If the process is sleeping, it wakes up. The entire operation happens in microseconds.

For a remote process: The node extracts the node name from the gen.PID, checks if a connection to that node exists, discovers the node's address if needed, establishes a connection pool if necessary, encodes your OrderRequest using EDF, wraps it in a protocol frame, sends it over TCP, and waits for the remote node to acknowledge delivery. The remote node receives the frame, decodes it, routes it to the recipient's mailbox, and sends an acknowledgment back. This takes milliseconds.

From your code's perspective, both operations look identical. The framework abstracts the complexity.

The Transparency Illusion

Network transparency is an illusion carefully maintained by the framework. Several mechanisms work together to create this effect.

Unified addressing - Every process has a gen.PID that includes the node name. Local and remote processes have the same identifier structure. You don't need different types for "local process" and "remote process". A gen.PID is just a gen.PID, and it works everywhere.

Automatic routing - When you send to a process, the framework examines the node portion of the identifier. If it matches the local node, the message is delivered locally. If it doesn't match, the framework initiates discovery to find the remote node and routes the message over the network. You don't trigger this logic explicitly - it happens automatically.

Location independence - You can receive a gen.PID from anywhere - as a return value, in a message, from a registry lookup - and immediately use it for communication. You don't need to check where it's from or set up connections. The framework handles it.

Failure semantics - When you send to a local process that doesn't exist, you get an error immediately. When you send to a remote process that doesn't exist, you get... nothing, by default. The message is sent over the network, and if nobody's listening, it's silently dropped. This asymmetry breaks the transparency illusion. The Important delivery flag fixes this: with Important enabled, sending to a missing remote process gives you an immediate error, just like local delivery. The framework makes the network behave like local memory.

How Messages Cross The Network

When you send a message to a remote process, what actually happens? The framework performs a complex series of operations to transform your Go value into bytes, transmit them over TCP, and reconstruct them on the receiving side. Understanding this flow helps you design efficient distributed systems and debug problems when they arise.

The sequence diagram below shows the complete message transmission pipeline, from the moment you call Send to the moment the recipient's HandleMessage is invoked:

When you send a message, the framework:

Encodes your value using EDF, transforming it into a byte sequence
Compresses it if the message exceeds the compression threshold (default 1024 bytes)
Frames it with protocol headers containing metadata (message type, sender, recipient, priority)

The remote node reverses this:

Reads the frame from the TCP connection
Decompresses if the compression flag is set
Decodes the bytes back into a Go value using EDF

This entire pipeline is invisible. You call Send, and the framework executes these steps. The receiving process calls HandleMessage, and it receives your value as if you'd passed it locally.

EDF: Ergo Data Format

EDF (Ergo Data Format) is a binary serialization format designed for distributed actor systems. It solves a fundamental problem: how do you serialize Go values - structs, slices, maps, framework types like gen.PID - across the network with the performance of code-generated serializers like Protocol Buffers, but without requiring code generation?

The answer is dynamic specialization. When you register a type, EDF analyzes its structure and builds specialized encoding and decoding functions specifically for that type. For structs, it creates functions for each field and composes them into a single encoder. This happens once at registration time, not during encoding. When you send a message, EDF uses these pre-built functions - no reflection, no runtime type analysis.

This approach delivers Protocol Buffers-class performance without .proto files or protoc code generation.

Registration happens at runtime - no build step, no generated files. You call edf.RegisterTypeOf() in your init() function, and EDF builds the optimized encoders. Framework types like gen.PID, gen.Ref, and gen.Event have native support with specialized encodings. During node handshake, both sides exchange their registered type lists and negotiate short numeric IDs, turning a full type name into 3 bytes on the wire. Field names aren't encoded - only field values in declaration order.

Performance benchmarks (see benchmarks/serial/) show encoding is 50-100% faster than Protocol Buffers, while decoding is 20-60% slower. The encoding advantage comes from the specialized functions built during registration.

EDF enforces strict type contracts - both nodes must register identical type definitions. Type identity is the full package path plus type name, not just the type name. For example, Order in package github.com/myapp/orders becomes #github.com/myapp/orders/Order. Two packages with the same type name Order are different types in EDF - this is Go's type system enforced at the protocol level.

This strict typing is a deliberate design choice that pushes version management to the application level. When you need to evolve a message type, you version it explicitly in your code:

Your actors handle both versions, routing logic based on the type received. This approach is essential for canary deployments where old and new versions coexist - each node declares what it understands, and the application code manages compatibility. Protocol-level backward compatibility would hide versioning from your code, making canary rollouts harder to control.

Type Constraints

EDF imposes size limits on certain types. These limits balance memory safety with practical message sizes.

Atoms (gen.Atom) - Maximum 255 bytes. Atoms are used for names - node names, process names, event names. Names longer than 255 bytes are uncommon and likely indicate a design issue. The 255-byte limit keeps name handling efficient.

Strings - Maximum 65,535 bytes (2^16-1). This covers most string use cases. For larger text (documents, logs, large payloads), use binary encoding ([]byte) instead, which supports up to 4GB.

Errors - Maximum 32,767 bytes (2^15-1). Error messages longer than 32KB are unusual. If you need to send detailed diagnostic information, use a separate field in your message struct.

Binary ([]byte) - Maximum 4,294,967,295 bytes (2^32-1, ~4GB). This is the largest single value EDF can encode. Messages containing multi-gigabyte binaries work but are inefficient. Consider chunking large data into multiple messages or using meta processes for streaming.

Collections (map, array, slice) - Maximum 2^32 elements. A map can have up to 4 billion entries. A slice can have 4 billion elements. These limits are unlikely to be hit in practice - a slice of 4 billion int64 values would consume 32GB of memory.

These limits are enforced during encoding. If you attempt to encode a 70,000 byte string, the encoder returns an error. The message isn't sent. On the receiving side, if a malicious sender tries to send an oversized value, the decoder rejects it and closes the connection.

Type Registration Requirements

For custom types to cross the network, both sending and receiving nodes must register them. Registration tells EDF how to encode and decode the type, and creates a numeric ID that's shared during handshake for efficient encoding.

Registration Requirements

Only exported fields - Structs must have all fields exported (starting with uppercase). This is by design: exported fields define your actor's contract. When actors communicate - locally or across the network - they exchange messages according to explicit contracts. Unexported fields are implementation details, internal state that shouldn't cross actor boundaries. If registration encounters unexported fields, it fails with "struct Order has unexported field(s)".

No pointer types - EDF rejects pointer types and structs containing pointer fields. This is by design: pointers are a local memory optimization and shouldn't be part of network contracts. A *Database field is meaningless to a remote actor - it can't dereference your memory address. Pointers express local sharing semantics that don't translate across address spaces.

For distributed references, use framework types designed for remote access: gen.PID (process reference), gen.Alias (named reference), gen.Ref (call reference). These work across nodes and provide location-independent semantics.

Nested types must be registered first - If your type contains other custom types, register the inner types before the outer type:

The order matters because registration builds the encoding schema by examining fields. When registering Person, EDF sees the Address field. If Address isn't registered yet, registration fails with "type Address must be registered first". If Address is already registered, EDF references its schema, creating an efficient nested encoding.

Custom Marshaling for Special Cases

If your type has unexported fields or needs special encoding, implement custom marshaling:

EDF supports both edf.Marshaler/Unmarshaler and Go's standard encoding.BinaryMarshaler/Unmarshaler interfaces. The key difference is performance: edf.Marshaler writes directly to EDF's internal buffer (io.Writer), avoiding intermediate allocations. When you call MarshalEDF(w), the io.Writer is EDF's reusable buffer - your bytes go straight to the wire. With encoding.BinaryMarshaler, you must allocate and return a []byte, which EDF then copies into its buffer.

For high-throughput message types, prefer edf.Marshaler. For types that implement standard interfaces or rarely-sent messages, encoding.BinaryMarshaler works fine.

Encoding Errors

Go's error type is an interface. Encoding an error requires special handling because interfaces don't have a fixed structure.

Framework errors (gen.ErrProcessUnknown, gen.TerminateReasonNormal, etc.) are pre-registered when the node starts. They have numeric IDs and encode compactly as 3 bytes: type tag 0x9c + 2-byte ID.

Custom errors need registration:

Registered errors encode as 3 bytes total (type tag + 2-byte ID where ID > 32767). Unregistered errors encode as type tag + 2-byte length + error string bytes. On decoding, the framework checks if it has a local error with that string. If it does, it returns the local error instance. If not, it creates a new error with fmt.Errorf(string).

This means error identity can be preserved across nodes if both sides register the error. If only one side registers it, you get an error with the correct message but not the same instance. Code comparing errors with errors.Is needs both sides to register for correct behavior.

Type Registration Timing

Type registration must happen before connection establishment. During handshake, nodes exchange their registered type lists and error lists. These lists become the encoding dictionaries for that connection.

If you register a type after a connection is established, that type isn't in the dictionary. Attempting to send a value of that type fails - the encoder can't find it in the shared schema. The only way to use the newly registered type is to disconnect and reconnect, forcing a new handshake that includes the type.

This is why registration typically happens in init() functions. The registration runs before main(), which runs before node startup, which runs before any connections are established. By the time connections form, all types are registered.

For dynamic type registration (registering types based on runtime configuration or plugin loading), you have limited options:

Register before node start - Load your configuration, determine which types you need, register them all, then start the node. This works but requires knowing all types upfront.

Coordinate reconnection - Register the new type, disconnect existing connections to nodes that need the type, wait for reconnection with new handshake. This is complex and causes temporary communication loss.

Use custom marshaling - Implement edf.Marshaler/Unmarshaler or encoding.BinaryMarshaler/Unmarshaler. These don't require pre-registration - they work immediately. The tradeoff is you write the encoding logic yourself.

Most applications register types statically in init() and avoid these complications.

Compression

Large messages are automatically compressed to reduce network bandwidth. Compression is transparent - you configure it on the process or node, and the framework applies it automatically when appropriate.

When compression is enabled, the framework checks the encoded message size before transmission. If it exceeds the compression threshold (default 1024 bytes), the message is compressed using the configured algorithm. The protocol frame's message type (byte 7) is set to 0xc8 (200, protoMessageZ) and byte 8 contains the compression type ID (100=LZW, 101=ZLIB, 102=GZIP), so the receiving node knows to decompress before decoding.

Configure compression in process options:

Or adjust it dynamically:

Type determines the compression algorithm. GZIP (ID=102) provides good compression ratios with reasonable speed. ZLIB (ID=101) is similar but with slightly different format. LZW (ID=100) is faster but produces lower compression. Choose based on your CPU/bandwidth tradeoff.

Level trades compression time for compression ratio. CompressionLevelBestSize produces smaller messages but takes longer. CompressionLevelBestSpeed compresses quickly but produces larger output. CompressionLevelDefault balances both.

Threshold sets the minimum size for compression. Messages smaller than the threshold aren't compressed, even if compression is enabled. Compressing tiny messages adds overhead without reducing size meaningfully. The default 1024 bytes is reasonable - messages below 1KB go uncompressed, larger messages get compressed.

Compression happens per-message. Each message is independently compressed or not, based on its size. This keeps compression stateless and allows the receiver to decode messages in any order.

Caching and Optimization

During handshake, nodes exchange caching dictionaries for frequently used values. This caching reduces message sizes significantly.

Atom caching - Node names, process names, event names - these atoms appear repeatedly in messages. Every gen.PID contains the node name. Every message frame contains sender and recipient identifiers. Instead of encoding "mynode@localhost" repeatedly (2-byte length + 17 bytes = 19 bytes), the handshake assigns it a numeric ID. Cached atoms encode as 2 bytes (uint16 ID, where ID > 255). All subsequent uses of that atom encode as the 2-byte ID.

Type caching - Registered types get numeric IDs. A User struct registered on both sides gets an agreed-upon ID. Messages containing User values encode the ID instead of the full type name and structure. A typical struct name like "#mypackage/User" might be 20-30 bytes - cached, it's 3 bytes (0x83 + 2-byte cache ID where ID > 4095).

Error caching - Registered errors get IDs. Framework errors are pre-registered with well-known IDs. Custom errors get IDs during handshake. Error responses that might encode as 50+ bytes (error string message) encode as 3 bytes with caching (type tag + 2-byte ID where ID > 32767).

The caches are bidirectional - both nodes maintain the same mappings. During encoding, the sender looks up the cache and uses IDs. During decoding, the receiver looks up IDs and reconstructs values. The cache persists for the connection lifetime. If the connection drops and reconnects, a new handshake creates a new cache.

This caching is automatic. You don't manage the cache or invalidate entries. The framework handles it. You just benefit from smaller messages.

Important Delivery

Network transparency breaks down when dealing with failures. Sending to a local process that doesn't exist returns an error immediately - the framework checks the process table and sees the PID isn't registered. Sending to a remote process that doesn't exist returns... nothing. The message is encoded, sent to the remote node, and the remote node silently drops it because there's no recipient. Your code doesn't know the process was missing.

This asymmetry makes debugging difficult. Is the remote process slow to respond, or does it not exist? Did the message get lost in the network, or was it never received? The fire-and-forget nature of normal Send provides no feedback.

The Important delivery flag fixes this:

With Important delivery:

The message is sent to the remote node with an Important flag in the frame (bit 7 of priority byte set)
The remote node attempts delivery to the recipient's mailbox
If delivery succeeds, the remote node sends an acknowledgment back

If the acknowledgment arrives, SendImportant returns nil. If an error response arrives, it returns the error. If the timeout expires, it returns gen.ErrTimeout.

This gives you the same semantics as local delivery: immediate error feedback when something goes wrong. The network becomes transparent for failures too, not just successes.

The cost is latency. Normal Send returns immediately - it queues the message and continues. SendImportant blocks until the remote node responds, adding a network round-trip. For messages that must be delivered, this cost is worth it. For best-effort messages where occasional loss is acceptable, stick with normal Send.

For detailed exploration of Important Delivery patterns, reliability guarantees, and protocols like RR-2PC and FR-2PC, see .

Protocol Frame Structure

EDF-encoded messages are wrapped in ENP (Ergo Network Protocol) frames for transmission over TCP.

Each frame has an 8-byte header:

Byte 0: Magic byte (78 for ENP)
Byte 1: Protocol version (1 for current version)
Bytes 2-5: Frame length (uint32, total size in bytes)

For PID messages, the frame contains:

Sender PID (8 bytes - just the ID, node is known from connection)
Priority byte (bits 0-6 = priority 0-2, bit 7 = Important delivery flag)
Optional reference (8 bytes - first uint64 of Ref.ID, only if Important)

The order byte (byte 6) preserves message ordering per sender. It's calculated as senderPID.ID % 255, ensuring messages from the same sender have the same order value. This guarantees sequential processing on the receiving side even if messages arrive on different TCP connections in the pool. Messages from different senders have different order values, enabling parallel processing.

When the receiving node reads a frame from TCP, it extracts the order byte and routes the frame to the appropriate receive queue. The connection creates 4 receive queues per TCP connection in the pool. So a 3-connection pool has 12 receive queues total. Frames are distributed to queues based on order_byte % queue_count. Each queue is processed by a dedicated goroutine that decodes frames and delivers messages to recipients. This parallel processing improves throughput while preserving per-sender ordering.

Limits of Transparency

Network transparency is powerful but not magical. The network has physical properties that can't be abstracted away.

Latency - Remote operations are slower. A local Send takes microseconds. A remote Send takes milliseconds. That's three orders of magnitude. For a single message, it's negligible. For thousands of messages, the difference is dramatic. Design systems to minimize remote calls, batch operations, and use asynchronous patterns.

Bandwidth - Network links have finite capacity. Sending millions of small messages can saturate a network connection. Encoding and decoding adds CPU overhead. Compression helps but costs CPU time. Be mindful of message volume and size. Local operations have effectively infinite bandwidth - remote operations don't.

Failures - Networks fail in ways local memory doesn't. Packets get lost. Connections drop. Nodes become unreachable. DNS fails. Firewalls block traffic. Local operations either succeed instantly or fail with a clear error. Remote operations can timeout, leaving you uncertain whether they succeeded. Design for these failure modes with timeouts, retries, and idempotent operations.

Partial failures - In a distributed system, some nodes can fail while others continue working. A local system either works entirely or crashes entirely. A distributed system can be partially operational - some nodes reachable, others not. This partial failure is the hardest aspect of distributed systems. The framework can't hide it entirely.

Ordering - Message ordering is preserved per-sender within a connection. Messages from process A to process B arrive in the order sent. But messages from different senders can interleave arbitrarily. And if a connection drops and reconnects, messages sent during disconnection are lost or delayed. Don't assume global ordering across the cluster.

Network transparency makes distributed programming feel local. But distributed programming has fundamental differences from local programming. The transparency is a tool that simplifies common cases - it doesn't eliminate the need to think about distributed system challenges.

Practical Implications

Understanding network transparency helps you design better distributed systems.

Use local clustering - Group processes that communicate frequently on the same node. If processes exchange hundreds of messages per second, put them locally. Their communication is microseconds instead of milliseconds, and you avoid network overhead.

Prefer async over sync - Use Send (asynchronous) instead of Call (synchronous) for remote communication when possible. Async messaging doesn't block the sender, improving throughput. Sync calls over the network tie up your process waiting for responses.

Design for message batching - Send one message with 100 items instead of 100 messages with 1 item each. Network overhead is per-message. Batching amortizes that overhead.

Handle failures explicitly - Use timeouts on sync calls. Use Important delivery for critical messages. Monitor connection health. Don't assume remote operations succeed - check errors and have fallback logic.

Keep messages small - Encoding and network transmission costs scale with message size. Large messages cause memory allocation, encoding overhead, network congestion. If you're sending megabytes of data, consider whether it belongs in messages or should use a different mechanism (file transfer, streaming, database).

Leverage compression - Enable compression for processes that send large messages. The CPU cost of compression is usually worth the network bandwidth savings. But don't compress tiny messages - the overhead exceeds the benefit.

Register types early - Do all type registration in init() functions before the node starts. Avoid dynamic type registration that requires connection cycling. Static registration is simpler and more reliable.

For details on how the network stack implements transparency, see . For understanding how nodes discover each other, see .

Web

HTTP and actors speak different languages. HTTP is fundamentally synchronous - a request arrives, blocks waiting for processing, gets a response, connection closes. The actor model is fundamentally asynchronous - messages arrive in mailboxes, get processed sequentially one at a time, responses are separate messages sent whenever ready.

Integrating these two worlds is possible, but the integration strategy matters. Choose wrong and you lose the benefits of both models. Choose right and you get HTTP's ubiquity with actors' concurrency and distribution capabilities.

This chapter shows two integration approaches, ordered from simple to complex. The simple approach works for most cases and keeps the entire HTTP ecosystem available. The meta-process approach trades tooling for deeper actor integration, enabling patterns impossible with standard HTTP stacks.

Before reaching for meta-processes, understand what you're giving up and what you're gaining. The simple approach might be all you need.

Simple Approach: Call from HTTP Handlers

The straightforward way: run a standard HTTP server, call actors from handlers using node.Call(), let network transparency distribute requests across the cluster.

This keeps HTTP and actors separate. HTTP handles protocol concerns - routing, middleware, headers, status codes. Actors handle business logic - state management, processing, coordination. Clean separation.

Basic Pattern

The HTTP server runs outside the actor system in a separate goroutine. Handlers call actors synchronously using node.Call(). Actors can be anywhere - same node, remote node, doesn't matter. Network transparency routes the call.

Why This Works

Call() blocks the HTTP handler goroutine, not an actor. Go's HTTP server creates one goroutine per connection. Blocking in a handler is normal - that goroutine waits, others continue serving requests.

The actor receiving the call processes it asynchronously in its own message loop. Multiple handlers can call the same actor concurrently. The actor processes one request at a time from its mailbox. This isolates the actor from HTTP concurrency.

Network transparency means the actor can be anywhere:

Change Node to move the actor. Code stays the same. Distribute load across nodes by routing different requests to different actors.

Cluster Load Distribution

Network transparency means actors can run anywhere in the cluster. The HTTP gateway becomes a router that distributes requests across backend nodes.

Simple consistent hashing distributes load evenly while maintaining request affinity:

Requests for user "alice" always go to the same backend node. That node caches alice's data in memory. Subsequent requests hit warm cache. Change clusterSize to add nodes - hashing redistributes load automatically while preserving most affinity.

For dynamic topology where nodes join and leave unpredictably, use application discovery. Central registrars (etcd, Saturn) track which nodes are running which applications in real-time:

Application discovery returns all nodes currently running the service. Each node reports its weight. Nodes with higher weights (more resources, better hardware, closer proximity) receive proportionally more traffic. Nodes that crash disappear from discovery immediately. New nodes appear as soon as they register. The HTTP gateway adapts to cluster topology changes without restarts.

For details on application discovery and central registrars, see .

Standard HTTP Tooling

This approach keeps the entire HTTP ecosystem available:

OpenAPI generation: Tools like swag/swaggo analyze HTTP handlers and generate OpenAPI specs. They see standard net/http handlers, so generation works normally.

Middleware: Standard HTTP middleware wraps handlers - authentication, logging, CORS, rate limiting. Actors are completely invisible to middleware.

Routing: Use any router - http.ServeMux (Go 1.22+), gorilla/mux, chi, echo. They all work with standard handlers.

Testing: Test HTTP handlers with httptest. Test actors separately with unit tests. Clean separation of concerns.

The actor system is an implementation detail. HTTP sees standard handlers. Clients see standard HTTP. Deployment tools see standard HTTP servers. Only the handler implementation uses actors internally.

When This Approach Works

Use this when:

You need standard HTTP tooling (OpenAPI, gRPC-gateway, middleware ecosystems)
Load balancing happens at the nginx/kubernetes level, not actor level
Backpressure from actors doesn't matter (actors process at their speed, HTTP clients wait)

This covers most HTTP/actor integration cases. The HTTP layer is stateless. Actors hold state and logic. HTTP routes requests to actors. Clean architecture.

For details on synchronous request handling in actors, see .

Meta-Process Approach: Deep Integration

Meta-processes convert HTTP into asynchronous actor messages. Instead of calling actors synchronously from handlers, requests become messages flowing into the actor system.

This approach enables:

Backpressure: actors control request rate through mailbox capacity
Addressable connections: each WebSocket/SSE connection becomes an independent actor with gen.Alias identifier - any actor anywhere in the cluster can send messages directly to specific client connections through network transparency. This is the killer feature for real-time systems (chat, multiplayer games, live dashboards, collaborative editing) where backend logic must push updates to specific clients across cluster nodes. Impossible with the simple approach.
Per-request routing: route to different actor pools based on request content

Standard HTTP routing and middleware still work - meta.WebHandler implements http.Handler and integrates with http.ServeMux or any router. You can wrap handlers in middleware for authentication, logging, CORS. What you lose is introspection-based tooling (OpenAPI generation, gRPC-gateway) because request processing happens inside actors, invisible to HTTP layer analysis tools.

Architecture

Two meta-processes work together:

meta.WebServer: External Reader runs http.Server.Serve(listener). Blocks there forever until listener fails. The http.Server creates its own goroutines for each HTTP connection - those goroutines call handlers, not the External Reader. Actor Handler never runs (no messages received).

meta.WebHandler: Implements http.Handler interface. External Reader blocks in Start() waiting for termination. When http.Server (running in WebServer) accepts a connection, it spawns a goroutine that calls handler.ServeHTTP(). Inside ServeHTTP():

Create context with timeout
Send meta.MessageWebRequest to worker actor
Block on <-ctx.Done() waiting for worker to call Done()

Actor Handler never runs - HandleMessage() and HandleCall() are empty stubs.

Worker actors receive meta.MessageWebRequest containing:

http.ResponseWriter - write response here
*http.Request - the HTTP request
Done() function - call this to unblock ServeHTTP

Data Flow

Compare this with typical meta-processes like TCP or UDP:

TCP/UDP meta-processes:

External Reader actively loops reading from socket, sends messages to actors
Actor Handler receives messages from actors, writes to socket
Both goroutines do real work - continuous bidirectional I/O

Web meta-processes:

WebServer's External Reader passively blocks in http.Server.Serve() doing nothing - http.Server does all the work internally
WebHandler's External Reader passively blocks on channel doing nothing - just waiting for termination
Neither has an active Actor Handler - no messages arrive in their mailboxes

Web meta-processes are unusual. They use the meta-process mechanism not for bidirectional I/O but for lifecycle management and integration with the actor system. The External Reader goroutines exist only to keep the meta-process alive while http.Server runs. The actual HTTP handling happens in goroutines spawned by http.Server, which are completely outside the meta-process architecture.

This works because http.Server already solves concurrency - it spawns goroutines per connection. The meta-process just wraps it for integration with actor lifecycle and messaging.

Basic Setup

When a request arrives:

WebServer's External Reader is blocked in http.Server.Serve()
http.Server accepts connection, spawns its own goroutine for this connection
That goroutine calls handler.ServeHTTP() (handler is WebHandler)

Critical: ServeHTTP() executes in http.Server goroutines, not in meta-process goroutines. WebHandler's External Reader remains blocked in Start() waiting for termination. WebHandler's Actor Handler never spawns because no messages arrive in its mailbox.

Worker Implementation

Workers receive meta.MessageWebRequest as regular messages in their mailbox:

The pattern: receive MessageWebRequest, process it, write to ResponseWriter, call Done(). The Done() call unblocks the ServeHTTP() goroutine waiting in WebHandler.

Using act.WebWorker: Framework provides act.WebWorker that automatically extracts MessageWebRequest, routes to HTTP-method-specific callbacks (HandleGet, HandlePost, etc.), and calls Done() after processing. Use this instead of manual message handling - it eliminates boilerplate and ensures Done() is always called. See for details.

Concurrent Processing with act.Pool

Single worker processes requests sequentially. Use act.Pool to process multiple requests concurrently:

Pool distributes incoming requests across 20 workers. Each worker processes one request at a time. System handles 20 concurrent requests.

Capacity control: PoolSize × WorkerMailboxSize defines maximum requests the backend accepts. With 20 workers and 10 mailbox size, system capacity is 220 requests (20 processing + 200 queued). Beyond this, requests are shed - pool cannot forward to workers with full mailboxes.

This limits load on backend systems. Database handles 20 concurrent queries maximum. External API gets 20 parallel requests maximum. Worker mailboxes buffer bursts without overwhelming downstream services.

Worker failures are handled automatically. Pool spawns replacement workers when crashes are detected. Other workers continue processing during restart.

Stateful Connections: WebSocket

HTTP request-response is stateless. WebSocket is the opposite - long-lived bidirectional connections remaining open for hours or days.

The framework provides WebSocket meta-process implementation in the extra library (ergo.services/meta/websocket). Each connection becomes an independent meta-process with gen.Alias identifier, addressable from anywhere in the cluster.

Each connection is an independent meta-process:

External Reader continuously reads messages from client
Actor Handler receives messages from backend actors, writes to client
Both operate simultaneously - full-duplex bidirectional communication

Killer feature: cluster-wide addressability. Any actor on any node can send messages directly to specific client connections:

Network transparency makes every WebSocket connection addressable like any other actor. Backend logic scattered across cluster nodes can push updates to specific clients without routing through intermediaries.

This is impossible with the simple approach. node.Call() is request-response. WebSocket requires continuous streaming both directions. Meta-processes provide the architecture: one goroutine reading from client, another writing to client, both operating on the same connection.

For WebSocket implementation and usage examples, see .

Choosing an Approach

Start with the simple approach. Use node.Call() from standard HTTP handlers. This works for most cases and keeps the entire HTTP ecosystem available - OpenAPI generation, middleware, familiar patterns.

Move to meta-processes when you specifically need:

WebSocket or long-lived connections: Each connection must be an addressable actor that backend logic can push updates to. The simple approach cannot do this - it's request-response only. Meta-processes make each connection an independent actor with cluster-wide addressability.

Capacity control through mailbox limits: Backend accepts exactly PoolSize × WorkerMailboxSize requests, no more. Beyond this, requests are rejected. This prevents memory exhaustion during overload. The simple approach queues unbounded requests in HTTP server.

The simple approach handles thousands of requests per second with proper actor distribution. Use meta-processes only when the simple approach cannot provide required capabilities.

Actor

The actor model requires sequential message processing - each actor handles one message at a time in a dedicated goroutine. This eliminates data races within the actor but shifts complexity to the message handling loop: reading from multiple mailbox queues in priority order, dispatching to different handlers based on message type, managing state transitions, converting exit signals to regular messages when trapping is enabled.

You could implement this yourself with gen.ProcessBehavior, but you'd rewrite the same logic for every actor. act.Actor solves this. It implements the low-level gen.ProcessBehavior interface and provides a higher-level act.ActorBehavior interface with straightforward callbacks: Init for initialization, HandleMessage for asynchronous messages, HandleCall for synchronous requests, Terminate for cleanup. You write business logic, act.Actor handles the mailbox mechanics.

Creating an Actor

Embed act.Actor in your struct and implement the act.ActorBehavior callbacks you need:

Spawn it like any process:

The factory function is called each time you spawn. Each process gets a fresh instance with its own state. This isolation is fundamental to the actor model - actors share nothing except messages.

Callback Interface

act.ActorBehavior defines the callbacks act.Actor will invoke:

All callbacks are optional. act.Actor provides default implementations that log warnings for unhandled messages. Implement only what you need.

Since act.Actor embeds gen.Process, you have direct access to all process methods: Send, Call, Spawn, Link, RegisterName, etc. No need to store references - they're built in.

Initialization

Init runs once when the process spawns. The args parameter contains whatever you passed to Spawn:

If Init returns an error, the process is cleaned up and removed. Spawn returns immediately with that error. Use this for validation: check arguments, verify resources, refuse to start if preconditions aren't met.

During Init, the process is in ProcessStateInit. All operations are available: Spawn, Send, SetEnv, RegisterName, CreateAlias, RegisterEvent, Link*, Monitor*, Call*, and property setters.

Any resources created during Init (names, aliases, events, links, monitors) are properly cleaned up if initialization fails.

Message Handling

Messages arrive in the mailbox and sit in one of four queues: Urgent, System, Main, or Log. act.Actor processes them in priority order:

Urgent - Maximum priority messages (MessagePriorityMax)
System - High priority messages (MessagePriorityHigh)
Main - Normal priority messages (MessagePriorityNormal

When a message arrives in Urgent, System, or Main, act.Actor calls HandleMessage:

The return value determines whether the actor continues or terminates:

Return nil to keep running
Return gen.TerminateReasonNormal for clean shutdown
Return any other error to terminate (logged as error)

The from parameter tells you who sent the message. Use it for replies. If you don't need replies, ignore it.

Synchronous Requests

When someone calls process.Call(pid, request), act.Actor invokes your HandleCall:

The error return value controls process termination, not the caller's response:

(result, nil) - Send result to caller, continue running
(result, gen.TerminateReasonNormal) - Send result, then terminate cleanly

To send an application error to the caller, return it as the result value:

This separation between transport errors (err return from Call) and application errors (result as error) is fundamental to actor communication. See for deeper discussion of error channels and when to use SendResponseError.

Asynchronous Handling of Synchronous Requests

Sometimes you can't respond immediately. Maybe you need to query another service, or delegate work to a pool of workers. Return (nil, nil) from HandleCall to defer the response:

The gen.Ref identifies the request. The caller blocks waiting for a response with that ref. You can send the response from any process - the one that received the request, a worker, or even a remote process. Just call SendResponse(callerPID, ref, result).

The ref has a deadline (from the caller's timeout). Check if it's still alive before doing expensive work:

Termination

To stop an actor, return a non-nil error from HandleMessage or HandleCall:

Termination reasons:

gen.TerminateReasonNormal - Clean shutdown, not logged as error
gen.TerminateReasonKill - Process was killed via node.Kill(pid)
gen.TerminateReasonPanic

After termination is triggered, act.Actor calls your Terminate callback:

At this point, the process is in ProcessStateTerminated and has been removed from the node. Most gen.Process methods return gen.ErrNotAllowed. You can still send messages (fire-and-forget), but you can't make calls, create links, or spawn children.

If a panic occurs during Init, HandleMessage, or HandleCall, the framework catches it, logs the stack trace, and terminates the process with gen.TerminateReasonPanic. The Terminate callback still runs, giving you a chance to clean up.

Trapping Exit Signals

By default, when an actor receives an exit signal (via SendExit or from a linked process), it terminates immediately. Enable TrapExit to convert exit signals into regular messages:

Exit signal messages:

gen.MessageExitPID - From a process (SendExit or link)
gen.MessageExitProcessID - From a named process link
gen.MessageExitAlias

Exception: Exit signals from the parent process cannot be trapped. If your parent terminates (and you created a link with LinkParent option or via Link/LinkPID), you terminate regardless of TrapExit. This ensures supervision trees can forcefully terminate subtrees.

Use TrapExit when you want to handle failures gracefully - log them, restart workers, switch to fallback services. Don't use it if you want standard supervision behavior (child fails → parent restarts it).

Split Handle

By default, HandleMessage and HandleCall are invoked regardless of how the process was addressed - by PID, by registered name, or by alias. Enable SetSplitHandle(true) to route based on address type:

The same split applies to HandleCall* variants. Use this when you want different behavior for internal communication (PID) versus public API (registered name) versus temporary sessions (alias).

Most actors don't need this. Leave split handle disabled and use HandleMessage/HandleCall for everything.

Specialized Callbacks

Logging

If your actor is registered as a logger (via node.AddLogger(pid, level)), it receives log messages in the Log queue:

Log messages have the lowest priority. They're processed after Urgent, System, and Main are empty. This prevents logging from starving regular message processing.

Events

If your actor subscribed to an event (via LinkEvent or MonitorEvent), it receives event messages:

Events arrive in the System queue (high priority). Use them for cross-cutting concerns where multiple actors need to react to the same occurrence.

Inspection

Actors can expose runtime state for monitoring and debugging via the HandleInspect callback:

Inspect the actor from within a process context or directly from the node:

Both methods only work for local processes (same node). Inspection requests go to the Urgent queue and bypass normal message processing. Keep HandleInspect implementation fast - don't do expensive computations or I/O. Return only string values (serialization limitation). The optional item parameters allow filtering which fields to return, though most implementations ignore them and return all fields.

Actor Pools

For workload distribution, use act.Pool instead of implementing manual worker management. See for details.

Patterns and Pitfalls

Don't spawn goroutines in callbacks. The actor model is sequential - one message at a time. Spawning goroutines breaks this, introducing data races on actor state. If you need concurrency, spawn child actors and send them messages.

Don't block on channels or mutexes. Callbacks run in the actor's goroutine. Blocking it starves message processing. Use async message passing (Send) instead of sync primitives.

Don't store gen.Process references. The embedded act.Actor provides all process methods. Storing additional references wastes memory and can cause confusion about which instance is authoritative.

Return errors for termination, not for caller responses. HandleCall's error return terminates the process. To send errors to callers, return them as the result value.

Use ref.IsAlive() before expensive async work. When handling calls asynchronously, check if the caller is still waiting before spending resources on the response.

Enable TrapExit only when needed. Default behavior (terminate on exit signal) works for most actors. Trap only when you have specific failure handling logic.

Handling Sync Requests

Handling synchronous requests in the asynchronous actor model

The actor model is fundamentally asynchronous. Processes send messages and continue immediately without waiting for responses. This asynchrony is core to the model - actors don't block, they process messages one at a time from their mailbox, and they scale because thousands of actors can run concurrently without threads blocking on I/O or responses.

But real systems often need synchronous patterns. A client makes a request and must wait for a response before continuing. An HTTP handler receives a request and can't return to the client until the response is ready. A database query needs to block until the data arrives. These synchronous requirements don't disappear just because your system uses actors.

The challenge is satisfying these synchronous requirements without actually blocking the actor. If an actor blocks waiting for a response, it can't process other messages in its mailbox. The actor becomes unresponsive to everything else. This defeats the purpose of the actor model - you want concurrent message processing, not sequential blocking.

This chapter explores how to handle synchronous-style requests while maintaining asynchronous actor behavior. You'll learn how the framework implements request-response, how to handle Call requests efficiently, and how to process them asynchronously even when the caller is blocked waiting.

The Nature of Synchronous Calls in Actors

In traditional synchronous code, when you call a function, you wait for it to return:

The calling thread stops. The operating system schedules other threads. Eventually the query completes, the thread wakes up, and execution continues. This is fine when you have many threads - some block, others run. But it's wasteful, and it doesn't scale to tens of thousands of concurrent operations.

In the actor model, you send a message and continue:

The sender doesn't block. The message goes into the database actor's mailbox. When the database actor processes it, it sends a response message back. The original sender handles that response later in its own message loop. This is how actors achieve massive concurrency - no actor ever blocks waiting, so you can run thousands of actors with a small thread pool.

But what if the sender legitimately needs to wait? What if it's an HTTP handler that can't return to the client until the query completes?

The framework provides Call for this:

From the caller's perspective, this looks synchronous - you call, you wait, you get a result. But from the system's perspective, it's asynchronous:

The caller sends a request message with a unique reference (gen.Ref)
The caller's goroutine blocks waiting for a response with that reference
The recipient receives the request as a HandleCall invocation

The caller blocks, but blocking is isolated to that one actor. The actor's goroutine is suspended (cheap), not spinning (expensive). Other actors run normally. The recipient processes the request whenever it gets to it in its mailbox, not immediately. The entire system remains asynchronous, but individual actors can use synchronous-style APIs when needed.

Basic HandleCall Implementation

When a process receives a Call request, the framework invokes HandleCall:

Critical distinction: The error you return from HandleCall is not the response to the caller - it's the termination reason for your process!

return result, nil - Send result to caller, continue running
return errorValue, nil - Send errorValue to caller, continue running

When you return a non-nil result from HandleCall, the framework automatically sends it as a response message to the caller. The caller's blocked Call unblocks and returns your result. Any value can be a result - integers, strings, structs, even errors.

If you need to send an error to the caller, return the error as the result value, not as the error return:

The second return value (error) is for terminating your process. Return gen.TerminateReasonNormal to gracefully stop, or any other error for abnormal termination. If you return both a result and gen.TerminateReasonNormal, the framework sends the result first, then terminates your process.

From the caller's side:

The caller blocks at Call until your HandleCall returns. This can be milliseconds (local, fast computation) or seconds (remote, slow operation). The caller can specify a timeout - if no response arrives within the timeout, Call returns nil, gen.ErrTimeout.

Note the distinction: err from Call is a framework-level error (timeout, network failure, process terminated). The result itself might be an error value sent by your HandleCall - that's application-level.

Why Not Just Use Channels?

You might wonder: why not just use Go channels for request-response?

This breaks the actor model in subtle ways:

Shared memory - Channels are shared memory. Passing a channel in a message creates a direct communication path outside the actor system. If the worker is on a remote node, the channel doesn't work (channels don't serialize). Your code becomes non-portable between local and remote.

Blocking semantics - Blocking on a channel blocks the actor's goroutine, but the actor is still "running" from the framework's perspective. The actor can't process other messages while blocked. With Call, the framework knows the actor is waiting for a response and can properly account for it (the actor is in ProcessStateWaitResponse).

Timeout coordination - Channels don't have built-in timeouts. You'd wrap them in select with time.After, but timeout cleanup is tricky. With Call, timeouts are built-in, and references have deadlines that the receiver can check.

No network transparency - Call works identically for local and remote processes. Channels don't. If you use channels for local request-response, your code won't work when you move to a distributed deployment.

The framework's Call mechanism is designed specifically for request-response in the actor model, works across the network, and integrates properly with the actor lifecycle.

Handling Requests with Worker Pools

A common pattern is a server process that receives many Call requests. If processing each request takes time (database query, HTTP call, complex computation), handling them sequentially in HandleCall creates a bottleneck. One slow request delays all subsequent requests.

The solution is act.Pool - a specialized actor that automatically distributes requests across a pool of worker actors:

Notice what's not in this code - there's no HandleCall for the Server. You don't need one.

act.Pool automatically intercepts all incoming Call requests and forwards them to workers. When you send a Call to the Server PID, the Pool:

Receives the Call request in its mailbox
Pops an available worker from the pool
Forwards the entire request (from, ref, message) to the worker

The worker receives the Call request with the original caller's PID and ref. When the worker returns a result from HandleCall, it goes directly to the original caller, bypassing the Pool entirely. The Pool is just a router.

From the caller's perspective:

This gives you concurrent request processing:

10 Call requests arrive at the Server simultaneously
Pool forwards each to a different worker
All 10 workers process concurrently

The caller's experience is unchanged - they call, they block, they get a result. They don't know about the pool. The concurrency is entirely internal to the server.

Worker resilience:

If a worker crashes or becomes unresponsive, the Pool automatically spawns a replacement worker. Worker failures don't affect the Pool's availability - other workers continue processing requests while the Pool restarts failed workers in the background.

If all workers are busy (mailboxes full), incoming requests queue up in the Pool's mailbox until a worker becomes available.

For more details on Pool configuration and advanced patterns, see .

Asynchronous Processing of Synchronous Requests

Sometimes you need to handle a Call request asynchronously within a single actor, without workers. Maybe you're waiting for a timer, or you need to make another Call before you can respond, or you want to batch multiple requests.

You can do this manually:

The pattern:

HandleCall stores from and ref for later
HandleCall returns (nil, nil) - async handling

You must respond eventually, or the caller will timeout. If you lose track of the ref or forget to respond, the caller waits until timeout and gets gen.ErrTimeout.

The result you send with SendResponse can be any value - strings, numbers, structs, even errors. If you want to send an error to the caller, just send it as a normal result value:

The caller receives it as result (first return value from Call) and can check if it's an error.

SendResponse vs SendResponseError: Two Channels for Results

When you handle Call requests asynchronously, you send responses later using SendResponse. But there's also SendResponseError. What's the difference, and when do you use each?

The difference is in which return value the caller receives from Call.

SendResponse sends to the result channel:

Whatever you send appears as the first return value (result). The second return value (err) is nil, meaning no framework error occurred. The result can be anything - strings, numbers, structs, even errors:

The caller must check if the result is an error:

SendResponseError sends to the error channel:

The error appears as the second return value (err), exactly where framework errors like timeout and network failures appear. The first return value (result) is nil.

From the caller's perspective, there's no difference between an error from SendResponseError and a framework error:

The problem with mixing channels

The framework uses the error channel for transport errors - problems with the messaging infrastructure. Your application uses it for business logic results. When you call SendResponseError, you're mixing these two concerns.

Consider a typical caller error handling:

This makes sense for transport errors - network glitches, temporary overload. But if the database actor uses SendResponseError for "record not found", the caller retries unnecessarily. The record won't appear in one second.

The caller has no way to distinguish. Both arrive through the error channel.

When mixing is justified

Despite this issue, SendResponseError has legitimate uses. The key is: use it for errors that should be handled like transport errors.

Imagine a database query actor. It receives queries, executes them against a database, and returns results. What errors can occur?

Application errors - problems with the query itself:

Bad SQL syntax
Permission denied
Constraint violation

These are not infrastructure problems. The actor is working fine, the database is up, the request was processed. The query just has issues. The caller should see these as results, not transport failures.

Infrastructure errors - problems with the database connection:

Database server is down
Network to database lost
Connection pool exhausted
Too many simultaneous connections

These are infrastructure problems. The actor couldn't process the request because a dependency is unavailable. From the caller's perspective, this is the same as if the actor itself were unreachable (timeout) or the node were down (network failure). The caller should handle all of these identically - retry, fallback, circuit breaking.

Here's how to implement this:

The caller handles both channels naturally:

This works because the caller wants to handle infrastructure failures identically, regardless of whether they originate from the framework (timeout, network) or from the application (database down). Both represent unavailable service, both trigger the same fallback logic.

Guideline

Use SendResponse for all normal cases, including expected errors (validation, not found, unauthorized). These are results - the request was processed, here's what happened.

Use SendResponseError only when the error represents an infrastructure failure that the caller should treat the same as transport errors - retry with backoff, circuit breaking, fallback to alternative services.

If in doubt, use SendResponse. It keeps transport and application concerns separate, giving the caller maximum clarity.

Using Ref.IsAlive for Timeout Awareness

When you handle requests asynchronously, the caller might timeout before you respond. Imagine:

Caller makes a Call with 5 second timeout
Your HandleCall stores the request, returns nil (async)
6 seconds pass
Caller's timeout fires,

Your response arrives after the caller stopped waiting. The caller won't receive it (it's not waiting on that ref anymore). Your work was wasted.

You can detect this with ref.IsAlive():

ref.IsAlive() checks the deadline embedded in the reference. When the caller made the Call with a timeout, the framework created a reference with MakeRefWithDeadline(now + timeout). The deadline is stored in ref.ID[2] as a unix timestamp. IsAlive() compares it to the current time - if the deadline passed, it returns false.

This lets you skip processing expired requests. If a request took too long to reach the front of the queue, and the caller already gave up, don't waste resources computing a response nobody will receive.

But be careful: IsAlive() returning false doesn't mean the caller is definitely gone. It means the deadline passed. The caller might have disappeared for other reasons (crash, network disconnect), or they might still exist but already moved on. It's a hint for optimization, not a guarantee about caller state.

If you send a response after the deadline, nothing bad happens. The response message arrives, the receiver checks if anyone is waiting for that ref, finds nobody, and drops the message. It's just wasted work - harmless but inefficient.

Common Patterns and Pitfalls

Pattern: Immediate vs deferred

Some requests you can answer immediately, others need async processing. Mix both in the same HandleCall based on the situation.

Pattern: Batch processing

Accumulate requests, process them together, respond to each individually. Efficient for operations with high setup cost (database connections, API requests with rate limits).

Pitfall: Losing references

You need both from and ref to send a response. Store them together.

Pitfall: Confusing result errors with termination errors

This is the most common mistake. Remember: the error return from HandleCall terminates your process, it doesn't go to the caller (except the special case of gen.TerminateReasonNormal with a non-nil result).

Pitfall: Blocking in HandleCall

Even though the caller is blocked waiting, your actor shouldn't block. If you sleep for 5 seconds, you can't handle other messages during that time. Other callers will queue up waiting. If this is unavoidable (calling a blocking API you don't control), spawn a worker to handle it or use act.Pool.

The Path to Important Delivery

Everything discussed so far assumes the response message arrives. But what if it doesn't? Networks drop packets. Remote processes crash. Connections fail.

When a response is lost, the caller blocks until timeout. Eventually Call returns gen.ErrTimeout, but you don't know if the request was processed or not. Did the receiver handle it and the response got lost? Or did the request itself get lost before reaching the receiver?

This uncertainty is a fundamental problem in distributed systems. The framework's Call mechanism gives you request-response semantics, but it doesn't guarantee the response arrives. It's "best effort" - works reliably for local calls and stable network connections, but no guarantees.

For many use cases, this is fine. Timeouts are acceptable. Callers can retry. Idempotent operations tolerate retries. But some operations can't tolerate uncertainty. A payment authorization must definitely succeed or definitely fail - timeout isn't acceptable.

The solution is Important Delivery. When you enable the Important flag, the framework changes from "best effort" to "confirmed delivery." Responses don't just get sent, they get acknowledged. If the response fails to deliver, you know immediately rather than waiting for timeout.

Important Delivery makes the network transparent for failures, not just successes. It turns request-response from "probably works" into "definitely works or definitely fails, no ambiguity."

We'll explore Important Delivery in depth in the next chapter. For now, understand that everything you've learned about Call and HandleCall still applies. Important Delivery is a layer on top, not a replacement. You'll still handle requests the same way - the framework just makes delivery more reliable.

For details on how messages and calls flow through the network, see . For understanding delivery guarantees, continue to .

Pub/Sub Internals

How the Pub/Sub system works internally

This document explains how Ergo Framework's pub/sub system works under the hood. It's written for developers who want to understand the architecture, network behavior, and performance characteristics when building distributed systems.

For basic usage, see Links and Monitors and Events. This document assumes you're familiar with those concepts and focuses on how the system works internally.

The Unified Architecture

Links, monitors, and events look like separate features when you use them. But underneath, they share the same mechanism. Understanding this unification explains why the system behaves consistently and why certain optimizations work.

The Core Concept

Every interaction in the pub/sub system follows one pattern:

A consumer subscribes to a target and receives notifications about that target.

This applies whether you're linking to a process, monitoring a registered name, or subscribing to an event stream. The differences are in what you subscribe to and what notifications you receive.

Three Components of Every Subscription

1. Consumer - The process creating the subscription. This is the process that will receive notifications when something happens to the target.

2. Target - What the consumer subscribes to. Targets come in several types:

Target Type

Example

What It Represents

3. Subscription Type - How the consumer wants to receive notifications:

Type

Creates

Notification

Effect on Consumer

The combination of target type and subscription type determines what message you receive:

Implicit vs Explicit Events

The targets divide into two categories based on what notifications they generate:

Implicit Events - Processes, names, aliases, and nodes generate termination notifications automatically. The target doesn't do anything special - when it terminates (or disconnects, for nodes), the framework generates notifications for all subscribers.

Explicit Events - Registered events generate both published messages AND termination notifications. A producer process explicitly registers an event and publishes messages to it. When the producer terminates or unregisters the event, subscribers also receive termination notification.

The key difference: implicit events give you one notification (termination). Explicit events give you N published messages plus termination notification.

Why Unification Matters

This unified architecture has practical benefits:

Consistent behavior - The same subscription and notification mechanics work for all target types. Once you understand how monitors work for processes, you understand how they work for events.

Shared optimizations - Network optimizations (covered later) apply to all subscription types. Whether you're monitoring 100 remote processes or subscribing 100 consumers to a remote event, the same sharing mechanism kicks in.

Predictable cleanup - Termination cleanup works identically for all subscriptions. When a process terminates, all its subscriptions are cleaned up using the same code path.

How Local Subscriptions Work

When you subscribe to a target on the same node, the operation is simple and fast.

What You Experience

The call returns instantly. There's no network communication, no blocking. The node records your subscription in memory.

What Happens Internally

The Target Manager maintains subscription records. When a process terminates, Target Manager looks up all subscribers and delivers notifications to their mailboxes. For links, notifications go to the Urgent queue. For monitors, they go to the System queue.

Guarantees

Instant subscription - No waiting, no blocking. The subscription is recorded synchronously.

Guaranteed notification - If the target terminates after you subscribe, you will receive notification. The notification mechanism is part of the termination process itself.

Asynchronous delivery - Notifications arrive in your mailbox like any other message. You process them in your HandleMessage callback.

Automatic cleanup - When the target terminates and you receive notification, the subscription is removed automatically. You don't need to unsubscribe.

How Remote Subscriptions Work

Remote subscriptions involve network communication but provide the same guarantees as local subscriptions.

What You Experience

The call blocks while the subscription request travels to the remote node and the response returns. This typically takes milliseconds on a local network.

What Happens Internally

The subscription request travels to the remote node's Target Manager. It validates that the target exists (returning an error if not), records the subscription, and sends confirmation. Once established, termination notifications travel back over the network.

Subscription Validation

Both local and remote subscriptions validate that the target exists:

For events, the target node validates the event is registered:

Guaranteed Notification Delivery

Remote subscriptions guarantee you receive exactly one termination notification. This guarantee holds even when networks fail. Two paths can deliver your notification:

Path 1: Normal Delivery

The target terminates normally. The remote node sends the notification over the network. You receive it with the actual termination reason:

Path 2: Connection Failure

The network connection fails before the notification arrives (or before the target even terminates). Your local node detects the disconnection and generates notifications for all subscriptions to targets on the failed node:

Why This Works

You're guaranteed notification through one of two mechanisms:

Remote node delivers it (normal case)
Local node generates it when detecting connection failure (failover)

The Reason field tells you which path occurred. Your code typically handles both the same way - the target is no longer accessible regardless of why.

This failover mechanism compensates for network unreliability. You write code assuming notifications always arrive, because they do.

Network Optimization: Shared Subscriptions

This section describes the optimization that makes distributed pub/sub practical at scale. Without it, many common patterns would be impractical.

The Problem

Consider a realistic scenario:

Naive implementation: Each MonitorPID call creates a separate network subscription. Result:

100 network round-trips to create subscriptions
100 subscription records on the remote node
100 network messages when the coordinator terminates

This doesn't scale. With 1000 workers, you'd have 1000 network messages just to deliver one termination notification.

What Actually Happens

The framework automatically detects when multiple local processes subscribe to the same remote target and shares the network subscription.

What you observe:

The first subscription to a remote target requires network communication. Every subsequent subscription from the same node to the same target returns instantly - it shares the existing network subscription.

How Notification Delivery Works

When the remote target terminates:

Remote node sends ONE notification message to your node
Your node receives it and looks up all local subscribers to that target
Your node delivers individual notifications to each subscriber's mailbox

Performance Characteristics

Operation

Without Sharing

With Sharing

Network cost comparison for 100 subscribers:

Phase

Without Sharing

With Sharing

Impact on Event Publishing

The same optimization applies to event publishing. When you publish an event with subscribers on multiple nodes:

The framework groups subscribers by node and sends ONE message per node:

What the producer sees:

What subscribers see:

Real-World Scale Example

Consider a market data feed with 1 million subscribers distributed across 10 nodes:

When the producer publishes one price update:

Approach

Network Messages

The optimization transforms O(N) network cost (where N = total subscribers) into O(M) cost (where M = number of nodes). For distributed systems with many subscribers per node, this is the difference between practical and impossible.

Actual benchmark results (from the benchmark):

Why This Matters for System Design

This optimization enables patterns that would be impractical otherwise:

Worker pools monitoring coordinators:

Distributed caching with invalidation:

Hierarchical supervision across nodes:

High-frequency event streaming:

The optimization applies when multiple processes on the SAME node subscribe to the SAME remote target.

These share:

These don't share:

Buffered Events: Partial Optimization

Buffered events receive partial optimization. The subscription is shared, but each subscriber must retrieve buffer contents individually.

Event buffers store recent messages for new subscribers:

When a subscriber joins, they receive the buffered messages:

The problem: different subscribers joining at different times need different buffer contents.

If subscriptions were fully shared, all subscribers would receive the same buffer - incorrect for late subscribers.

What Actually Happens

First subscriber: Network round-trip to create subscription AND retrieve buffer.

Subsequent subscribers: Network round-trip to retrieve current buffer (subscription already exists).

Published messages: Still optimized - one network message per node, distributed locally.

Performance Comparison

Aspect

Unbuffered Event

Buffered Event

When to Use Buffers

Use buffers when:

New subscribers need recent history (last N configuration updates)
Subscribers might miss messages during brief disconnections
State can be reconstructed from recent messages

Avoid buffers when:

Real-time streaming where history isn't useful
High subscriber count across many nodes (each pays network cost)
Messages are only meaningful at publish time

Practical guidance:

Producer Notifications

Producers can receive notifications when subscriber interest changes. This enables demand-driven data production.

Enabling Notifications

What You Receive

When Notifications Arrive

Transition

Notification

You only receive notifications when crossing the zero threshold. The notifications answer: "is anyone listening?" - not "how many are listening?"

Practical Use Case: On-Demand Data Production

The producer idles when nobody's listening, avoiding unnecessary API calls and resource usage. When subscribers appear, it starts producing. When all subscribers leave, it stops.

Network Transparency

Notifications work across nodes. Remote subscribers count toward "someone is listening":

The producer doesn't know or care whether subscribers are local or remote. The notification mechanism handles it transparently.

Multiple Events

Each event tracks subscribers independently:

Automatic Cleanup

Subscriptions clean up automatically when any participant terminates. This eliminates resource leaks from forgotten subscriptions.

When Target Terminates

All subscribers receive notification. The subscription ceases to exist - there's nothing to unsubscribe from.

When Subscriber Terminates

Your subscriptions are removed from:

Local subscription records
Remote nodes (for remote subscriptions)

If you were the last local subscriber to a remote target, the network subscription is removed. Otherwise, it stays for remaining local subscribers.

When Event Producer Terminates

Subscribers can't distinguish explicit UnregisterEvent from producer termination - both deliver termination notification with reason gen.ErrUnregistered.

When Network Connection Fails

All subscriptions involving the failed node are cleaned up. If the node reconnects later, you need to re-subscribe - the framework doesn't automatically restore subscriptions.

Explicit Unsubscription

You can explicitly remove subscriptions:

Explicit unsubscription is useful when:

You want to stop watching before termination
You're switching to a different target
You're implementing connection retry logic

But in most cases, you don't need explicit unsubscription. Let termination handle cleanup.

Cleanup Order Guarantees

When a process terminates, cleanup happens in a specific order:

Process state changes to Terminated
All outgoing subscriptions (where process is consumer) are removed
All incoming subscriptions (where process is target) generate notifications

This ordering ensures:

You don't receive notifications after your process starts terminating
Subscribers to you receive notifications before your resources are freed
No race conditions between notification delivery and cleanup

Summary

Concept

How It Works

Key Performance Insights

For subscriptions:

First subscription to remote target: network round-trip
Additional subscriptions to same target: instant
Unbuffered events: full sharing

For notifications:

One network message per subscriber node
Local distribution to all subscribers on that node
Cost scales with number of nodes, not number of subscribers

For cleanup:

Automatic on any termination
No resource leaks possible
No manual unsubscription required

Project Structure

How to Structure Projects Built with Ergo Framework

The same codebase can run as a monolith on your laptop or as distributed services across a data center. This flexibility comes from one principle: applications are the unit of composition. How you organize your project determines whether you can use this flexibility or fight against it.

This chapter covers project organization, message isolation patterns, deployment strategies, and evolution paths. The goal is a structure that supports both development simplicity and production scalability without code changes.

The Flexibility Promise

Ergo's network transparency means a process doesn't know if it's talking to a neighbor in the same node or a remote process across the network. The same Send() call works either way. But this only helps if your code is organized to take advantage of it.

Consider two deployment scenarios:

Development: All applications in one process for fast iteration.

Production: Applications distributed across nodes for scalability.

The application code is identical in both cases. Only the entry point changes - which applications start on which nodes.

This works because:

Applications are self-contained functional units
Messages define contracts between applications
The framework handles routing transparently

Your project structure must preserve these properties. Mix them up, and you lose deployment flexibility.

Directory Layout

A well-structured project separates entry points from applications from shared code:

Entry Points (`cmd/`)

Each directory in cmd/ produces a different binary with a different deployment topology.

Monolith - everything together:

Distributed - each application on its own node:

The application code (apps/api, apps/worker) is identical. The entry point decides what runs where.

Applications (`apps/`)

Each subdirectory in apps/ is a self-contained application. An application is:

A cohesive functional unit
Deployable independently
Composed of actors with a supervision tree
Communicating via messages

Application structure:

Application definition:

Applications should not import each other. If apps/api imports apps/worker, you've created a compile-time dependency that limits deployment flexibility.

Service-Level Types (`types/`)

When applications need to communicate, they need shared message types. The types/ directory holds these contracts:

Both apps/orders and apps/shipping can import types without importing each other. This breaks the circular dependency while maintaining strong typing.

Shared Libraries (`lib/`)

Non-actor code that multiple applications use goes in lib/:

Libraries must be:

Stateless - no global variables, no goroutines
Pure - same inputs produce same outputs
Actor-agnostic - no dependency on gen.Process

Libraries are safe to call from actor callbacks because they don't block or manage state.

Message Isolation Levels

Messages define contracts between actors. The visibility of message types controls who can send them and where they can travel. Ergo uses Go's export rules plus EDF serialization requirements to create four isolation levels.

Understanding these levels is critical for proper encapsulation.

Level 1: Application-Internal (Same Node)

Messages used only within a single application instance on one node.

Characteristics:

Type is unexported (scheduleTask)
Fields are unexported (taskID, not TaskID)
Cannot be imported by other packages

Use when:

Communication between actors in the same application
Messages never leave the local node
Implementation details that shouldn't be exposed

Level 2: Application-Cluster (Same Application, Multiple Nodes)

Messages between instances of the same application across nodes.

Characteristics:

Type is unexported (replicateState)
Fields are exported (Version, not version)
Cannot be imported by other packages

Use when:

Replication between application instances
Cluster-internal coordination
Messages that other applications shouldn't see

Level 3: Cross-Application (Same Node Only)

Messages between different applications on the same node.

Characteristics:

Type is exported (StatusQuery)
Fields are unexported (taskID, not TaskID)
CAN be imported by other packages

Use when:

Local service queries
Same-node optimization paths
Explicitly preventing network transmission

This level is intentionally restrictive. If someone tries to send StatusQuery to a remote node, serialization fails. The unexported fields act as a compile-time guard against accidental network use.

Level 4: Service-Level (Everywhere)

Messages that form public contracts between applications across the cluster.

Characteristics:

Type is exported (ProcessTask)
Fields are exported (TaskID)
CAN be imported by any package

Use when:

Public API between applications
Events that multiple applications subscribe to
Commands sent across application boundaries

Summary Table

Level

Scope

Type

Fields

Serializable

Import

Choosing the Right Level

Start with Level 1 (maximum restriction). Only increase visibility when needed:

Does another application need this message?
- No → Keep type unexported (Level 1 or 2)
- Yes → Export type (Level 3 or 4)

Application Design Patterns

Supervision Structure

Applications typically have a supervision tree:

Configuration via Options

Applications accept configuration through an Options struct:

Entry points configure options based on deployment:

Inter-Application Communication

Applications discover each other through application names, not node names:

When running as monolith, routes returns the local node. When distributed, it returns remote nodes. The code doesn't change.

Event Publishing

Applications publish events for loose coupling:

Events decouple applications. Orders doesn't know who listens. Shipping doesn't know where Orders runs.

Deployment Patterns

Pattern 1: Development Monolith

Everything in one process for fast iteration:

Benefits:

Single binary to run
No network setup
Easy debugging
Fast startup

Pattern 2: Distributed Production

Each application on dedicated nodes:

Each binary runs one application:

Benefits:

Independent scaling per tier
Fault isolation
Resource optimization
Zero-downtime updates

Pattern 3: Hybrid Deployment

Group related applications for efficiency:

Benefits:

Reduced network hops for common paths
Fewer nodes to manage
Right-sized for actual traffic patterns

Testing Strategies

Unit Testing Actors

Test actors in isolation using the testing framework:

Integration Testing Applications

Test complete applications:

Testing Distributed Scenarios

Test multiple nodes:

Evolution and Refactoring

Starting Simple

Begin with a monolith:

Extracting Applications

When the monolith grows, extract bounded contexts:

Step 1: Identify boundaries in the combined application.

Step 2: Create separate application packages.

Step 3: Update the entry point.

Step 4: When ready, create distributed entry points.

The application code never changes. Only entry points and deployment.

Merging Applications

If you over-distributed:

No application code changes. Just different composition.

Best Practices

Application Boundaries

Do:

One application per bounded context
Applications that scale together can be one application
Applications that deploy together can be one application

Don't:

Create applications for single actors
Split applications by technical layer (web/service/data)
Create circular dependencies between applications

Good:

Bad:

Message Design

Do:

Start with Level 1 (most restrictive)
Increase visibility only when needed
Document which level each message uses

Don't:

Default to Level 4 for everything
Mix isolation levels arbitrarily
Use any or interface{} for messages

Dependencies

Do:

Applications import types/ for shared contracts
Applications import lib/ for utilities
Entry points import applications

Don't:

Applications import other applications
Libraries depend on applications
Create import cycles

Configuration

Do:

Use Options structs for application config
Validate in CreateApp or Load
Provide sensible defaults

Don't:

Hard-code configuration in actors
Read os.Getenv directly in actors
Store configuration in global variables

What's Next

This article covered project organization for flexible deployment. As your system grows into a distributed cluster, two topics become essential:

- service discovery, load balancing, failover, and observability
- evolving message contracts during rolling upgrades

Unit

A zero-dependency library for testing Ergo Framework actors with fluent API

Introduced for Ergo Framework 3.1.0 and above (not yet released. available in v310 branch)

The Ergo Unit Testing Library makes testing actor-based systems simple and reliable. It provides specialized tools designed specifically for the unique challenges of testing actors, with zero external dependencies and an intuitive, readable API.

What You'll Learn

This guide takes you from simple actor tests to complex distributed scenarios. Here's the journey:

Getting Started (You Are Here!)

Your First Test - Simple echo and counter examples
Built-in Assertions - Simple tools for common checks
Basic Message Testing - Verify actors send the right messages

Intermediate Skills (Next Steps)

Configuration Testing - Test environment-driven behavior
Complex Message Patterns - Handle sophisticated message flows
Basic Process Spawning - Test actor creation and lifecycle

Advanced Features (When You Need Them)

Actor Termination - Test error handling and graceful shutdowns
Exit Signals - Manage process lifecycles in supervision trees
Scheduled Operations - Test cron jobs and time-based behavior

Expert Level (Complex Scenarios)

Dynamic Value Capture - Handle generated IDs, timestamps, and random data
Complex Workflows - Test multi-step business processes
Performance & Load Testing - Verify behavior under stress

Tip: The documentation follows this learning path. You can jump to advanced topics if needed, but starting from the beginning ensures you understand the foundations.

Why Testing Actors is Different

Traditional testing tools don't work well with actors. Here's why:

The Challenge: Actors Are Not Functions

Regular code testing follows a simple pattern:

But actors are fundamentally different:

They run asynchronously - you send a message and the response comes later
They maintain state - previous messages affect future behavior
They spawn other actors - creating complex hierarchies

What Makes Actor Testing Hard

Asynchronous Communication

Message Flow Complexity

Dynamic Process Creation

State Changes Over Time

How This Library Solves Actor Testing

The Ergo Unit Testing Library addresses each of these challenges:

Event Capture - See Everything Your Actor Does

Instead of guessing what happened, the library automatically captures every actor operation:

Fluent Assertions - Test What Matters

Express your test intentions clearly:

Dynamic Value Handling - Work With Generated Data

Capture and reuse dynamically generated values:

State Testing Through Behavior - Verify State Changes

Test state indirectly by verifying behavioral changes:

Why Zero Dependencies Matters

Actor testing is complex enough without dependency management headaches:

No version conflicts - Works with any Go testing setup
No external tools - Everything needed is built-in
Simple imports - Just import "ergo.services/ergo/testing/unit"

Core Concepts

Now that you understand why actor testing is different, let's explore the key concepts that make this library work.

The Event-Driven Testing Model

Everything your actor does becomes a testable "event".

When you run this simple test:

Here's what happens behind the scenes:

Your actor receives the message - Normal actor behavior
Your actor sends a response - Normal actor behavior
The library captures a SendEvent - Testing magic

The library automatically captures these events:

SendEvent - When your actor sends a message
SpawnEvent - When your actor creates child processes
LogEvent - When your actor writes log messages

Why Events Matter

Events solve the fundamental challenge of testing asynchronous systems:

Instead of this (impossible):

You do this (works perfectly):

The Fluent Assertion API

The library provides a readable, chainable API that expresses test intentions clearly:

Benefits of the fluent API:

Readable - Tests read like English sentences
Discoverable - IDE autocomplete guides you through options
Flexible - Chain only the validations you need

Installation

Your First Actor Test

Let's start with the simplest possible actor test to understand the basics:

A Simple Echo Actor

Testing the Echo Actor

What Just Happened?

This simple test demonstrates the core pattern:

unit.Spawn() - Creates a test actor in an isolated environment
actor.SendMessage() - Sends a message to your actor (like prod would)
actor.ShouldSend() - Verifies that your actor sent the expected message

Key insight: You're not testing internal state - you're testing behavior. You verify what the actor does (sends messages) rather than what it contains (internal variables).

Why This Works

The testing library automatically captures everything your actor does:

Every message sent by your actor
Every process spawned by your actor
Every log message written by your actor

Then it provides fluent assertions to verify these captured events.

Adding Slightly More Complexity

Let's test an actor that maintains some state:

This shows how you test stateful behavior without accessing internal state - by observing how the actor's responses change over time.

Built-in Assertions

Before diving into complex actor testing, let's cover the simple assertion utilities you'll use throughout your tests.

Why Built-in Assertions Matter for Actor Testing:

Actor tests often need to verify simple conditions alongside complex event assertions. Rather than forcing you to import external testing libraries (which could conflict with your project dependencies), the unit testing library provides everything you need:

Available Assertions

Equality Testing:

Boolean Testing:

Nil Testing:

String Testing:

Type Testing:

Why Zero Dependencies Matter

No Import Conflicts:

Consistent Error Messages: All assertions provide clear, consistent error messages that integrate well with the actor testing output.

Framework Agnostic: Works with any Go testing setup - standard go test, IDE test runners, CI/CD systems, etc.

Basic Message Testing

Now that you understand the fundamentals, let's explore message testing in more depth.

What Comes Next

Now you'll learn how to test different aspects of actor behavior, building from simple to complex:

Fundamentals (You're here!)

Basic message sending and receiving
Simple process creation
Logging and observability
Configuration testing

Intermediate Skills

Complex message patterns
Event inspection and debugging
Actor lifecycle and termination
Error handling and recovery

Advanced Features

Scheduled operations (cron jobs)
Network and distribution
Performance and load testing

Basic Logging Testing

Logging is crucial for production actors - it provides visibility into what your actors are doing and helps with debugging. Let's learn how to test logging behavior.

Why Test Logging?

Logging tests ensure:

Your actors provide sufficient information for monitoring
Debug information is available when needed
Log levels are respected (don't log debug in production)

Simple Logging Test

Testing Different Log Levels

Testing Log Content

Logging Best Practices for Testing

Structure your log messages to make them easy to test:

Test log levels appropriately:

Error - Test that errors are logged when they occur
Warning - Test that concerning but non-fatal events are captured
Info - Test that important business events are recorded

Intermediate Skills

Now that you've mastered the basics, let's tackle more complex testing scenarios.

Configuration and Environment Testing

Real actors often behave differently based on configuration. Let's test this:

The Spawn function creates an isolated testing environment for your actor. Unlike production actors that run in a complex node environment, test actors run in a controlled sandbox where every operation is captured for verification.

Key Benefits:

Isolation: Each test actor runs independently without affecting other tests
Deterministic: Test outcomes are predictable and repeatable
Observable: All actor operations are automatically captured as events

Example Actor:

Test Implementation:

Configuration Options - Fine-Tuning the Test Environment

Test configuration allows you to simulate different runtime conditions without requiring complex setup:

Environment Variables (WithEnv): Test how your actors behave with different configurations without changing production code. Useful for testing feature flags, database URLs, timeout values, and other configuration-driven behavior.

Log Levels (WithLogLevel): Control the verbosity of test output and verify that your actors log appropriately at different levels. Critical for testing monitoring and debugging capabilities.

Process Hierarchy (WithParent, WithRegister): Test actors that need to interact with parent processes or require specific naming for registration-based lookups.

Message Testing

`ShouldSend()` - Verifying Actor Communication

Message testing is the heart of actor validation. Since actors communicate exclusively through messages, verifying message flow is crucial for ensuring correct behavior.

Why Message Testing Matters:

Validates Integration: Ensures actors communicate correctly with their dependencies
Confirms Business Logic: Verifies that the right messages are sent in response to inputs
Detects Side Effects: Catches unintended message sends that could cause bugs

Example Actor:

Test Implementation:

Advanced Message Matching - Flexible Validation Patterns

When testing complex message structures or dynamic content, the library provides powerful matching capabilities:

Pattern Matching Benefits:

Partial Validation: Test only the fields that matter for your specific test case
Dynamic Content Handling: Validate messages with timestamps, UUIDs, or generated IDs
Type Safety: Ensure messages are of the correct type even when content varies

Process Spawning

`ShouldSpawn()` - Testing Process Lifecycle Management

Process spawning is a fundamental actor pattern for building hierarchical systems. The testing library provides comprehensive tools for verifying that actors create, configure, and manage child processes correctly.

Why Process Spawning Tests Matter:

Resource Management: Ensure actors don't spawn too many or too few processes
Configuration Propagation: Verify that child processes receive correct configuration
Error Handling: Test behavior when process spawning fails

Example Actor:

Test Implementation:

Dynamic Process Testing - Handling Generated Values

Real-world actors often generate dynamic values like session IDs, request tokens, or timestamps. The library provides sophisticated tools for capturing and validating these dynamic values.

Dynamic Value Testing Scenarios:

Session Management: Test actors that create sessions with generated IDs
Request Tracking: Verify that request tokens are properly generated and used
Time-based Operations: Validate actors that schedule work or create timestamps

Remote Spawn Testing

`ShouldRemoteSpawn()` - Testing Distributed Actor Creation

Remote spawn testing allows you to verify that actors correctly create processes on remote nodes in a distributed system. The testing library captures RemoteSpawnEvent operations and provides fluent assertions for validation.

Why Test Remote Spawning:

Distribution Logic: Ensure actors spawn processes on the correct remote nodes
Load Distribution: Verify round-robin or other distribution strategies work correctly
Error Handling: Test behavior when remote nodes are unavailable

Example Actor:

Test Implementation:

Advanced Remote Spawn Patterns:

Multi-Node Distribution: Test round-robin or other distribution strategies across multiple nodes
Error Scenarios: Verify proper error handling when nodes are unavailable
Event Inspection: Direct inspection of RemoteSpawnEvent for detailed validation

Actor Termination Testing

`ShouldTerminate()` - Testing Actor Lifecycle Completion

Actor termination is a critical aspect of actor systems. Actors can terminate for various reasons: normal completion, explicit shutdown, or errors. The testing library provides comprehensive tools for validating termination behavior and ensuring proper cleanup.

Why Test Actor Termination:

Resource Cleanup: Ensure actors properly clean up resources when terminating
Error Propagation: Verify that errors are handled correctly and lead to appropriate termination
Graceful Shutdown: Test that actors respond correctly to shutdown signals

Termination Reasons:

gen.TerminateReasonNormal - Normal completion of actor work
gen.TerminateReasonShutdown - Graceful shutdown request
Custom errors - Abnormal termination due to specific errors

Example Actor:

Test Implementation:

Advanced Termination Patterns:

Exit Signal Testing

`ShouldSendExit()` - Testing Graceful Process Termination

Exit signals (SendExit and SendExitMeta) are used to gracefully terminate other processes. This is different from actor self-termination - it's about one actor telling another to exit. The testing library provides comprehensive assertions for validating exit signal behavior.

Why Test Exit Signals:

Graceful Shutdown: Ensure supervisors can properly terminate child processes
Resource Cleanup: Verify that exit signals trigger proper cleanup in target processes
Error Propagation: Test that failure conditions are communicated via exit signals

Example Actor:

Test Implementation:

Exit Signal Testing Methods

Basic Exit Signal Assertions:

Advanced Exit Signal Patterns:

Cron Testing

`ShouldAddCronJob()`, `ShouldExecuteCronJob()` - Testing Scheduled Operations

Cron job testing allows you to validate scheduled operations in your actors without waiting for real time to pass. The testing library provides comprehensive mock time support and detailed cron job lifecycle management.

Why Test Cron Jobs:

Schedule Validation: Ensure cron expressions are correct and jobs run at expected times
Job Management: Test job addition, removal, enabling, and disabling operations
Execution Logic: Verify that scheduled operations perform correctly when triggered

Cron Testing Features:

Mock Time Support: Control time flow for deterministic testing
Job Lifecycle Testing: Validate job creation, scheduling, execution, and cleanup
Event Tracking: Monitor all cron-related operations and state changes

Example Actor:

Test Implementation:

Cron Testing Methods

Job Lifecycle Assertions:

Mock Time Control:

Advanced Cron Patterns:

Built-in Assertions

The library includes a comprehensive set of zero-dependency assertion functions that cover common testing scenarios without requiring external testing frameworks:

Why Built-in Assertions:

Zero Dependencies: Avoid version conflicts and complex dependency management
Consistent Interface: All assertions follow the same pattern and error reporting
Testing Framework Agnostic: Works with any Go testing approach

Advanced Features

Dynamic Value Capture - Testing Generated Content

Real-world actors frequently generate dynamic values like timestamps, UUIDs, session IDs, or auto-incrementing counters. Traditional testing approaches struggle with these values because they're unpredictable. The library provides sophisticated capture mechanisms to handle these scenarios elegantly.

The Challenge of Dynamic Values:

Timestamps: Created at runtime, impossible to predict exact values
UUIDs: Randomly generated, different in every test run
Auto-incrementing IDs: Dependent on execution order and system state

The Solution - Value Capture:

Capture Strategies:

Immediate Capture: Capture values as soon as they're generated
Pattern Matching: Use validation functions to identify and validate dynamic content
Structured Matching: Validate message structure while ignoring specific dynamic fields

Event Inspection - Deep System Analysis

For complex testing scenarios or debugging difficult issues, the library provides direct access to the complete event timeline. This allows you to perform sophisticated analysis of actor behavior beyond what's possible with standard assertions.

`Events()` - Complete Event History

Access all captured events for detailed analysis:

`LastEvent()` - Most Recent Operation

Get the most recently captured event:

`ClearEvents()` - Reset Event History

Clear all captured events, useful for isolating test phases:

Event Inspection Use Cases:

Performance Analysis: Count operations to identify performance bottlenecks
Workflow Validation: Ensure complex multi-step processes execute in the correct order
Error Investigation: Analyze the complete event sequence leading to failures

Timeout Support - Assertion Timing Control

The library provides timeout support for assertions that might need time-based validation:

Timeout Function Usage:

Assertion Wrapping: Wrap assertion functions to add timeout behavior
Integration Testing: Useful when testing with external systems that might have delays
Performance Validation: Ensure assertions complete within expected time limits

Testing Patterns and Best Practices

Test Organization Strategies

Single Responsibility Testing: Each test should focus on one specific behavior or scenario. This makes tests easier to understand, debug, and maintain.

State Isolation: Each test should start with a clean state and not depend on other tests. Use actor.ClearEvents() when needed to reset event history between test phases.

Error Path Testing: Don't just test the happy path. Actor systems need robust error handling, so test failure scenarios thoroughly:

Message Design for Testability

Structured Messages: Design your messages to be easily testable by using structured types rather than primitive values:

Predictable vs Dynamic Content: Separate predictable content from dynamic content in your messages to make testing easier:

Performance Testing Considerations

Event Overhead: While event capture is lightweight, be aware that every operation creates events. For performance-critical tests, you can:

Clear events periodically with ClearEvents()
Focus assertions on specific time windows
Use event inspection to identify performance bottlenecks

Scaling Testing: Test how your actors behave under load by simulating multiple concurrent operations:

Best Practices

Use descriptive test names that clearly indicate what behavior is being tested
Test all message types your actor handles, including edge cases
Capture dynamic values early using the Capture() method for generated IDs

This testing library provides comprehensive coverage for all Ergo Framework actor patterns while maintaining zero external dependencies and excellent readability. By following these patterns and practices, you can build robust, well-tested actor systems that behave correctly in both simple and complex scenarios.

Complete Examples and Use Cases

The library includes comprehensive test examples organized into feature-specific files that demonstrate all capabilities through real-world scenarios:

Feature-Based Test Files

basic_test.go - Fundamental Actor Testing

Basic actor functionality and message handling
Dynamic value capture and validation
Built-in assertions and event tracking

network_test.go - Distributed System Testing

Remote node simulation and connectivity
Network configuration and route management
Remote spawn operations and event capture

workflow_test.go - Complex Business Logic

Multi-step order processing workflows
State machine validation and transitions
Business process orchestration

call_test.go - Synchronous Communication

Call operations and response handling
Async call patterns and timeouts
Send/response communication flows

cron_test.go - Scheduled Operations

Cron job lifecycle management
Mock time control and schedule validation
Job execution tracking and assertions

termination_test.go - Actor Lifecycle Management

Actor termination handling and cleanup
Exit signal testing (SendExit/SendExitMeta)
Normal vs abnormal termination scenarios

Comprehensive Test Examples

Complex State Machine Testing (workflow_test.go)
- Multi-step order processing workflow
- Validation, payment, and fulfillment pipeline

Getting Started with Examples

Learning Path

Start with Basic Examples: basic_test.go - Core functionality and patterns
Explore Message Testing: basic_test.go - Message flow and assertions
Learn Process Management: basic_test.go

Each test file provides complete, working implementations of specific actor patterns and demonstrates best practices for testing each scenario. All tests include comprehensive comments explaining the testing strategy and validation approach.

Configuration and Environment Testing

Real actors often behave differently based on configuration. Let's test this:

Complex Message Patterns

As your actors become more sophisticated, your message testing needs to handle more complex scenarios:

Testing Message Sequences

Testing Conditional Logic

Basic Process Spawning

Many actors need to create child processes. Here's how to test this:

Capturing Dynamic Process IDs

When actors spawn processes, you often need to use the generated PID in subsequent tests:

Event Inspection for Debugging

When tests fail, you need to understand what actually happened:

Failure Injection Testing

Overview

The Ergo Unit Testing Library includes a failure injection system that allows you to test how your actors handle various error conditions. This is essential for building robust actor systems that can gracefully handle failures in production.

Method Failure Injection

Access failure injection through the actor's Process() method:

Available Failure Methods

The failure injection system provides several methods on TestProcess:

Common Use Cases

Testing Spawn Failures

Testing Message Send Failures

Testing Intermittent Failures

Testing Pattern-Based Failures

Testing One-Time Failures

Advanced Testing Scenarios

Testing Supervisor Restart Strategies

Testing Method Call Tracking

Best Practices

Clear Events Between Test Phases: Use ClearEvents() when transitioning between test phases to avoid assertion confusion.
Test Recovery: Always test that your actors can recover after failures are cleared or when using one-time failures.
Verify Call Counts: Use GetMethodCallCount()

Common Pitfalls

Event Accumulation: Events accumulate across multiple operations. Use ClearEvents() to reset between test phases.
Timing Issues: Some assertions may need time to complete. Use appropriate timeouts and consider async patterns.
Message Ordering: In high-throughput scenarios, message ordering might not be guaranteed. Test for this explicitly.

Conclusion

The Ergo Framework unit testing library provides comprehensive tools for testing actor-based systems. From simple message exchanges to complex distributed workflows, you can validate every aspect of your actor behavior with confidence.

Key Takeaways:

Start Simple: Begin with basic message testing and gradually add complexity
Test Comprehensively: Cover happy paths, error conditions, and edge cases
Use Fluent Assertions: Take advantage of the readable assertion API

The library's zero-dependency design, comprehensive feature set, and integration with Go's testing framework make it the ideal choice for building robust, well-tested actor systems with the Ergo Framework.

Next Steps:

Explore the complete test examples in the framework repository
Start with simple actors and gradually build complexity
Integrate testing into your development workflow

Happy testing!

Ergo Framework Documentation

Overview

hashtagCore Components

hashtagNetwork Transparency

hashtagWhat This Enables

hashtagPerformance

hashtagZero Dependencies

Basics

Actor Model

hashtagThe Fundamental Concept

Supervision Tree

hashtagThe Supervision Principle

hashtagHow Supervision Works

hashtagFault Tolerance Through Isolation

hashtagSupervision in Ergo Framework

hashtagBuilding Reliable Systems

hashtagWhere to Go From Here

Generic Types

hashtagIdentifiers and Names

Node

hashtagWhat a Node Provides

Process

Links And Monitors

Events

hashtagRegistering Events

hashtagPublishing Events

hashtagSubscribing to Events

hashtagEvent Lifecycle

hashtagNetwork Transparency

hashtagToken Delegation

hashtagEvent Messages

hashtagPractical Patterns

Cron

CertManager

Actors

Meta Processes

Networking

Mutual TLS

hashtagConfiguration

hashtagCertAuthManager

hashtagRuntime Certificate Rotation

hashtagTroubleshooting

Testing

Advanced

extra library

Actors

Applications

Observer

Meta-Processes

Loggers

Colored

hashtagVisual Organization

Rotate

hashtagFile Rotation Mechanics

hashtagAsynchronous Writing

hashtagConfiguration

hashtagBasic Usage

Registrars

hashtagAvailable Registrars

Saturn Сlient

Network Protocols

Tools

Saturn - Central Registrar

Overview

hashtagCore Components

hashtagNetwork Transparency

hashtagWhat This Enables

hashtagPerformance

hashtagZero Dependencies

Network Protocols

Loggers

Meta-Processes

Applications

Registrars

hashtagAvailable Registrars

Actors

CertManager

Observer

hashtagUpdating Certificates

hashtagCertificate Lifecycle

Core Components

Network Transparency

What This Enables

Performance

Zero Dependencies

The Fundamental Concept

The Supervision Principle

How Supervision Works

Fault Tolerance Through Isolation

Supervision in Ergo Framework

Building Reliable Systems

Where to Go From Here

Identifiers and Names

What a Node Provides

Registering Events

Publishing Events

Subscribing to Events

Event Lifecycle

Network Transparency

Token Delegation

Event Messages

Practical Patterns

Configuration

CertAuthManager

Runtime Certificate Rotation

Troubleshooting

Visual Organization

File Rotation Mechanics

Asynchronous Writing

Configuration

Basic Usage

Available Registrars

Core Components

Network Transparency

What This Enables

Performance

Zero Dependencies

Available Registrars

Updating Certificates

Certificate Lifecycle

Mutual TLS

The Supervision Principle

How Supervision Works

Fault Tolerance Through Isolation

Supervision in Ergo Framework

Building Reliable Systems

Where to Go From Here

The Fundamental Concept

Links: Coupling Lifecycles

The Unidirectional Nature

Monitors: Observation Without Coupling

Network Transparency in Practice

Removing Links and Monitors

Practical Usage Patterns

The Difference That Matters

What Makes an Actor

Why Sequential Processing Matters

Location Transparency

Real-World Implementations

How This Applies to Go

The Actor Mindset

Moving Forward

Installation

Starting Saturn

Configuration file structure

Service Discovery

Registering Events

Publishing Events

Subscribing to Events

Event Lifecycle

Network Transparency

Token Delegation

Event Messages

Practical Patterns

File Rotation Mechanics

Asynchronous Writing

Configuration

Basic Usage

Configuration

CertAuthManager