An extra library of meta-process implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy.

Building reliable concurrent and distributed systems is hard. In Go, you might start with goroutines and channels. As the system grows, you add mutexes to protect shared state. Then you need to coordinate across multiple services, so you introduce message queues or RPC. Before long, you're managing synchronization primitives, handling partial failures, and debugging race conditions that only appear under load.

Ergo Framework offers a different foundation. Think of it as making goroutines addressable and message-passing-only, then extending that model across a cluster. Processes are like goroutines - lightweight, multiplexed onto OS threads - but isolated and communicating only through messages. Each process has an identifier that works whether the process is local or on a remote node. Sending a message looks the same either way.

The actor model isn't new. Erlang proved these patterns work for systems requiring massive concurrency and high reliability. Ergo brings them to Go: no external dependencies, familiar Go idioms, and performance that doesn't sacrifice correctness for speed.

Core Components

The framework consists of a few fundamental pieces that work together.

A node provides the runtime environment. It manages process lifecycles, routes messages, handles network connections, and provides services like logging and scheduled tasks. When you start a node, you get infrastructure. When you spawn a process, the node handles the mechanics.

Processes are lightweight actors. Each has a mailbox where messages queue up, priority-sorted into urgent, system, main, and log queues. The process handles messages one at a time in its own goroutine. When the mailbox empties, the goroutine sleeps. This makes processes efficient - you can have thousands without resource problems. It also makes them safe - sequential message handling means no race conditions within a process.

Supervision trees provide fault tolerance. Supervisors monitor worker processes. When a worker crashes, the supervisor restarts it according to a configured strategy. Supervisors can supervise other supervisors, creating a hierarchy. Failures are isolated to subtrees. The rest of the system continues running while the failed part recovers.

Meta processes solve a specific problem: integrating blocking I/O with the actor model. HTTP servers block waiting for requests. TCP servers block accepting connections. A meta process uses two goroutines - one runs your blocking code (like http.ListenAndServe), the other handles messages from other actors. This bridges synchronous APIs with asynchronous actor communication.

The framework treats local and remote processes identically. Send a message to a process on the same node or a process on a remote node - the code is the same. The framework handles the difference.

When you send to a remote process, the node extracts the target node from the process identifier, discovers that node's address (through static routes or a registrar), establishes a connection if needed, encodes the message, and sends it. The remote node receives it, decodes it, and delivers it to the target process's mailbox. This happens automatically. Your code just sends a message.

This transparency extends to failure detection. Use the Important delivery flag and you get the same error semantics for remote processes as for local ones. Without it, a message to a missing remote process times out (was it slow or dead?). With it, you get immediate error notification (process doesn't exist), just like local delivery. The network becomes transparent not just for success cases but for failures too.

Nodes discover each other through a registrar. By default, each node runs a minimal registrar. Nodes on the same host find each other through localhost. For remote nodes, the framework queries the registrar on the remote host. For production clusters, configure an external registrar like etcd or Saturn for centralized discovery, cluster configuration, and application deployment tracking.

You write business logic using message passing between processes. The framework handles concurrency (processes run in parallel but each is sequential internally), fault tolerance (supervisors restart failures), and distribution (messages route automatically to remote processes). You're not writing code to manage connections, encode messages, or handle network failures explicitly. Those are solved problems handled by the framework.

Systems built this way have useful properties. They scale by adding nodes and distributing processes across them. The code doesn't change - deployment topology is operational configuration. They handle failures through supervision rather than defensive programming everywhere. They evolve through composition - add new process types, adjust supervision strategies, change message flows - without restructuring the foundation.

The development experience differs from typical microservices. No REST endpoints to define. No service discovery to configure (it's built in). No serialization libraries to manage (the framework handles it). No retry logic scattered throughout (supervision handles recovery). You model your domain as processes exchanging messages, and the framework provides the infrastructure.

Lock-free queues in process mailboxes avoid contention. Processes sleep when idle, consuming no CPU. Connection pooling uses multiple TCP connections per remote node for parallel delivery. These design choices add up to performance comparable to hand-written concurrent code, but without the complexity.

The real performance benefit is development velocity. You're not debugging race conditions or deadlocks. You're not coordinating distributed transactions. You're not managing connection pools or implementing retry logic. The framework handles those concerns, leaving you to focus on what your system does.

Benchmarks measuring message passing, network communication, and serialization performance are available at .

The framework uses only the Go standard library. No external dependencies means no version conflicts, no supply chain vulnerabilities, no surprise breaking changes from third-party packages. The requirement is just Go 1.21 or higher.

This isn't ideological purity. It's practical stability. The framework's behavior depends only on Go itself. Updates are predictable. Supply chain is simple. The code you write today will compile and run the same way years from now, assuming Go maintains backward compatibility (which it does).

For detailed explanations of these concepts, start with and explore the section. For API documentation, see the godoc comments in the source code.

The Ergo Framework allows nodes to run with various network stacks. You can replace the default network stack or add it as an additional stack. For more information, refer to the section.

This library contains implementations of network stacks that are not part of the standard Ergo Framework library.

An extra library of logger implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy.

Terminal output with ANSI colors. Highlights Ergo types (PIDs, Atoms, Refs) and colorizes log levels for visual clarity. Synchronous writes to stdout with immediate formatting.

Use cases: Local development, interactive debugging, fast visual scanning of logs in a terminal.

File logger with automatic time-based rotation and optional gzip compression. Asynchronous writes via background goroutine. Configurable retention policy.

Use cases: Production long-running services, time-windowed log archives, on-host log retention with bounded disk usage.

Forwards panics and errors to a Sentry project. Captures panic stack traces pointing at the panic origin and tags events by ergo subsystem (node, network, application, meta, process). Asynchronous, non-blocking.

Use cases: Centralized error tracking, panic alerting in production, grouping recurring failures by root cause.

The additional application library for Ergo Framework contains packages with a narrow specialization or external dependencies since Ergo Framework adheres to the "zero dependencies" principle.

You can find the source code of these applications in the application library repository at .

An extra library of registrars or client implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework follows a "zero dependency" policy.

A client library for the central registrar. Provides service discovery, configuration management, and real-time cluster event notifications through a centralized registrar service.

Features:

• Centralized service discovery

• Real-time event notifications

An extra library of actors implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy.

Kubernetes health probe actor that serves /health/live, /health/ready, and /health/startup endpoints. Actors register named signals with probe type and optional heartbeat timeout. When a signal goes down, the corresponding probe endpoint returns 503.

Use cases: Kubernetes liveness/readiness/startup probes, container orchestration integration, dependency health tracking, graceful degradation.

Distributed leader election actor implementing Raft-inspired consensus algorithm. Provides coordination primitives for building systems that require single leader selection across a cluster.

Use cases: Task schedulers, resource managers, single-writer databases, distributed locks, cluster coordinators.

Network communication in production systems needs encryption. TLS provides this, but managing TLS certificates introduces operational challenges. Certificates expire. Security incidents require rotation. Updating certificates traditionally means restarting services, causing downtime.

The naive approach loads certificates at startup from files. When you need to update a certificate, you replace the file and restart the service. For a single service, this works. For distributed systems with dozens of nodes and services, coordinating restarts for certificate updates becomes an operational burden.

Ergo Framework provides gen.CertManager for live certificate updates. Load a certificate at startup, and you can update it later without restarting. All components using that certificate manager - node acceptors, web servers, TCP servers - automatically use the updated certificate for new connections.

Create a certificate manager with an initial certificate:

For development or testing, generate a self-signed certificate:

Note: Self-signed certificates require setting InsecureSkipVerify: true in network options to bypass certificate validation. This is acceptable for development but never use it in production.

The actor model is a computational approach to building concurrent systems, first proposed in the 1970s. At its core is a simple yet powerful idea: instead of having program components share memory and coordinate through locks, they communicate by sending messages to each other.

In the actor model, everything is an actor. An actor is an independent entity that has its own private state and processes incoming messages one at a time. Actors never directly access each other's state. Instead, they send messages and wait for responses if needed.

This might seem like a constraint, but it's actually what makes the model powerful. By eliminating shared state, we eliminate entire classes of concurrency bugs that plague traditional multi-threaded programs.

An actor consists of three things:

Private State - Data that belongs exclusively to this actor. No other actor can read or modify it directly.

Behavior - The logic that determines how the actor responds to messages. This can change over time as the actor processes different messages.

Mailbox - A queue where incoming messages wait to be processed. The actor pulls messages from this queue one at a time.

Building reliable systems from independent processes requires solving a fundamental coordination problem. When a process terminates - whether from a crash, graceful shutdown, or network failure - other processes that depend on it or supervise it need to know. Without this knowledge, a supervisor can't restart failed workers, dependent processes continue attempting to use unavailable services, and the system degrades silently.

The challenge is detecting termination without breaking isolation. Processes can't share memory or directly observe each other's state. The traditional approach in distributed systems uses heartbeats: processes periodically signal they're alive, and silence implies failure. But heartbeats introduce overhead, timing sensitivity, and the fundamental ambiguity of distinguishing "slow" from "dead."

Ergo Framework provides a different mechanism. Processes explicitly declare relationships - links and monitors - and the framework delivers termination notifications through these channels. When a process terminates, the node automatically notifies all processes that established relationships with it. The notification is immediate, deterministic, and part of the normal message flow.

Links and monitors both deliver termination notifications, but they differ in what happens next. A link couples your lifecycle to the target's - when it terminates, you terminate. A monitor simply informs you of termination, leaving the response up to you. The choice depends on whether you need failure propagation or just failure awareness.

Links: Coupling Lifecycles

Creating a link to another process declares a dependency. You're stating that your operation depends on the target's continued existence. When the target terminates, you receive an exit signal - a high-priority message that typically causes your termination as well.

Exit signals arrive in the Urgent queue, bypassing normal message ordering. The default behavior is immediate termination when an exit signal arrives. This cascading failure makes sense in many scenarios. If a worker's connection to a critical service is gone, the worker has nothing useful to do and should terminate cleanly.

But sometimes you want to handle exit signals explicitly. Actors can enable exit signal trapping through act.Actor. When trapping is enabled, exit signals are delivered as gen.MessageExit* messages to your HandleMessage callback. You can examine the signal, check the termination reason, and decide whether to terminate or attempt recovery.

The process-level exit messages carry the termination reason in a Reason field. The reason tells you what happened: normal shutdown (gen.TerminateReasonNormal), abnormal crash, panic (gen.TerminateReasonPanic) or forced kill (gen.TerminateReasonKill). This context lets you make informed decisions about how to react.

gen.MessageExitNode is the exception: it carries only Name, the node that went away. A lost connection has no reason to report beyond itself.

The framework provides linking methods for different identification schemes. LinkPID takes a process identifier and links to that specific process instance. When it terminates, you receive gen.MessageExitPID. LinkProcessID links to a registered name rather than a specific instance. If the process terminates or unregisters the name, you receive gen.MessageExitProcessID. LinkAlias works with process aliases - termination or alias deletion triggers gen.MessageExitAlias.

You can also link to node connections with LinkNode. If the connection to the specified node is lost, you receive gen.MessageExitNode. This is useful for processes that can't operate when a particular remote node is unavailable.

The generic Link method takes four target types and dispatches to the typed method for each: gen.PID, gen.ProcessID, gen.Alias, and gen.Atom - which it reads as a registered name on the local node, not as a node name. Anything else returns gen.ErrUnsupported. Monitor, Unlink and Demonitor behave the same way.

That means node and event targets are not reachable through the generic form: Link(gen.Atom("other@host")) looks for a local process registered under that name and fails with gen.ErrProcessUnknown, and Link(gen.Event{...}) returns gen.ErrUnsupported. Call LinkNode and LinkEvent directly.

Links in Ergo are unidirectional, and this deserves emphasis because it differs from Erlang.

When you execute process.LinkPID(target), you establish a relationship where target's termination affects you. The link points from you to the target. If the target terminates, you receive an exit signal. But if you terminate, the target is unaffected. The link doesn't point backward.

Erlang's links are bidirectional. If process A links to process B in Erlang, either terminating causes the other to terminate. This symmetry can be useful, but it also creates unexpected cascading failures. In Ergo, if you want bidirectional coupling, you create two links: A links to B, and B links to A.

The unidirectional design gives you precise control. Consider a shared service with multiple workers. Each worker links to the service (if the service dies, workers should too). But the service doesn't link back to workers (a worker crash shouldn't kill the service). Unidirectional links express this asymmetric dependency naturally.

Monitors provide lifecycle awareness without lifecycle coupling. You track when something terminates, but you don't terminate yourself.

The quintessential monitor use case is supervision. A supervisor monitors worker processes. When a worker terminates, the supervisor receives a down message. The message includes the worker's PID or identifier and the termination reason. The supervisor examines this information, consults its restart strategy, and decides whether to spawn a replacement. The supervisor continues running regardless of how many workers have crashed.

Down messages arrive in the System queue with high priority (but lower than Urgent exit signals). Every gen.MessageDown* type includes a Reason field except gen.MessageDownNode, which carries only the node name. For MonitorPID, you receive gen.MessageDownPID with the target's PID and reason. For MonitorProcessID, you receive gen.MessageDownProcessID with the registered name and reason. The reason might indicate normal termination, a crash, or a special case like name unregistration (gen.ErrUnregistered).

Monitoring registered names or aliases handles invalidation gracefully. If you monitor a process by name and that process unregisters its name, you receive a down message with reason gen.ErrUnregistered. The process might still be running, but it's no longer accessible by that name, which is what you were monitoring. Same logic applies to alias deletion - you're notified that the thing you were monitoring is no longer valid.

Node monitoring tracks connection health. MonitorNode sends you gen.MessageDownNode when the connection to a remote node is lost. It has one field, Name - there is no reason to read, because losing the connection is the whole of the news. This is useful for detecting network partitions or remote node crashes without linking (which would terminate your process).

Links and monitors work across nodes without changing their semantics or your code.

When you link or monitor a remote target, the framework sends a request to the remote node. The remote node records that your process is watching the target. This setup happens during your Link* or Monitor* call and involves a network round-trip. The operation can fail if the remote node is unreachable or the target doesn't exist - check the error return.

Once established, the remote node tracks your subscription. When the target terminates on the remote node, the remote node sends a notification message back to your node. Your node routes it to your process's mailbox. From your perspective, it's just another message - you don't see the network mechanics.

Network failures complicate this. If the connection to the remote node fails while your link or monitor is active, your local node detects the disconnection. It looks up which local processes had links or monitors to targets on that failed node. For links, it sends exit signals with reason gen.ErrNoConnection. For monitors, it sends down messages with the same reason.

This unified handling means you write the same error handling code for local and remote targets. The notification mechanism is consistent. The reason field distinguishes between target termination and network failure, but the notification path is identical.

Links and monitors aren't permanent. You can remove them explicitly or they're removed automatically when participants terminate.

To remove a link, use the corresponding Unlink* method with the same target. UnlinkPID, UnlinkProcessID, UnlinkAlias, UnlinkNode each remove the link created by their Link* counterpart. If you never created the link, unlinking returns an error. For monitors, the Demonitor* methods work the same way.

When the target terminates and you receive notification, the link or monitor is automatically removed. You receive one notification per relationship. If the target is later restarted (by a supervisor), you won't receive notification about that new instance unless you create a new link or monitor to it.

When you terminate (the process that created the link or monitor), your relationships are cleaned up automatically. The target doesn't receive notification that you stopped watching. This asymmetry is intentional - the target doesn't track who's watching it, so it doesn't care when watchers go away.

Several common patterns emerge from combining links and monitors.

Workers often link to infrastructure processes they depend on. A worker processing HTTP requests might link to a database connection pool process. If the pool terminates (perhaps during a deployment), the worker receives an exit signal and terminates. The worker's supervisor detects the termination, waits a moment (hoping the database pool restarts), and spawns a new worker. The new worker links to the (now running) pool and resumes processing.

Supervisors monitor their children. Each worker termination triggers a down message. The supervisor checks the reason. If it's gen.TerminateReasonNormal, the worker finished its task and doesn't need restart. If it's an error or panic, the supervisor spawns a replacement. The supervisor's continued operation despite worker failures is the whole point of the supervisor pattern.

Load balancers monitor backend processes. Each backend termination updates the balancer's routing table. The balancer continues routing to available backends. When a backend restarts, it might need to register with the balancer, which would then monitor it again.

Parent-child relationships often use LinkChild and LinkParent options in gen.ProcessOptions. These provide a convenient way to create links automatically after process initialization completes. You can also call Link methods directly during initialization if needed. If either participant terminates, the other receives an exit signal.

Links propagate failure. Monitors report failure. Choose based on whether the watcher should terminate when the target terminates.

If continued operation without the target is meaningless, use a link. If you can adapt to the target's absence (by finding a replacement, degrading gracefully, or restarting the target), use a monitor.

The unidirectional nature of links matters more than you might initially think. It lets you express asymmetric dependencies precisely. Workers depend on services, but services don't depend on individual workers. Clients depend on servers, but servers don't depend on individual clients. Links point from the dependent to the dependency, making the relationship clear.

For event-based publish/subscribe patterns using links and monitors, see the chapter. For supervision trees built on monitors, see .

The rotate logger writes log messages to files with automatic rotation based on time intervals. Instead of a single growing log file that eventually fills the disk, the logger creates new files periodically and optionally compresses old ones. This keeps disk usage predictable and makes log files manageable for analysis and archival.

The logger operates asynchronously - log messages enter a queue and a background goroutine writes them to the file. This design prevents blocking your processes when disk I/O is slow. Logging happens in the background while your actors continue processing messages without waiting for disk writes to complete.

File Rotation Mechanics

Rotation happens based on time periods. You configure a duration - one minute, one hour, one day - and the logger creates a new file every period. The active file is always named <Prefix>.log. When the period ends, the logger:

1. Copies the active file to a timestamped filename: <Prefix>.YYYYMMDDHHmi.log

2. Optionally compresses it with gzip: <Prefix>.YYYYMMDDHHmi.log.gz

3. Truncates the active file to start fresh for the new period

4. Deletes old files if depth limit is configured

This approach ensures the active file always has the same name. You can tail it (tail -f <Prefix>.log) and it works across rotations. The timestamped copies accumulate in the log directory, creating a chronological archive.

The timestamp format is YYYYMMDDHHmi - year, month, day, hour, minute. This format sorts lexicographically, so ls -l shows files in chronological order. It's compact but human-readable.

The logger uses an internal lock-free queue (MPSC - multi-producer single-consumer). When any process logs a message, it pushes to the queue and returns immediately. A single background goroutine pops messages from the queue and writes them to the file.

This design has several advantages:

Non-blocking - Logging never blocks your process. If the disk is slow or the file system stalls, your actors continue running. The queue absorbs bursts of messages.

Ordering - Messages from a single producer maintain order. The queue preserves submission order, so logs reflect the actual sequence of events within each process.

Batching - The background goroutine processes messages continuously. If multiple messages arrive quickly, it writes them in a tight loop, reducing syscall overhead.

The logger requires a rotation period and accepts several optional parameters:

Period - The rotation interval. Minimum is time.Minute. Smaller periods create more files with less data each. Larger periods create fewer files with more data each. Choose based on how you analyze logs - if you search specific time ranges, shorter periods help. If you archive logs by day, use 24 * time.Hour.

Path - Directory for log files. Defaults to ./logs relative to the executable. The logger creates the directory if it doesn't exist. Supports ~ for home directory expansion (~/logs becomes /home/user/logs). Use absolute paths in production to avoid ambiguity.

Prefix - Filename prefix. Defaults to the executable name. The active file is <Prefix>.log, rotated files are <Prefix>.YYYYMMDDHHmi.log[.gz]. Use meaningful prefixes if multiple services log to the same directory.

Compress - Enables gzip compression for rotated files. The active file stays uncompressed for fast writing. When rotating, the logger compresses the copy, reducing disk usage by 5-10x for text logs. Compressed files have .log.gz extension. Use compression if disk space matters more than CPU for compression.

Depth - Limits the number of retained log files. When rotating, if the number of files exceeds Depth, the logger deletes the oldest file. Set to 0 (default) for unlimited retention. Set to a specific number (e.g., 24) to keep the last 24 periods. This prevents unbounded disk usage.

TimeFormat - Timestamp format in log messages. Same as colored logger - any format from time package or custom layout. Empty string uses nanosecond timestamps. Choose based on readability vs. precision.

IncludeName - Includes registered process names in log messages. Helps identify which process logged what.

IncludeBehavior - Includes behavior type names in log messages. Useful during development to understand code flow.

ShortLevelName - Uses abbreviated level names ([TRC], [DBG], etc.) instead of full names. Saves space in log files.

Configure the rotate logger in node options:

This configuration:

• Rotates every hour

• Stores logs in /var/log/myapp/

• Names files myapp.log (active) and myapp.202411191200.log.gz (rotated with compression)

Depth is not retention by age, and not by what is in the directory. The logger keeps an in-memory list of the files it has rotated during its own lifetime and removes the oldest only once that list grows past Depth. It never scans the directory at startup, so files left by an earlier run are neither counted nor deleted - restart the node every hour with Depth: 24 and the directory grows without bound. To bound disk use across restarts, do it outside the logger: logrotate, a cron sweep, or a volume with its own retention.

For detailed logger configuration options, see the rotate.Options struct in the package. For understanding how loggers integrate with the framework, see .

saturn is a tool designed to simplify the management of clusters of nodes created using the Ergo Framework. It offers the following features:

• A unified registry for node registration within a cluster.

• The ability to manage multiple clusters simultaneously.

• The capability to manage the configuration of the entire cluster without restarting the nodes connected to Saturn (configuration changes are applied on the fly).

• Notifications to all cluster participants about changes in the status of applications running on nodes connected to Saturn.

The source code of the saturn tool is available on the project's page: .

To install saturn, use the following command:

Available arguments:

• host: Specifies the hostname to use for incoming connections.

• port: Port number for incoming connections. The default value is 4499.

• path: Path to the configuration file

To start Saturn, a configuration file named saturn.yaml is required. By default, Saturn expects this file to be located in the current directory. You can specify a different location for the configuration file using the -path argument.

You can find an example configuration file in the project's Git repository.

The saturn.yaml configuration file contains two root elements:

1. Saturn: This section includes settings for the Saturn server.

   • You can configure the Token for access by remote nodes and specify certificate files for TLS connections.

   • By default, a self-signed certificate is used. For clients to accept this certificate, they must enable the InsecureSkipVerify

If the name of a configuration element ends with the suffix .file, the value of that element is treated as a file. The content of this file is then sent to the nodes as a []byte.

To configure settings for all nodes in all clusters, use the Clusters section in the saturn.yaml configuration file. Here, you can define global settings that will apply to every node within every cluster managed by Saturn:

in this example:

• Var1, Var2, Var3, and Var4 will be applied to all nodes in all clusters.

• However, the value of Var1 for nodes named node@host.local in any cluster will be overridden with the value 456.

If nodes are registered without specifying a Cluster in saturn.Options, they become part of the general cluster. Configuration for the general cluster should be provided in the Cluster@ section

In the example above:

• The variable Var1 is set to 789 for the general cluster (all nodes in the general cluster will receive Var1: 789).

• However, for the node node@host.local within the general cluster, Var1 will be overridden to 456.

Thus, all nodes in the general cluster will inherit Var1: 789, except for node@host.local, which will specifically have Var1: 456. Other nodes in the general cluster will retain the default values from the Cluster@ section unless they are explicitly overridden in the configuration.

To specify settings for a particular cluster, use the element name Cluster@<cluster name> in the configuration file:

Saturn can manage multiple clusters simultaneously, but resolve requests from nodes are handled only within their own cluster.

The name of a registered node must be unique within its cluster.

When a node registers, it informs the registrar which cluster it belongs to. Additionally, the node reports the applications running on it. Other nodes in the same cluster receive notifications about the newly connected node and its applications. Any changes in application statuses are also reported to the registrar, which in turn notifies all participants in the cluster.

For more details, see the section.

The colored logger provides visual clarity for console output by applying color highlighting to log messages. Instead of monochrome text where errors blend with informational messages, each log level gets a distinct color, and framework types are highlighted automatically. This makes it easier to scan logs during development and debugging.

The logger writes directly to standard output with immediate formatting - no buffering, no delays. When a process logs a message, it appears instantly in your terminal with colors applied. This synchronous approach keeps logs simple and predictable during interactive development.

Visual Organization

Color helps your eyes parse logs quickly. Log levels use consistent colors:

• Trace - Faint white (low importance, background noise)

• Debug - Magenta (development information)

• Info - White (normal operation)

• Warning - Yellow (attention needed)

• Error - Red bold (problems occurred)

• Panic - White on red background bold (critical failures)

Framework types also get color highlighting:

• gen.Atom - Green (names and identifiers)

• gen.PID - Blue (process identifiers)

• gen.ProcessID - Blue (named processes)

When you log process.Log().Info("started %s", pid), the PID renders in blue automatically. You don't annotate it - the logger detects the type and applies color. This works for any framework type used as an argument.

Each log message follows a consistent structure:

Timestamp appears first. By default, it's the Unix timestamp in nanoseconds. You can configure any format from Go's time package, or define your own. Nanosecond timestamps are sortable and precise, useful when correlating logs with traces or metrics.

Level shows the severity. The bracket format [INFO] or short form [INF] makes levels easy to grep. Color reinforces the level visually - you don't need to read the text to know something is an error.

Source identifies where the message originated:

• Node logs - Show the node name in green (CRC32 hash for compactness)

• Network logs - Show both local and peer node names

• Process logs - Show PID in blue, optionally the registered name in green, optionally the behavior type

• Meta-process logs - Show alias in cyan, optionally the behavior type

The optional components (name, behavior) are controlled by configuration. During development, you might want behavior names to understand which actor logged something. In production, you might omit them to reduce output.

Message is your formatted string with arguments. Framework types in arguments get color highlighting automatically.

The logger accepts several options during creation:

TimeFormat - Sets timestamp format. Any format from time package works (time.RFC3339, time.Kitchen, custom layouts). Leave empty for nanosecond timestamps. Nanoseconds are precise but hard to read. RFC3339 is human-friendly but verbose. Choose based on your use case.

ShortLevelName - Uses abbreviated level names: [TRC], [DBG], [INF], [WRN], [ERR], [PNC]. Saves horizontal space in the terminal. Full names are clearer for people unfamiliar with the abbreviations.

IncludeName - Adds the registered process name to the source. If a process registers as "worker", logs show the name in green next to the PID. Helpful when you have many processes and want to identify them by role rather than PID.

IncludeBehavior - Adds the behavior type name to the source. Logs show which actor implementation generated the message. Useful during development to understand code flow. In production, this adds noise if you have good message content.

IncludeFields - Includes structured logging fields in the output. Fields appear below the message with faint color. Useful when your log messages use context fields for correlation (request IDs, user IDs, etc.).

DisableBanner - Disables the Ergo logo banner on startup. The banner announces framework version and adds visual flair. Disable it in production or when running tests where the banner clutters output.

Register the colored logger in node options:

The default logger writes to stdout too, but without colors. If you don't disable it, you get each message twice - once colored, once plain. Disabling the default logger ensures only the colored version appears.

For detailed logger configuration options, see the colored.Options struct in the package. For understanding how loggers integrate with the framework, see .

This package implements the gen.Registrar interface and serves as a client library for the central registrar, Saturn. In addition to the primary Service Discovery function, it automatically notifies all connected nodes about cluster configuration changes.

To create a client, use the Create function from the saturn package. The function requires:

• The hostname where the central registrar is running (default port: 4499, unless specified in saturn.Options)

• A token for connecting to Saturn

• a set of options saturn.Options

Then, set this client in the gen.NetworkOption.Registrar options

Using saturn.Options, you can specify:

• Cluster - The cluster name for your node

• Port - The port number for the central Saturn registrar

• KeepAlive - The keep-alive parameter for the TCP connection with Saturn

When the node starts, it will register with the central registrar in the specified cluster.

Additionally, this library registers a gen.Event and generates messages based on events received from the central Saturn registrar within the specified cluster. This allows the node to stay informed of any updates or changes within the cluster, ensuring real-time event-driven communication and responsiveness to cluster configurations:

• saturn.EventNodeJoined - Triggered when another node is registered in the same cluster.

• saturn.EventNodeLeft - Triggered when a node disconnects from the central registrar

• saturn.EventApplicationLoaded - An application was loaded on a remote node. Use ResolveApplication from the gen.Resolver

To receive such messages, you need to subscribe to Saturn client events using the LinkEvent or MonitorEvent methods from the gen.Process interface. You can obtain the name of the registered event using the Event method from the gen.Registrar interface. This allows your node to listen for important cluster events like node joins, application starts, configuration updates, and more, ensuring real-time updates and handling of cluster changes.

Using the saturn.EventApplication* events and the feature, you can dynamically manage the functionality of your cluster. The saturn.EventConfigUpdate events allow you to adjust the cluster configuration on the fly without restarting nodes, such as updating the cookie value for all nodes or refreshing the TLS certificate. Refer to the section for more details.

You can also use the Config and ConfigItem methods from the gen.Registrar interface to retrieve configuration parameters from the registrar.

To get information about available applications in the cluster, use the ResolveApplication method from the gen.Resolver interface, which returns a list of gen.ApplicationRoute structures:

• Name The name of the application

• Node The name of the node where the application is loaded or running

• Weight The weight assigned to the application in gen.ApplicationSpec

You can access the gen.Resolver interface using the Resolver method from the gen.Registrar interface.

An actor is not a function. You cannot call it and inspect a return value. It runs on its own goroutine, communicates only through messages, keeps private state that evolves from one message to the next, spawns children, and can be terminated or restarted by a supervisor. A test that reaches into an actor's fields tests the wrong thing and breaks the isolation the model depends on.

So the testing tools observe an actor the way the rest of the system does: through what it does. Every outward action - a message sent, a process spawned, a log line written, an exit signal, a timer scheduled - is captured as a record. You drive the actor with inputs - deliver a message, fire a timer, deliver an exit - and assert on the records it produced. You test behavior, not state. That single idea runs through all four packages this section covers.

The Observation Model

The shape of every test is the same: drive an input, then assert on an output.

sub.SendMessage(client, "ping")
sub.ShouldSend().To(client).Message("pong").Once().Assert()

The actor received a message and sent one back. The harness recorded the send, and the assertion checks that it happened exactly once, to the right target, with the right payload. The same fluent grammar - a Should... builder, filters, a count, a terminal - describes every kind of action, and it reads the same whether the actor runs in a mock or on a live node. There is no result := actor.Process(msg) to inspect, because actors do not work that way; instead you verify the messages an actor sends, the children it spawns, the events it emits, and how it terminates.

Four packages cover the span from a single function up to a cluster of real nodes. They share one assertion grammar and line up along one axis: how much of the real system is present.

check is the language, and nothing more. It defines the record types and the fluent Should... grammar every test is written in. You rarely import it directly - unit and stage expose its assertions on their own handles - but reading it once teaches the vocabulary the other layers reuse: cardinalities, filters, scoping, and value capture.

mock adds a controllable dependency, for code that is not an actor. It provides a standalone fake of each gen.* interface - Node, Process, MetaProcess, Log, Cron, Network, RemoteNode, Registrar, Resolver - to inject into ordinary code that consumes one: a helper, a resolver, a constructor. Override the methods you care about; the rest return safe defaults.

A second constructor with a T suffix also records every call, so you can assert on what the code did with the dependency:

unit brings one actor to life against a mock node. It spawns a single behavior, lets you drive its callbacks - deliver messages, fire timers, deliver exits and downs - and asserts on what it does. It runs synchronously: no real goroutines, no clock, fully deterministic and fast. This is where most actor logic is tested.

stage brings the whole runtime to life. It starts real nodes, runs real actors and applications, lets them talk over the real network, and observes what the live runtime does. Use it for what unit deliberately leaves out: the real scheduler, supervision and restarts, links and monitors across nodes, remote spawn, disconnects. Because everything runs for real and concurrently, assertions wait with Within instead of reading a snapshot.

• Testing one actor's message handling, spawning, logging, or lifecycle: use unit.

• Testing a helper or component that takes a gen.Node or gen.Process and you need to control what it returns: use mock.

• Testing behavior that needs the real scheduler, the real network, or more than one node: use stage.

unit and stage are not two ways to write the same test; they answer different questions. A typical project tests the bulk of its actor logic with unit, where tests run in microseconds and never flake, and reserves stage for the cross-node and supervision scenarios that only the real runtime exhibits.

Read on in order: for the grammar every test is written in, then , , and , each building on the one before.

Standard TLS provides server authentication - the client verifies the server's certificate. Mutual TLS (mTLS) adds client authentication - both sides present and verify certificates. Only clients with certificates signed by a trusted CA can connect.

NodeOptions.CertManager is used for:

• Default acceptor (created automatically on gen.DefaultPort, 11144)

• All outgoing connections

The actor model rests on rules the Go compiler cannot check. A message must not share memory with its sender. A callback must not block without a bound. A goroutine started inside a callback must not reach back into actor state. Break any of these and the code compiles, the tests pass, and the failure arrives later - as a race under load, as a mailbox that never drains, as a node that reports itself healthy while serving nothing.

argus reads your packages and reports those breaks. It is a vet tool, so it runs the way go vet runs: over the build graph, with your build tags, cached per package.

In production, as a vet tool. The go command drives it and caches the results:

During development, directly over package patterns:

Package patterns resolve against the main module of the working directory, exactly as the go command resolves them. To analyse a module you are not standing in, move there first:

Every finding carries a tier and a rule ID. The tier says how much to trust it, and decides whether it fails your build:

The sentry logger forwards panics and errors to a Sentry project, complete with stack traces pointing at the panic origin. When a production node recovers a panic in an actor callback, that event is usually written to a file and noticed hours later, if at all. Sentry-backed logging surfaces the same event in your issue tracker within seconds, grouped by panic value, with the originating frames marked as application code.

The logger does not replace your console or file logger. It runs alongside them and captures a narrow slice of the log stream: every panic from anywhere in the framework, and errors from the higher-level subsystems (node, network, application). Meta processes and individual actors are off by default to keep noisy actor-level errors from drowning your Sentry quota, and can be turned on per source when needed.

What Gets Sent

Two severities are forwarded; the others stay local. The matrix maps source against level:

Every panic event reaches Sentry regardless of which subsystem produced it. The framework recovers panics in actor callbacks, supervisors, pools, web workers, meta processes, applications, cron jobs and network handlers; each recovery site formats a Log().Panic() message which the logger sees and captures.

Error-level events from the node, network and application subsystems are forwarded by default. These are the operational layer: a network handshake failure, an application that refuses to start, a node configuration error. They tend to be infrequent and worth a Sentry issue every time.

Error-level events from meta processes and individual actors are opt-in via CaptureMetaErrors and CaptureProcessErrors. A misbehaving actor can produce many errors per second, and you usually want to investigate those in your local logs first rather than pay for ingest in Sentry. Turn the flag on when you have specific signals you want to track centrally.

Lower levels (warning, info, debug, trace) are never forwarded. Sentry is for things that need attention, not a general log sink.

For panic events the logger captures the goroutine stack at the moment the panic was recovered. The captured stack points at the line of code that actually panicked rather than at the framework's recovery wrapper. In Sentry this appears as an Exception with Type: "panic", value set to the recovered panic value, and a stack trace with caller-first ordering.

Frames are marked as application code by default; frames from the Go runtime, the Sentry SDK and ergo.services/* are tagged as framework code so Sentry's UI collapses them out of the way and highlights the lines that belong to your project.

Sentry groups events by the Exception.Value. Since that value is the recovered panic message (nil pointer dereference, index out of range, your own panic string), every reoccurrence of the same root cause ends up in the same issue rather than scattered by the surrounding format string.

If you wrap Log().Panic() in a helper layer of your own, increase SkipFrames so the captured stack still trims your wrapper out and starts where the panic actually originated.

Every event carries a source tag identifying which subsystem produced it: node, network, application, meta or process. Additional tags depend on the source: node and network events include the node name (and peer name for network); application events include the application name and run mode; meta and process events include their identifier (alias or PID) and optionally the behavior type.

Structured fields attached via Log().AddFields() arrive in Sentry as Extra data. The same fields you use to correlate logs locally show up in the Sentry event panel, so you can filter or pivot on request_id, user_id, or any other context you have already wired through your logging path.

The Log method is called synchronously by the framework. To avoid holding up the logging path, the sentry logger accepts the message and hands it off to a background worker that builds and ships the Sentry envelope. The framework continues without waiting for network I/O to Sentry.

When the internal queue is full, new events are dropped silently. The queue cap exists to bound memory under a panic storm; on a well-behaved system it stays empty most of the time. If you see drops in practice, raise QueueLimit or investigate the actor that keeps panicking.

Terminate() is given a bounded amount of time to drain the queue and flush events that the Sentry SDK has already buffered for transport. After that window the logger gives up and the node exits.

The logger creates its own Sentry client and hub. It does not call sentry.Init() and does not touch the global sentry.CurrentHub(). If you already use the Sentry SDK directly elsewhere in your process, the integrations do not interfere with each other.

The logger accepts the following options:

DSN - Sentry project DSN. Leave empty to fall back to the SENTRY_DSN environment variable as handled by the Sentry SDK. Required either inline or via environment.

Environment - Tag attached to every event. Common values are production, staging, development. Sentry uses this to filter and segment issues.

Release - Version of the running binary. When set, Sentry can link an issue to the deploy that introduced it and surface regressions across releases.

ServerName - Override for the auto-detected hostname. Useful when running in containers where the default hostname is meaningless.

CaptureMetaErrors - Forwards error-level events from meta processes. Off by default. Turn on when meta-process errors are operationally relevant.

CaptureProcessErrors - Forwards error-level events from individual actors. Off by default. Turn on selectively; an actor in a bad state can produce many errors per second.

QueueLimit - Cap on the internal event queue. Events past the cap are dropped. The default is conservative; raise it if you expect panic storms and want to keep more events in flight rather than drop them.

FlushTimeout - Time budget for Terminate() to drain the queue and flush the SDK's transport buffer before the logger returns. The default is enough for normal shutdown; reduce it if your node has very strict shutdown deadlines.

SkipFrames - Top frames trimmed from captured panic stacks. The default matches the standard call chain. Increase if you wrap Log().Panic() in your own helpers, decrease if you call into the logger from a place closer to the panic.

BeforeSend - Hook forwarded to the Sentry SDK. Runs for every outgoing event. Return nil to drop, return the (possibly modified) event to forward. Useful for scrubbing sensitive fields or applying additional filters.

Register the sentry logger alongside your console or file logger in node options:

The default logger continues to write to stdout; the sentry logger receives the same stream and forwards the subset described above. To limit Sentry to just errors and panics without producing log work for the rest of the levels, register the logger with an explicit level filter:

For detailed logger configuration options, see the sentry.Options struct in the package. For understanding how loggers integrate with the framework, see .

A node is the runtime environment where your actors live. Think of it as the container that hosts processes, routes messages between them, and handles the complexities of distributed communication.

When you start a node, you're launching a complete system with several subsystems working together: process management, message routing, networking, and logging. Each subsystem has a specific responsibility, and they coordinate to provide the foundation for your application.

What a Node Provides

Process Management - The node tracks every process running on it. When you spawn a process, the node assigns it a unique PID, registers it in the process table, and manages its lifecycle. When a process terminates, the node cleans up its resources and notifies any processes that were linked or monitoring it. The node provides ProcessRangeShortInfo for efficient iteration over all processes with their current state, including mailbox latency when built with -tags=latency.

Message Routing - When a process sends a message, the node figures out where it needs to go. Local process? Route it directly to the mailbox. Remote process? Establish a network connection if needed and send it there. The sender doesn't need to know these details.

Network Stack - The node handles all network communication. It discovers other nodes, establishes connections, encodes messages, and manages the complexity of distributed communication. This is what makes network transparency possible.

Pub/Sub System - Links, monitors, and events all work through a publisher/subscriber mechanism in the node core. When a process terminates or an event fires, the node knows who's subscribed and delivers the notifications. The node provides EventInfo to query statistics for a specific event and EventRangeInfo for callback-based iteration over all registered events with their per-event counters (messages published, local/remote deliveries).

Logging - Every log message goes through the node, which fans it out to registered loggers. This centralized logging makes it easy to capture, filter, and route log output.

A node needs a name. The format is name@hostname, where the hostname determines which network interface to use for incoming connections.

The name must be unique on the host. Two nodes with the same name can't run on the same machine, but nodes with different names can coexist.

The gen.NodeOptions parameter configures the node: which applications to start, environment variables, network settings, logging configuration. If you specify applications in the options, the node loads and starts them automatically. If any application fails to start, the entire node startup fails - this ensures you don't end up in a partially initialized state.

The node manages the complete process lifecycle.

When you spawn a process, the node creates it, registers it in the process table, calls its ProcessInit callback, and transitions it to the sleep state. The process is now live and can receive messages.

When the process terminates (either naturally or through an exit signal), the node calls ProcessTerminate, removes it from the process table, and notifies any processes that were linked or monitoring. Resources are cleaned up, and the gen.PID becomes invalid.

Processes can register names, making them addressable by name rather than PID. This is useful for well-known processes that other parts of the system need to find. The node maintains a name registry, ensuring each name maps to exactly one process.

Message routing is one of the node's core responsibilities.

When a process sends a message locally, the node simply places it in the recipient's mailbox. The recipient's goroutine wakes up (if it was sleeping), processes the message, and goes back to sleep if no more messages are waiting.

When the message goes to a remote process, things are more interesting. The node checks if a connection exists to the remote node. If not, it discovers the remote node's address (through the registrar or static routes) and establishes a connection. The message is encoded into the Ergo Data Format, optionally compressed, and sent over the network. The remote node receives it, decodes it, and delivers it to the recipient's mailbox.

From the sender's perspective, both paths look identical. That's network transparency.

Making remote message delivery work like local delivery requires solving three problems: finding remote nodes, establishing connections, and ensuring compatibility.

The first problem is discovery. When you send to a remote process, the node extracts which node that process belongs to from its identifier. Every node runs a small registrar service by default. For nodes on the same host, you query the local registrar. For nodes on different hosts, you query the registrar on that remote host - the framework derives the hostname from the node name and sends the query there. The registrar responds with connection information.

This default approach works for simple setups but has limitations. You're querying individual hosts, which requires them to be directly reachable. There's no cluster-wide view, no centralized configuration, no way to discover which applications are running where.

That's where etcd or Saturn come in. Instead of each node being its own island with a local registrar, you run a centralized registry service. All nodes register there when they start. All discovery queries go there. The central registrar becomes the source of truth for the cluster, providing not just discovery but configuration management, application tracking, and topology change notifications. It transforms independent nodes into a coordinated cluster.

Once a node is discovered, connections are established. Multiple TCP connections form a pool to that node, enabling parallel message delivery. The connections negotiate protocol details during handshake: which protocol version to use, whether compression is supported, what features are enabled. This negotiation allows nodes with different capabilities to work together.

Nodes have environment variables that all processes inherit. This provides a way to configure behavior without hardcoding values. A process can override inherited variables or add its own, creating a hierarchy: process environment overrides parent, which overrides leader, which overrides node.

Environment variables are case-insensitive. Whether you set "database_url" or "DATABASE_URL", the process sees the same value. This eliminates a common source of configuration bugs.

Stopping a node can be graceful or forced.

Graceful shutdown stops the running applications first, each one through its own Stop and Terminate callbacks, then sends exit signals to whatever processes are left. Processes receive gen.TerminateReasonShutdown and can save state, close connections, or send final messages before terminating, and the node waits for their ProcessTerminate callbacks to return rather than only for them to leave the process table. Once everything has stopped, the network stack shuts down, the loggers are closed, and the node exits.

Forced shutdown kills all processes immediately without waiting for cleanup. This is useful when you need to stop quickly, but processes don't get a chance to clean up properly.

One subtlety: if you call Stop from within a process, you create a deadlock. The process can't terminate because it's waiting for Stop to complete, but Stop is waiting for all processes (including this one) to terminate. The solution is either to call Stop in a separate goroutine or use StopForce, which doesn't wait.

Graceful shutdown can hang indefinitely if a process is stuck - perhaps blocked on a channel, waiting for an external resource, or caught in incorrect logic. To prevent this, the node has a shutdown timeout.

When the timeout expires, the node escalates: it force-kills any remaining processes and waits a short settle window (5 seconds) for them to unregister. If they still refuse to die, the node hard-exits with error code 1 after logging the surviving processes.

The default timeout is 3 minutes. You can change it globally through gen.NodeOptions:

or per call via StopWithTimeout, which overrides the node-level value for that invocation:

During shutdown, the node logs which processes are still running. Every 5 seconds, it prints a warning with the first 10 pending processes, showing their PID, registered name (if any), behavior type, state, and mailbox queue length. This diagnostic output helps identify what's blocking the shutdown:

The state tells you what the process is doing: running means it's handling a message, sleep means it's idle waiting for messages. The queue count shows how many messages are waiting. A process stuck in running with a growing queue indicates it's blocked in a callback and not processing its mailbox.

When the timeout fires, the same listing is printed at error level along with the force-kill notice, and the post-settle hard-exit report (if reached) shows whichever processes survived the kill so you can identify culprits that ignore termination signals.

Every node has a creation timestamp assigned when it starts. This timestamp is embedded in every gen.PID, gen.Ref, and gen.Alias that the node creates.

When two nodes connect, they exchange their creation timestamps during the handshake. Each connection stores the remote node's creation value.

Before sending any message to a remote process, the framework compares the target's Creation field against the stored creation of that remote node. If they differ, the operation returns gen.ErrProcessIncarnation immediately - no network message is sent.

This mechanism handles a common distributed systems problem: what happens when a remote node restarts? After restart, the node gets a new creation timestamp. Any gen.PID or gen.Alias from before the restart now contains the old creation value. When you try to send a message using that stale identifier, the framework detects the mismatch and returns an error instead of delivering the message to a wrong process.

The check applies to all remote operations: Send, Call, Link, Unlink, Monitor, Demonitor, SendExit, and SendResponse.

The node is infrastructure, not application logic. It provides the mechanisms - process management, message routing, networking - that your actors use to accomplish work.

This separation is important. Your actors focus on application logic: handling requests, processing data, managing state. The node handles the plumbing: routing messages, establishing connections, managing lifecycles. You don't write code to discover remote nodes or encode messages. The node does that.

This is what makes the framework approachable. You write actors that send and receive messages, and the node makes it all work, whether processes are local or distributed across a cluster.

The following chapters dive into specific node capabilities. explains the actor lifecycle and operations. covers distributed communication. explains how processes track each other.

WebWorker is a specialized actor for handling HTTP requests sent as meta.MessageWebRequest messages. It automatically routes requests to HTTP-method-specific callbacks and ensures the request completion signal is called.

Used with meta.WebHandler to convert HTTP requests into actor messages. See for integration approaches.

When WebHandler sends meta.MessageWebRequest to an actor, that actor must:

1. Extract the message from mailbox

Applications often need tasks to run periodically. Generate a daily report at midnight. Clean up expired sessions every hour. Send weekly summary emails. Poll an external API every five minutes.

You could implement this yourself - spawn a process that sleeps, wakes up, performs the task, and sleeps again. But then you're managing wake times, handling timezone changes, accounting for daylight saving time transitions, and ensuring the scheduler itself stays alive. The scheduling logic becomes scattered across your application.

Cron provides scheduled task execution as a framework service. You declare what should run and when using the familiar crontab syntax. The framework handles timing, execution, and all the edge cases around time-based scheduling.

Every minute, the cron system wakes up and evaluates all job specifications against the current time. Jobs whose specifications match the current minute are queued for execution. Each queued job then runs in its own goroutine.

This design is stateless - no pre-calculated schedules, no complex data structures to maintain. When you add a job, it participates in the next evaluation. When you remove a job, it stops participating. Timezone and daylight saving time transitions are handled naturally because each evaluation uses current time rules.

The stateless approach has implications. Multiple executions of the same job can run concurrently if the job takes longer than its interval. A job scheduled every minute that takes two minutes to complete will have two instances running simultaneously. If your job can't handle concurrent execution, implement serialization in the action itself - for example, send a message to a named process that processes requests sequentially.

A process is an actor - a lightweight entity that handles messages sequentially in its own goroutine. It's the fundamental building block of an Ergo application.

Every process has a mailbox where incoming messages wait to be processed. The mailbox contains four queues with different priorities: Urgent for critical system messages, System for framework control, Main for regular application messages, and Log for logging. When the process wakes up to handle messages, it processes them in priority order, taking from Urgent first, then System, then Main, and finally Log.

When built with -tags=latency, each queue tracks the age of its oldest unprocessed message. ProcessMailbox.Latency() returns the maximum latency across all four queues in nanoseconds, or -1 if the tag is not enabled. This helps identify processes that are falling behind on message processing. See for details.

The process runs only when it has messages to handle. When the mailbox is empty, the process sleeps, consuming no CPU. When a message arrives, the process wakes, handles the message, and sleeps again if nothing else is waiting. This efficiency is why you can have thousands of processes in a single application.

A process identifier (gen.PID

Not all code that touches the framework is an actor. A helper that takes a gen.Process to make a call, a custom resolver that implements gen.Resolver, a constructor that reads configuration from a gen.Node - these are ordinary functions, and you test them the ordinary way: give them a dependency you control, run them, and check what happened. mock provides that dependency. For each of the framework's interfaces it offers a standalone fake you can hand to the code under test in place of the real thing.

These fakes are deliberately dumb, and it helps to say plainly what they are not: a mock is not the harness in disguise. It runs no actor, starts no goroutine, and never fails your test on its own. It implements an interface, lets you override the methods you care about, and returns safe defaults for the rest. Each example names the type doing the work, so it is always clear which mock you are looking at.

mock.NewProcess returns a value that satisfies

You cannot test an actor the way you test a function. There is no result := actor.Handle(msg) to inspect: an actor runs on its own goroutine, keeps private state, and speaks only in messages. What you can see is what it does - the messages it sends, the children it spawns, the way it terminates. So the testing tools watch the thing under test and record every such action, and a test then asks questions of that recording: did it send this, how many times, to whom, and after what.

check is the language those questions are written in. It is not a harness you run on its own; it is the shared vocabulary - the record types and the assertion grammar - that both and hand you. You will rarely import it directly. You call its assertions on a unit Subject or a stage Node, and because both layers expose the very same grammar, learning it once here lets you assert anything in either of them. The examples below use a handle - sub for a unit subject, node

An actor's state is private by design. Nothing outside it can read a field, and that is what makes the actor model safe: no locks, no shared memory, no races. It is also what makes a running actor system opaque. A debugger can pause one goroutine, but a node is thousands of them, and the interesting question is rarely about one process in isolation.

HandleInspect is the one sanctioned way out of that. It is not an API for other processes to use, not a message handler, and not a place to compute anything. It is the actor's answer to a single question, asked from the outside at an arbitrary moment: what do you currently believe?

func (w *Worker) HandleInspect(from gen.PID, item ...string) map[string]string {
    return map[string]string{
        "state":       w.state,
        "queue_depth": fmt.Sprintf("%d", len(w.queue)),
        "last_error":  w.lastError,
    }
}

The mechanics are covered in Actor: requests arrive on the Urgent queue, values are strings, and the callback must return immediately. This page is about the part the mechanics do not tell you - what to put in there, and what it buys.

The division of labour is worth stating plainly, because it is unusual. The framework does not define what belongs in that map. It has no schema for it, no field registry, no notion of which keys are meaningful. All it does is instrument: deliver the request, call your callback, and carry the result to whoever asked - the observer renders it, its MCP surface returns it as a tool result, a sibling process gets it from process.Inspect. The content is entirely yours, and so is the diagnostic value. An actor whose callback returns three convenient fields is an actor that cannot be diagnosed, and no amount of tooling above it changes that.

This is the whole of it. Diagnosis is limited to the fields you chose to expose, and that choice is made months before the incident by someone who does not know what the incident will be.

The failure mode is specific and easy to walk into. An actor exposes the values that were easy to format - a name, a counter, a boolean - and omits the ones that carry the decision it just made. Everything looks reasonable in the inspection output, and the actual state is unreachable.

A concrete case. A leader-election actor exposed four fields: cluster id, term, a boolean "am I the leader", and a peer count. Every one of them is true and none of them is enough. The actor has three roles, not two - follower, candidate, leader - and a boolean cannot say which of the first two it is. So a replica stuck as a candidate for hours, campaigning and never winning, was indistinguishable from a healthy follower on every surface the system offered. The peer count said 3 without saying which three, so a stale entry pointing at a process that no longer existed looked identical to a live peer. Nothing reported whether an election timer was still armed, which is the difference between a replica that will try again and one that has stopped trying.

The state was recoverable, but only sideways: by comparing per-connection message counters between nodes to prove that a leader had emitted nothing for a day, and by taking four inspection readings twenty minutes apart to prove a term had stopped advancing. Each of those would have been one field.

The rule that follows is not "expose more". It is: expose the state your code branches on. If a line of your actor reads a field to decide what to do, an operator will eventually need to read the same field to understand what was done.

In practice, five kinds of field carry almost all the diagnostic value.

The derived role, not the raw flags. If your actor has three states and you store two booleans, expose the resolved state as a word. Whoever reads it should not have to reconstruct your state machine from its parts.

Identities, not counts. peers: 3 cannot be checked against reality. peers: n1@host,n2@host,n3@host can, and it is how a stale or duplicated entry becomes visible.

Things you dropped silently. Every place where your code decides to ignore a message is invisible by construction. A counter per reason turns a silent drop into evidence: dropped: cluster_id_mismatch=2,stale_heartbeat=17. This is usually the highest-value field in the map, because a message that was discarded leaves no other trace anywhere.

Whether your timers are armed. For any actor driven by SendAfter, "waiting to act" and "has stopped acting" look the same from outside. One boolean separates them. Note that testing the gen.CancelFunc you stored does not answer it: SendAfter fires once, and nothing clears your variable when it does, so a spent handle still looks armed. Track the arming explicitly or the field will lie.

When the last transition happened. A value plus the timestamp it last changed answers "is this stuck?" in one reading. Without the timestamp it takes two readings and a guess about how long to wait between them.

Keep it cheap. The callback runs on the actor's own goroutine, so while it runs the mailbox is not being drained. Format what you already hold; do not compute, do not call out, do not touch the network.

act.Actor, act.Pool, act.Router, act.Supervisor and the extra-library actors all have state of their own worth reporting - pool statistics, restart counts, election state. If your implementation overrides HandleInspect, that state must not disappear.

The base behaviors handle this for you: they compute their own fields first and merge yours on top.

So a consumer adds fields, and may deliberately replace one, but cannot erase the rest. If you are implementing HandleInspect on top of one, you can rely on it: the framework's fields will be there beside yours.

The base behaviors namespace their own keys with a reserved ergo: prefix - ergo:pool_size, ergo:children_total, ergo:state and so on. That is what makes the merge safe in both directions: a field of yours cannot collide with one of theirs by accident, and you can still override one deliberately by naming it with the prefix. If you write a base behavior of your own, follow the same shape and the same prefix; if you are the consumer, keep your keys unprefixed and they will never clash.

Everything above assumes the answer fits in a map. Plenty of actors hold state that does not: a registry with a hundred thousand sessions, a scheduler with a deep queue, a cache with a million keys. Returning all of it is not an option - the callback must be cheap, and nobody can read it anyway.

The item arguments are the way out, and they are more than a field filter. Treat them as a small query vocabulary with three tiers.

A bounded summary by default. With no items, answer with aggregates only: totals, distribution, the extremes. The size of this answer must not depend on the size of the state.

A help item that names what can be asked. This is what makes the surface self-describing. A reader - human or agent - does not need to know your schema in advance; it asks once and learns the vocabulary. Nothing in the framework enforces this or knows about it: it is a convention you implement, which is precisely why it is worth implementing - without it the vocabulary exists only in your source.

Parameterised items that drill into one entity. session <id>, user <id>, top slowest - each returns detail about a small, named part of the state.

Four things that matter in that shape.

Cap every answer. A query that can match a million entries must return the first n and say so. An unbounded answer reintroduces the problem the queries exist to avoid.

Never scan. user <id> above reads an index the actor already maintains. If answering a query means walking the whole state, either keep the index or do not offer the query - the callback holds the actor's own goroutine while it runs.

Report an unknown item. Returning nothing for a key the caller asked about is indistinguishable from a value that happens to be empty. <unknown item> costs one line and removes the ambiguity.

Keep it read-only. A query language over item is a good idea; a command language over it is not. Callers treat inspection as free - tools cache and retry it, an agent exploring a symptom calls it dozens of times - so anything that changes state belongs in HandleCall instead.

Answering queries also changes what the vocabulary itself tells a reader. session <id>, user <id>, top slowest says that this actor is indexed by session and by user and tracks latency, before anyone looks at a single value. That is diagnostic information in its own right: two actors with the same number of fields can differ enormously in how much they will let you ask, and help is where the difference becomes visible.

A single inspection call is a snapshot of one actor. What makes the callback worth designing carefully is that everything above it is built from those snapshots.

One node, read by a human. shows a process list, its tree, and the inspection output of whichever process you open, updating live. This is the view for "I know roughly where the problem is".

A whole cluster, read by an AI. The exposes the same inspection as resources and tools an agent asks for on demand: enumerate processes across the cluster, inspect any of them, follow the topology, capture a profile. Every node runs the built-in system application that answers those questions, and the one node serving MCP reaches all of them, so a single conversation covers the cluster.

The difference between the two is not convenience, it is method. A dashboard answers questions decided in advance. An agent holding the whole inspection surface can work the other way round: start from a symptom, enumerate what exists, read the state of the processes that look implicated, correlate across nodes, and narrow down. Point diagnosis becomes system diagnosis, because nothing has to be selected up front.

And it reads your source. This is the part that changes the character of the work. An agent that can inspect live actor state and read the code that produced it is looking at cause and effect at once. The state says what the system believes; the source says which branch produced that belief and what it will do next. Neither alone is enough - a field value without the code is a number, and code without runtime state is a hypothesis - and together they close the loop that a debugger closes for a single-threaded program, but across a live distributed system that cannot be paused.

That is the reason to treat HandleInspect as a design surface rather than a debug convenience. The value of every layer above it - the observer view, the cluster-wide diagnostic, the agent that explains what it found - is bounded by whether the field it needed was exposed.

Follow the query vocabulary to its conclusion and a summary plus a handful of drill-downs stops being a debug aid. Look up one entity by id, list what belongs to one tenant, show the worst ten by latency - that is the read half of an admin panel, over live state, for the cost of a switch statement. No separate service, no query layer, no HTTP handlers, no second deployment, and no copy of the data: the actor already holds the state, and it already maintains the indexes because it needs them to do its job. The observer and MCP supply the front end.

Worth being clear about the boundaries, so nobody builds the wrong thing on it.

It is the read half only. Commands go through HandleCall, for the reasons above.

It is for operators, not end users. There is no authentication on it beyond access to the node, no per-tenant scoping, and no audit trail. Whoever can reach the observer or the MCP entry point can ask anything the actor answers. That is the right trade for an engineering tool and the wrong one for a customer-facing feature.

It is a diagnostic surface, not a data API. Values are strings, keys are yours to rename, and nothing versions them. If another system needs this data, give it a proper request and response instead of parsing an inspection map.

Before shipping an actor, read your own HandleInspect against these:

• Does it report the role or phase as a word, rather than the flags it is derived from?

• Does it name the peers, children or targets it holds, rather than counting them?

• Does every branch that silently drops or ignores something have a counter here?

• If the actor is timer-driven, can a reader tell "waiting" from "stopped"?

• - the callback's mechanics and constraints

• - the human-facing view of the same data

• - the same data as an agent-facing MCP surface

• - using Ergo as both runtime and diagnostic surface

Remote application starting means launching an application on another node from your code. The remote node has the application loaded but not running. You send a start request, and the application starts on that node with the mode and options you specify. The application runs under the remote node's supervision, part of the remote node's application tree.

This capability enables dynamic application deployment and orchestration. You have a cluster of nodes, each with applications loaded but waiting. A coordinator node decides which applications should run where, based on load, topology, or scheduling logic. Remote application starting makes this coordination explicit and controllable.

Like remote spawning, remote application starting isn't automatic. Security matters. You don't want arbitrary nodes starting arbitrary applications. The framework requires explicit permission - the remote node must enable each application individually and can restrict which nodes are allowed to start it.

Security Model

The gate is the application registry, not the flag. EnableRemoteApplicationStart is on in gen.DefaultNetworkFlags, and a node that configures no flags of its own is given those defaults, so on an ordinary node the flag is already true. What actually stops a peer is that no application can be started remotely until network.EnableApplicationStart(name) has allowed it.

Set the flag when you want the transport-level door shut - a node that must never accept a remote application start, whatever it has enabled. Note that it is only consulted when Flags.Enable is true, and that supplying Flags yourself replaces the whole default set rather than adjusting one member of it:

This flag is a global switch. With it disabled, all remote application start requests fail immediately with gen.ErrNotAllowed. With it enabled, requests proceed to per-application permission.

Even with EnableRemoteApplicationStart turned on, remote nodes can't start anything until you explicitly enable specific applications:

Now remote nodes can request starting the "workers" application. The application must be loaded on this node (via node.ApplicationLoad). If it's not loaded, remote start requests fail with gen.ErrApplicationUnknown. If it's already running, remote start requests fail because you can't start a running application again.

The application name is the permission token. Remote nodes must use this exact name when requesting starts. If they request "workers" and you haven't enabled it, the request fails. If they request "admin_app" without permission, it fails. You control what's startable remotely.

By default, EnableApplicationStart allows all nodes to start the application. But you can restrict it to specific nodes:

Now only those two nodes can start the workers application. Requests from other nodes fail with gen.ErrNotAllowed.

You can update the access list dynamically:

Calling EnableApplicationStart again with the same application name updates the access list.

To remove nodes from the access list:

This removes scheduler@node2 from the allowed list. Other nodes in the list remain allowed.

To completely disable remote starting for an application:

Without any node arguments, DisableApplicationStart removes the permission entirely. All future start requests for that application fail.

To re-enable with an open access list (any node can start):

This is the explicit "allow all nodes" configuration.

To start an application on a remote node, first get a gen.RemoteNode interface:

With the remote node handle, start an application:

The application starts on the remote node. The start is synchronous - the call blocks until the remote node confirms the application started or returns an error.

The mode is the application's termination policy: it decides when the termination of a group member takes the whole application down. For remote starts you can state it explicitly, overriding the mode the spec declared:

If you use ApplicationStart without specifying a mode, the application starts with the mode it was loaded with (set during ApplicationLoad).

Nothing restarts the application itself. The mode decides when it stops, never whether it comes back, and that holds for a remote start exactly as it does for a local one. Choose it by how much of the group must be alive for the service to mean anything: a permanent application is one where a missing member makes the rest pointless.

For details on application modes, see .

When an application starts remotely, parent tracking is set at multiple levels:

Application Parent: Set to the requesting node name:

Process Parent for Group Members: Processes started directly by the application (listed in Group) receive the requesting node's core PID as their parent:

Process Parent for Descendants: If those processes spawn children, the children receive their spawning process PID as parent (normal process hierarchy):

Only the first-level processes (application group members) have the cross-node parent relationship. Subsequent generations follow standard process parent-child relationships within the local node.

This parent information is for tracking and auditing, not supervision. The application is supervised by the local application supervisor on the remote node. Terminating the requesting node does not affect the running application.

By default, remote applications don't inherit environment variables from the requesting node. To enable environment inheritance:

Now when you start an application remotely, the application's processes receive a copy of the requesting node's core environment. This enables configuration propagation - your scheduler node has configuration in its environment, and applications started remotely inherit it.

Important: Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via node.Network().RegisterType (see for details on the type registry; the legacy edf.RegisterTypeOf still works but is deprecated). If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote application start fails entirely with an error like "no encoder for type <type>". The framework doesn't skip problematic variables: any non-serializable value causes the entire start request to fail.

When you call remote.ApplicationStart:

1. Check capabilities - The local node checks if the remote node's EnableRemoteApplicationStart flag is true (learned during handshake). If false, fail immediately.

2. Create start message - Package the application name, startup mode, and options into a MessageApplicationStart protocol message. Include a reference for tracking the response.

3. Send request - Encode and send the message to the remote node. Wait for a response (this is synchronous - remote application start blocks until the remote node replies).

If anything fails (application not found, access denied, already running, remote node terminating), the error is returned to the caller. The entire operation is synchronous - you call ApplicationStart and block until the application is running or an error occurs.

Idempotency - Starting an already-running application returns an error. If you're unsure of the application's state, query it first using remote.ApplicationInfo to check if it's already running. Or handle the error gracefully and treat "already running" as success.

Startup time - Some applications take time to start - they might load configuration, establish connections, initialize state. The remote start call blocks during this entire startup sequence. If startup is slow, the caller waits. For long-running startup logic, consider using async patterns or monitoring application state separately.

Failure modes - Remote application start can fail in ways local start can't. The network connection can drop mid-request. The remote node can crash before responding. The application might fail to start for reasons specific to that node (missing dependencies, configuration issues). Handle errors explicitly.

Resource contention - An application starting on a remote node consumes that node's resources (CPU, memory, file descriptors). If multiple nodes simultaneously request starting applications on the same remote node, it could become resource-constrained. Coordinate start requests to avoid overwhelming nodes.

Application lifecycle - Once started remotely, the application runs until explicitly stopped or until the remote node terminates. The requesting node has no automatic control over the running application. gen.RemoteNode has no stop counterpart: to stop it later, either ask the node's system application through manage.RequestDoAppStop, or coordinate it with your own messages.

Supervision independence - The application is supervised by the remote node, not by the requesting node. If the requesting node crashes, the application keeps running. If the remote node crashes, the application terminates. This independence is important for operational reasoning - the application's lifecycle is tied to where it runs, not to who started it.

Configuration management - Applications often need configuration. With ExposeEnvRemoteApplicationStart, you can propagate environment variables. But this creates coupling - the application depends on the requesting node's configuration. Consider whether configuration should come from the remote node's local environment, from a centralized configuration service, or from the requesting node. The right answer depends on your architecture.

Dynamic orchestration - A coordinator node decides which applications should run on which nodes based on cluster state, resource availability, or scheduling logic. The coordinator starts applications dynamically as needed.

Staged deployment - Applications are pre-loaded on nodes but not started. A deployment controller starts them in a specific order, waiting for health checks between stages. This enables controlled rollouts.

Capacity management - Some applications run only during high-load periods. A resource manager monitors load and starts applications on additional nodes when needed, then stops them when load decreases.

Geographic distribution - Applications are loaded across multiple regions. A traffic manager starts applications in specific regions based on user distribution, latency requirements, or failover needs.

Testing and validation - Test frameworks load applications on test nodes but don't start them until test execution. Tests start applications with specific configurations, run scenarios, then stop them. This enables repeatable, isolated testing.

Maintenance windows - During maintenance, you stop applications on a node, perform updates, then start them again. Remote start enables coordinated maintenance across a cluster without manually SSHing to each node.

Remote application starting is about control and coordination. If your cluster has static application deployment (applications always run on specific nodes), you don't need this feature - use supervision trees and let supervisors start applications automatically. If your cluster has dynamic application deployment (applications move between nodes based on conditions), remote application starting enables that flexibility.

For understanding the underlying network mechanics, see . For controlling connections to remote nodes, see . For understanding application lifecycle and modes, see .

Building reliable systems means accepting an uncomfortable truth: failures will happen. Hardware fails. Networks partition. Bugs exist in code. The question isn't whether your processes will crash, but what happens when they do.

The supervision tree model provides an answer. Instead of trying to prevent all failures, you structure your system so failures are expected, isolated, and automatically recovered from.

The Supervision Principle

The model divides processes into two distinct roles:

Workers do the actual work. They handle requests, process data, manage state, and inevitably, sometimes crash when things go wrong.

Supervisors watch over workers. Their only job is to start child processes and restart them when they fail. Supervisors don't do application work - they manage lifecycle.

This separation is crucial. If workers handled their own restart logic, a bug in that logic would prevent recovery. By moving restart responsibility to a separate supervisor, you ensure that failures in workers can always be recovered.

How Supervision Works

A supervisor starts its children and monitors them. When a child crashes, the supervisor decides what to do based on its restart strategy. Should it restart just this one child? Restart all children? Restart all children in a specific order?

The strategy depends on the relationships between children. If they're independent, restart just the failed one. If they depend on each other, restart all of them to ensure consistent state. If they have startup dependencies, restart in order.

Supervisors can supervise other supervisors, forming a tree. At the top might be an application supervisor. Below it, supervisors for different subsystems. Below those, the actual workers. When a worker crashes, only its portion of the tree is affected. The rest of the system continues running.

This tree structure creates fault isolation boundaries. A crashed database worker doesn't affect the HTTP handler workers. A failed cache process doesn't take down the authentication processes. Each supervision subtree handles its own failures without cascading them upward.

The Erlang community calls this "let it crash." It sounds reckless, but it's actually disciplined. Instead of defensive programming trying to handle every possible error, you let processes fail and rely on supervisors to restart them in a clean state. Often, a fresh restart clears transient problems that would be difficult to handle explicitly.

Ergo Framework implements supervision through the act.Supervisor actor. When you create a supervisor, you specify its children and restart strategy. The framework handles the monitoring and restart logic.

Workers are typically act.Actor implementations - regular actors that do application work. Supervisors are act.Supervisor implementations - actors whose behavior is managing children.

Because supervisors are also actors, they can be supervised. This is how you build the tree: supervisors supervising supervisors supervising workers, all the way down.

The tree structure emerges from how you compose supervisors and workers. There's no special tree-building API. You just nest supervisors, and the tree forms naturally.

The supervision tree model leads to systems with interesting properties.

Self-healing - Failures trigger automatic recovery. Most transient problems resolve themselves through restart.

Graceful degradation - When a subsystem fails, only that part stops working. The rest continues serving requests.

Operational simplicity - Instead of complex error handling throughout your code, you centralize recovery logic in supervisors.

The trade-off is that you need to design processes that can restart cleanly. State that must survive restarts needs to be externalized - in databases, in other processes, or rebuilt from messages. But this discipline leads to more robust designs anyway.

Understanding supervision requires seeing it in practice. The chapter covers the specifics: restart strategies, child specifications, and practical patterns for structuring your application.

The combination of the actor model (isolated processes, message passing) and supervision trees (automatic recovery) gives you the tools to build systems that handle failures gracefully. It's a different approach than traditional error handling, but one that scales well to distributed systems where failures are inevitable.

WebSocket provides persistent bidirectional connections between clients and servers. Unlike HTTP request-response, a WebSocket connection remains open for extended periods, allowing both client and server to send messages at any time.

The framework provides WebSocket meta-process implementation that integrates WebSocket connections with the actor model. Each connection becomes an independent actor addressable from anywhere in the cluster.

WebSocket connections need two capabilities simultaneously:

Continuous reading: Connection must block reading messages from the client. When a message arrives, forward it to application actors for processing.

Asynchronous writing: Backend actors must be able to push messages to the client at any time - notifications, updates, events from the actor system.

This is exactly what meta-processes solve. External Reader continuously reads from the WebSocket. Actor Handler receives messages from backend actors and writes to the WebSocket. Both operate concurrently on the same connection.

Two meta-processes work together:

WebSocket Handler: Implements

A single actor processes messages sequentially. This is fundamental to the actor model - it eliminates race conditions and makes reasoning about state straightforward. But it also means one actor can become a bottleneck. If messages arrive faster than the actor can process them, the mailbox grows, latency increases, and eventually the system stalls.

The standard solution is to run multiple workers. Instead of sending requests to one actor, distribute them across several identical actors processing in parallel. This works, but now you need routing logic: pick a worker, check if it's alive, handle mailbox overflow, restart dead workers. This boilerplate appears in every pool implementation.

act.Pool solves this. It's an actor that manages a pool of worker actors and automatically distributes incoming messages and requests across them. You send to the pool's PID, the pool forwards to an available worker. The pool handles worker lifecycle, automatic restarts, and load balancing. From the sender's perspective, it's just one actor. Under the hood, it's N workers processing in parallel.

Like act.Actor provides callbacks for regular actors, act.Pool uses the act.PoolBehavior

The actor model excels at point-to-point communication. Process A sends a message to process B. Process C makes a request to process D. Each interaction has a specific sender and receiver.

But some scenarios need one-to-many communication. A price feed updates and dozens of trading strategies need the new price. A user logs in and multiple subsystems need notification. A sensor reading arrives and various monitoring processes need to react. You could send individual messages to each interested process, but then the producer needs to track all consumers. When consumers come and go, the producer's consumer list becomes a maintenance burden.

Events solve this with publish/subscribe semantics. A producer registers an event and publishes values to it. Consumers subscribe to the event without the producer knowing who they are. The framework handles message distribution - when the producer publishes an event, all current subscribers receive it. Subscribers can come and go dynamically, and the producer's code doesn't change.

A process becomes an event producer by calling RegisterEvent with an event name and options. The call returns a token - a unique reference that proves ownership. Only the process holding this token (or a process it delegates to) can publish events under this name.

The Notify option controls whether the producer receives notifications about subscriber changes. When enabled, the producer receives