Inspecting With Observer
Real-time inspection and management of Ergo nodes
A running Ergo node is not one program you can step through with a debugger. It is thousands of processes, each on its own goroutine, each handling its own messages, being restarted by supervisors as failures happen. You cannot pause one and read its variables the way you would a function, and adding print statements changes the very timing you are trying to understand.
Observer takes a different approach. It watches the live node the way the system itself does: what each process is doing right now, how messages travel between them, and where CPU and memory go. It embeds into the node, streams updates to your browser as they happen, and needs no changes to the processes it observes. And it is not read-only: from the same screen you can change a setting, send a message, or stop a process on the running system.
You deploy Observer on a single node and inspect the whole cluster from it. Every Ergo node runs the built-in system application, which is what Observer talks to, so switching to another node needs nothing extra deployed there. For installation and options, see Observer Application.
To try it against a live cluster:
git clone https://github.com/ergo-services/examples
cd examples/observability
make upThis starts a multi-node cluster with Observer, tracing, health probes, Prometheus metrics, and Grafana dashboards. Open http://localhost:9911 for Observer and http://localhost:8888/dashboards for Grafana.
How to read this page
The tour goes from the outside in. First how to move around the interface at all, then the node at a glance, its applications, and finding the one process that matters among thousands. Then everything about that process and how to act on it. After that come the specialized views, each answering a specific question: how messages flow, how the node is connected, what the logs say, where memory and time go, and how a single request travels across the system. Then how to point any of these views at any node in the cluster, and last, the same data read by an AI agent rather than by you.
Screenshots are collapsed. Open the one you need; the page stays light if you do not.

Getting around
Everything lives behind a sidebar with eight pages, and the page you are on is only half the interface. The other half is floating windows: click a PID, an event, a connection, or an application anywhere in the UI and it opens in its own window on top of the page. Windows drag, resize, minimize, and maximize on a double-click of the title bar. Several can be open at once and they survive page switches, so you can keep a suspect process in view while you read the log or a trace somewhere else.
The sidebar lists every open window below the navigation, so you always know what you have open even when a window is minimized or buried. A window whose subject is gone (the process terminated, the connection dropped) stays open with its name struck through and marked "gone", so a disappearance is something you notice rather than something that silently vanishes. One button minimizes them all.
Each window has a copy link action. The link encodes the node and the window, so pasting it into a chat gives a colleague the same process detail window on the same node, not just the page it was on. Opening such a link switches Observer to the right node first, then reopens the window. Links exist for processes, meta processes, connections, applications (optionally at a subtree root), and event streams.
The top bar carries the node selector on the left and the connection indicator on the right. The indicator is not decoration: it reports whether the stream is actually live, and clicking it opens the traffic counters and the list of active subscriptions, which is how you tell "nothing is happening" apart from "the stream is dead". A theme toggle and an About dialog sit at the bottom of the sidebar, which also collapses to icons when you need the width.
The node at a glance: Info
The Info page answers the first question you ask: is this node healthy, and what is it running? Three glance cards and three live charts across the top track the numbers that move: process count split into total, running and zombie; event throughput published against received; memory used against the GOMEMLIMIT ceiling; and per-second CPU split into user and system time. A climbing memory line or a growing process count shows up here first.
Below the glance, Node identifies it (name, CRC32, framework version, operating system and architecture), Node Runtime carries the figures the Go runtime reports (goroutines, live heap, the next GC threshold, GC cycles and the CPU fraction spent collecting), and Node Config holds two controls that take effect immediately: the log level, and the tracing sampler that decides whether the node starts new traces for messages sent through node.Send() and node.Call(). Turn the sampler up to investigate, and back off when you are done.
The rest of the page summarizes the node without leaving it. Registry counts registered names, aliases and events. Delivery Errors splits failures four ways, send and call, local and remote, which separates "the target is gone" from "the network is broken". Events shows published, received, local sent and remote sent. Loggers shows the log volume per level beside the loggers currently registered, and Tracing counts spans by kind (send, call, request, response, spawn, terminate) beside the registered exporters. Both say plainly when nothing is registered.
Managing applications: Applications
An Ergo node runs its work as applications, each a supervised group of processes. A strip across the top counts them by state (total, running, transitioning, loaded) and every application is a card, and the card is a summary of the contract it declared: its state (loaded, running, stopping) and mode, its description, version, parent and uptime, whether it depends on the network, and the applications it depends on. Two fields feed service discovery: its weight, the priority the registrar applies when several nodes offer the same application, and its tags, the labels a caller selects on (see Tags for Instance Selection).
Two numbers on the card are easy to confuse and worth separating. Members is how many processes the application declared in its group. Processes is how many exist in its tree right now, the members plus everything they spawned, and the bar draws that as a share of the whole node. An application declaring four members and owning twelve hundred processes is not a contradiction; it is a supervisor tree that grew, and the bar tells you how much of the node it accounts for.
The card also shows the role map when the application defines one, which decouples logical roles from process names (see Process Role Mapping), and the application environment when it carries any.
The lifecycle controls sit on the card: start in a chosen mode, stop, force-stop when a graceful stop will not complete, or unload. These are the coarse controls; when something is wrong at the application level, this is where you intervene. System applications are protected from all four.
The supervision tree
A running application opens its process tree in a floating window, and the tree is more than a diagram.
Colour carries a signal you choose. Kind paints each node by what the process is, which the framework reports for its own behaviors and your actors can opt into (see the Kind column below). The other five modes are heatmaps: mailbox depth, mailbox latency, utilization (the share of a process's lifetime spent inside callbacks), throughput, and state. An entire application lights up by the signal you picked, so a hot or backlogged branch becomes obvious at a glance. Each mode has a "how to read" explanation next to the selector, so the colour scale is never a guess.
The tree is navigable rather than static. Search by name or PID and step through matches with Enter and Shift+Enter, zoom in and out or fit the whole tree to the window, collapse branches beyond a depth you set, and adjust node width and label truncation when names are long. Any node can be opened as a subtree, either in place or in a new window, and a control takes you back to the application root. The node cap bounds how much is fetched at once, and the tree refreshes either on demand or on an interval you choose. Display options are remembered across sessions; per-window ones reset when the window closes.
Finding the process that matters: Processes
The processes page is where you spend most of your time. Every process on the node appears in a table that refreshes every second, and the columns are less a list of facts than a set of signals you learn to read.
Mailbox depth turning yellow then red is a backlog forming: messages are arriving faster than the process handles them.
A state stuck in "running" for tens of seconds is a process blocked inside a handler, usually on something it should not be doing there.
High running time relative to uptime marks a hot process that spends its life executing handlers rather than waiting.
A green delta like "+42" next to Messages In shows who is active right now, in the last second.
Because the columns are signals, sorting turns the table into a diagnostic tool. Sort by Messages In to find the busiest processes, by Mailbox to bring the most backlogged to the top, by Running Time to see where handler time goes. Fourteen columns cover identification (PID, name, kind, behavior, application), messaging (messages in and out, mailbox depth, latency), and lifecycle (running time, init time, wakeups, uptime, state), which is enough to answer most questions without opening a single process.
Kind deserves a note. It classifies a process by what it is for rather than what type implements it: actor, supervisor, pool and router describe structure; fsm, saga, worker, scheduler, queue, producer, consumer and coordinator describe behavior; web, gateway, proxy, stream, broker and client sit on a boundary; store, cache and session hold data; metrics, leader, follower, health, monitor and logger are operational. The base behaviors report their own kind; your actors opt in either statically, by implementing ProcessKind(), or at runtime with SetProcessKind. Any string is valid, and anything outside the list renders as custom. Colour encodes the category and the icon the specific kind, the same way in the table and in the supervision tree; a reference gallery is one click away from either.
Three cards across the top summarize the node rather than the scope: process counters (total, spawned, terminated, spawn failures), a States bar showing how they split across running, sleep, wait and zombie, and delivery errors split local and remote. Clicking a segment of the States bar filters the table to it. Below them, three charts follow the current scope: messages in and out, the distribution of utilization, and mailbox latency. Click any PID to open a floating detail window.
Choosing what the table shows: Scope
A node can run tens of thousands of processes, so the table never shows all of them at once. The Scope panel decides what the node sends to the browser, and it works in two modes.
In the default mode you pick a window into the process list: "first 500" returns the 500 oldest processes, "last 500" the 500 newest, and entering a PID starts the window there. The node scans only that window, which stays fast no matter how many processes exist because it never looks beyond the range you asked for.
The "All" mode switches to a full scan: the node walks every process, applies your filters as it goes, and returns up to 10,000 matches. Because a full unfiltered scan could flood the browser, this mode requires at least one filter.
Filters narrow by name, behavior type, application, state, or minimum mailbox depth, and appear as removable chips in the toolbar. A separate search field adds quick matching over PID, name, behavior and application on top of the returned rows, for ad-hoc lookups without changing the scope.
Everything about one process: Process Details
Once the table points you at a suspect, the floating detail window tells you everything about it. A header line carries its state, PID, kind, behavior and application, four cards count messages in and out, mailbox depth and wakeups, and four tabs hold the detail.
The overview tab shows two live charts: incoming and outgoing message rates over the last minute, and the depths of the four mailbox queues (Main, System, Urgent, Log). Cards below show running time, init time, state time and uptime, the same signals as in the table but plotted over time. The parent and leader processes appear as links that open their own windows.
The relations tab reveals how the process is wired into the rest of the system: the aliases it registered, the meta processes it owns, the events it created, and its links and monitors grouped by type (see Links and Monitors). This answers a question that matters before you touch anything: who else is affected if this process terminates. Events registered on other nodes can be opened from here as remote event streams. From this tab you can also open the process's supervision subtree as a floating tree, with the same colour modes as the application tree, to see where it sits and which branch below it is busy.
The inspect tab shows whatever the process chooses to publish about itself through its HandleInspect callback, as live key-value pairs, refreshed on demand or on an interval you pick, with a filter over the keys. If your actor implements that method, it can surface any internal state you care about: queue lengths, cache sizes, open connection counts, the current step of a job. This is your own window into a process that the framework cannot see on its own. A process that implements nothing says so rather than showing an empty box.
Acting on a live process
The detail window is also where Observer stops being read-only.
The config tab changes settings that take effect on the running process immediately: raise its log level to get more detail from just that process, adjust message priority, compression (with its algorithm, level and threshold), important delivery and network ordering for its outgoing messages, or turn on the tracing sampler for a targeted look. The fallback setting shows where messages go when the mailbox overflows. The environment section appears when the node has ExposeEnvInfo enabled in its security settings.
Three icons in the window header intervene directly, from whichever tab you are on. Send Message delivers a message to the process. Send Exit sends an exit signal with a reason you choose, normal, shutdown or your own. Kill terminates it immediately. These act on the real system, so each asks for confirmation and names the target; they are disabled for system processes.
Processes the framework does not schedule: Meta
A meta process is not an actor with a mailbox loop; it is a goroutine the framework supervises on an actor's behalf, which is how a TCP connection, a web handler or a port stays attached to a process without blocking it. They appear on the relations tab of their owner and open into their own window.
That window is the actor window in miniature and for the same reasons: its two mailbox queues against their size limit, message counters in and out, uptime, the log level and message priority as live controls, and the meta process's own HandleInspect state. You can send it a message or an exit signal from there, which is how you close one connection out of thousands without touching the process that owns them.
Watching messages flow: Events
Processes rarely work alone; they publish and subscribe to events. The events page makes that traffic visible.
Five cards across the top count the events registered and the messages published, received, delivered locally and sent to remote nodes. Below them, like the processes page, the table shows only what the scope defines. Each row names the event, its producer process, when it was registered, how many subscribers it has, and publication statistics, with delta indicators marking events that are actively publishing. The From control chooses First (oldest registered) or Last (newest), and filters narrow by name, notify mode, buffered mode, open mode, and minimum subscriber count. Three charts above the table summarize the scope: event throughput, event utilization, and how recently each event last published.
Beyond the table, you can open a live stream of a single event and watch the actual messages as the producer publishes them, in real time. Filters match the message type and its content, with an exclude mode to hide noise, so you can answer questions the counters cannot: what exactly is this producer emitting, and does it match what subscribers expect. The window states which of the two it is doing: observing a producer that publishes regardless, or acting as the first subscriber of an event whose producer is notified.
How the node is connected: Network
The network page shows how the node reaches the rest of the cluster and how much traffic flows.
The top of the page is configuration rather than telemetry. Parameters holds the network mode, max message size, handshake and protocol versions, and the negotiated flags. Registrar shows the service discovery backend, its endpoints, and which of its optional capabilities are available (proxy, application routes, config, events), so a node that cannot find its peers tells you here whether it even has a registrar to ask. Acceptors lists the node's listeners with their addresses, TLS configuration and per-acceptor flags. Below that, the page splits into five tabs.
The Connections tab is the default. Four live charts plot aggregate traffic across all connections: messages per second, bytes per second, compression operations, and fragmentation operations, each in and out. A connection table with its own scope shows every connection with delta indicators: direction, TLS or plain, node and connection uptimes, messages and bytes each way, pool size, reconnections and measured clock skew. Click a row for a detailed window. A cluster nodes section lists everything known through the registrar or an active connection, which is the quick picture of the topology.
The Routes tab shows configured static routes and proxy routes side by side: static routes tell the node where to dial when a name matches, proxy routes describe how to reach nodes through an intermediary. A node with neither says so, which is the expected state when discovery is doing the work.
The Types tab is a snapshot of the wire-format type registry, the set of message types the node knows how to serialize. Each row shows its registration ID, owning protocol version, kind, the wire size of a zero value, and canonical name; expand a row for the inferred Go shape of the type. Two filters narrow by name and by schema content. Refresh re-fetches the registry, which rarely changes after startup, so this tab does not stream. Errors and Atoms are the same idea for the other two registries: the error sentinels and the atoms this node knows how to put on the wire. With the node built using -tags=typestats, extra columns show per-type encode and decode counts and wire-byte totals; The average bytes per operation is what to sort on when picking compression candidates: a high average is worth compressing, a low one is not worth the framing overhead. See The typestats Tag.
Connection details
Clicking a connection row opens a window with the full picture of one connection. Metric cards show messages and bytes in each direction. The identity section shows node and connection uptimes, framework and protocol versions, max message size, and the negotiated network flags as colored pills (Remote Spawn, Fragmentation, Important Delivery, and so on), each green when both nodes agreed to enable it. Below, the pool size and reconnection counter appear, and for outgoing connections the Pool DSN lists the addresses of the pooled TCP connections.
Two live charts track messages and bytes per second each way, and a proxy transit section adds a third when the connection carries traffic on behalf of other nodes. The compression and fragmentation sections show how many messages were compressed or fragmented, the ratios, the bytes saved and the reassembly timeouts, which tells you whether those features are helping or adding overhead. A "Switch observer to this node" button re-points Observer at the remote node.
What the node is saying: Log
The log page captures log messages in real time from every source on the node: processes, meta processes, the node itself, and the network stack.
Each entry shows a timestamp, a color-coded severity, the source, its registered name and behavior, and the message. The source column tells you where the message came from (a process PID, a meta-process alias, the node, or a network peer), and with the rich source toggle it becomes clickable, opening the detail window for whatever generated it. Long messages collapse and expand on click, structured fields appear as key=value pairs beneath the text, and any message can be copied on its own.
The Scope panel controls what the node captures, and the filtering happens on the server: disabling the debug level means the node stops collecting debug messages entirely, so filtering reduces load rather than just hiding rows. You can also match on source, behavior, field names and values, and message text, with an exclude mode to remove noise, and set the ring-buffer size with the limit.
The Play/Pause button stops the stream without disconnecting, so you can freeze the view and read what is there while the node keeps running. If the buffer fills and the node has to drop messages, a log storm warning appears with the suppressed count; raise the limit or narrow the scope if you see it often.
Where memory and time go: Profiler
The profiler answers the two questions the process table cannot: why is memory growing, and why is something stuck. A GC Pressure section stays live at the top with four charts, allocation rate, dead rate, live ratio, and the fraction of CPU spent in garbage collection, so you can see memory pressure building before it becomes a problem. That section streams; the two tabs below it do not.
Both tabs work on a snapshot you ask for. Press Capture and the node profiles itself once, or set an interval and let it recapture on a schedule. Neither needs a restart or a special build flag. A heap capture costs the node a garbage collection, which is stated on the button, so it is cheap enough to use during an incident and not something to leave on a one-second timer.
Heap
The heap snapshot lists allocations sorted by in-use bytes, and can be re-sorted by in-use objects, allocated bytes, or allocated objects: the difference between "what is alive" and "what has been churned" is often the whole answer. For each entry you get the in-use and total bytes and objects, and the function responsible, meaning the first non-runtime function in the allocation stack. Expand a row for the full stack trace, or switch from the table to the flamegraph to see which call paths own the memory by area rather than by row. Filter by function name and by a minimum byte threshold to drop the noise.
Reach for this when memory grows unexpectedly: the stacks point straight at the code paths doing the allocating, and if one dominates the in-use bytes, that is where to start.
Goroutines
The goroutine snapshot groups goroutines by call stack, so 500 goroutines blocked on the same channel receive show up as one group of 500, with the state (running, chan receive, select, sleep, and so on), how long they have waited (green under a minute, yellow under five, red beyond), and where each was spawned versus where it is now. Expand a group for the full stack and goroutine IDs, or switch to the flamegraph view for the same snapshot arranged by stack. Filter by stack content, by state, and by a minimum wait time.
This is how you find deadlocks and blocking. Filter to "chan receive", search for a package name to isolate a specific actor, and a large group stuck for a long time in a state that should be brief usually points right at the problem.
Following a request across the system: Tracing
The hardest question in a message-passing system is what actually happened when a request came in, because the work spreads across many processes and often many nodes. Tracing answers it. When tracing is on, Observer collects traces continuously, so data is already waiting when you open the page. For how tracing works, see Distributed Tracing.
Because Observer connects to one node at a time, it shows the observations recorded on that node. For a single trace stitched across the whole cluster, export to Pulse with Grafana Tempo or Jaeger.
Trace list
Traces are listed newest first. Each row shows a copyable trace ID, the root process and the message that started the trace, an error marker if any span failed, the span count, a bar showing this trace's duration against the longest in view, and the total duration. The search field matches across trace and span fields alike (IDs, from, to, message text, attributes), Pause holds new traces back, and Clear empties the buffer.
Waterfall
Click a trace to expand its waterfall. It groups the observation points for each message (Sent, Delivered, Processed) into one row and arranges the rows into a tree by parent and child, so the indentation is the causal chain of who triggered whom.
Each row carries a color-coded kind (SEND, CALL, RESP, SPAWN, TERM, and SPAN for a business span opened with StartTracingSpan), the sender and receiver with their behaviors, the message type, and a timeline bar split into a lighter transit segment (Sent to Delivered) and a solid processing segment (Delivered to Processed). Hovering shows the node at each point and the exact durations. For a message that crosses nodes, the transit time subtracts the measured clock skew between them, so the number reflects real travel time rather than clock drift. Local PIDs are clickable and open detail windows, and clicking a row opens a panel with every field of the span and the custom attributes merged from all of its observation points.
Scope
The Scope panel toggles which span kinds (SEND, CALL, RESP, SPAWN, TERM) and observation points (Sent, Delivered, Processed) are collected, and a message pattern filter matches message type and error text with an optional exclude. The buffer limit sets how many traces are kept. Collecting fewer kinds is not just less noise on screen: the node stops recording what you switched off.
Inspecting the whole cluster
Everything above applies to one node, but you rarely run just one. The node selector in the top bar lists every node Observer can see through the registrar, searchable by name or by CRC32; pick one and all the views above re-point to it. Nothing needs to be deployed on the target: it already runs the system application that Observer talks to, and the list updates live as nodes join and leave the cluster. The node you are on is part of the URL, which is what makes a copied window link land on the right node.
You can also reach a node that Observer is not yet connected to. If the registrar knows it, selecting it is enough; otherwise you give its name, host, port and cookie, optionally over TLS, and Observer establishes the connection. The "Switch observer to this node" button on a connection detail window does the same for a peer you are already looking at. From a single browser tab, you move freely across the entire cluster.
The map of the whole cluster, every node at once with the traffic between them, is not part of this UI. It belongs to the cloud interface at ergo.observer, which reads the same observer over the same API. What the embedded UI gives you is one node at a time, with free movement between them.
The same node, read by an agent
Every view on this page is a layout somebody decided on in advance, which is what makes it fast to read and useless for a question nobody anticipated. The same observer serves a second surface for exactly those questions: an MCP endpoint where an AI agent reads the node as addressable resources and calls tools on demand.
It is the same data, the same node, and the same authorization: a read-only ceiling refuses an agent's kill for the same reason it hides yours. What differs is who chooses the next question. See Inspecting With an AI Agent.
Last updated
Was this helpful?





















