Inspecting With Observer
Last updated
Last updated
To install the observer tool, you need to have Golang compiler version 1.20 or higher. Run the following command:
Available arguments for starting observer:
-help: displays information about the available arguments.
-version: prints the current version of the Observer tool.
-host: specifies the interface name for the Web server to run on (default: "localhost").
-port: defines the port number for the Web server (default: 9911).
-cookie: sets the default cookie value used for connecting to other nodes.
If you are running observer on a server for continuous operation, it is recommended to use the environment variable COOKIE instead of the -cookie argument. Using sensitive data in command-line arguments is insecure.
After starting observer, it initially has no connections to other nodes, so you will be prompted to specify the node you want to connect to.
Once you establish a connection with a remote node, the Observer application main page will open, displaying information about that node.
If you have integrated the Observer application into your node, upon opening the Observer page, you will immediately land on the main page showing information about the node where the Observer application was launched.
On this tab, you will find general information about the node and the ability to manage its logging level. Changing the logging level only affects the node itself and any newly started processes, but it does not impact processes that are already running.
Graphs provide real-time information over the last 60 seconds, including the total number of processes, the number of processes in the running state, and memory usage data. Memory usage is divided into used, which indicates how much memory was reserved from the operating system, and allocated, which shows how much of that reserved memory is currently being used by the Golang runtime.
In addition to these details, you can view information about the available loggers on the node and their respective logging levels. For more details, refer to the Logging section. Environment variables will also be displayed here, but only if the ExposeEnvInfo option was enabled in the gen.NodeOptions.Security settings when the inspected node was started.
The Network tab displays information about the node's network stack.
The Mode indicates how the network stack was started (enabled, hidden, or disabled).
The Registrar section shows the properties of the registrar in use, including its capabilities. Embedded Server indicates whether the registrar is running in server mode, while the Server field shows the address and port number of the registrar with which the node is registered.
Additionally, the tab provides information about the default handshake and protocol versions used for outgoing connections.
The Flags section lists the set of flags that define the functionality available to remote nodes.
The Acceptors section lists the node's acceptors, with detailed information available for each. This list will be empty if the network stack is running in hidden mode.
Since the node can work with multiple network stacks simultaneously, some acceptors may have different registrar parameters and handshake/protocol versions. For an example of simultaneous usage of the Erlang and Ergo Framework network stacks, refer to the Erlang section.
The Connected Nodes section displays a list of active connections with remote nodes. For each connection, you can view detailed information, including the version of the handshake used when the connection was established and the protocol currently in use. The Flags section shows which features are available to the node when interacting with the remote node.
Since the ENP protocol supports a pool of TCP connections within a single network connection, you will find information about the Pool Size (the number of TCP connections). The Pool DSN field will be empty if this is an incoming connection for the node or if the protocol does not support TCP connection pooling.
Graphs provide a summary of the number of received/sent messages and network traffic over the last 60 seconds, offering a quick overview of communication activity and data flow.
On the Processes List tab, you can view general information about the processes running on the node. The number of processes displayed is controlled by the Start from and Limit parameters.
By default, the list is sorted by the process identifier. However, you can choose different sorting options:
Top Running: displays processes that have spent the most time in the running state.
Top Messaging: sorts processes by the number of sent/received messages in descending order.
Top Mailbox: helps identify processes with the highest number of messages in their mailbox, which can be an indication that the process is struggling to handle the load efficiently.
For each process, you can view brief information:
The Behavior field shows the type of object that the process represents.
Application field indicates the application to which the process belongs. This property is inherited from the parent, so all processes started within an application and their child processes will share the same value.
Mailbox Messages displays the total number of messages across all queues in the process's mailbox.
Running Time shows the total time the process has spent in the running state, which occurs when the process is actively handling messages from its queue.
By clicking on the process identifier, you will be directed to a page with more detailed information about that specific process.
All log messages from the node, processes, network stack, or meta-processes are displayed here. When you connect to the Observer via a browser, the Observer's backend sends a request to the inspector to start a log process with specified logging levels (this log process is visible on the main Info tab).
When you change the set of logging levels, the Observer's backend requests the start of a new log process (the old log process will automatically terminate).
To reduce the load on the browser, the number of displayed log messages is limited, but you can adjust this by setting the desired number in the Last field.
The Play/Pause button allows you to stop or resume the log process, which is useful if you want to halt the flow of log messages and focus on examining the already received logs in more detail.
This page displays detailed information about the process, including its state, uptime, and other key metrics.
The fallback parameters specify which process will receive redirected messages in case the current process's mailbox becomes full. However, if the Mailbox Size is unlimited, these fallback parameters are ignored.
The Message Priority field shows the priority level used for messages sent by this process.
Keep Network Order is a parameter applied only to messages sent over the network. It ensures that all messages sent by this process to a remote process are delivered in the same order as they were sent. This parameter is enabled by default, but it can be disabled in certain cases to improve performance.
The Important Delivery setting indicates whether the important flag is enabled for messages sent to remote nodes. Enabling this option forces the remote node to send an acknowledgment confirming that the message was successfully delivered to the recipient's mailbox.
The Compression parameters allow you to enable message compression for network transmissions and define the compression settings.
Graphs on this page help you assess the load on the process, displaying data over the last 60 seconds.
Additionally, you can find detailed information about any aliases, links, and monitors created by this process, as well as any registered events and started meta-processes.
The list of environment variables is displayed only if the ExposeEnvInfo option was enabled in the node's gen.NodeOptions.Security settings.
Additionally, on this page, you can send a message to the process, send an exit signal, or even forcibly stop the process using the kill command. These options are available in the context menu.
If the behavior of this process implements the HandleInspect method, the response from the process to the inspect request will be displayed here. The Observer sends these requests once per second while you are on this tab.
In the example screenshot above, you can see the inspection of a process based on act.Pool. Upon receiving the inspect request, it returns information about the pool of processes and metrics such as the number of messages processed.
The Log tab on the process information page displays a list of log messages generated by the specific process.
Please note that since the Observer uses a single stream for logging, any changes to the logging levels will also affect the content of the Log tab on the main page.
On this page, you'll find detailed information about the meta-process, along with graphs showing data for the last 60 seconds related to incoming/outgoing messages and the number of messages in its mailbox. The meta-process has only two message queues: main and system.
You can also send a message to the meta-process or issue an exit signal. However, it is not possible to forcibly stop the meta-process using the kill command.
If the meta-process's behavior implements the HandleInspect method, the response from the meta-process to the inspect request will be displayed on this tab. The Observer sends this request once per second while you are on the tab.
On the Log tab of the meta-process, you will see log messages generated by that specific meta-process. Changing the logging levels will also affect the content of the Log tab on the main page.
