Management

Once the server has recorded telemetry, an administrator needs a way to read it back: to search the log for an error, to look up how a message was delivered, or to retrieve raw metric values for a report. Three objects give this access, one for each kind of record:

Log for log entries.
Trace for stored delivery traces.
Metric for stored metrics.

These objects can be listed and read in the WebUI, with the CLI and through the JMAP API. They cannot be created, changed or deleted, because the records belong to the server: the server writes them, and it removes stored traces and metrics when their retention period ends.

Logs

A Log object (found in the WebUI under Management › Observability › Logs) is one line of a log file. Each entry has a timestamp, a level, the event name and the details of the line. The WebUI log viewer and stalwart-cli query Log search the entries for a text.

The entries are not kept in a database. They are read from the files themselves, so no TracingStore is needed to view them. The server reads every file in the directory of the first enabled Log tracer, on the node that serves the request. That directory should therefore contain only Stalwart’s log files.

Two further points concern the files. Without an enabled Log tracer there are no files to read, and the request fails with No log tracers configured on the server. Entries written in the multiline format show only their first line, so the Log tracer used for viewing logs should write single-line entries.

In a cluster, each node reads its own files. The WebUI therefore shows the log of the node that served the WebUI request, not a combined log of the cluster.

Traces

A Trace object (found in the WebUI under Management › Emails › History › Inbound Delivery, Management › Emails › History › Outbound Delivery) is a stored delivery trace. It holds the events of one SMTP session that received a message, or of one outbound delivery attempt, together with the timestamp, the sender (from), the recipients (to) and the message size.

Traces are recorded unless the TracingStore is Disabled, as described in History. They can be searched by queue identifier, event type and keywords. The WebUI shows them as the inbound and outbound delivery history.

Metrics

A Metric object is one stored data point of a metric. A node records one at each collection interval, unless the MetricsStore is Disabled. Its @type is Counter, Gauge or Histogram, and it carries the metric name, the timestamp and the nodeId of the cluster node that recorded it (0 on a single node).

The WebUI dashboard draws its charts from these records, and shows the node in its list view. Raw data points can also be queried directly, with x:Metric/query or with the CLI, which can filter by node: stalwart-cli query Metric --where nodeId=1.

Values are stored as changes since the previous data point of the same node, not as running totals. What count means depends on the type:

For a Counter, count is the number of events in the interval.
For a Histogram, count is the number of observations in the interval and sum their total, so sum divided by count is the average for the interval.
For a Gauge, count is the value at the time of recording.

Because each data point covers only its own interval, a total over a period is obtained by summing the data points of each node in that period. The first data point after a node restarts covers the time since the process started.

The metrics store holds a fixed selection of metrics, including the cache, memory, search indexing and task metrics. The cache.swap-pending gauge is not among them.