History

Logs and metric exporters show what the server is doing now, or send it elsewhere. Telemetry history instead keeps records of past activity in the server’s own database, so that an administrator can examine them after the fact, for example to find out what happened to a message that a user reported missing. Two kinds of record are kept:

• Delivery traces: the events of each SMTP session that received a message and of each outbound delivery attempt, kept together as one trace per session. They drive the WebUI’s inbound and outbound delivery history, in which an administrator can follow what happened to a message.
• Metrics: the values of the server’s metrics, recorded by each node at a regular interval. They drive the WebUI dashboard.

The stored records are read through the objects described in Telemetry management.

Enterprise feature

This feature is available in the Enterprise edition of Stalwart.

Delivery traces

Tracing store

Delivery traces are kept in the store selected by the TracingStore object (found in the WebUI under Settings › Storage › Tracing Store). The choice is about where the write load and the space of the traces should fall:

• Default uses the data store, so no further database is needed.
• FoundationDb, PostgreSql or MySql selects a separate database, with the same fields as the corresponding storage backends.
• Disabled turns recording off.

A new installation uses Default. With an Enterprise license, traces are therefore recorded in the data store from the start.

Contents of a trace

A trace is recorded for each SMTP session and delivery attempt that involves a queued message. It contains the session’s protocol, authentication, filtering and delivery events, up to 2048 events per trace, and leaves out protocol data and other low-level events. Other sessions are not recorded: neither IMAP, POP3, JMAP and HTTP sessions, nor SMTP sessions that end without a message being queued.

Searching traces

So that the history can be searched by queue identifier, event type and keywords such as addresses, each trace is also added to the search index. The indexed fields are fixed and cannot be configured.

This indexing is controlled by indexTelemetry on the Search object. With indexTelemetry disabled, traces are still stored but not indexed, so a search of the history finds nothing. A change to the setting takes effect after the ReloadSettings action and needs no restart.

Retention

Traces are kept for the period set by holdTracesFor on the DataRetention object (found in the WebUI under Settings › Storage › Data Retention › Archiving, Settings › Storage › Data Retention › Data Cleanup, Settings › Storage › Data Retention › Auto-Expunge, Settings › Storage › Data Retention › Telemetry), 30 days by default. The daily data purge then deletes them, as described in Data retention.

Metrics

Metrics are kept in the store selected by the MetricsStore object (found in the WebUI under Settings › Storage › Metrics Store). It offers the same variants as TracingStore and has the same default, Default.

Each node records a fixed set of metrics on the schedule set by metricsCollectionInterval, hourly, on the hour by default. The records are kept for holdMetricsFor, 90 days by default.

Storage cost

The histories are written continuously and grow with the server’s activity, so they need planning on a busy server.

Delivery traces take most of the space. Each message received or delivered adds a trace with its events, and an entry in the search index. The space they use is therefore proportional to the mail volume multiplied by holdTracesFor. Metrics, by contrast, add one record per metric, node and interval, which is small and nearly constant.

Two settings keep the cost under control. A separate TracingStore keeps the write load and growth of the traces away from the data store, and a shorter holdTracesFor reduces the space they use. The space taken by deleted traces is returned according to the rules of the backend, described in Disk space by backend.

The following TracingStore keeps delivery traces in a separate PostgreSQL database:

{
  "@type": "PostgreSql",
  "host": "telemetry-db.example.com",
  "database": "stalwart_traces",
  "authUsername": "stalwart",
  "authSecret": {"@type": "EnvironmentVariable", "variableName": "TRACES_DB_PASSWORD"}
}

The following change to DataRetention shortens the retention of delivery traces to 14 days:

{
  "holdTracesFor": 1209600000
}