OpenTelemetry

Many organisations collect the logs and traces of all their systems in one observability backend. OpenTelemetry is the vendor-neutral standard that makes this possible: applications send their telemetry to a collector, and the collector forwards logs, traces and metrics to the backends in use. Exporting Stalwart’s events to an OpenTelemetry collector places them alongside the telemetry of the rest of the infrastructure.

The export maps Stalwart’s events onto the two OpenTelemetry signals that fit them. Each event becomes a log record. Each session, such as an SMTP connection, becomes a span, the OpenTelemetry unit for an operation with a start and an end. Exporting metrics is described separately in OpenTelemetry metrics.

Configuration

The exporter is a variant of the Tracer object (found in the WebUI under Settings › Telemetry › Tracers). OTLP, the OpenTelemetry protocol, runs over two transports, and each has its own variant: OtelHttp for OTLP over HTTP and OtelGrpc for OTLP over gRPC. Both have the common tracer settings described in Tracing, and the following fields:

• url is the address of the collector.
• enableLogExporter and enableSpanExporter select whether log records and spans are exported. Both are enabled by default.
• throttle is the minimum interval between two requests to the collector. Events produced in the meantime are sent together.
• timeout limits each request.

HTTP transport

With OtelHttp, url is required, and the tracer uses it as given, with no path appended. This matters because OTLP collectors receive log records and spans at different paths, /v1/logs and /v1/traces, while one tracer has only one URL. To export both signals, define two tracers, each with its own URL: one with only enableLogExporter, and one with only enableSpanExporter.

httpAuth sets the authentication: Unauthenticated, Basic with username and secret, or Bearer with bearerToken. httpHeaders adds other headers, such as a tenant identifier that the backend requires. allowInvalidCerts accepts an invalid TLS certificate.

The following pair of tracers exports spans and log records to a collector that requires a bearer token. The first sends spans to /v1/traces:

{
  "@type": "OtelHttp",
  "url": "https://otel.example.com:4318/v1/traces",
  "enableLogExporter": false,
  "httpAuth": {
    "@type": "Bearer",
    "bearerToken": {"@type": "EnvironmentVariable", "variableName": "OTEL_TOKEN"}
  },
  "level": "info"
}

The second sends log records to /v1/logs:

{
  "@type": "OtelHttp",
  "url": "https://otel.example.com:4318/v1/logs",
  "enableSpanExporter": false,
  "httpAuth": {
    "@type": "Bearer",
    "bearerToken": {"@type": "EnvironmentVariable", "variableName": "OTEL_TOKEN"}
  },
  "level": "info"
}

gRPC transport

With OtelGrpc, url is the address of the collector’s gRPC endpoint, without a path, such as http://otel.example.com:4317. When url is empty, the tracer falls back to the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variables, and to http://localhost:4317 if none is set. httpAuth and httpHeaders apply as with the HTTP transport, and are sent as gRPC request metadata.

The following tracer exports to a collector on the same host:

{
  "@type": "OtelGrpc",
  "url": "http://127.0.0.1:4317",
  "httpAuth": {"@type": "Unauthenticated"},
  "level": "info"
}

Export failures

Export failures are reported as telemetry.otel-exporter-error events. These events are never sent to an OpenTelemetry tracer, so only a local tracer, such as a Log or console tracer, shows them. A server that exports only to OpenTelemetry should keep a local tracer for this reason.