Apache Kafka / Redpanda

Apache Kafka is a distributed event streaming platform, and Redpanda is a Kafka-compatible implementation. Either can serve as the coordinator of a cluster. This suits organisations that already operate Kafka as their messaging infrastructure.

Kafka support is not part of the official binaries. Using it requires compiling Stalwart with the kafka feature, and an official binary on which Kafka is selected logs Binary was not compiled with the selected coordinator backend. Kafka also does not replace Redis. A cluster coordinated through Kafka still requires a Redis-based in-memory store, configured before the coordinator as described in Coordination mechanism.

Configuration

Kafka coordination is selected with the Kafka variant of the Coordinator object (found in the WebUI under Settings › Cluster › Coordinator).

hosts lists the bootstrap brokers, the brokers a client first contacts to discover the rest of the Kafka cluster, as host:port pairs. At least one is required.

Kafka delivers each message once to every consumer group, so the way nodes form groups decides who receives what. groupId is the prefix of the consumer groups. Each node joins a group of its own, named <groupId>-<nodeId> after the prefix and its node identifier, so that every node receives every notification. A new group starts reading at the latest offset, so a node that joins does not replay earlier notifications.

Two timeouts govern the connection to the brokers. sessionTimeout is the consumer session timeout: when the broker receives no heartbeat from a consumer within this time, it considers the consumer failed. messageTimeout is the time allowed for a message to be delivered to the brokers.

useTls enables TLS for the broker connections, and allowInvalidCerts accepts an invalid broker certificate.

The following coordinator connects to two brokers over TLS:

{
  "@type": "Kafka",
  "hosts": {"kafka1.example.com:9093": true, "kafka2.example.com:9093": true},
  "groupId": "stalwart-cluster",
  "useTls": true
}