Node ID

Nodes of a cluster create records independently, and two records created at the same moment on different nodes must never receive the same identifier. Stalwart prevents such collisions by giving every node a node identifier, a number from 0 to 511 that is unique within the cluster. Each node embeds its node identifier in the identifiers it generates for new records. Since there are 512 possible values, a cluster can have at most 512 nodes.

Identifier assignment

Node identifiers are assigned automatically, through leases in the shared data store. When a node starts, it takes a lease on an identifier and records its hostname with the lease. The hostname comes from the STALWART_HOSTNAME environment variable when it is set, and from the operating system otherwise.

A node looks for an identifier in the following order:

1. An identifier already leased under its own hostname, so that a restarted node keeps its identifier.
2. An identifier whose lease is stale, that is, not renewed for more than one hour.
3. The lowest identifier that is not leased.

A running node renews its lease every 30 minutes. When all 512 identifiers are held by active nodes, a new node cannot obtain one, so it does not start and reports Failed to obtain a node id: all 512 ids are leased by active nodes.

The lease is tied to the hostname, and this has an important consequence: every node must have a unique hostname. Two nodes with the same hostname receive the same identifier, and they can then generate colliding record identifiers. In containers, several replicas may report the same hostname, or the hostname may be an IP address, which is treated as localhost. In these cases, set STALWART_HOSTNAME to a distinct value for each node. The hostname that clients see is a separate setting, described in Load balancing.

The clocks of all nodes should be synchronised, for example with NTP, for two reasons. Record identifiers include the time. In addition, each node judges whether another node’s lease is stale by comparing the renewal time with its own clock.

Node status

The ClusterNode object (found in the WebUI under Management › Cluster) lists the leases. Each entry shows the nodeId and the hostname, the time of the last renewal in lastRenewal, and the status derived from it:

Status	Meaning
active	The lease was renewed within the last hour.
stale	The lease has not been renewed for more than one hour. The node is down or cannot reach the data store, and its identifier can be taken by another node.
inactive	The lease has not been renewed for more than 24 hours. The entry is removed by the next store maintenance run or the next node start.

The stale status needs interpreting. A node that has been stopped shows as stale an hour later, which is expected. A running node that shows as stale, on the other hand, is failing to renew its lease, typically because it cannot write to the data store. Its log then contains Failed to renew node ID lease.

The following entry describes an active node:

{
  "nodeId": 1,
  "hostname": "mx1.example.org",
  "lastRenewal": "2026-01-01T00:00:00Z",
  "status": "active"
}

ClusterNode is read-only. The nodes create and update its entries themselves, and administrators cannot edit them. The entries can be queried with stalwart-cli query ClusterNode.

Nodes that return after a long absence

A node that restarts with the same hostname reclaims its previous identifier, as long as its lease entry still exists. Once the entry has been taken by another node, or removed after 24 hours, the returning node receives a different identifier. This has no effect on stored data, since the identifier only distinguishes the records each node creates. A node whose hostname changes is treated as a new node, and it receives a new identifier.

One situation does need action. A node that loses contact with the data store for more than an hour can find, when contact returns, that another node has taken its stale identifier. Its lease renewal then fails with Node id lease is held by another host. The node keeps running with the same identifier, however, so both nodes can generate colliding record identifiers. Restart such a node, so that it obtains a new identifier.