S3-compatible

Object storage keeps large amounts of data as objects in buckets, accessed over HTTP, and it is well suited to the bulk of a mail server’s data: message contents that are written once and read occasionally. Amazon S3 defined the most widely used object storage API, and many services implement it, such as MinIO, Ceph, Cloudflare R2, Backblaze B2, Wasabi and DigitalOcean Spaces. Any of them can serve as the blob store. Object storage suits large volumes of message contents, and all the nodes of a cluster can share it.

Configuration

The backend is selected with the S3 variant of the BlobStore object (found in the WebUI under Settings › Storage › Blob Store). bucket names the bucket, which must already exist. keyPrefix prepends a fixed prefix to every object key, so that several servers can share one bucket without their objects colliding.

region selects the service endpoint, in one of two forms. For AWS, DigitalOcean, Wasabi and Yandex, it can name one of their predefined regions, such as {"@type": "EuCentral1"}. For any other service it is Custom, with the endpoint in url and the region name in region; this is the form used for MinIO, Ceph and other self-hosted services. Requests use path-style addressing (https://<endpoint>/<bucket>/<key>).

The credentials are given in accessKeyId and secretKey. Temporary credentials, such as those issued by AWS STS, also come with a session token. sessionToken carries it, and Stalwart sends it in the x-amz-security-token header. For permanent credentials the field is set to None. As an alternative, profile reads the credentials of a named profile from a shared credentials file. Secrets can be given as a value, read from an environment variable or read from a file.

Two further settings concern the connection. timeout limits the duration of each request, and allowInvalidCerts accepts an invalid TLS certificate on the endpoint.

The following blob store uses a MinIO server, with the secret key read from an environment variable:

{
  "@type": "S3",
  "bucket": "stalwart",
  "region": {
    "@type": "Custom",
    "url": "https://minio.example.com:9000",
    "region": "us-east-1"
  },
  "accessKeyId": {"@type": "Value", "value": "stalwart"},
  "secretKey": {"@type": "EnvironmentVariable", "variableName": "S3_SECRET_KEY"},
  "sessionToken": {"@type": "None"}
}

Failures and retries

Stalwart retries a request that fails with a status of 408, 429, 500, 502, 503 or 504, or with one of the S3 error codes InternalError, ServiceUnavailable, SlowDown and RequestTimeout. Each such request is retried up to maxRetries times, and 0 disables retries. The delay before each retry is chosen at random, up to a limit that doubles from one second to at most 20 seconds. A request that fails with a connection error is retried once, and other statuses are not retried.

When a write still fails, the operation that needed it fails too. An SMTP message being received, for example, is refused with the temporary error 451 4.3.5 Unable to accept message at this time., so the sending server tries again later. Reads depend on the service in the same way: while it is unreachable, the messages whose contents are stored in it cannot be read.

Stalwart writes each blob to one bucket only. Redundancy across sites or providers therefore has to come from the storage service itself, for example through bucket replication.

Verifying writes

Not every S3-compatible service keeps its promise when it acknowledges a write. Some answer 200 OK before the data is durably stored, and can lose it later without any error reaching the client. This has been observed on self-hosted Ceph clusters under load, on some low-cost providers, and behind proxies that drop the request body.

A message accepted from a sender cannot be requested again, so a lost write means a lost message. For this reason verifyAfterWrite is enabled out of the box. After every successful write, Stalwart reads the object’s metadata with a HEAD request to confirm that it is stored, and retries the write otherwise. Services that guarantee durability on acknowledgement, such as AWS S3, Cloudflare R2 and Backblaze B2, do not need the check, and disabling it for them saves one request per write.