Collaborative digests

A single server sees only its own share of a spam run, and a message whose content is new to it gives its other checks little to work with. Collaborative spam detection networks pool what many servers see. Their participants report the fingerprints of messages they consider spam, and the network keeps count.

A mail server that takes part computes the fingerprint, or digest, of each message it receives, and asks the network how often that digest has been reported. A message that many participants have reported is part of a spam run, even if its content is new to the server. Stalwart supports Pyzor, an open network of this kind.

Pyzor

Pyzor is configured on the SpamPyzor singleton, found in the WebUI under Settings › Spam Filter › Pyzor. It is disabled by default.

Digest

The digest has to match across copies of a spam run that differ in small details, so Stalwart computes it from a normalised version of the text. It takes the plain-text and HTML body parts and removes URLs, email addresses, IP addresses, long tokens and short lines. It then hashes a few selected lines with SHA-1. A message without enough text produces no digest, and is not checked.

Privacy

The digest is sent to the Pyzor server in a single UDP request, as the anonymous Pyzor user, and the message itself is never sent. Enabling Pyzor nevertheless discloses some information to the operator of the Pyzor server: for every message received, a fingerprint of its text and the address of the querying server.

Answer and tag

The server answers with two counts: how many times the digest has been reported as spam, and how many times it has been reported as legitimate (whitelisted). Both counts matter, because a digest that many participants consider legitimate should not be treated as spam merely because some have reported it.

The message receives the PYZOR tag, which scores 3.5 in the rule set, when the server’s answer is successful and two conditions hold. The spam count must be greater than blockCount. In addition, either the whitelist count is less than allowCount, or the whitelist count divided by the spam count is less than ratio.

With the defaults, a digest reported as spam more than 5 times is tagged. It escapes the tag only when it has also been whitelisted at least 10 times and the whitelist reports amount to at least 0.2 of the spam reports.

A request that fails or times out adds no tag, so an unreachable network never counts against a message. The failure is logged as spam.pyzor-error.

Enabling Pyzor

Out of the box, the filter queries the public server public.pyzor.org on UDP port 24441. Enabling Pyzor with the public network therefore needs only enable. The following object body does that:

{
  "enable": true
}

An organisation that runs its own Pyzor server points the filter to it by setting host and port.

Pyzor adds a network round trip to every message. Each message waits for the Pyzor answer, up to timeout, before its score is final. Tuning mail throughput describes the effect of external lookups on capacity.

The participation is one-way. Stalwart queries Pyzor but does not report messages to it, so the server contributes nothing back to the network.