Fighting SPAM is not easy. Here’s an example of how we configured an ISP mail server cluster to reduce the SPAM load…
Introduction
ISPs must provide a mail relay service to their end users. However, mail relay isn’t a revenue generating service and ISPs can’t allocate unlimited staff resources to these services. The problem administrators face is trying to find a balance between unobstructively allowing their clients to send legitimate mail but to stop them from spamming.
The main source of SPAM from clients (SMTP clients in the ISPs IP space) is typically computers infected with bots / malware behind the users router. It may be their own or even a neighbor taking advantage of an unsecured wireless network. There is a smaller secondary source als0 – spammers who have someone gotten hold of a customers SMTP authentication details and are logging in from outside of the ISPs IP space to relay mail through.
Now, the problem this causes is that when the ISP mail servers pass on this SPAM to its destination such as a Yahoo! or GMail recipient, these companies examine the ratio of SPAM to HAM coming from the ISPs servers. When this ratio goes the wrong way, they legitimately block the mail servers meaning that customers of the ISP cannot get their mail delivered to these destinations. This is a big problem.
In this article, I’ll describe a new mail cluster set-up for an ISP to try and ensure their SPAM:HAM ratio stays on the right side.
Solution Overview
The mail cluster comprises a number of SMTP servers all configured identically (in fact all booting from the same image using PXE and an NFS root file system with dedicated local /var partitions). They are running Debian Stable (Lenny at the time of writing) and the mail server is the excellent Postfix.
Postfix is configured to accept authentication using SASL with Dovecot (configured only for authentication, no POP3 or IMAP services) over TLS on port 25 or SSL on port 465. Users on the ISP’s address space do not need to authenticate to relay mail. This is actually a legacy configuration and quite common among ISPs. If we were to go back many years starting all over again, I’d certainly try and evaluate the merits of only allowing relay after authentication irrespective of your ISP.
Mail server server selection is made via DNS round robin but this could easily be changed to a load balancer. As part of this retrofit, we took the opportunity to IPv6 enable all the servers.
We now need to look at securing these boxes to ensure the level of SPAM is controller. There are three points at which we examine incoming mail:
- before it hits the SMTP daemon at all;
- after the SMTP headers (
MAIL FROM, RCPT TO) have been passed to the daemon but before the mail has been accepted (i.e. before the SMTP DATA command);
- after the daemon has accepted the mail.
We’ll examine those in each of the sections below.
Rate Limiting Connections
In order to limit repeated connections to the mail server – such as from a SPAM bot that creates a new SMTP session for every mail (rather than sending many mails in one session), we use iptables recent match module.
This module keeps count of the number of connections in a given time period allow us to take alternative action if the number is above a threshold. In our case, we just drop additional connections until the number of connections drops below that threshold. For example:
-A INPUT -p tcp -m state --state NEW -m multiport
       --dports 25,465 -j SMTP_IN
-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --set
-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --update
       --seconds 60 ! --hitcount 10 -j SMTP_IN_LIMIT
-A SMTP_IN -j ACCEPT
-A SMTP_IN_LIMIT -j LOG --log-prefix "SMTP_IN: LIMIT EXCEEDED:: "
-A SMTP_IN_LIMIT -j DROP
We have some additional rules where, for example, we have whitelisted core infrastructure IP addresses and also mail from non ISP IP addresses (the purpose here is to stop SPAM bots from within the network).
Checks Before Accepting the Mail
Once the client connects to Postfix we check a number of other items.
We of course use some of Postfix’s own policy checks (such as reject_non_fqdn_sender, reject_non_fqdn_recipient, etc) but that’s pretty par for the course and not discussed below.
The ISP uses a commercial subscription to the Spamhaus data feed service. This is a very reliable database of IP addresses that are known to be sending SPAM (and other classifications). This is the first thing we check – customer or not. If you’re on this list, we do not accept mail from you.
If you’re not coming from the ISP address space, we then initiate gray listing using a Debian package called postfix-gld (see this site) which uses a MySQL database as its storage engine. See greylisting.org for more details on this useful anti-SPAM measure.
Now, if you’ve gotten this far, we use a package called postfix-policyd (see this site) to initiate rate limiting. Clients are limited in an hourly window on how many mails they can send and on how many recipients in total these mails can reach. The exact numbers are a moving goal post for now while we tweak it.
We could of course have used policyd for gray listing also but this posed a significant problem. Our Postfix configuration looks like this:
smtpd_recipient_restrictions =
   ...
   check_policy_service inet:127.0.0.1:10031,
   permit_mynetworks,
   permit_sasl_authenticated,
   check_policy_service inet:127.0.0.1:2525
   ...
As you can see, the issue is that we do not want to gray list our own IP range or people with a valid username and password but we do want to rate limit these. Using policyd for both is not possible unfortunately.
At this point, your mail will be accepted for relay. We not start the next batch of checks.
Content Filtering
We now need to examine these mails to discard SPAM and mails infected with a virus. For this we use the excellent amavisd-new package with SpamAssassin and ClamAV installed on the servers.
We’re still tweaking the rules for best effect but as it stands, if a virus is detected in the mail it’s silently discarded. A mail with a reasonable SPAM score is marked as SPAM in the subject but mails with a high score are also just discarded. DSNs are not sent.
If a mail passes all these checks, it’ll be relayed onwards.
Fighting SPAM is far from easy but the above will make a big dent. Unfortunately we can’t stop just there. We also have scripts to analyse the logs and rate limiting database to pull out the bad guys so the ISP’s tech support team can chase these up retroactively and instruct them on protecting their machines.
