IPv6 – Are You Ready?
An excellent ‘viral’ video from Cisco.
Thoughts, ramblings and rants…
An excellent ‘viral’ video from Cisco.
UPDATED VERSION AVAILABLE:Â https://www.barryodonovan.com/index.php/2013/02/12/irish-radio-stations-on-linux-2013
I’m a bit of a newstalk junky and like to have the radio on in the back ground. It’s quite painful jumping between websites and even more painful getting them all to work under Linux so I have some simple Bash aliases for VLC and RTE Radio 1, Today FM and Newstalk:
alias 2fm='cvlc http://dynamic.rte.ie/av/live/radio/2fm.asx' alias newstalk='cvlc http://newstalk.fmstreams.com:8008/listen.pls' alias rteradio1='cvlc http://dynamic.rte.ie/av/live/radio/radio1.asx' alias todayfm='cvlc http://audiostore.todayfm.com/audio/todayfmIRL_64K.asx'
UPDATED 2011-02-07: 2FM added.
I complained about bad experiences with FAS on this blog lately (it actually got worse than that lately) so I thought I’d mention two good experiences.
I was doing a bit of research for a potential project at work and I needed some statistics on businesses in Ireland. I went to the CSO’s online information request and got a response a couple of hours later. They didn’t have the information I needed but suggested a contact in the Revenue Commissioners.
I mailed Revenue then and got a very helpful call back this morning (less the three working hours later) with the information I asked for as well as some additional information to help me better understand the information behind the figures.
Creating a database schema directly in the database or in YAML can be very frustrating and difficult. I’ve looked at tools like MySQL Workbench in the paste and have not been impressed – especially as I need an export to YAML function for Doctrine (a PHP ORM project which I’m a big fan of).
I recently came across ORM Designer and have quickly become a big fan. It’s interface is nice, it has excellent support for Doctrine including behaviors and I’ve ran through many update / export to YAML / create models from YAML cycles without issue.
I’d strongly recommend for anyone using Doctrine – plus there’s a 30 day evaluation without any limiting of functionality so you can examine it for yourselves.
Fighting SPAM is not easy. Here’s an example of how we configured an ISP mail server cluster to reduce the SPAM load…
ISPs must provide a mail relay service to their end users. However, mail relay isn’t a revenue generating service and ISPs can’t allocate unlimited staff resources to these services. The problem administrators face is trying to find a balance between unobstructively allowing their clients to send legitimate mail but to stop them from spamming.
The main source of SPAM from clients (SMTP clients in the ISPs IP space) is typically computers infected with bots / malware behind the users router. It may be their own or even a neighbor taking advantage of an unsecured wireless network. There is a smaller secondary source als0 – spammers who have someone gotten hold of a customers SMTP authentication details and are logging in from outside of the ISPs IP space to relay mail through.
Now, the problem this causes is that when the ISP mail servers pass on this SPAM to its destination such as a Yahoo! or GMail recipient, these companies examine the ratio of SPAM to HAM coming from the ISPs servers. When this ratio goes the wrong way, they legitimately block the mail servers meaning that customers of the ISP cannot get their mail delivered to these destinations. This is a big problem.
In this article, I’ll describe a new mail cluster set-up for an ISP to try and ensure their SPAM:HAM ratio stays on the right side.
The mail cluster comprises a number of SMTP servers all configured identically (in fact all booting from the same image using PXE and an NFS root file system with dedicated local /var
partitions). They are running Debian Stable (Lenny at the time of writing) and the mail server is the excellent Postfix.
Postfix is configured to accept authentication using SASL with Dovecot (configured only for authentication, no POP3 or IMAP services) over TLS on port 25 or SSL on port 465. Users on the ISP’s address space do not need to authenticate to relay mail. This is actually a legacy configuration and quite common among ISPs. If we were to go back many years starting all over again, I’d certainly try and evaluate the merits of only allowing relay after authentication irrespective of your ISP.
Mail server server selection is made via DNS round robin but this could easily be changed to a load balancer. As part of this retrofit, we took the opportunity to IPv6 enable all the servers.
We now need to look at securing these boxes to ensure the level of SPAM is controller. There are three points at which we examine incoming mail:
MAIL FROM
, RCPT TO
) have been passed to the daemon but before the mail has been accepted (i.e. before the SMTP DATA
command);We’ll examine those in each of the sections below.
In order to limit repeated connections to the mail server – such as from a SPAM bot that creates a new SMTP session for every mail (rather than sending many mails in one session), we use iptables recent match module.
This module keeps count of the number of connections in a given time period allow us to take alternative action if the number is above a threshold. In our case, we just drop additional connections until the number of connections drops below that threshold. For example:
-A INPUT -p tcp -m state --state NEW -m multiport        --dports 25,465 -j SMTP_IN -A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --set -A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --update        --seconds 60 ! --hitcount 10 -j SMTP_IN_LIMIT -A SMTP_IN -j ACCEPT -A SMTP_IN_LIMIT -j LOG --log-prefix "SMTP_IN: LIMIT EXCEEDED:: " -A SMTP_IN_LIMIT -j DROP
We have some additional rules where, for example, we have whitelisted core infrastructure IP addresses and also mail from non ISP IP addresses (the purpose here is to stop SPAM bots from within the network).
Once the client connects to Postfix we check a number of other items.
We of course use some of Postfix’s own policy checks (such as reject_non_fqdn_sender
, reject_non_fqdn_recipient
, etc) but that’s pretty par for the course and not discussed below.
The ISP uses a commercial subscription to the Spamhaus data feed service. This is a very reliable database of IP addresses that are known to be sending SPAM (and other classifications). This is the first thing we check – customer or not. If you’re on this list, we do not accept mail from you.
If you’re not coming from the ISP address space, we then initiate gray listing using a Debian package called postfix-gld
(see this site) which uses a MySQL database as its storage engine. See greylisting.org for more details on this useful anti-SPAM measure.
Now, if you’ve gotten this far, we use a package called postfix-policyd
(see this site) to initiate rate limiting. Clients are limited in an hourly window on how many mails they can send and on how many recipients in total these mails can reach. The exact numbers are a moving goal post for now while we tweak it.
We could of course have used policyd for gray listing also but this posed a significant problem. Our Postfix configuration looks like this:
smtpd_recipient_restrictions = Â Â Â ... Â Â Â check_policy_service inet:127.0.0.1:10031, Â Â Â permit_mynetworks, Â Â Â permit_sasl_authenticated, Â Â Â check_policy_service inet:127.0.0.1:2525 Â Â Â ...
As you can see, the issue is that we do not want to gray list our own IP range or people with a valid username and password but we do want to rate limit these. Using policyd for both is not possible unfortunately.
At this point, your mail will be accepted for relay. We not start the next batch of checks.
We now need to examine these mails to discard SPAM and mails infected with a virus. For this we use the excellent amavisd-new package with SpamAssassin and ClamAV installed on the servers.
We’re still tweaking the rules for best effect but as it stands, if a virus is detected in the mail it’s silently discarded. A mail with a reasonable SPAM score is marked as SPAM in the subject but mails with a high score are also just discarded. DSNs are not sent.
If a mail passes all these checks, it’ll be relayed onwards.
Fighting SPAM is far from easy but the above will make a big dent. Unfortunately we can’t stop just there. We also have scripts to analyse the logs and rate limiting database to pull out the bad guys so the ISP’s tech support team can chase these up retroactively and instruct them on protecting their machines.
I’ve recently performed a complete upgrade of Open Solutions’ mail servers and I’ve now moved onto doing likewise for one of our ISP customers with a lot of users.
These retrofits include installing virus and SPAM filters to protect both ourselves and the ISP customers but also to stop customers who have infected computers from spewing these emails out.
When everything’s up and appears to be working, I like to test both filtering systems to ensure they’re working. Quoting from eicar:
Using real viruses for testing in the real world is rather like setting fire to the dustbin in your office to see whether the smoke detector is working. Such a test will give meaningful results, but with unappealing, unacceptable risks.
Fortunately, test files exist for virus checkers and SpamAssassin:
You probably won’t have noticed but this site is now available over IPv6:
$ host www.barryodonovan.com www.barryodonovan.com has address 87.232.16.35 www.barryodonovan.com has IPv6 address 2a01:268:3002::35
I spend a lot of my working hours doing a lot with IPv6 and, as any sys admin knows, it’s quite often the case that you get around to doing these things for yourself last. In our case, there was a bit of work involved as we had to first get our ISP’s core network dual stacked with IPv6 – luckily they’re a customer of ours 😉
Stay tuned here and over on the company blog for upcoming IPv6 posts and announcements.
In the meantime, if your ISP isn’t offering IPv6 to end users yet, head on over to SixXS where you can get an IPv6 tunnel for free. If you’re based in Ireland be sure to chose HEAnet or Airwire as your PoP as they’re both based in Ireland and members of INEX so your latency will be as low as possible.
UPDATE: Much more on this an why over on the company blog: We’re IPv6 Ready – Finally!
At work, we’ve always used and advocated MySQL for high availability mission critical applications thanks to it’s built in binary replication features.
This was a feature PostgreSQL sorely lacked but it is some thing that has also been rectified in the 9.0 release which, among many other features, includes:
We’ll be studying and testing this new release carefully as, despite the previous lack of the above features, PostgreSQL had a lot of other features over MySQL and was also ahead of MySQL with such taken for granted features as views, triggers and stored procedures.
See my article on the company blog for a discussion on this, and a how to on using Fail2ban to help stop these attacks.