Category: Recipes

“Go Faster” Websites – Introducing Minify

We’ve been minifying and bundling CSS and JS for years to ensure quick page loads of the applications we build. We’ve now generalised, documented and packaged the tool we use for this and released it under a BSD license so others can benefit.

Most web developers know that including lots of JS and CSS files in their sites slow page load times down. Most also know that these files should be minified and bundled into one file on production sites. Most developers don’t do this though. It’s a lot of extra steps in putting your new changes live.

Also, using CDNs or setting expiry times into the future for mostly static files such as CSS and JS also significantly improves page load as clients will grab these files once and use their local cache until their expire. This also poses issues for web developers that is easily overcome by versioning these files – literally adding a version number to the bundles – for example min.bundle-v6.css would be version 6 of the CSS minified and bundled file.

We’ve been doing both of these for a long time with the sites we build. We’ve now generalised, documented and packaged the tool we use for this and released it under a BSD license so others can benefit. See our page on GitHub to download this tool and for examples of its use:

https://github.com/opensolutions/Minify

This tool will:

  • automatically find all CSS/JS files in a given directory named xxx-blah.css where xxx is a three digit ordering / sequence number;
  • minify these files and create a single file bundle including them in the correct order;
  • automatically generate template include files allowing production / development mode (i.e. use individual CSS/JS or bundles based on an application option);
  • versioning for those using CDNs, future expiry dates, etc to ensure clients load fresh JS/CSS bundles.

If you use it, please drop us a note to let us know how you get on! 

Asterisk with SS7 via Wanpipe, Dahdi and libss7 – with ISUP SAM support

A howto on installing Asterisk with SS7 supported via libss7 on Sangoma hardware along with support for ISUP SAM messages.

After much head banging in bringing up an SS7 link with SAM support, I am documented what worked here.

Firstly, what is SAM support? One end of an SS7 link initialises a new call by sending an Initial Address Message (IAM). All SS7 software stacks support this and usually it’s enough. One case where it’s not enough is when one wants to address a phone number with more than the E.164 standard max length of 16  (usually to pass additional information tacked on the start, end of or even replacing an A or B number). In this scenario, SS7 uses a Subsequent Address Message (SAM) to send the additional digits. Most / all mainstream Asterisk SS7 software stacks do not support this.

The platform and software used is as follows:

  • Ubuntu 10.04 LTS standard CLI install;
  • dahdi-linux-complete-2.4.0 from the archives (direct link);
  • a patched version of libss7 supporting SAM via SVN (see below);
  • a patched version of chan-dahdi via SVN (see below);
  • Asterisk 1.6.0.28 from the archives (direct link);
  • Asterisk Addons 1.6.0.6 from the archives (direct link);
  • Sangoma Wanpipe 3.5.24 (direct link).

Yes, I know some of the above are obsolete. Unfortunately it’s what’s required for everything to play nicely together. And, guess what, it works!

Before You Start

Ensure you have all the necessary packages for building the packages:

apt-get install build-essential libnewt-dev subversion    \
   libsqlite3-dev libsnmp-dev bison libtool flex          \
   autoconf2.13 libmysqlclient-dev

You can assume the working directory is /usr/local/src in the following.

Installing dahdi-linux-complete

Download and extract dahdi-linux-complete into /usr/local/src. Then, in the same directory, create some symbolic links:

ln -s dahdi-linux-complete-2.4.0+2.4.0/linux dahdi-linux
ln -s dahdi-linux-complete-2.4.0+2.4.0/tools dahdi-tools

Now build dahdi-linux:

cd dahdi-linux
make
make install

Now move onto dahdi-tools:

cd dahdi-tools
./configure
make menuconfig # (and select tools or accept defaults)
make
make install
make config

Installing libss7

There is a patched version of Digium’s libss7 referred to as Attila’s libss7 available via SVN. Download and install it:

svn co https://observer.router.hu/repos_pub/libss7/trunk alibss7
cd alibss7
make
make install

Installing Asterisk

Download and extract the Asterisk 1.6.0.28 package reference above.

cd asterisk-1.6.0.28

Now this next bit is very important and easy to miss – you also need a patched version of chan_dahdi.c that is compatible with the patched version of libss7. This is also available from SVN:

svn co https://observer.router.hu/repos_pub/chan_dahdi/trunk achan_dahdi

Then copy it to your Asterisk source:

cp achan_dahdi/chan_dahdi.c asterisk-1.6.0.28/channels/chan_dahdi.c

Now continue with the Asterisk installation:

./configure
make menuconfig
make
make install
make samples
make config

Install Asterisk Addons (optional)

Download and unpack the above referenced add ons package and build and install:

cd asterisk-addons-1.6.0.6
./configure
make menuconfig
make
make install
make samples

Sangoma Wanpipe

I’m using one of Sangoma’s E1 / T1 interface cards and so I need Wanpipe also. I’m using version 3.5.24 and preceed as follows after unpacking:

cd wanpipe-3.5.24./Setup install

During the install, follow these prompts:

  • select option 2 => Asterisk/Dahdi Support;
  • enter path /usr/local/src/dahdi-linux (for Zaptel path prompt);
  • select defaults for everything else;
  • you DO want to install start-up scripts;
  • you DO to configure wanpipe devices for DAHDI;
  • you DO want to generate /etc/asterisk/chan_dahdi.conf and:
    • select E1 / T1 as appropriate;
    • select line framing and encoding;
    • choose clock source;
    • select Zaptel/Dahdi – PRI CPE as signalling;
    • select National ISDN 2 as switch type;
    • do not enable hardware DTMF detection;
    • use all channels;
    • select dial plan context as appropriate;
  • and continue for other ports as necessary;
  • finally, choose Save cfg: Stop Asterisk & Wanpipe now
  • you would like wanrouter to start on system boot;
  • and you would like to execute ‘dahdi_cfg’ each time wanrouter starts.

Configuration Steps

We now need to set various options in Wanpipe, Dahdi and Asterisk for SS7 as it’s PRI/ISDN by default.

Edit all /etc/wanpipe/wanpipeX.conf files as necessary and change:

 TDMV_DCHAN              = 16

to

TDMV_DCHAN              = 0

Now edit /etc/dahdi/system.conf and change (for example):

span=1,1,0,ccs,hdb3,crc4
bchan=1-15,17-31
echocanceller=mg2,1-15,17-31
hardhdlc=16

to

span=1,1,0,ccs,hdb3,crc4
bchan=2-31
echocanceller=mg2,2-31
mtp2=1

which of course assumes signalling is on channel 1. If you have voice only links, you might need something like:

span=1,0,0,ccs,hdb3,crc4bchan=1-31
echocanceller=mg2,2-31

Lastly, we need to configure Asterisk. Replace lines such as:

;Sangoma A102 port 1 [slot:4 bus:5 span:1]
switchtype=national
context=from-pstn
group=0
echocancel=yes
signalling=pri_cpe
channel =>1-15,17-31

with an appropriate configuration. Mine follows below with some edits and some important notes at the end:

;Sangoma A102 port 1 [slot:4 bus:5 span:1]
switchtype=national
context=from-pstn
group=0
echocancel=yes

signaling=ss7
ss7type=itu
ss7_called_nai=dynamic
ss7_calling_nai=dynamic
ss7_internationalprefix=00
ss7_nationalprefix=0
ss7_subscriberprefix=
ss7_unknownprefix=
linkset=1
pointcode=1
adjpointcode=2
defaultdpc=3
networkindicator=national_spare
cicbeginswith=2
channel=2-31
sigchan=1
  • Set pointcode, adjpointcode and defaultdpc as appropriate;
  • set networkindicator as appropriate and ensure it matches the other end (you can see what you’re being sent and what you’re sending via ss7 debug;
  • cicsbeginwith is normally 1 but the telco on my end are starting at 2 – this was groping in the dark diagnostics and issues such as no audio, CICs not in service when both sides claim they are, etc may point to misaligned CICs;
  • make sure you have configured from-pstn or the appropriate context in yourextensions.conf.

Confirming Your Link Is Up

Now start wanrouter (/etc/init.d/wanrouter start); dahdi (/etc/init.d/dahdi start); and Asterisk (/etc/init.d/asterisk start). You should see your link come up via logs available with the dmesg command. Launch the Asterisk console and check the status of your links:

ast-deg1-1*CLI> ss7 show cics 1
  CIC   DPC  DAHDI        STATE   BLOCKING
    2    3      2         Idle
    3    3      3         Idle
    4    3      4         Idle
    5    3      5         Idle
    6    3      6         Idle
    7    3      7         Idle
    8    3      8         Idle
    9    3      9         Idle
   10    3     10         Idle
   11    3     11         Idle
   12    3     12         Idle
   13    3     13         Idle
   14    3     14         Idle
   15    3     15         Idle
   16    3     16         Idle
   17    3     17         Idle
   18    3     18         Idle
   19    3     19         Idle
   20    3     20         Idle
   21    3     21         Idle
   22    3     22         Idle
   23    3     23         Idle
   24    3     24         Idle
   25    3     25         Idle
   26    3     26         Idle
   27    3     27         Idle
   28    3     28         Idle
   29    3     29         Idle
   30    3     30         Idle
   31    3     31         Idle

You should now be okay to make test calls.

Do You Need Professional Support / Consultancy?

While I will try to respond to comments and questions on this blog, I don’t have the time to provide one on one assistance pro-bono. Professional consultancy on Asterisk and SS7 is available worldwide through my company, Open Solutions with contact details here.

References

For posterity, I have added Domjan Attila patched libss7 and chan_dahdi to GitHub:

We’ve Released Some of our Nagios Plugins

We create a lot of Nagios installations for our own systems over, for customer systems which we manage and as a service over at Open Solutions. We’ve written a lot of custom Nagios plugins over the years as part of this process.

We are now making a concerted effort to find them, clean them, maintain them centrally and release them for the good of others.

To that end, we have created a repository on GitHub for the task with a detailed readme file:

They main goal of Nagios plugins that we write and release are:

  • BSD (or BSD like) license so you can hack away to wield into something that may be more suitable for your own environment;
  • scalable in that if we are polling power supply units (PSUs) in a Cisco switch then it should not matter if there is one or a hundred – the script should handle them all;
  • WARNINGs are designed for email notifications during working hours; CRITICAL means an out of hours text / SMS message;
  • each script should be an independant unit with no dependancies on each other or unusual Perl module requirements;
  • the scripts should all be run with the --verbose on new kit. This will provide an inventory of what it finds as well as show anything that is being skipped. OIDs searched for by the script but reported as not supported on the target device should really be skipped via various --skip-xxx options.
  • useful help available via --help or -?

Using Doctrine ORM with Zend Application

In this first of a serious of articles where we delve into some of the hidden treasures in our ViMbAdmin application, we look at how to integrate Doctrine ORM with Zend – and specifically Zend_Application and Zend_Controller.

In this article we delve into our ViMbAdmin application and we look at how to integrate Doctrine ORM with Zend – and specifically Zend_Application and Zend_Controller.

The first assumption (and requirement) we are going to make is that you are using Zend_Application. If you want to see a working application set up and configured for this, please checkout or browse our ViMbAdmin source code – which we’ll reference throughout this document.

Zend Application has a resource framework which allows us to bootstrap various resources on demand. We have created a Doctrine resource for this very purpose which you can download from here (and you may need to edit the class name and change the plugin path in the config code below to match your setup). Our implementation does many things:

  • instantiates the Doctrine object;
  • sets up an autoloader for Doctrine models;
  • instantiates the Doctrine manager;
  • opens the connection to the database;
  • sets all collations and character sets to UTF8 (this is hard coded but can easily be changed);
  • sets various hard coded Doctrine attributes which can also be changed.

We the add various configuration parameters to the application.ini file:

 

Or the following where $application is the instance of Zend_Application:

$application->getBootstrap()->bootstrap( 'doctrine' );

From that, you can use Doctrine to your hearts content!

We also have a Doctrine CLI script which works from the same resource. See:

http://code.google.com/p/vimbadmin/source/browse/trunk/bin/doctrine-cli.php

 

Doctrine ORM – Find Many to Many Objects Without a Relationship

Hmmm, does the title of this post make sense? Probably not but it’s not an easy concept to squeeze into a few words.

Here’s the scenario, I have two tables A and B in Doctrine ORM with a many-to-many relationship defined in table AB.

Now, I want to find all objects in A that do not have a relationship with an object in B via AB.

Here’s what I have:

Doctrine_Query::create()
    ->from( 'A a' )
    ->leftJoin( 'A.AB ab' )
    ->where( 'ab.id IS NULL' )
    ->fetchArray()

This works but is it the best way?

 

Querying for DNS Glue Records (using dig)

On a project I’m working on, I need to establish if a domain has IPv6 glue records or not. If I had to do it on a once off, a whois lookup would answer that nicely:

$ /usr/bin/whois opensolutions.ie
<snip>
nserver:     dns1.dns.opensolutions.ie 87.232.1.40 2a01:268:4::40
nserver:     dns2.dns.opensolutions.ie 87.232.1.41 2a01:268:4::41
nserver:     dns3.dns.opensolutions.ie 87.232.16.61 2a01:268:3002::61

However, in this case, I will need to do it many times on many domains and do not need to have to worry about whois servers limiting the queries or parsing the output from different whois servers.

After some digging, it looks like the nameservers of TLDs return glue records in the additional section. Let’s look by example on opensolutions.ie. First, find the TLD servers for .ie:

$ dig NS ie
<snip>
;; ANSWER SECTION:
ie.                     172800  IN      NS      gns1.domainregistry.ie.
ie.                     172800  IN      NS      uucp-gw-1.pa.dec.com.
ie.                     172800  IN      NS      uucp-gw-2.pa.dec.com.
ie.                     172800  IN      NS      ns3.ns.esat.net.
ie.                     172800  IN      NS      banba.domainregistry.ie.
ie.                     172800  IN      NS      ice.netsource.ie.
ie.                     172800  IN      NS      gns2.domainregistry.ie.
ie.                     172800  IN      NS      ns-ie.nic.fr.
ie.                     172800  IN      NS      b.iedr.ie.

Now query one of these for the nameservers for opensolutions.ie:

$ dig NS opensolutions.ie @banba.domainregistry.ie.
<snip>
;; AUTHORITY SECTION:
opensolutions.ie.       172800  IN      NS      dns3.dns.opensolutions.ie.
opensolutions.ie.       172800  IN      NS      dns2.dns.opensolutions.ie.
opensolutions.ie.       172800  IN      NS      dns1.dns.opensolutions.ie.

;; ADDITIONAL SECTION:
dns1.dns.opensolutions.ie. 172800 IN    A       87.232.1.40
dns1.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:4::40
dns2.dns.opensolutions.ie. 172800 IN    A       87.232.1.41
dns2.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:4::41
dns3.dns.opensolutions.ie. 172800 IN    A       87.232.16.61
dns3.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:3002::61

As you can see, the authority section contains the nameservers for opensolutions.ie which are all on the opensolutions.ie domain. We then find the glue records for these nameservers in the additional section.

Useful RANCID Debugging Tips

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

Note that in the following, I use clogin and rancid which assumes a Cisco device. Change to the appropriate variations if you’re not trying to work with a Cisco.

  1. Test logging into a device: 
    > clogin rtr1.example.com
  2. Test logging into a device and a single command: 
    > clogin -t 90 -c"show version" rtr1.example.com
  3. Test logging into a device and run a sequence of commands: 
    > clogin -t 90 -c"show version;show calendar" rtr1.example.com
  4. Show what RANCID does with debugging output: 
    > rancid -d rtr1.example.com

    If the above throws some errors (especially a list of missed commands, and if you’re using TACACS, ensure you have authorisation to run all the commands RANCID tries but logging into the router as the RANCID user and executing them one at a time.

  5. Same as (4) but record all router / switch output for analysis: 
    > setenv NOPIPE YES
> rancid -d rtr1.example.com

    and then complete output can be found in the file: rtr1.example.com.raw (in this example).

  6. Run RANCID on a single switch / router tree rather than all: 
    > /usr/local/bin/rancid-run [tree]
  7. Run RANCID normally:
> /usr/local/bin/rancid-run
  1. Don’t forget that logs are available in RANCID’s logs/ directory.

Changing a User’s UID on Apple XServe / Snow Leopard

Due to exporting NFS file systems from Linux boxes to an XServe, I had need to match the new users’ UID on the XServe to the Linux UIDs. Unfortunately this was not so obvious.

There’s a good how-to here:

http://www.inteller.net/notes/change-user-id-on-snow-leopard

Irish Radio Stations on Linux

UPDATED VERSION AVAILABLE: https://www.barryodonovan.com/index.php/2013/02/12/irish-radio-stations-on-linux-2013

I’m a bit of a newstalk junky and like to have the radio on in the back ground. It’s quite painful jumping between websites and even more painful getting them all to work under Linux so I have some simple Bash aliases for VLC and RTE Radio 1, Today FM and Newstalk:

alias 2fm='cvlc http://dynamic.rte.ie/av/live/radio/2fm.asx'
alias newstalk='cvlc http://newstalk.fmstreams.com:8008/listen.pls'
alias rteradio1='cvlc http://dynamic.rte.ie/av/live/radio/radio1.asx'
alias todayfm='cvlc http://audiostore.todayfm.com/audio/todayfmIRL_64K.asx'

UPDATED 2011-02-07: 2FM added.

Combatting SPAM in ISP Mail Servers

Fighting SPAM is not easy. Here’s an example of how we configured an ISP mail server cluster to reduce the SPAM load…

Introduction

ISPs must provide a mail relay service to their end users. However, mail relay isn’t a revenue generating service and ISPs can’t allocate unlimited staff resources to these services. The problem administrators face is trying to find a balance between unobstructively allowing their clients to send legitimate mail but to stop them from spamming.

The main source of SPAM from clients (SMTP clients in the ISPs IP space) is typically computers infected with bots / malware behind the users router. It may be their own or even a neighbor taking advantage of an unsecured wireless network. There is a smaller secondary source als0 – spammers who have someone gotten hold of a customers SMTP authentication details and are logging in from outside of the ISPs IP space to relay mail through.

Now, the problem this causes is that when the ISP mail servers pass on this SPAM to its destination such as a Yahoo! or GMail recipient, these companies examine the ratio of SPAM to HAM coming from the ISPs servers. When this ratio goes the wrong way, they legitimately block the mail servers meaning that customers of the ISP cannot get their mail delivered to these destinations. This is a big problem.

In this article, I’ll describe a new mail cluster set-up for an ISP to try and ensure their SPAM:HAM ratio stays on the right side.

Solution  Overview

The mail cluster comprises a number of SMTP servers all configured identically (in fact all booting from the same image using PXE and an NFS root file system with dedicated local /var partitions). They are running Debian Stable (Lenny at the time of writing) and the mail server is the excellent Postfix.

Postfix is configured to accept authentication using SASL with Dovecot (configured only for authentication, no POP3 or IMAP services) over TLS on port 25 or SSL on port 465. Users on the ISP’s address space do not need to authenticate to relay mail. This is actually a legacy configuration and quite common among ISPs. If we were to go back many years starting all over again, I’d certainly try and evaluate the merits of only allowing relay after authentication irrespective of your ISP.

Mail server server selection is made via DNS round robin but this could easily be changed to a load balancer. As part of this retrofit, we took the opportunity to IPv6 enable all the servers.

We now need to look at securing these boxes to ensure the level of SPAM is controller. There are three points at which we examine incoming mail:

  1. before it hits the SMTP daemon at all;
  2. after the SMTP headers (MAIL FROM, RCPT TO) have been passed to the daemon but before the mail has been accepted (i.e. before the SMTP DATA command);
  3. after the daemon has accepted the mail.

We’ll examine those in each of the sections below.

Rate Limiting Connections

In order to limit repeated connections to the mail server – such as from a SPAM bot that creates a new SMTP session for every mail (rather than sending many mails in one session), we use iptables recent match module.

This module keeps count of the number of connections in a given time period allow us to take alternative action if the number is above a threshold. In our case, we just drop additional connections until the number of connections drops below that threshold. For example:

-A INPUT -p tcp -m state --state NEW -m multiport
        --dports 25,465 -j SMTP_IN

-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --set
-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --update
        --seconds 60 ! --hitcount 10 -j SMTP_IN_LIMIT
-A SMTP_IN -j ACCEPT

-A SMTP_IN_LIMIT -j LOG --log-prefix "SMTP_IN: LIMIT EXCEEDED:: "
-A SMTP_IN_LIMIT -j DROP

We have some additional rules where, for example, we have whitelisted core infrastructure IP addresses and also mail from non ISP IP addresses (the purpose here is to stop SPAM bots from within the network).

Checks Before Accepting the Mail

Once the client connects to Postfix we check a number of other items.

We of course use some of Postfix’s own policy checks (such as reject_non_fqdn_sender, reject_non_fqdn_recipient, etc) but that’s pretty par for the course and not discussed below.

The ISP uses a commercial subscription to the Spamhaus data feed service. This is a very reliable database of IP addresses that are known to be sending SPAM (and other classifications). This is the first thing we check – customer or not. If you’re on this list, we do not accept mail from you.

If you’re not coming from the ISP address space, we then initiate gray listing using a Debian package called postfix-gld (see this site) which uses a MySQL database as its storage engine. See greylisting.org for more details on this useful anti-SPAM measure.

Now, if you’ve gotten this far, we use a package called postfix-policyd (see this site) to initiate rate limiting. Clients are limited in an hourly window on how many mails they can send and on how many recipients in total these mails can reach. The exact numbers are a moving goal post for now while we tweak it.

We could of course have used policyd for gray listing also but this posed a significant problem. Our Postfix configuration looks like this:

smtpd_recipient_restrictions =
    ...
    check_policy_service inet:127.0.0.1:10031,
    permit_mynetworks,
    permit_sasl_authenticated,
    check_policy_service inet:127.0.0.1:2525
    ...

As you can see, the issue is that we do not want to gray list our own IP range or people with a valid username and password but we do want to rate limit these. Using policyd for both is not possible unfortunately.

At this point, your mail will be accepted for relay. We not start the next batch of checks.

Content Filtering

We now need to examine these mails to discard SPAM and mails infected with a virus. For this we use the excellent amavisd-new package with SpamAssassin and ClamAV installed on the servers.

We’re still tweaking the rules for best effect but as it stands, if a virus is detected in the mail it’s silently discarded. A mail with a reasonable SPAM score is marked as SPAM in the subject but mails with a high score are also just discarded. DSNs are not sent.

If a mail passes all these checks, it’ll be relayed onwards.

Fighting SPAM is far from easy but the above will make a big dent. Unfortunately we can’t stop just there. We also have scripts to analyse the logs and rate limiting database to pull out the bad guys so the ISP’s tech support team can chase these up retroactively and instruct them on protecting their machines.