Tag: linux

Virtual Mail with Ubuntu, Postfix, Dovecot and ViMbAdmin

As part of pushing our new release of ViMbAdmin, I wrote up a mini how-to for setting up a virtual email system on Ubuntu where the components are:

  • Postfix as the SMTP engine;
  • Dovecot for IMAP. POP3, Sieve and LMTP;
  • ViMbAdmin as the domain / mailbox / alias management system via web interface.

It supports a number of features including mailbox archival and deletion, quota support and display of mailbox sizes (as well as per domain totals).

Find the how-to at:

Bird / Quagga with MD5 Support for IPv4/6 on FreeBSD & Linux

Over in INEX we run a route server cluster which alleviates the burden of setting up bilateral peering sessions for the more than 80% of the members that use them. The current hardware is now about six years old and we have a forklift upgrade in the works.

BGP allows for MD5 authentication between clients (using the TCP MD5 signature option, see RFC 2385) and – while recently obsoleted in RFC 5925 – it is still widely used in shared LAN mediums such as IXPs; primarily to prevent packet spoofing and session hijacking via recycled IP addresses.

Our current route server implementation runs on FreeBSD which does not support TCP MD5 in its stock kernel (you are required to compile a custom kernel – see below for details). Additionally, specifying the session MD5 is not done in the BGP daemon configuration but separately in the IPsec configuration. Lastly, our current FreeBSD version has no support for TCP MD5  over IPv6. These have all led to unnecessarily complex configurations and a degree of confusion.

Because of this, we decided to test up to date Linux and FreeBSD versions for native IPv4 and IPv6 TCP MD5 support with Bird and Quagga (our route server daemons of choice).

In each case, BGP sessions were tested for:

  • no MD5 on each end (expected to work);
  • same MD5 on each end (expected to work);
  • different MD5 on each end (expected not to work); and
  • MD5 on one end with no MD5 on the other end (expected not to work).

For Linux, the platform chosen was Ubuntu 12.04 LTS with the stock 3.2.0-40-generic kernel.

  • Sessions were tested for Quagga to Quagga and Quagga to Bird;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga and Bird from the 12.04 apt repositories were used.

The results - everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6.

Summary: Linux will support TCP MD5 nativily for IPv4 and IPv6 when using Quagga or Bird.

For FreeBSD, we used the latest production release of 9.1. TCP MD5 support is not compiled in by default so a custom kernel must be built with the additional options of:

options   TCP_SIGNATURE
options   IPSEC
device    crypto
device    cryptodev

In addition to this, the MD5 shared secrets need to be added to the IPsec SA/SD database via the setkey utility or, preferably, via the /etc/ipsec.conf file which, for example, would contain entries for IPv4 and IPv6 addresses such as:

add 192.0.2.1 192.0.2.2 tcp 0x1000 -A tcp-md5 "supersecret1";
add 2001:db8::1 2001:db8::2 tcp 0x1000 -A tcp-md5 "supersecret2";

where the addresses ending in .1/:1 are local and .2/:2 are the BGP neighbor addresses. This file can be processed by setting ipsec_enable="YES" in /etc/rc.conf and executing /etc/rc.d/ipsec reload.

  • Sessions were tested for Quagga/Linux to Quagga/FreeBSD and  from Quagga/Linux to Bird/FreeBSD;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga from the 12.04 apt repositories and stock Quagga and Bird from FreeBSD ports were used.

The results – almost everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6;
  • one odd but expected behavior – you only need to set the MD5 via setkey / ipsec.conf – setting it (or not) in the Quagga and Bird config has no effect so long as it is set via setkey (but is useful for documentation purposes). However, trying to set it in Quagga without having rebuilt the kernel will result in an error.

Summary: FreeBSD will support TCP MD5 via a custom kernel and setkey / ipsec.conf for IPv4 and IPv6. Note that there is an additional complexity when changing or removing MD5 passwords as these need to be amended / deleted via setkey which can put an extra burden on automatic route server configuration generators.

Enabling External Commands in Nagios / Ubuntu

I get caught by the following quite often (too many Nagios installations!):

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands. An error occurred while attempting to commit your command for processing.

The correct way to fix this in Ubuntu is:

service nagios3 stop
dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
service nagios3 start

Nagios Plugin for Checking Backups via rsnapshot

We’ve just added a check_rsnapshot.php script to our nagios-plugins bundle on Github. This script will verify rsnapshot backups via Nagios using a number of checks / tests:

  • minfiles – checks the number of files in a snapshot against a minimum expected number;
  • minsize – checks the size of a snapshot against a minimum expected size;
  • log – parses the rsnapshot log to ensure the most recent runs for each retention period completed successfully;
  • timestamp – checks for files created server side containing a timestamp and thus ensuring snapshots are succeeding;
  • rotation – checks that retention directories are being rotated; and
  • dir-creation – checks that retention directories are being created.

Please see this Github wiki page for more information including instructions.

Analysing MySQL Slow Query Logs

MySQL has a really useful feature that allows it to log slow queries where slow is a minimum time defined by you in micro seconds. It helps a lot is diagnosing website outages or slow responsiveness issues after the fact.

Unfortunately I couldn’t find any nice graphical tools for analysing these but there are a few command line tools:

mysqldumpslow

MySQL’s own tool, mysqldumpslow, which aggregates queries and allows you to sort them by: query time or average query time; lock time or average lock time; rows sent or average rows sent; or the number of queries.

Percona’s MySQL Slow Query Log Analyser

Dating from 2006, Percona’s Peter Zaitsev wrote about their own version of a slow query log analyser (local copy) which has given me good results. Note that their micro time patch has since been incorporated into MySQL mainstream.

One of the main differences over MySQL’s own version is that as well as printing the aggregated query (with number and string literals wildcarded), it also prints a real example of the query allowing a copy and paste to MySQL for execution with EXPLAIN.

Example output with query details redacted:

### 230 Queries 
### Total time: 4708.948293, Average time: 20.4736882304348
### Taking 0.093420 to 203.693466 seconds to complete
### Rows analyzed 0 - 141008
SET timestamp=XXX;
SELECT ... FROM ... AS A 
        INNER JOIN ... AS C ON C.item_id = A.item_id 
    WHERE XXX AND C.item_lang = 'XXX' AND ... 
    ORDER BY CATALOG.item_sort LIMIT XXX;

SET timestamp=1348032761;
SELECT ... FROM ... AS A 
        INNER JOIN ... AS C ON C.item_id = A.item_id 
    WHERE 1 AND C.item_lang = '1' AND ... 
    ORDER BY C.item_sort LIMIT 1;

 

Centralised Logging

I’m currently looking at some centralised logging tools and the following stand out:

  • Octopussy – one I cam across a long time ago but looking at some of the others below it may be past its sell by date?
  • Graylog2 – GSOH (in dating parlance) – “Manage your logs in the dark and have lasers going and make it look like you’re from space.“
  • logstash – “a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.“
  • Kibana - You have logs. Billions of lines of data. You shipped, dated it, parsed it and stored it. Now what do you do with it? Now you make sense of it… Kibana is an alternative browser based interface for Logstash… that allows you to efficiently search, graph, analyze and otherwise make sense of a mountain of logs.

Kibana has a Bootstrap UI and is written in PHP which immediately bumps it up my list 😉

Ubuntu 12.04 (Precise Pangolin) and PHP 5.4 (again)

My previous post, Ubuntu 12.04 (Precise Pangolin) and PHP 5.4, has been extremely popular but I left some work for the user to figure out.

In a nutshell, here is how you install PHP 5.4 in Ubuntu 12.04 (Precise Pangolin).

1. Install the signing key for the PPA (which also adds the sources to apt):

add-apt-repository ppa:ondrej/php5

If the above command is not available, install it using:

apt-get install python-software-properties

2. Now update the package database and then upgrade the system. As part of upgrading, PHP 5.4 will be installed automatically:

apt-get update
apt-get upgrade
apt-get dist-upgrade

Now, you should have PHP 5.4 installed:

# php -v
PHP 5.4.3-4~precise+1 (cli) (built: May 17 2012 13:00:25)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies

As an aside, at the time of my previous post, PHP Memcache packages were not available from the PPA but that has since been rectified.

Looking at New Features in PHP 5.4

PHP 5.4 was released at the start of March and heralds a key new feature which I have long bemoaned the lack of: traits.

Traits (see PHP’s documentation and Wikipedia’s description) allow programmers to define a set of functions which can be shared among different classes in ways that normal inheritance cannot do.

A Trait is intended to reduce some limitations of single inheritance by enabling a developer to reuse sets of methods freely in several independent classes living in different class hierarchies.

I’ve banged my head against the desk a number of times trying to work around the lack of traits in PHP and I hope PHP 5.4 makes its way into mainstream Linux distributions a lot quicker that 5.3 did.

There are also a number of other interesting features which look really cool:

  • Simpler array dereferencing: $secondElement = getArray()[1];
  • A shortened array syntax: $a = [ "foo" => "bar", "bar" => "foo" ];
  • A built in web server (for development) which could be really interesting – see here.

 

We’ve Just Launched TallyStick

TallyStick allows you to focus on what you love to do while it takes care of tracking and accumulating the time you spend on projects. As of today, TallyStick is live and open for business. Please sign up, take advantage of the one month’s free trial (with no credit card details required) and tell us what you think.

[TallyStick Logo]

Overt at Open Solutions, we manage networks, build Internet infrastructure, consult and develop web applications for many customers. Tracking the time our engineers spent on different projects – especially those on the front line hopping from customer to customer – was becoming an issue.

And we had enough. So we built TallyStick. Necessity, as the saying goes, is the mother of invention.

We believe in web applications that are simple, beautiful and functional. And after an exhaustive search, we couldn’t find one to fit our needs… so we built it. And who better to build a time tracking application for IT and other consulting professionals than those same professionals!

TallyStick allows you to focus on what you love to do while it takes care of tracking and accumulating the time you spend on projects. We believe it’s simple, efficient and quick. And we hope you agree.

As of today, TallyStick is live and open for business. Please sign up, take advantage of the one month’s free trial (with no credit card details required) and tell us what you think.

Also, all users signing up during the beta period will receive a lifetime discount of 10% on that account!

Useful RANCID Debugging Tips

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

Note that in the following, I use clogin and rancid which assumes a Cisco device. Change to the appropriate variations if you’re not trying to work with a Cisco.

  1. Test logging into a device: 
    > clogin rtr1.example.com
  2. Test logging into a device and a single command: 
    > clogin -t 90 -c"show version" rtr1.example.com
  3. Test logging into a device and run a sequence of commands: 
    > clogin -t 90 -c"show version;show calendar" rtr1.example.com
  4. Show what RANCID does with debugging output: 
    > rancid -d rtr1.example.com

    If the above throws some errors (especially a list of missed commands, and if you’re using TACACS, ensure you have authorisation to run all the commands RANCID tries but logging into the router as the RANCID user and executing them one at a time.

  5. Same as (4) but record all router / switch output for analysis: 
    > setenv NOPIPE YES
> rancid -d rtr1.example.com

    and then complete output can be found in the file: rtr1.example.com.raw (in this example).

  6. Run RANCID on a single switch / router tree rather than all: 
    > /usr/local/bin/rancid-run [tree]
  7. Run RANCID normally:
> /usr/local/bin/rancid-run
  1. Don’t forget that logs are available in RANCID’s logs/ directory.