Category: Work

Well, there goes my day…

I arrived to work this morning to find that the Heartbleed Bug happened.

This is one of the biggest security issues to crop up in a long time – allowing the data normally protected by TLS/SSL to be compromised. This is the kind of data that normally passes securely between clients and protected websites, email services, instant messaging, etc.

Upgrade all your systems now. This is where my well planned day went.

Be sure to restart all services that use OpenSSL (or reboot your servers). A useful command [source] for this post-upgrade is:

grep -l 'libssl.*deleted' /proc/*/maps | tr -cd 0-9\\n | xargs -r ps u

A useful Python script for testing your web servers can be found in this Gist. NB: it’s not just web servers affected – any services with SSL/TLS may be affected.

Virtual Mail with Ubuntu, Postfix, Dovecot and ViMbAdmin

As part of pushing our new release of ViMbAdmin, I wrote up a mini how-to for setting up a virtual email system on Ubuntu where the components are:

  • Postfix as the SMTP engine;
  • Dovecot for IMAP. POP3, Sieve and LMTP;
  • ViMbAdmin as the domain / mailbox / alias management system via web interface.

It supports a number of features including mailbox archival and deletion, quota support and display of mailbox sizes (as well as per domain totals).

Find the how-to at:

Doctrine2 Provider for Laravel 4 Authentication

I’ve just added to the Doctrine2 service provider for Laravel by adding a UserProvider allowing Doctrine2Bridge to provide a driver for Laravel’s authentication system.

Simply put – this allows a Doctrine2 database table stroing users’ usernames and passwords to be used as the backend for Laravel 4 authentication.

Full documentation and examples can be found here. Available on Packagist and forkable on GitHub.

Doctrine2 Service Provider for Laravel 4

I’ve just released a Laravel 4 package which contains a service provider to the Doctine2 entity manager and the Doctrine2 cache. These are made available via facades named D2EM and D2Cache respectively.

Currently it uses the XML schema method for defining entities but this can easily be augmented with the other methods. Also, ArrayCache and MemcacheCache are fully supported caching interfaces. Any other cache that requires no configuration is also supported but some trivial coding will be required for caching backends required configuration.

Combining this with TwigBridge, we have an excellent framework with Laravel 4!

Querying Cisco MST Port Roles via SNMP with OSS_SNMP

OSS_SNMP is a PHP SNMP library written by myself for people who hate SNMP. After a customer migration from PVST to MST (Multiple Spanning Tree), I have added a number of MST functions / MIBs to OSS_SNMP:

During a fairly significant network migration involving breaking / connecting a number of links, I wanted to be able to monitor the MST port role of significant ports at a glance. For this purpose, I wrote the mst-port-roles.php script and have committed it as an example to OSS_SNMP. First, here is what it looks like when run on the command line (with hostnames obfuscated):

MST Port RolesFrom a very simple array of port details at the top of the script, it will poll all switches and for each port print:

  • device and port name;
  • port state and speed;
  • port role for each applicable MST instance.

I run it on bash and use bash colouring. The script is well documented and can easily be repurposed for other networks. You’ll find the source here.

Useful RIPE Database Links

Some very useful RIPE database links that I was recently shown include:

Making a Console Cable for Soekris Boxes

Connecting to a Soekris box from your computer via a console cable is a DTE to DTE connection and so you need to purchase or make up a null modem cable. This page on RS232 serial null modem cable wiring proved invaluable.

With the documentation above, I created a null modem with loop back handshaking cable using two wire-able female DB9-to-RJ45 converters and a standard straight-through network cable following the pin positions in the above link.

To locally connect and loop back  pins  1+4+6 and 7+8, I snipped those wires, striped the ends and just twisted together and covered with electrical tape. Pin 5 is connected to its opposite number and you only then need to ensure that pins 2 and 3 are crossed over.

This worked a charm with my USB serial port. After I tried about a dozen various unlabeled console cables I have lying around that is…

Pastebin Alternatives

Update 2019: we now use PrivateBin a lot. It is a self-hosted, minimalist, open source online pastebin where the server has zero knowledge of pasted data.

Pastebin has been a valuable tool for years – to the extent that pastebin it has entered the common lexicon of sysadmins, network engineers and developers.

There are, however, a few notable alternatives:

  • GitHub Gists – what’s particularly cool about these is that each Gist (which is just pasted text) is also a fully fledged Git repository with versioning and the ability for fork. There’s also syntax highlighting and a nice UI. If you’re a GitHub user, your own Gists are also linked to your account.
  • p.ip.fi – this scores big points for its pure simplicity. You’d argue that a pastebin doesn’t really need a complex UI and p.ip.fi is laudable in its complete lack of a UI. Just paste and hit Ctrl-S and you’re done. Very nice. (Credit to Nick for pointing this out).
  • sprunge.us – this is a command line pastebin which should appeal directly to sysadmins and network engineers. (Credit to dnolan for leading me to this one). This is best demonstrated via an epic traceroute:
$ traceroute -m 60 216.81.59.173 | curl -F 'sprunge=<-' http://sprunge.us
http://sprunge.us/THie

Check out the result at http://sprunge.us/THie.

Bird / Quagga with MD5 Support for IPv4/6 on FreeBSD & Linux

Over in INEX we run a route server cluster which alleviates the burden of setting up bilateral peering sessions for the more than 80% of the members that use them. The current hardware is now about six years old and we have a forklift upgrade in the works.

BGP allows for MD5 authentication between clients (using the TCP MD5 signature option, see RFC 2385) and – while recently obsoleted in RFC 5925 – it is still widely used in shared LAN mediums such as IXPs; primarily to prevent packet spoofing and session hijacking via recycled IP addresses.

Our current route server implementation runs on FreeBSD which does not support TCP MD5 in its stock kernel (you are required to compile a custom kernel – see below for details). Additionally, specifying the session MD5 is not done in the BGP daemon configuration but separately in the IPsec configuration. Lastly, our current FreeBSD version has no support for TCP MD5  over IPv6. These have all led to unnecessarily complex configurations and a degree of confusion.

Because of this, we decided to test up to date Linux and FreeBSD versions for native IPv4 and IPv6 TCP MD5 support with Bird and Quagga (our route server daemons of choice).

In each case, BGP sessions were tested for:

  • no MD5 on each end (expected to work);
  • same MD5 on each end (expected to work);
  • different MD5 on each end (expected not to work); and
  • MD5 on one end with no MD5 on the other end (expected not to work).

For Linux, the platform chosen was Ubuntu 12.04 LTS with the stock 3.2.0-40-generic kernel.

  • Sessions were tested for Quagga to Quagga and Quagga to Bird;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga and Bird from the 12.04 apt repositories were used.

The results - everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6.

Summary: Linux will support TCP MD5 nativily for IPv4 and IPv6 when using Quagga or Bird.

For FreeBSD, we used the latest production release of 9.1. TCP MD5 support is not compiled in by default so a custom kernel must be built with the additional options of:

options   TCP_SIGNATURE
options   IPSEC
device    crypto
device    cryptodev

In addition to this, the MD5 shared secrets need to be added to the IPsec SA/SD database via the setkey utility or, preferably, via the /etc/ipsec.conf file which, for example, would contain entries for IPv4 and IPv6 addresses such as:

add 192.0.2.1 192.0.2.2 tcp 0x1000 -A tcp-md5 "supersecret1";
add 2001:db8::1 2001:db8::2 tcp 0x1000 -A tcp-md5 "supersecret2";

where the addresses ending in .1/:1 are local and .2/:2 are the BGP neighbor addresses. This file can be processed by setting ipsec_enable="YES" in /etc/rc.conf and executing /etc/rc.d/ipsec reload.

  • Sessions were tested for Quagga/Linux to Quagga/FreeBSD and  from Quagga/Linux to Bird/FreeBSD;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga from the 12.04 apt repositories and stock Quagga and Bird from FreeBSD ports were used.

The results – almost everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6;
  • one odd but expected behavior – you only need to set the MD5 via setkey / ipsec.conf – setting it (or not) in the Quagga and Bird config has no effect so long as it is set via setkey (but is useful for documentation purposes). However, trying to set it in Quagga without having rebuilt the kernel will result in an error.

Summary: FreeBSD will support TCP MD5 via a custom kernel and setkey / ipsec.conf for IPv4 and IPv6. Note that there is an additional complexity when changing or removing MD5 passwords as these need to be amended / deleted via setkey which can put an extra burden on automatic route server configuration generators.

Enabling External Commands in Nagios / Ubuntu

I get caught by the following quite often (too many Nagios installations!):

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands. An error occurred while attempting to commit your command for processing.

The correct way to fix this in Ubuntu is:

service nagios3 stop
dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
service nagios3 start