Author: Barry O'Donovan

lft :: Layer Four Trace

Colin pointed out a useful utility called lft in response to a question on IIU. lft looks like a useful alternative traceroute application as it claims to have the ability to identify stateful inspection firewalls and other useful information.

What I found immediately attractive was the -A option which displays the AS numbers of addresses along the path and also the -N which looks up and displays the network names.

e.g.

# lft -S -A  www.yahoo.com

TTL  LFT trace to f1.us.www.vip.ird.yahoo.com (87.248.113.14):80/tcp
 ...
 ...
 3   [AS35272] lns3.net.imagine.ie (87.232.0.26) 27.3ms
 4   [AS35272] ve5.core.net.imagine.ie (87.232.0.129) 9.0ms
 5   [AS35272] ge0-0.border1.net.imagine.ie (87.232.0.1) 8.6ms
 6   [AS3257] ge-2-0-0-207.dub20.ip.tiscali.net (213.200.67.145) 13.8ms
 7   [AS3257] yahoo-overture-gw2.dub20.ip.tiscali.net (213.200.67.202) 13.9ms
 8   [AS34010] ge-1-4.bas-b1.ird.yahoo.com (87.248.101.13) 10.9ms
 9   [AS34010] [target] f1.us.www.vip.ird.yahoo.com (87.248.113.14):80 12.6ms

and

# lft -S -N www.heanet.ie

TTL  LFT trace to www.heanet.ie (193.1.219.79):80/tcp
 ...
 ...
 3   [87-RIPE/IMAGINE-IRL] lns1.net.imagine.ie (87.232.0.24) 24.0ms
 4   [87-RIPE/IMAGINE-IRL] ve5.core.net.imagine.ie (87.232.0.129) 22.3ms
 5   [87-RIPE/IMAGINE-IRL] ge0-0.border1.net.imagine.ie (87.232.0.1) 60.6ms
 6   [RIPE-CBLK/IE-INEX-IPV4-PI-NETBLK1] gige6-1-cr1-cwt.hea.net (193.242.111.16) 8.7ms
 7   [RIPE-CBLK/HEANET-EXT] gige6-1-ar1-cwt.hea.net (193.1.195.177) 45.4ms
 8   [RIPE-CBLK/HEANET-EXT] blanch-sr1-po1.services.hea.net (193.1.195.139) 25.6ms
 9   [RIPE-CBLK/HEANET-LAN] [target] www.heanet.ie (193.1.219.79):80 9.4ms

 

Want to Publish Your Own Book?

I was purchasing a book online today and I was brought to Lulu.com – Self Publishing to complete the transaction.

Lulu claims to be an easy and free way to publish and sell easily within minutes. The free part is obviously misleading but it appears to do what it says on the tin: allow you to publish and print your book to order and the cost of printing does not seem to be too prohibitive.

Nagios Plugin for the Promise VTrak 200i

For a project I was working on, I installed a Promise VTrak M200i disk shelf (i for iSCSI but then that’s a whole other blog post!) and needed to add it into the customers management systems.

Unfortunately there didn’t seem to be a lot of information out there on Promise’s SNMP MIBs so with a bit of playing about, I was able to dig out the ones I needed. The Nagios plug-in I wrote and am making available here will monitor the shelf via SNMP and alert on the following chassis issues:

  • critical if any of the shelf’s disk states changes from “OK”;
  • warning if the battery state changes from “FullyCharged”;
  • critical if either of the PSU states change from “Powered On and Functional”;
  • critical is any of the cooling devices (fans) change from “Functional”;
  • critical if any of the temperature sensors’ states change from “normal”;
  • critical if any of the drives go offline or are missing; and
  • warning if any of the drives go into the rebuilding state or have their PFA flag set.

While this is specifically designed for a single M200i, it should be easily customisable for other models.

It can be downloaded from here (http://www.opensolutions.ie/). It will also appear on the development section of this site and Nagios Plugins.

OIDs Used

1.3.6.1.4.1.7933.1.10.2.1.1.1.8
The table of physical disk statuses.
.1.3.6.1.4.1.7933.2.1.7.1.1.14.1.1
The battery status.
.1.3.6.1.4.1.7933.2.1.4.1.1.2.1
The table of Power Supply Unit statuses.
.1.3.6.1.4.1.7933.2.1.3.1.1.3.1
The table of cooling device/fan statuses.
.1.3.6.1.4.1.7933.2.1.5.1.1.3
The table of temperature sensor statuses.
.1.3.6.1.4.1.7933.1.10.1.2.1.1.22.1
The number of drives that are offline.
.1.3.6.1.4.1.7933.1.10.1.2.1.1.23.1
The number of drives in the PFA status set.
.1.3.6.1.4.1.7933.1.10.1.2.1.1.24.1
The number of drives in rebuild status.
.1.3.6.1.4.1.7933.1.10.1.2.1.1.25.1
The number of drives that are missing.

Nagios Alerts via SMS with Kapow

I have a client who required a Nagios installation with alerting via SMS (*). They use Kapow as their SMS gateway.

There were two aspects required:

  1. The sending of alerts via the SMS gateway;
  2. The monitoring of available credits on the SMS gateway;

 

1. Send Alerts via SMS Gateway

The sendsms script is:

#! /bin/bash

USERNAME=username
PASSWORD=password
SENDSMSADDRESS="https://www.kapow.co.uk/scripts/sendsms.php"
MAXMSGLENGTH=320

read -n $MAXMSGLENGTH -r MSG

MSG=`php -r "echo urlencode( \"$MSG\" );"`

wget -q -O - "$SENDSMSADDRESS?username=$USERNAME&password=$PASSWORD&mobile=$1&sms=$MSG"

I use a quick hack with PHP to URL encode the string. I didn’t know a shell command off hand but I’m open to suggestions. This can be tested with:

echo This is a test message | sendsms 353861234567

Edit /etc/nagios/misccommands.cfg to include the following:

# 'host-notify-by-sms' command definition
define command{
        command_name    host-notify-by-sms
        command_line    /usr/bin/printf "%b" "Host '$HOSTALIAS$' is $HOSTSTATE$: $OUTPUT$" | /usr/local/bin/sendsms $CONTACTPAGER$
        }

# 'notify-by-sms' command definition
define command{
        command_name    notify-by-sms
        command_line    /usr/bin/printf "%b" "$NOTIFICATIONTYPE$: $SERVICEDESC$@$HOSTNAME$: $SERVICESTATE$ ($OUTPUT$)" | /usr/local/bin/sendsms $CONTACTPAGER$
        }

Ensure your /etc/nagios/contacts.cfg is updated to include notification by SMS with your mobile number:

define contact{
        contact_name                    barryo
        alias                           Barry O'Donovan
        service_notification_period     barryoworkhours
        host_notification_period        barryoworkhours
        service_notification_options    w,u,c,r
        host_notification_options       d,u,r
        service_notification_commands   notify-by-email,notify-by-sms
        host_notification_commands      host-notify-by-email,host-notify-by-sms
        email                           joe@bloggs.com
        pager                           353868765432
}

Sin é.

 

2. Monitor SMS Gateway Credits

The plugin code is:

#! /bin/bash

USERNAME=username
PASSWORD=password
CHECKCREDITSADDRES="https://www.kapow.co.uk/scripts/chk_credit.php"

CRIT=$1
WARN=$2

CREDITS=`wget -q -O - "$CHECKCREDITSADDRES?username=$USERNAME&password=$PASSWORD"`

if [[ -z $CREDITS || ! $CREDITS -ge 0 ]]; then
        echo -e "$CREDITS\\n";
        exit 3;
elif [[ $CREDITS -le $CRIT ]]; then
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 2;
elif [[ $CREDITS -le $WARN ]]; then
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 1;
else
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 0;
fi

Create a plugin configuration file for Nagios, say /etc/nagios-plugins/config/sms_credits.cfg:

# 'check_sms_credits' command definition
define command{
        command_name    check_sms_credits
        command_line    /usr/local/bin/check_sms_credit $ARG2$ $ARG1$
        }

Where $ARG1$ is the warning threshold and $ARG2$ is the critical threshold.

I add the service to the Nagios monitoring box via /etc/nagios/config/sms_credit.cfg:

#
# check sms credits on Kapow - barryo 20070519
#

define service{
        use                             core-service
        host_name                       noc
        service_description             SMS Credits
        check_command                   check_sms_credits!50!100
}

And I believe that’s it.

*) The monitoring box is in a different country to the servers it monitors so a network failure will not prevent the alert getting out.

O2 XDA Exec as a Modem

I recently purchased an O2 XDA Exec to fulfill by on-call/support obligations. With a trip away this weekend, I decided to play it safe and just make sure it worked as a modem for an install of Windows hidden away on a tiny partition of my laptop (which of course includes PuTTY).

I have actually installed PuTTY on the XDA as well and it works great but it’s only really practical for quick fixes or real emergencies when it’ll “have to do”.

The use of the XDA as a modem for accessing the Internet from a PC via USB is not very intuitive but fortunately someone else had done all the work: http://www.pcurtis.com/xda.htm. Their instructions are reproduced below for prosperity. The Internet access point for O2 in Ireland is internet and the GPRS username/password is gprs/gprs.

Use of XDA as a Modem for accessing the Internet from a PC via USB

Although the comprehensive handbook and advertising material indicated this was possible it took considerable research on the Internet to implement the use of the XDA as a modem for GPRS/3G access from a PC. This was not entirely unexpected as it took a very long time to find out how to do the same with our Sony T610 phone. We have finally succeeded and are providing detailed instructions which work for a USB cable.

The following procedures have only been tested with the XDA so far but should be applicable to any PDA running Windows Mobile 5. I have therefore used PDA instead of XDA throughout this section.

Firstly one must understand that use of PDA as a modem for accessing the Internet involves running a built in program called Wireless Modem on the PDA which simulates a modem on the PDA USB port (or on the Bluetooth or IR connections). When this program is running it is just like an old fashioned external modem box with a screen display with an online and data lights – when the program is running the USB connection is completely different and when it is plugged in it will be recognised as a new modem device. One must therefore install a USB modem on your computer which uses the special driver supplied on the applications disk with the PDA. Only then can you set up a Dial-up internet connection using the PDA.

The use of GPRS/3G requires the setting up the PDA modem by a special initialisation string with details of the GPRS access point or, in some cases it will be already set if you have just made a connection using the required access point. You then connect (dial) using a magic code of *99# instead of an ordinary telephone number. You need to know the access point, username and password for you providers GPRS/3G service to do this in the case of O2 the access point is mobile.o2.co.uk , the user name is web, o2web or faster and the password for all of them is password. In the case of Vodafone UK prepay the access point (APN) is pp.vodafone.co.uk , the username is wap and the password is also wap . Try Ross Barkman’s Page or http://www.formatc.de/roaming/gprs.htm for complete sets of GPRS settings world wide.

So in detail:

  1. Disconnect the PDA USB cable from your computer if it is plugged in
  2. If you have installed ActiveSync on the computer it should be disabled by Right clicking the ActiveSync icon in your computer tooltray, open “Connection Settings” and Uncheck “Allow USB Connection”.
  3. On your PDA: Go to “Beam” (in Settings/Connections), and uncheck “Receive all Incoming beams”.
  4. Turn on the telephone in Wireless settings and check you have a signal.
  5. Then in Programs, click on “Wireless Modem” click on Tools/Options and Check the Preset GPRS connections and write in the APN, your network provider’s internet access point (mobile.o2.co.uk for O2 and pp.vodafone.co.uk for Vodafone) and click OK. Set the Connection type to USB and click on “Start” and leave the program running.
  6. Now, connect your PDA to the computer with the USB cable and after a few seconds the “Found New Hardware Wizard” should start on your computer. Click “Install from a specific location”, then “Don’t search. I will choose.”, then click on “Modems”, and “Have Disk.” and go to the folder on the CD provided which contains the HTC USB Driver. The HTC USB Modem will be shown in the compatible hardware: click to install. Continue if you receive the alert of the software not passing the Windows XP compatibility test (Continue anyway). Click Finish.
  7. Now, in Control Panel of your computer, open “Phone and Modem Options”. Select the “HTC USB Modem” and click on “Properties”. In the “Advanced” tab type or cut and paste into “Extra initialization commands” the following string AT+CGDCONT=1,”IP”,”xxx”,””,0,0 where xxx is your internet access point ie for Vodafone the string is AT+CGDCONT=1,”IP”,”pp.vodafone.co.uk”,””,0,0 . Click Ok to finish.
  8. Finally, on your computer, go to “Network Connections” and click “Create a new connection”. Choose “Connect to the internet”, then “Set up my connection manually”. Click “Connect using a dial-up modem”, then give a name to your ISP. In phone number type *99#. Get username and password from your network provider – for Vodafone UK use wap for both
  9. Make a shortcut of this connection on your desktop for quick access.

    10. NB ALWAYS use the same USB connector or you will need to set up another!

    Every time you want to use the PDA as a modem:

    1. Disconnect PDA USB cable if connected
    2. If necessary Disable ActiveSync on your computer as above by unchecking the “Allow USB connection”
    3. If necessary turn on the telephone and check for a signal.
    4. Open Wireless Modem program on your device and push on Start (with the Connection Type on USB) and leave it running.
    5. Connect your device USB cable (always using the same connector on the PC )
    6. Dial from your desktop using the shortcut you created

    After you have finished:

    1. Close the connection on the computer to stop being charged then
    2. Remove the USB cable before you
    3. Close the Wireless Modem program but remember to
    4. Enable ActiveSync by re-enabling the USB connection before you
    5. reconnect the USB cable (always using the same connector on the PC) when you want to start synchronisation.

Coming soon: O2 XDA Exec, GPRS, USB and Linux.

Putting /etc Under Subversion (SVN)

A Google for the above took some work to locate the exact recipe I wanted for this. The problem is that one really needs to do an ‘in-place’ import. The solution was from Subversion‘s own FAQs (specifically this) which is reproduced here with some changes:

# svn mkdir svn+ssh://user@host/srv/svn-repository/hosts/host1/etc \
         -m "Make a directory in the repository to correspond to /etc for this host"
# cd /etc
# svn checkout svn+ssh://user@host/srv/svn-repository/hosts/host1/etc .
# svn add *
# svn commit -m "Initial version of this host's config files"

 

OpenVPN “Just Works”

When it comes to OSS, it very often happens that I find something I like and stick with it.

OpenVPN is a good example of this.

I have a number of OpenVPN installations for various purposes and today I had need of yet another for a new client.

I often thought about writing a how-to for OpenVPN. But why bother? It’s quick and easy to implement and they already have a brief but comprehensive how-to which always does the job for me – once you’ve set it up once, the next time will take just 30 minutes.

OpenVPN just works. It does what it says on the tin and it’s reliable and robust.

IPMI Sensor Data on Dell 1850s and 2850s via SNMP and Cacti

I use Cacti to monitor a lot of Dell servers, primarily 1850s and 2850s but also the newer models of same (1950s and 2950s). One itch that I’ve meant to scratch for a while is graphing some of the information available through the servers’ IPMI interface; specifically the servers’ various temperatures and and fan speeds.

IPMI Details

There are patches available for the Linux kernel to allow the IPMI information to be read via the lm_sensors project but I chose to avoid this (at least for now) as I’d have to schedule downtime to reboot the servers for a new kernel. It’d also ruin their uptime – most of the servers (serving many thousands of users daily) have almost two years of uptime. (The kernels are monolithic.)

Instead, I went with the already compiled in Linux IPMI Driver (see kernel source: Documentation/IPMI.txt) which is available in the ‘Character Devices’ menu. I specifically needed the following options for the Dells:

  • drivers/char/ipmi/ipmi_msghandler
  • drivers/char/ipmi/ipmi_devintf
  • drivers/char/ipmi/ipmi_si

In order to read information from the IPMI, you need the ipmitool utility which is available on most recent Linux distributions or from here.

Lastly, I needed to create a character special file to interface with the IPMI:

mknod /dev/ipmi0 c 254 0

The sensor information was then available via:

# ipmitool sensor
Temp             | 30.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 90.000    | na
Temp             | 34.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 90.000    | na
Ambient Temp     | 16.000     | degrees C  | ok    | na        | 3.000     | 8.000     | 42.000    | 47.000    | na
...

Making IPMI Sensor Information Available via SNMP

I make the IPMI sensor information available over SNMP by adding the following to the snmpd.conf file:

# Monitor IPMI Temperature and Fan stats
exec    .1.3.6.1.4.1.X.1000 ipmitemp        /usr/local/sbin/ipmi-temp-stats
exec    .1.3.6.1.4.1.X.1001 ipmifan         /usr/local/sbin/ipmi-fan-stats

(Replace X above as appropriate.)

The scripts referenced are: /usr/local/sbin/ipmi-temp-stats:

#! /bin/sh

PATH=/usr/bin:/bin
STATS=/tmp/ipmisensor-snmp

printf "%f\n" `cat $STATS | grep Temp | cut -s -d "|" -f 2`

And /usr/local/sbin/ipmi-fan-stats:

#! /bin/sh

PATH=/usr/bin:/bin
STATS=/tmp/ipmisensor-snmp

printf "%f\n" `cat $STATS | grep FAN | cut -s -d "|" -f 2`

The file they reference is generated every 5mins (Cacti polling interval) via a cron entry in the file /etc/cron.d/ipmitool:

*/5 * * * * root /usr/bin/ipmitool sensor >/tmp/ipmisensor-snmp

After restarting SNMP and allowing the cron job to execute at least once, you can test the results via:

# snmpwalk -c <community> -v <version> <ip/hostname> .1.3.6.1.4.1.X.1000
SNMPv2-SMI::enterprises.X.1000.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.X.1000.2.1 = STRING: "ipmitemp"
SNMPv2-SMI::enterprises.X.1000.3.1 = STRING: "/usr/local/sbin/ipmi-temp-stats"
SNMPv2-SMI::enterprises.X.1000.100.1 = INTEGER: 0
SNMPv2-SMI::enterprises.X.1000.101.1 = STRING: "37.000000"
SNMPv2-SMI::enterprises.X.1000.101.2 = STRING: "39.000000"
SNMPv2-SMI::enterprises.X.1000.101.3 = STRING: "23.000000"
SNMPv2-SMI::enterprises.X.1000.101.4 = STRING: "36.000000"
...
SNMPv2-SMI::enterprises.X.1000.102.1 = INTEGER: 0
SNMPv2-SMI::enterprises.X.1000.103.1 = ""

Graphing This Information in Cacti

Finally, I graph this information on Cacti (see end of post for examples).

I am making six templates available here which can be imported into Cacti (these were generated using version 0.8.6j) for graphing the above:

  1. Cacti graph template for Dell 1850 temperatures (see first image below);
  2. Cacti graph template for Dell 2850 temperatures (see second image below);
  3. Cacti graph template for Dell 1850 fan speeds (see third image below);
  4. Cacti graph template for Dell 2850 fan speeds (see fourth image below);
  5. Cacti host template for Dell 1850; and
  6. Cacti host template for Dell 2850.

The last two templates available are host templates for Dell 1850s and 2850s (I’m sure they’ll work fine with 1950s and 2950s also). These templates include:

  • Host MIB – Logged in Users;
  • Host MIB – Processes;
  • IPMI Fan Speeds (Dell x850) (from above);
  • IPMI Temperatures (Cel) (Dell x850) (from above);
  • ucd/net – CPU Usage;
  • ucd/net – Load Average;
  • ucd/net – Memory Usage;
  • SNMP – Get Mounted Partitions (data query); and
  • SNMP – Interface Statistics (data query).

Example graphs are shown below; they’re not the cleanest given the amount of information they contain but they serve my purposes.

[Dell 1850 Temps]

[Dell 2850 Temps]

[Dell 1850 Fan Speeds]

[Dell 2850 Fan Speeds]

© 2007 Barry O’Donovan. All text is licensed under a Creative Commons Attribution 3.0 License. All scripts and Cacti templates are licensed under the MIT License.

GCC Optimisations per CPU

Pádraig Brady wrote a very useful script for getting the optimum gcc options for your CPU which I keep coming back to (and forgetting where to find it).

The last version can be found at http://www.pixelbeat.org/scripts/gcccpuopt along with a lot of other useful scripts and scripting examples here.

IPv6 Blasts Into Orbit

http://www.dmcii.com/news.htm:

On Thursday, 29 March 2007, a Cisco Systems router, flying in low Earth Orbit onboard the UK-DMC satellite built by Surrey Satellite Technology Ltd (SSTL), was successfully configured by NASA Glenn Research Center to use IPsec and IPv6 technologies in space.

…

The five DMC satellites in orbit rely on standard IP networking to send mission-critical imagery to ground stations and to interact with terrestrial networks.

…

Working together, Cisco Systems, NASA Glenn Research Center and SSTL are the first to configure and test IPsec and IPv6 on a satellite.