Category: Networking

NOCtools – A Mixed Bag of Tools and Utilities for NOC Engineers

NOCtools is a mixed bag of tools and utilities that are useful for NOC engineers. This project originally started out as a way to highlight and utilise our OSS_SNMP library (a PHP SNMP library for people who hate SNMP).

It since grew into a way to graphically present information on network topology that is normally difficult and cumbersome to do by logging into individual devices. Such information includes a discovered L2 topology by CDP, using this to present rapid-PVST port roles and so forth.

From the company blog:

Today, we are introducing NOCtools which uses this library to provide a number of useful tools including:

  • CDP Neighbours: for a given CDP enabled device, display its CDP neighbours with information and also a graph showing connected ports.
  • CDP L2 Topology: graph the layer 2 network topology based on a recursive crawl of CDP neighbours.
  • RSTP Topology & Port Roles: similar to CDP L2 Topology above but this takes a specific VLAN and identifies and graphs Per-VLAN Spanning Tree port roles.
  • Per-VLAN RSTP Port Roles: a tool that will display the per-VLAN Rapid STP port roles for a given VLAN (or for all VLANs) on a device.
  • Inter-Device VLAN Comparison: a tool that will compare VLANs available (and their respective names) across selected devices allowing you to ensure consistency as well as perform simple security audits.

Follow the links about for screen shots and more details. We are releasing this under a GNU GPL license in the hope that the wider networking community will benefit from them.

A PHP SNMP Library for People Who HATE SNMP, MIBs and OIDs!

I hate SNMP! But I have to use it on a daily basis with my company, Open Solutions.

Don’t get me wrong, it’s an essential tool in the trade of network engineering but it’s also a serious PITA. Finding MIBs, OIBs, making them work, translating them, cross-vendor translations, etc, blah, blah. And then, when you do find what you need, you’ll have forgotten it months later when you need it again.

Anyway, while trying to create some automatic L2 topology graphing tools (via Cisco/Foundry Discovery Protocol for example) and also some per VLAN RSTP tools to show port states, I started writing this library. As I wrote, I realised it was actually very useful and present it here now in the hopes that the wider network engineering community will find it useful and also contribute back MIBs.

An example may best illustrate the library:

First, we need to instantiate an SNMP object with a hostname / IP address and a community string:

$ciscosw = new \OSS\SNMP( $ip, $community );

Assuming the above is a standard Cisco switch, let’s say we want to get an associate array of VLAN names indexed by the VLAN ids:

print_r( $ciscosw->useCisco_VTP()->vlanNames() );

This yields something like the following:

Array (
    [1] => default
    [2] => mgmt
    [100] => cust-widgets
    [1002] => fddi-default
    ...
)

It really is that easy.

See the GitHub project page: https://github.com/opensolutions/OSS_SNMP

What the Hell is INEX? An IXP?

In a few recent posts, I’ve mentioned INEX.

INEX is a neutral, industry-owned association, founded in 1996, that provides IP peering facilities for its members. INEX membership is open to all organisations that can benefit from peering their IP traffic and there are currently 57 members.

INEX can also be considered Ireland’s IP Peering Hub. INEX membership provides high-speed, reliable and resilient IP traffic exchange facilities for both Irish and International organisations, allowing them to route IP traffic efficiently thereby providing faster, more reliable and lower-latency internet access for their customers.

So what the hell is an IXP? Well, Euro-IX commissioned the following, the Internet Revealed: A file about IXPs, a couple of years ago which brilliantly explains IXPs.

Follow Up – IPv6 Statistics at INEX

A couple of days ago, I was talking about World IPv6 day with some notes on the Irish context.

INEX is a neutral, industry-owned association, founded in 1996, that provides IP peering facilities for its members. INEX membership is open to all organisations that can benefit from peering their IP traffic and there are currently 57 members.

INEX can also be considered Ireland’s IP Peering Hub. INEX membership provides high-speed, reliable and resilient IP traffic exchange facilities for both Irish and International organisations, allowing them to route IP traffic efficiently thereby providing faster, more reliable and lower-latency internet access for their customers.

As a follow up to the previous post, here’s a like for like comparison of IPv4 and IPv6 traffic over peering LAN 1 of the exchange:

Notes:

  • As a layer 2 exchange, traffic over INEX is symmetrical – traffic originating from one member is destined for another.
  • INEX runs two peering LANs for resiliency. The IPv6 traffic on LAN 2 was negligible over the same period. See the public statistics and the weathermaps of each LAN showing the network topology.

 

World IPv6 Day with Irish Statistics

In case it passed you by, today was World IPv6 Day. In a nutshell: “Major Internet service providers (ISPs), home networking equipment manufacturers, and web companies around the world are coming together to permanently enable IPv6 for their products and services by 6 June 2012.” This includes top content providers such as Facebook (see under their hood), Google (read what they had to say), Yahoo! and Microsoft. In fact, you may not even have noticed but Google were advertising it front and centre on their search page:

Google Announcing World IPv6 Day on Their Search Page

Over at INEX, we were unable to pull out IPv6 traffic statistics on the exchange until recently and my colleague just got the first pass of that project complete this week in time for World IPv6 Day. Here’s how it looked over the hours leading up to and into World IPv6 Day:

Now, the peek of almost 40Mbps is, most assuredly, small compared to the overall peek of 24Gbps, but there is a very pronounced jump in IPv6 traffic which is certainly a good sign and a move in the right direction. The overall peering statistics at INEX are public and we’ll be breaking out IPv4 and IPv6 into separate graphs shortly also.

Why does IPv6 amount to < 0.2% of the traffic at the exchange? Well there are two main factors:

  • Until today, there has been very little mass or popular content available over IPv6. So, even if you were IPv6 enabled, there was very little for you access.
  • None of the large ISPs in Ireland are providing IPv6 connectivity to end users outside of certain closed test programs.

This is the classic chicken and egg problem: with no content available the ISPs were not motivated to provide IPv6 connectivity; and, conversely, with no IPv6 enabled eyeballs the content providers were not motivated to make their services available over IPv6.

While today was not necessarily a content provider only day, I’m unaware of any Irish ISPs that got involved. But, now that we have significant content available over IPv6, hopefully the ISPs will begin to ramp up their own programs. And – to be fair – it’s not all bad news with the ISPs in Ireland. Most have their core and edge networks IPv6 enabled, it’s the access layer that’s the issue (and it’s a really really big issue and a very difficult issue).

AMS-IX (the Amsterdam Internet Exchange) is in the top three IXPs in the world by traffic volume and they also make their IPv6 statistics public. As a second demonstration of traffic levels on World IPv6 Day, here is the week to date showing a huge differential for today:

If you’re not sure what all this is about, well then here are a few words from the creator of the Internet himself:

And if you’re keen to start experimenting with IPv6, first email and ask your ISP. They’ll say no, but do it anyway! Then head over to SixXS (and be sure to choose either HEAnet or Digiweb as your PoP as both are INEX members and as such you’ll have the lowest possible latency).

Some New Nagios Plugins

Over the past ten years I have left many many new and hacked Nagios plugins on many servers around the globe. I’m now making a concerted effort to find them, clean them, maintain them centrally and release them.

To that end, I have created a repository on GitHub for the task with a detailed readme file:

As a starting point, there are four plugins available now:

  • check_chassis_cisco.pl – a script to poll a Cisco switch or router and check if the device was recently rebooted; its temperature sensors; its fans; its PSU; its CPU utilisation; and its memory usage.

 

  • check_chassis_server.pl – a script to poll a Linux / BSD server and check its load average; memory and swap usage; and if it has been recently rebooted.

 

  • check_portsecurity.pl – a script to check all ports on a Cisco switch and issues a critical alert if port security has been triggered resulting in a shutdown port on the device.

 

  • check_portstatus.pl – a script which will issue warnings if the port status on any Ethernet (by default) port on a Cisco switch has changed within the last hour (by default). I.e. a port up or a port down event.

Benchmarking the Mikrotik Routerboards RB750 and RB750G

Continuing on from today’s earlier post, Benchmarking the Mikrotik Routerboard RB1100, I now present some results for the RB750 and RB750G using the same methodology and platform.

The RB750 and the RB750G are two identical looking routers intended for the SOHO environment:

The specifications for the RB750 (with differences for the RB750G in italics and parenthesis) are:

  • five FastEthernet 100Mbps (Gigabit 1Gbps) ports;
  • 32MB DDR SDRAM ;
  • 64MB on board NAND storage;
  • Atheros AR7240 400MHz (AR7161 680MHz) CPU;
  • powered by PoE or power jack;
  • up to 3W (6W) power consumption;
  • ports 2-5 share dedicated switch chip allowing full 100Mbps (1Gbps) throughput;
  • all ports can be individually configured.
  • €31.73 (€54.61) from Wireless Connect.

Both routers come with an L4 license of Mikritik’s RouterOS which is built on the Linux kernel so anyone familiar with Linux networking will get up to speed on these boxes in no time.

As a disclaimer in case it is not clear, all routing tests are done using just two ports – one for the traffic generator and one for the receiver – with the device under testing routing the packets between two networks. As such, on the RB750, the maximum throughput we could achieve would be 100Mbps.

I ran tests for plain routing and also, in evaluating it for certain uses, over a VPN tunnel.

All results are presented below. Given the wealth of features, I think these are super boxes at a super price. So far I’ve put them on the end of an Imagine DSL line providing IPv4 and v6 over PPPoE and the end of a 30Mb UPC line taking its UPC IP via DHCP. They provide firewall, NAT, port forwarding, OpenVPN tunnels, QoS, DHCP, DNS caching and VLANs for phone / VoIP and managment networks.

 

 

 

Benchmarking the Mikrotik Routerboard RB1100

I attended and gave a talk at the recent Irish Wireless Conf & Expo on behalf of INEX. I don’t get to do much with wireless links and as such I found many of the talks and exhibitors very interesting. One company that had a large presence through both Wireless Connect in Dublin and Irish Wireless in Shannon was Mikrotik – a company manufacturing routers built on Linux and some kit that I had been meaning to look at for some time.

Following the conference I picked up some RB750’s and RB750G’s and was very impressed. So much so, that I picked up a RB1100 also. The RB1100 specifications include:

  • 13 individual 1Gbps ports;
  • 2 x 5 port switch groups;
  • 800MHz Power PC MPC8544E processor;
  • SODIMM RAM slot with up to 1.5GB RAM;
  • 1 x microSD card slot;
  • 1U rack mount case.

I decided to benchmark this to see at just what rate it could route packets.

Benchmark Methodology and Tests

I used two PCs running Linux with iperf to measure TCP throughout with different packet sizes. To establish a baseline, I ran the same tests with the two PCs directly connected (this is the Direct Connection results below). The maximum achievable result with this is 1Gbps.

An example command line for the test which runs for 10 secs by default and for a packet size of 64 bytes is:

iperf -f m -i 1 -c 10.0.0.1 -l 64

Then I ran four test sets routing traffic between two networks as follows:

  1. No c/t, no f/w: connection tracking disabled and firewall set to allow all;
  2. No c/t, f/w: connection tracking disabled but with some simple firewall rules;
  3. C/t, no f/w: connection tracking enabled but firewall set to allow all;
  4. C/t, f/w: connection tracking enabled and stateful firewall rules.

In addition, I ran the above four tests with the RB1100 configured as a OpenVPN server:

/interface ovpn-server serverset auth=sha1,md5 certificate=cert1 \
cipher=blowfish128,aes128,aes192,aes256                          \
default-profile=your_profile enabled=yes                         \
keepalive-timeout=disabled mac-address=FE:50:A7:D5:FE:B7         \
max-mtu=1500 mode=ip netmask=24 port=1194                        \
require-client-certificate=no

One of the PCs was connected to the RB1100 as a VPN client pushing traffic to the other server on a non-VPN connect with all traffic routed through the RB1100. I also did a baseline test by running the VPN server with the same encryption on one of the PCs with a direct connect to the other and then pushing traffic over the VPN link.

Results:

The results can be seen in the following graph:

Without connection tracking and firewall, full line rate is achievable for packet sizes of 256bytes and higher – all in all, an excellent result. That said, no connection tracking and no firewall would be an unusual configuration and with these, the box maxes out at around 525Mbps – still an excellent result for less than €400.

The VPN tests yielded:

VPN throughput primarily relies on CPU horse power and the PCs used for the Direct Connection baseline test are pretty modern.

We’re IPv6 Ready! Are you?

IPv6 ReadyOver in INEX, we just launched a new initiative to promote and increase awareness of IPv6 among content owners and businesses generating revenue from an online presence.

This project is called IPv6 Ready and it is essential a certification program for websites that are IPv6 ready to one of two standards:

Gold: The website has a AAAA (IPv6) DNS record; and

Platinum: At least one of the websites DNS name servers is additionally IPv6 enabled.

IPv6 PendingFor those websites that are not IPv6 enabled (and in many cases this is dependent on a third party hosting company), we also have a very cool IPv6 Pending badge which you can use to let your customers know that you are IPv6 aware.

The badges shown here are the large versions but we also have an extra large, medium and small so you’ll find an appropriate one for your site.

How do you get your badges? Easy, just head over to IPv6Ready.ie and register your site. Once you complete the simple process, you’ll be emailed all four personalised badges!

Help us make this a success! Please repost, blog, tweet and spread the word any way you can to help us raise awareness and push IPv6 forward – even just a little. If nothing else, please register and display a badge! You’ll also get a link such as this to your own certificate!

Querying for DNS Glue Records (using dig)

On a project I’m working on, I need to establish if a domain has IPv6 glue records or not. If I had to do it on a once off, a whois lookup would answer that nicely:

$ /usr/bin/whois opensolutions.ie
<snip>
nserver:     dns1.dns.opensolutions.ie 87.232.1.40 2a01:268:4::40
nserver:     dns2.dns.opensolutions.ie 87.232.1.41 2a01:268:4::41
nserver:     dns3.dns.opensolutions.ie 87.232.16.61 2a01:268:3002::61

However, in this case, I will need to do it many times on many domains and do not need to have to worry about whois servers limiting the queries or parsing the output from different whois servers.

After some digging, it looks like the nameservers of TLDs return glue records in the additional section. Let’s look by example on opensolutions.ie. First, find the TLD servers for .ie:

$ dig NS ie
<snip>
;; ANSWER SECTION:
ie.                     172800  IN      NS      gns1.domainregistry.ie.
ie.                     172800  IN      NS      uucp-gw-1.pa.dec.com.
ie.                     172800  IN      NS      uucp-gw-2.pa.dec.com.
ie.                     172800  IN      NS      ns3.ns.esat.net.
ie.                     172800  IN      NS      banba.domainregistry.ie.
ie.                     172800  IN      NS      ice.netsource.ie.
ie.                     172800  IN      NS      gns2.domainregistry.ie.
ie.                     172800  IN      NS      ns-ie.nic.fr.
ie.                     172800  IN      NS      b.iedr.ie.

Now query one of these for the nameservers for opensolutions.ie:

$ dig NS opensolutions.ie @banba.domainregistry.ie.
<snip>
;; AUTHORITY SECTION:
opensolutions.ie.       172800  IN      NS      dns3.dns.opensolutions.ie.
opensolutions.ie.       172800  IN      NS      dns2.dns.opensolutions.ie.
opensolutions.ie.       172800  IN      NS      dns1.dns.opensolutions.ie.

;; ADDITIONAL SECTION:
dns1.dns.opensolutions.ie. 172800 IN    A       87.232.1.40
dns1.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:4::40
dns2.dns.opensolutions.ie. 172800 IN    A       87.232.1.41
dns2.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:4::41
dns3.dns.opensolutions.ie. 172800 IN    A       87.232.16.61
dns3.dns.opensolutions.ie. 172800 IN    AAAA    2a01:268:3002::61

As you can see, the authority section contains the nameservers for opensolutions.ie which are all on the opensolutions.ie domain. We then find the glue records for these nameservers in the additional section.