We are now close to releasing IXP Manager v7, and it has become apparent that the single documentation site is suboptimal and confusing for people who have not upgraded but are reading about newer features. The best solution for this is to introduce major.minor versioning on the documentation site.
Material supports versioning via another tool called mike. For a new site, this is quite easy to set up and run, but we had to do the following for an existing site.
Quick side note: we are using a Python venv for mkdocs, mike, etc. and not installing system wide. One thing to watch out for is that mike calls mkdocs but this may not be in your path. This can be solved by updating the PATH environment variable or running mike as follows:
PATH=./venv/bin:$PATH mike deploy --push --update-aliases 7.0 dev
Install mike per its and Material’s documentation.
Delete the old documentation site: mike delete --all
Deploy your current docs as the latest version: mike deploy --push --update-aliases 6.4 latest
Deploy the docs you are working on for the upcoming version as the dev tag: mike deploy --push --update-aliases 7.0 dev
Set the default docs: mike set-default --push latest
We were using a custom CNAME for Github pages and a custom 404 page. These would have been removed with step (2) above but are both available in your documentation’s Githib repo under the gh-pages branch history. To restore these you need to place them back in the root folder of the gh-pages branch:
I was delighted to help celebrate Euro-IX’s 20th anniversary today by giving a talk on Our Community and IXP Manager. My goal was to show that collaboration is a two-way street with IXP Manager by taking a look at how our community has helped the project and reflect on how the project helps our community.
I was delighted to help celebrate Euro-IX’s 20th anniversary today by giving a talk on Our Community and IXP Manager. My goal was to show that cooperation is a two-way street with IXP Manager by taking a look at how our community has helped the project and reflect on how the project helps our community. The event was of course online and you can find full details here: Euro-IX – 20 Years of Cooperation.
Euro-IX is my favourite event in our industry and I attended my first one on behalf of INEX when it was in Rome in 2009, accompanying the late Barry Rhodes. Since then, I’ve attended at least one of the two every year.
I began with a brief update on the current status of the IXP Manager project. An image I and the team at INEX take great pride in sharing is the following world view with 166 green dots – each dot representing an IXP that uses IXP Manager.
We, INEX and the IXP Manager project, launched a new website in June 2020. This site has a database backend that stores the 166 IXPs noted above and generates the above image dynamically. On a nightly basis, it also polls these IXPs in one of two ways:
If the IX-F Member Export is enabled and publically available, we will pull the current data from this. This is the case for 98 of the 166 and this data is real-time and accurate.
For the rest, we’ll use PeeringDB which requires networks (members of IXPs) to assert that they peer at an exchange and so these numbers will be a significant undercount.
The resulting stats, as of 27th June 2021, were presented next:
Now we’re getting into the meat of the presentation by looking at some of the things that the Euro-IX – and wider – community give IXP Manager. The first tangible item is community contributions which mostly cover the costs of a full-time developer. This program is currently in its fifth year. Rather than repeating what I said, please see this page on the website for details.
The next slide was about knowledge. This is possibly an under-discussed and under-valued aspect of Euro-IX in general. Knowledge is a very valuable commodity in most industries that is aggressively protected with NDAs, patents, and contracts. Yet, at Euro-IX forums, the operators of IXPs come together to learn; to create knowledge; and to share knowledge – openly and freely. The best of this community knowledge gets distilled into IXP Manager which, as we always say, teaches and implements best practices – and these practices have in part come from this creation and sharing.
One of the things I’m most proud of in our new IXP Manager website is the contributor wall of fame. Anyone who makes at least five contributions to the project through the GitHub platform will appear here which, as of today’s date, has just over 40 people. Any of the following will count towards a contribution:
Possibly the hardest: contribute code through a pull request.
Open an issue (bug report or feature request).
Simply comment on an existing issue (or your own!).
Contribute editorial fixes/content to the documentation.
This section was rounded out with a shortlist of other sources of community collaboration. The first was the coffee breaks between sessions and the conversations over drinks at the socials at Euro-IX forums. These relaxed environments allowed for more long-form free-flowing discussions with IXP Manager users to help flesh out potential features and to understand where the problems, grit, and rough edges exist. The information learned from these conversations then feeds into later versions of IXP Manager. I miss these face-to-face engagements and, hopefully, we can get back to them soon.
Finally, I wanted to give a specific mention to the open-source community in general. Before I was ever involved in IXPs, INEX, or IXP Manager, I had (and continue to have) an involvement in other open-source projects. For IXP Manager, the FOSS community has given us wide-ranging tools from the PHP language itself to Laravel – the web application framework we use, and right through to front-end libraries such as TailwindCSS and jQuery.
Of course, when we mention open-source in the IXP community, we have to give a nod to Bird, the internet routing daemon, which is the powerhouse of so many of our route server implementations. Hopefully, we’ll also find the time in 2021 to add OpenBGPd support to IXP Manager.
IXP Manager – Giving to Our Community
The is the other side of the two-way street and I wanted to touch on some of the things we hope IXP Manager gives our community. This isn’t about new releases, or functionality or features. And it’s not about being boastful or giving ourselves a pat on the back. It’s aspirational – it’s about what we think – what we hope – IXP Manager gives the community. It’s also not something we can claim success for – it’s up to the community to decide if we’ve succeeded here.
This first element of this was our mission statement – our vision / hope for the project and why we open-sourced it:
Our vision for IXP Manager and the basis for making it a free-to-use, open-source project was that it might enable the creation of IXPs where they are required.
The existence of these IXPs would, in turn, create a stronger, open, more robust and better-connected internet.
This very much goes back to the kind of people involved in IXPs. For most of us, the notion of for the good of the internet is part of our DNA.
The next thing I hope IXP Manager gives the exchanges that use it is independence. The independence to build a local community and form the best and most appropriate community-led IXP for the region in which it exists. This is also an anti-colonialism / anti-imperialism measure as we’ve seen a number of the larger IXPs establish IXPs well outside their original countries and regions. To my mind, in most cases, this is not the best way to build IXPs and usually leads to transactional IXPs rather than communities.
I also like to think of IXPs as facilitators – if not even guardians to an extent – of net neutrality. When you have for-profit IXPs then the business motive is wealth creation. That is the very definition of for-profit. When this is your motivation then the decisions you make may not necessarily be to the benefit of all your members – especially the weaker / smaller members.
Something I’m fairly satisfied that IXP Manager does give new IXPs is the best possible start. As well as a good leg-up for smaller IXPs that move to the platform. The concepts of secure by design and best current practices are baked into what IXP Manager does. Of course, an IXP will always need staff with network engineering and system admin skills – but we hope that the steep learning curve is very much flattened by the use of IXP Manager. To the extent that the challenges in starting a new IXP should no longer be technical but rather environmental: regional political and regulatory issues, commercial issues such as funding and data centre contracts, and finding members.
Finally, we hope IXP Manager gives new IXPs credibility. The reason (so they’ve told us) that Facebook, Amazon AWS and Netflix (historically) have supported IXP Manager through patronage is that they see the value in coming to a new exchange that has installed and is making good use of IXP Manager. These content networks know that much of the technical elements will be “done right”.
There’s also a reverse angle to this – reputation protection for the rest of us. There are only a few hundred active IXPs worldwide and we’ve all worked really hard to build confidence in our abilities – most as community-led not-for-profit enterprises – to deliver secure, reliable services. It only takes a small few bad actors (such as cowboys looking for a quick profit or inexperienced operators) to sow the seeds of doubt. I hope IXP Manager helps put a large insurance blanket over the rest of us by helping these new IXPs start out in the best possible way.
Epilogue
Our Euro-IX community not only supports IXP Manager. We all support each other – through the forums, mailing lists and virtual meet-ups. That we are all so willing to share information and experiences so openly is the wonderful thing about Euro-IX. Not just an association but a community. It’s a community I’m proud to be a part of and call so many of its members by friends. Here’s to the next 20 years!
Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.
In case you missed it, we have a new modern website for IXP Manager – find it at https://www.ixpmanager.org/. One of the features of this new website is that we now gather IXP Manager usage statistics on a daily basis – including the distribution of versions in use.
Reassuringly, of the installs we can poll for version used, ~65% are using one of the latest three minor versions (5.5.0 – 5.7.0). This is reassuring for a number of reasons including: knowing that IXPs stay current; knowing that IXPs are concerned about security updates; and knowing that the upgrade process is not especially difficult.
Of the 145 installs we know about, we can poll 116 and collect the version is use which yields the following table:
5.x.y
83
4.x.y
26
3.x.y
7
Distribution of major IXP Manager versions in use as at September 8th 2020.
Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.
Community IX Atlanta (CIX-ATL) are in the process of upgrading from 4.9.3 to the latest (5.7.0) and they graciously allowed me to record the process:
The video is a real-life experience where it wasn’t planned in advance allowing the viewer to see the mistakes and thought processes throughout. Also, if you weren’t aware of it, we have an on-going series of IXP Manager tutorials here.
When considering a legacy upgrade, there are two main approaches:
Build a new IXP Manager installation on a new (modern) server and migrate the database (this is what we’ve done here).
Attempt an in place upgrade alternating between IXP Manager upgrades and operating system upgrades. This is probably more awkward with more scope for issues to crop up (especially on non-IXP Manager applications which may be on the same server).
Remember, what’s covered here is “just” the IXP Manager and database upgrade. There’s a bunch of other things that would also need to be done including:
Working through the various upgrade actions in the release notes (mentioned throughout the video). Essentially you’ll need to step through each set of release notes for the versions you cycle through (and jump over).
If building a new server, pointing elements such as route server cron jobs and other API consumers at the new server.
Migrating other applications from the legacy server (e.g. maybe you have mrtg co-installed there).
In a production environment, my goal would be to build the new IXP Manager installation with the copied and upgraded database and run them in parallel. NB: either avoid or duplicate changes made in the UI across both installations of IXP Manager for this period of time.
Once the new installation of IXP Manager is ready for production use, you will then step through all external tools that consume data from it (sflow, mrtg, route servers, route collectors, etc.) and migrate them to the new installation. Sometimes simply updating DNS can achieve most of this but you’ll probably want to take it piece-meal and ensure each external service works as expected.
Take particular care with essential services such as route servers. This is an opportune time to upgrade to Bird v2 and add RPKI. What we did at INEX was do one route server at a time with 1-2 weeks between upgrades. This allowed time to ensure the new system was stable and also to ensure no member issues due to RPKI filtering, etc. (spoiler alert: it was uneventful!).
As you complete the migration, you can also consider if some services should be left on the “old” server. Separating tasks between different servers is good practice and so ask youself if everything should be migrated over to the new server.
More than anything, I hope this video entices you to keep current with your IXP Manager installations!
We call IXP Manager’s statistics and graphing architecture Grapher. It’s a backend agnostic way to collect and present data. Out of the box, we support MRTG for standard interface graphs, sflow for peer to peer and per-protocol graphs, and Smokeping for latency/packet loss graphs. You can see some of this in action on INEX’s public statistics section.
Internet Exchange Points (IXPs) play a significant role in national internet infrastructures and IXP Manager is used in nearly 100 of these IXPs worldwide. In the last couple weeks we have got a number of queries from those IXPs asking for suggestions on how they can extract traffic data to address queries from their national Governments, regulators, media and members. We just published our own analysis of this for traffic over INEX here.
Grapher has a basic API interface (documented here) which we use to help those IXP Manager users address the queries they are getting. What we have provided to date are mostly quick rough-and-ready solutions but we will pull all these together over the weeks (and months) to come to see which of them might be useful permanent features in IXP Manager.
How to Use These Examples
The code snippets below are expected to be placed in a PHP file in the base directory of your IXP Manager installation (e.g. /srv/ixpmanager) and executed on the command line (e.g. php myscript.php).
Each of these scripts need the following header which is not included below for brevity:
We’ve placed a working API endpoint for INEX above – change this for your own IXP / scenario.
Data Volume Growth
An IXP was asked by their largest national newspaper to provide daily statistics of traffic growth due to COVID-19. For historical reasons linked to MRTG graph images, the periods in IXP Manager for this data is such that: day is last 33.3 hours; week is last 8.33 days; month is last 33.33 days; and year is last 366 days.
This is fine within IXP Manager when comparing averages and maximums as we are always comparing like with like. But if we’re looking to sum up the data exchanged in a proper 24hr day then we need to process this differently. For that we use the following loop:
$start = new Carbon('2020-01-01 00:00:00');
$bits = 0;
$last = $data[0][0];
$startu = $start->format('U');
$end = $start->copy()->addDay()->format('U');
foreach( $data as $d ) {
// if the row is before our start time, skip
if( $d[0] < $startu ) { $last = $d[0]; continue; }
if( $d[0] > $end ) {
// if the row is for the next day break out and print the data
echo $start->format('Y-m-d') . ','
. $bits/8 / 1024/1024/1024/1024 . "\n";
// and reset for next day
$bits = $d[1] * ($d[0] - $last);
$startu = $start->addDay()->format('U');
$end = $start->copy()->addDay()->format('U');
} else {
$bits += $d[1] * ($d[0] - $last);
}
$last = $d[0];
}
The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in TBs via 8/1024/1024/1024/1024). This can, for example, be pasted into Excel to create a simple graph:
The elements of the $d[] array mirror what you would expect to find in a MRTG log file (but the data unit represents the API request – e.g. bits/sec, pkts/sec, etc.):
d[0] – the UNIX timestamp of the data sample.
$d[1] and $d[2] – the average incoming and outgoing transfer rate in bits per second. This is valid for the time between the $d[0] value of the current entry and the $d[0] value of the previous entry. For an IXP where traffic is exchanged, we expect to see $d[1] roughly the same as $d[2].
$d[3] and $d[4] – the maximum incoming and outgoing transfer rate in bits per second for the current interval. This is calculated from all the updates which have occured in the current interval. If the current interval is 1 hour, and updates have occured every 5 minutes, it will be the biggest 5 minute transfer rate seen during the hour.
Traffic Peaks
The above snippet uses the average traffic values and the time between samples to calculate the overall volume of traffic exchanged. If you just want to know the traffic peaks in bits/sec on a daily basis, you can do something like this:
The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in Gbps via 1000/1000/1000). This can also be pasted into Excel to create a simple graph:
Import to Carbon / Graphite / Grafana
Something that is on our development list for IXP Manager is to integrate Graphite as a Grapher backend. Using this stack, we could create much more visually appealing graphs as well as time-shift comparisons. In fact this is how we created the graphs for this article on INEX’s website which includes graphs such as:
To create this, we need to get the data into Carbon (Graphite’s time-series database). Carbon accepts data via UDP so we used a script of the form:
The Carbon / Graphite / Grafana stack is quite complex so unless you are familiar with it, this option for graphing could prove difficult. To get up and running quickly, we used the docker-grafana-graphite Docker image. Beware that the default graphite/storage-schemas.conf in this image limits data retention to only 7 days.
We’ve just released IXP Manager v5.3.0. The headline feature in this release is two-factor authentication (2fa) and user session management. This blog post overviews the PHP elements on how we did that.
While IXP Manager is a Laravel framework application, it uses Doctrine ORM as its database layer via the Laravel Doctrine bridge. For those curious, this really is a carry over from when IXP Manager was a Zend Framework application. For the migration, we concentrated on the controller and view elements of the MVC stack leaving the model layer on Doctrine. Over time we’ll probably migrate the model layer over to Laravel’s Eloquent.
Before reading on, it would be useful to first read the official documentation we have written aroud 2fa and user session management:
Hopefully the how we did this will be useful for anyone else in the same boat or even just trying to understand the Laravel authentication stack.
Two factor authentication (2fa) strengthens access security by requiring two methods (also referred to as factors) to verify your identity. Two factor authentication protects against phishing, social engineering and password brute force attacks and secures your logins from attackers exploiting weak or stolen credentials.
User session management allows a user to be logged in and remembered from multiple browsers / devices and to manage those sessions from within IXP Manager.
For 2fa, we used the antonioribeiro/google2fa-laravel package which is built on antonioribeiro/google2fa. If we were 100% in Laravel’s eco-system the would have been easier but because we use Doctrine, we needed to override a number of classes.
Structurally we need a database table to indicate if a user has 2fa enabled and to hold their 2fa secret – for this we created Entities\User2FA. Similarly, we have a controller to handle the UI interaction of enabling, configuring and disabling 2fa: User2FAController – this also includes generating QR codes for the typical 2fa activation process.
On the user session management side, we created Entities\UserRememberToken to hold multiple tokens per user (rather than Laravel’s default single token in a column in the user’s user database entry. For the frontend UI, UserRememberTokenController allows a user to view their active sessions and invalidate (delete) them if required.
The actual mechanism of enforcing 2fa is via middleware: IXP\Http\Middleware\Google2FA. This is added, as appropriate, to web routes via the RouteServiceProvider. This will check the user’s session and if 2fa is enabled but has not been completed, then the middleware will enforce 2fa before granting access to any routes covered by it.
Note that because we also implemented user session management via long-lived cookies and because the fact that a user has passed 2fa or not is held in the session, we need to persistently store the fact in the user’s specific remember token database entry. This is done via the Google2FALoginSucceeded listener. This is then later checked in the SessionGuard – where, if we log a user in via the long-lived cookie, we also make them as having passed 2fa if so set.
Speaking of the SessionGuard, this was one of the bigger changes we had to make – we overrode the Illuminate\Auth\SessionGuard as we needed to replace a few functions to make 2fa and user session management work. We have kept these to a minimum:
The user() function – Laravel’s long lived session uses a single token but we require a token per device / browser. We also need to side-step 2fa for existing sessions as discussed above and allow for features such as allowing a user to delete other long-lived sessions and to provide functionality to allow these sessions to expire.
The above constitutes a bulk to the changes. Because 2fa can be enforced via middleware, it doesn’t really touch the core Laravel authentication process. The user session management was more invasive and responsible for the bulk of the changes required in the DoctrineUserProvider and SessionGuard.
Ubuntu 18.04 ships with PHP 7.2 by default but there are various reasons why you may wish to upgrade to newer versions. For example, active support for it ends later this year – far sooner than the 2023 support window for the OS.
In addition, applications will be released that will require newer versions in that 2018 – 2023 window. For IXP Manager, we are releasing v5 this month and mandating PHP 7.3 support. We do this to stay current and to prevent developer apathy – insisting on legacy frameworks and packages that have been EOL’d provides a major stumbling block for bringing on new developers and contributors. There’s also a real opportunity cost – I have a couple free hours, will I work on project A or project B? If project A uses an old stale toolchain where everything is that much more awkward that project B then which would you choose?
So, from a typical LAMP stack install of Ubuntu 18.04, you’ll find something like the following packages for PHP:
root@ubuntu:/var/www/html# dpkg -l | grep php | cut - -b 1-65
ii libapache2-mod-php 1:7.2+60ubuntu1
ii libapache2-mod-php7.2 7.2.17-0ubuntu0.18.04.1
ii php-common 1:60ubuntu1
ii php-mysql 1:7.2+60ubuntu1
ii php7.2-cli 7.2.17-0ubuntu0.18.04.1
ii php7.2-common 7.2.17-0ubuntu0.18.04.1
ii php7.2-json 7.2.17-0ubuntu0.18.04.1
ii php7.2-mysql 7.2.17-0ubuntu0.18.04.1
ii php7.2-opcache 7.2.17-0ubuntu0.18.04.1
ii php7.2-readline 7.2.17-0ubuntu0.18.04.1
Obviously your exact list will vary depending on what you installed. I find the easiest way to upgrade is to start by removing all installed PHP packages. Based on the above:
The goto place for current versions of PHP on Ubuntu is OndÅ™ej Surý’s PPA (Personal Package Archive). OndÅ™ej maintains this in his own time so don’t be afraid to tip him here.
It’s easy to add this to 18.04 as follows:
add-apt-repository ppa:ondrej/php
apt-get update
Then install the PHP 7.3 packages you want / need. For example we can just take the package removal line above and install the 7.3 equivalents with:
Copy of an article I wrote on INEX’s own blog for longevity – original published here on April 10 2019.
In this article, we talk about the new route servers that we deployed across all three peering platforms at INEX during February 2019 and, particularly, RPKI support.
Most route server instances at internet exchanges (IXPs) perform prefix filtering based on route/route6 objects published by internet routing registries (IRRDBs). INEX members would be used to creating these through RIPE’s database. However there are many other registries and the data quality of some of these IRRDB objects is often poor, with problems relating to missing, stale and incorrectly duplicated information.
A typical IRRDB entry would resemble the following:
RPKI is a public key infrastructure framework designed to secure the internet’s routing infrastructure in a way that replaces IRRs with a database where trust is assigned by the resource holder. The equivalent of a route object in RPKI is called a ROA (Route Origin Authorisation). It is a cryptographically secure triplet of information which represents a route, the AS that originates it and the maximum prefix length that may be advertised. An example of an IPv4 and an IPv6 ROA would be:
ROAs are typically created through your own RIR (so, RIPE for most INEX members). These RIRs are called trust anchors in RPKI. RIPE have created an extremely easy wizard for creating ROAs through the LIR Portal.
To implement RPKI in a router, the router needs to build and maintain a table of verified ROAs from the five RIRs/trust anchors. The easiest way of doing this is to use a local cache server which pulls and validates the ROAs from the trust anchors and uses a new protocol called RPKI-RTR to feed that information to routers. Currently there are three validators: RIPE’s RPKI Validator v3; Routinator 3000 from NLnetLabs; and Cloudflare’s GoRTR. INEX currently uses the former two.
RPKI validation of a route against the table of ROAs yields one of three possible results:
VALID: a ROA exists for the route and both the prefix length is within the allowed range and the origin ASN matches.
INVALID: a ROA exists for the route but either (or both) the prefix length is outside the allowed range and/or the origin ASN is different.
UNKNOWN: no ROA exists for the route.
UNKNOWN is a common response as the database has only a fraction of the prefix coverage as IRR databases do. We are now in a multi-year transition from IRR to RPKI route validation while ROAs are created.
Bird V2
As well as RPKI support, we have also upgrading all route servers to Bird v2.
This is a significant rewrite to Bird which, for v1, maintained separate code and daemons for IPv4 and IPv6. Bird v2 merges these code bases and also introduces support for new SAFIs such as l3vpns / mpls.
Overall, the configuration changes required were minimal and INEX continues to run separate daemons of Bird v2 for IPv4 and IPv6 daemons. Route servers are CPU intensive and separate daemons allows for maximum stability, keeps the configuration clean and fits into the existing deployment processes we have built up with IXP Manager.
Route Server Filtering Flow
Our work on the new route servers will be released to the community as part of IXP Manager v5 shortly. The new filtering flow is enumerated below. One of the key new features is that if any route fails a step, we use internal large community tagging to indicate this and the specific reason to our members through the IXP Manager looking glass (more on that later).
Filter small prefixes (>/24 for IPv4, >/48 for IPv6).
Filter martian / bogon ranges.
Sanity check to ensure the AS path has at least one ASN and no more than 64.
Sanity check to ensure the peer ASN is the same as first ASN in the prefix’s AS path.
Prevent next-hop hijacking (where a member advertises a route but puts the next hop as another member’s router rather than their own). We do allow same-AS’s to specify their other router(s).
Ensure that the origin AS is in set of ASNs from member’s AS-SET. See below for some additional detail on this.
RPKI validation. If it is RPKI VALID, accept the route. If it is RPKI INVALID then filter it.
If the route is RPKI UNKNOWN, revert to standard IRRDB filtering.
Regarding step 7 above, an AS-SET is another type of IRRDB database entry where a network which also acts as a transit provider for other networks can enumerate the AS numbers of those downstream networks. This is something RPKI does not yet support but it is being worked on – see AS-Cones.
Lastly we have enhanced the BGP large community support to allow our members request as-path prepends on announcements to specific members over the route servers. For these and more supported communities, see the INEX route server page here.
Bird’s Eye and the Looking Glass
As well as IXP Manager, INEX has also written and open sourced a secure micro service for querying Bird daemons called Bird’s Eye. IXP Manager uses this to provide a web-based looking glass into our route collectors and servers. We have recently released v1.2.1 of Bird’s Eye which adds support for Bird v2.
We have greatly enhanced IXP Manager’s looking glass to support both Bird v2 and the large communities we use to tag filtered reasons. You can explore any of INEX’s route servers to see this yourself – for example this is route server #1 for IPv4 on INEX LAN1. When members log into IXP Manager they will also find a new Filtered Prefixes tool which will summarise any filtered routes across all 12 of INEX’s route server instances.
More Information
We have spoken about this at a number of conferences recently:
I gave a talk on migrating legacy web applications to Laravel at last year’s Laravel Live UK conference in London. Those that rated it gave it high marks which is always brilliant feedback to receive.
2018 was Laravel Live UK’s inaugural conference and it was a packed house. They’ve just announced the dates for 2019 and I would strongly recommend attending for anyone using or interested in use Laravel.
Following the conference, I wrote up the talk as an article and it has just been published in the March 2019 edition of php[architect]. This is an excellent magazine which I’ve subscribed to for a few years now – the digital edition is very reasonable and comes as a DRM-free PDF onto an app on your phone/pad or downloaded to your computer.
As it happens, they chose this article as the teaser for this issue and so it is freely available online here and downloadable as a PDF here. But seriously, if you are a PHP developer, you need to subscribe to this magazine.
Lastly, if you are interested in the slide deck from the conference, you can download them here – but the article is a much better way to understand this material.
We use standard PHPUnit tests for IXP Manager for some mission critical aspects. These take data from a test database filled with known sample data (representing a range of different member configurations). The tests then use this information to generate configurations and compare these against known-good configurations.
This way, we know that for a given set of data, we will get a predictable output so long as we haven’t accidentally broken anything in development.
But, as an end user, how do you know that what you stick in a web-based form gets put into the database correctly? And conversely, how do you know that form represents the database data correctly when editing?
To test the frontend, we turn to Laravel Dusk – an expressive, easy-to-use browser automation and testing API. What this actually means is that we can right code like this:
We have now added Dusk tests for UI elements that involve adding, editing and deleting all aspects of a member interface and all aspects of adding, editing and delete a router object. Here’s an example of the latter:
Doctrine2 ORM is a fantastic and powerful object relational mapper (ORM) for PHP. We use it for IXP Manager to great effect and we only support MySQL so our hands are not tied to pure Doctrine2 DQL supported functions. We also use the excellent Laravel Doctrine project with the Berberlei extensions.
Sometimes time is against you as a developer and the documentation (and StackOverflow!) lacks the obvious solutions you need and you end up solving what could be a single elegant query very inefficiently in code with iterative database queries. Yuck.
I spent a bit of time last night trying to unravel one very bad example of this where the solution would require DQL that could:
group / concatenate multiple column results from a one-to-many relationship;
join a table without a relationship;
ensure the joining of the table without the relationship would not exclude results where the joint table had no matches;
provide a default value for (3).
Each of these was solved as follows:
via MySQL’s GROUP_CONCAT() aggregator. The specific example here is that when a MAC address associated with a virtual interface can be visible in multiple switch ports. We want to present the switch ports to the user and GROUP_CONCAT() allows us to aggregate these as a comma separated concatenated string (e.g. "Ethernet1,Ethernet8,Ethernet9").
Normally with Doctrine2, all relationships would be well-defined with foreign keys. This is not always practical and sometimes we need to join tables on the result of some equality test. We can do this using a DQL construct such as: JOIN Entities\OUI o WITH SUBSTRING( m.mac, 1, 6 ) = o.oui.
This is as simple as ensuring you LEFT JOIN.
The COALESCE() function is used for this: COALESCE( o.organisation, 'Unknown' ) AS organisation.
We have not yet pushed the updated code into IXP Manager mainline but the above referenced function / code is not replaced with the DQL query:
SELECT m.mac AS mac, vi.id AS viid, m.id AS id,
m.firstseen AS firstseen, m.lastseen AS lastseen,
c.id AS customerid, c.abbreviatedName AS customer,
s.name AS switchname,
GROUP_CONCAT( sp.name ) AS switchport,
GROUP_CONCAT( DISTINCT ipv4.address ) AS ip4,
GROUP_CONCAT( DISTINCT ipv6.address ) AS ip6,
COALESCE( o.organisation, 'Unknown' ) AS organisation
FROM Entities\\MACAddress m
JOIN m.VirtualInterface vi
JOIN vi.VlanInterfaces vli
LEFT JOIN vli.IPv4Address ipv4
LEFT JOIN vli.IPv6Address ipv6
JOIN vi.Customer c
LEFT JOIN vi.PhysicalInterfaces pi
LEFT JOIN pi.SwitchPort sp
LEFT JOIN sp.Switcher s
LEFT JOIN Entities\\OUI o WITH SUBSTRING( m.mac, 1, 6 ) = o.oui
GROUP BY m.mac, vi.id, m.id, m.firstseen, m.lastseen,
c.id, c.abbreviatedName, s.name, o.organisation
ORDER BY c.abbreviatedName ASC
