Author: Barry O'Donovan

Synchronising Microsoft Exchange to Another IMAP Server

This post is much less of a detailed how-to but rather some useful links. We were tasked with the job of sync’ing about 1,000 MS Exchange mailboxes to a Dovecot server. This needed to be done via an administrator account on the Exchange end as individual user passwords were not available.

The tool of choice for this is imapsync.  Unfortunately, there is not a single formula that will work for all as it can depend on the Exchange configuration and version as well as the use of domains on the Exchange and ActiveDirectory servers.

To help understand the various combinations of logins for Exchange, I found the following invaluable: Understanding login strings with POP3/IMAP.

Also invaluable is the imapsync FAQ – just search for mentions of Exchange.

In the end, the following worked for me (but your mileage will most definitely vary!):

./imapsync --host1 exchange-server 
    --user1 'domain/adminuser/user' --password1 'admin-password' 
    --authmech1 LOGIN 
    --host2 dovecot-server --user2 user@example.com 
    --password2 userpassword

One key element here is that when logging into Exchange as an individual user I had to use --authmech1 NTLM but if you use this auth method with the above user string, you will always end up logging into the admin’s mailbox, not the user’s. That, at least, was my experience.

MySQL 5.6 – Memcached / NoSQL Support and More

MySQL 5.6 has been released with some interesting new features and performance increases:

  • What’s New in MySQL 5.6
  • DBA and Developer Guide to MySQL 5.6
  • InnoDB Integration with memcached:MySQL 5.6 includes a NoSQL interface, using an integrated memcached daemon that can automatically store data and retrieve it from InnoDB tables, turning the MySQL server into a fast “key-value store” for single-row insert, update, or delete operations. You can still also access the same tables through SQL for convenience, complex queries, bulk operations, application compatibility, and other strengths of traditional database software.

    With this NoSQL interface, you use the familiar memcached API to speed up database operations, letting InnoDB handle memory caching using its buffer pool mechanism. Data modified through memcached operations such as ADD, SET, INCR are stored to disk, using the familiar InnoDB mechanisms such as change buffering, the doublewrite buffer, and crash recovery. The combination of memcached simplicity and InnoDB durability provides users with the best of both worlds.

  • Multi-threaded Slaves
  • Improved IPv6 Support – both in the bind to address option and the INET_ATON() function.
  • Replication improvements.

All in all, some nice new features. Especially the memcached integration.

That said, MariaDB seems to be making inroads on MySQL with some distributions considering a switch. Some interesting reading from that project includes:

Woot! We hit 50Gbps at INEX!

 

Monitoring LDAP – Example with Munin

Following up from my articles on Creating an LDAP Addressbook / Directory, then Securing LDAP with TLS / SSL and Multi-Master LDAP Replication; I’ll now look at monitoring LDAP with Munin as an immediate example and Nagios to follow.

First we need to enable monitoring on LDAP – execute:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: {2}back_monitor.la
EOF

after ensuring {2} is the appropriate next sequence for oldModuleLoad. You can check this my running:

ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=module{0},cn=config

Now create a user with access to the monitoring information:

cat <<EOF | ldapadd -H ldapi:/// -D cn=admin,dc=nodomain -w h.TDVyELBjm0g
dn: cn=monitor,dc=nodomain
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: monitor
description: LDAP monitor
userPassword: cA.5rMfzHw9vw
EOF

Lastly, configure the monitor database:

cat <<EOF | ldapadd -Y EXTERNAL -H ldapi:///
dn: olcDatabase={2}Monitor,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMonitorConfig
olcDatabase: {2}Monitor
olcAccess: {0}to dn.subtree="cn=Monitor" 
  by dn.base="cn=monitor,dc=nodomain" read by * none
EOF

The monitoring module should now be active and you can test with:

ldapsearch -D cn=monitor,dc=nodomain -w cA.5rMfzHw9vw -H ldapi:/// -b cn=Monitor

Configuring Munin

“Munin is a networked resource monitoring tool that can help analyze resource trends and “what just happened to kill our performance?” problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.”

On Ubuntu, you can install Munin and the required packages for LDAP monitoring with:

apt-get install munin-node libnet-ldap-perl

Then edit /etc/munin/plugin-conf.d/munin-node and add a section such as:

[slapd_*]
env.server 127.0.0.1
env.binddn cn=monitor,dc=nodomain
env.bindpw cA.5rMfzHw9vw

During the install, Munin may have detected OpenLDAP and added appropriate symlinks. If it didn’t, you can possibly do it from the output of:

munin-node-configure --suggest --shell

For me (Ubuntu 12.10), slapd showed up with an error Wrong amount of autoconf which I haven’t debugged. Instead I just created the symlinks manually:

ln -s /usr/share/munin/plugins/slapd_ slapd_statistics_bytes
ln -s /usr/share/munin/plugins/slapd_ slapd_statistics_pdu
ln -s /usr/share/munin/plugins/slapd_ slapd_statistics_referrals
ln -s /usr/share/munin/plugins/slapd_ slapd_operations_diff
ln -s /usr/share/munin/plugins/slapd_ slapd_statistics_entries
ln -s /usr/share/munin/plugins/slapd_ slapd_connections
ln -s /usr/share/munin/plugins/slapd_ slapd_waiters
ln -s /usr/share/munin/plugins/slapd_ slapd_operations

And restart Munin:

service munin-node restart

Adventures with LDAP (OpenLDAP) – SSL, Multi-Master Replication and Monitoring

In my career to date, I successfully managed to avoid all but the periphery engagement in OpenLDAP. Until recently that is – we had to build a Microsoft Exchange like environment with open source software in a way that was closely integrated and easily managed. But, more on that another time. For anyone else diving into OpenLDAP, here are some articles on my experiences that I have penned:

Multi-Master LDAP Replication

Following up from my articles on Creating an LDAP Addressbook / Directory and then Securing LDAP with TLS / SSL, I’ll now focus on multi-master replication. Actually, this example will focus on master-master but it can easily be extended out to multi-master.

If you’ve been reading the other articles, then some caveats and differences apply here:

  • if you plan to set up replication, I recommend you do it from the beginning which is what this article looks at;
  • in the Addressbook articale, we created a new dedicated database for the addressbook. Herein however, I replicate the default database. I’ll explain how to replicate any given database below too.

For your environment, ensure you have DNS names registered or that you are using named hosts defined in the /etc/hosts file. For our case, let’s assume we have a hosts file entry as follows:

10.20.30.40    ldap1
10.20.30.41    ldap2

and, for each of the two hosts, we have respectively included the following in the SLAPD_SERVICES variable in /etc/defaults/slapd of each host (change for ldap2):

SLAPD_SERVICES="ldap://ldap1/ ...."

I’m going to write each of the following LDIFs as commands you can copy and paste.

We’re going to start by setting server IDs, loading the syncprov module and creating a user for syncing the config database. On ldap1:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: cn=config
changetype: modify
add: olcServerID
olcServerID: 1
EOF

Repeat above on ldap2 but change the server ID to 2. Then, on both:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: {1}syncprov.la
EOF

On the above, ensure {1} is the next available module sequence by running the following first:

ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=module{0},cn=config

Now, again on both servers:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcRootPW
olcRootPW: h.TDVyELBjm0g
EOF

We now need to update the server IDs and those of our peers. So, on both servers, run:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: cn=config
changetype: modify
replace: olcServerID
olcServerID: 1 ldap://ldap1/
olcServerID: 2 ldap://ldap2/
EOF

To get the replication running for the config database, we run the following on both servers:

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
EOF
cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001 provider=ldap://ldap1/ binddn="cn=config" 
  bindmethod=simple credentials=h.TDVyELBjm0g 
  searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=002 provider=ldap://ldap2/ binddn="cn=config" 
  bindmethod=simple credentials=h.TDVyELBjm0g 
  searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE
EOF

You now have 2-way master-master replication of the configuration database. Make sure you check the logs for any issues and you can easily test by changing a config option on first, verifying on the second, reverting on the second and verifying again on the first.

We can now replicate any other database by using similar changes to the above. Let’s say we want to replicate the database olcDatabase={1}hdb,cn=config, then execute the following on one server – remember, your configuration is now replicated!

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcLimits
olcLimits: dn.exact="cn=admin,dc=nodomain" time.soft=unlimited 
  time.hard=unlimited size.soft=unlimited size.hard=unlimited
-
add: olcSyncRepl
olcSyncRepl: rid=004 provider=ldap://ldap1/ binddn="cn=admin,dc=nodomain" 
  bindmethod=simple credentials=O4PbIOzA9gvEQ searchbase="dc=nodomain" 
  type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncRepl: rid=005 provider=ldap://ldap2/ binddn="cn=admin,dc=nodomain" 
  bindmethod=simple credentials=O4PbIOzA9gvEQ searchbase="dc=nodomain" 
  type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1
-
add: olcDbIndex
olcDbIndex: entryUUID  eq
-
add: olcDbIndex
olcDbIndex: entryCSN  eq
-
add: olcMirrorMode
olcMirrorMode: TRUE
EOF

NB: ensure you change the admin user and password above as appropriate for your database. Specifically, it should be the olcRootDN and oldRootPW as listed in the olcDatabase={1}hdb,cn=config object. Finally, execute the following on one server.

cat <<EOF | ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
EOF

References

Securing LDAP with TLS / SSL

This is a continuation of a previous post, Creating an LDAP Addressbook / Directory where we add SSL encryption to the directory.

In our case, we used a signed Unified Communications Certificate (UCC) (also known as a Subject Alternative Names (SAN) Certificate) from GoDaddy. The following will work for those as well as standard signed certificates. I have not tested with wildcard certificates. If you want to use a self-signed certificate, see the TLS and SSL section of Ubuntu’s OpenLDAP documentation as well as notes at the end of this document.

GoDaddy (or any other signing authority) will, when presented with a CSR (Certificate Signing Request), return a signed certificate as well as their own CA cert. You will already have your private key which you used to generate the CSR. With this information, prepare a file called tls.ldif with (for example):

dn: cn=config
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/gd_bundle.crt
-
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/webmail.opensolutions.ie.crt
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/webmail.opensolutions.ie.key

And apply the change via:

ldapmodify -Y EXTERNAL -H ldapi:/// -f tls.ldif

On Ubuntu (you own distribution may vary here), you need to add the SSL service by editing /etc/default/slapd and updating the SLAPD_SERVICES line to read:

SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"

and then restart the server (/etc/init.d/slapd restart). You should now consider firewalling the standard port (389) to force users to use the encrypted SSL port.

Following our example with Thunderbird, you can now update your LDAP directory configuration by setting the hostname to match the subject name in your UCC / certificate (e.g. abook.opensolutions.ie) and the port to 636.

Notes for Self Signed Certificates

If you are using a self-signed certificate, you need to ensure a couple of things. Let’s assume you created a self-signed certificate for abook.opensolutions.ie. Clients need a special configuration parameter for untrusted / self-signed certificates. Copy your self-signed certificate (e.g. /etc/ssl/webmail.opensolutions.ie.crt above) to the client machine(s) – say /etc/ssl/certs/abook.crt.

Now, on the client machine, add the following line to /etc/ldap/ldap.conf:

TLS_CACERT /etc/ssl/certs/abook.crt

Secondly, the hostname you use to access the LDAP server must also match the certificate subject name – i.e. use abook.opensolutions.ie in this example rather than an IP address / alternative hostname.

Creating an LDAP Addressbook / Directory

This article will describe my experiences in creating a read-only LDAP address book (with Thunderbird as a proof of concept); also known as a corporate directory. This is written by someone who has (to put it mildly) hated LDAP for years and dies a little every time he reads an introduction to LDAP that describes it in terms of DNS.

There is one important point to make before we start – while these instructions should apply to any *nix distribution, it uses OpenLDAP/slapd version 2.4 which uses the newer runtime dynamic configuration engine. All of the below performed on Ubuntu 12.10.

Installing OpenLDAP is as easy as (root user is assumed in all of the following):

apt-get install slapd ldap-utils

As part of this process, you’ll be asked to enter an admin password – record this as it will be stored in hashed format.

You can immediately run some LDAP queries to test / get to know your system:

  • Dump your entire configuration:
ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=config
  • List all configuration objects:
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=config dn
  • List all installed schemas:
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=schema,cn=config dn

In the output from the last command, you’ll see core, cosine, nis and inetorgperson. These are all we need for an address book directory. If you are so inclined, there is a published but neglected schema for Thunderbird specifically but it is not a standard and those fields may not (and probably will not) be supported by other clients.

One thing you might want to do before you start is up the logging level (from none by default) as follows. Don’t forget to change it back when you’re up and running as your logs will fill up fast.

cat <ldapmodify -Y EXTERNAL -H ldapi:///
dn: cn=config
changetype: modify
replace: olcLogLevel
olcLogLevel: 296
EOF

The installation will have created an organisation object based on your domain (or nodomain). E.g.

 ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b dc=nodomain
dn: dc=nodomain
objectClass: top
objectClass: dcObject
objectClass: organization
o: nodomain
dc: nodomain

dn: cn=admin,dc=nodomain
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator

You can find out what domain yours is under by examining the olcSuffix field of the output of:

ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b olcDatabase={1}hdb,cn=config

You may want to modify this to suit or add new objects. We’re going to add new objects – which will work fine as long as the new olcSuffix does not conflict with the output from the above.

Let’s start with creating a database for our directory. First we need a directory on the filesystem:

mkdir -p /var/lib/ldap/opensolutions
chown openldap: /var/lib/ldap/opensolutions

Now create a file (say db-create.ldif) with something like:

# Database creation
dn: olcDatabase=hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: hdb
olcSuffix: dc=opensolutions,dc=ie
olcDbDirectory: /var/lib/ldap/opensolutions
olcRootDN: cn=admin,dc=opensolutions,dc=ie
olcRootPW: gOeBTo5vfBdUs
olcDbConfig: set_cachesize 0 2097152 0
olcDbConfig: set_lk_max_objects 1500
olcDbConfig: set_lk_max_locks 1500
olcDbConfig: set_lk_max_lockers 1500
olcDbIndex: cn,sn,uid,mail pres,eq,approx,sub
olcDbIndex: objectClass eq
olcLastMod: TRUE
olcDbCheckpoint: 512 30
olcAccess: to attrs=userPassword 
  by dn="cn=ldapadmin,dc=opensolutions,dc=ie" write 
  by anonymous auth 
  by self write 
  by * none
olcAccess: to attrs=shadowLastChange 
  by self write 
  by * read
olcAccess: to dn.base="" by * read
olcAccess: to * 
  by dn="cn=admin,dc=opensolutions,dc=ie" write 
  by * read

And instruct LDAP to create the database:

ldapadd -Y EXTERNAL -H ldapi:/// -f db-create.ldif

The above creates a new LDAP database with some useful indexes. You can ignore the olcAccess for now as we’ll come back and address this later. What is above is fairly typically of a default installation.

Now, we need to add an organization object, an admin user to manage that (i.e. add, edit and remove entries from the corporate database) and an organisationalUnit object to hold our staff information. Create a file (say opensolutions.ldif) containing:

# Organisation object
dn: dc=opensolutions,dc=ie
dc: opensolutions
description: Open Solutions Corporate Directory
objectClass: top
objectClass: dcObject
objectClass: organization
o: Open Source Solutions Limited

# Admin user
dn: cn=ldapadmin,dc=opensolutions,dc=ie
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: ldapadmin
description: Corporate Directory Administrator
userPassword: Jh90Ckb.c.Tp6

# Unit for our corporate directory
dn: ou=people,dc=opensolutions,dc=ie
ou: people
description: All people in Open Solutions
objectclass: organizationalUnit

And add these objects to the database:

ldapadd -x -D cn=admin,dc=opensolutions,dc=ie -W -f opensolutions.ldif

Note the password is as specified in the database creation object (ie. gOeBTo5vfBdUs in this case).

A quick work on security and access control. By default, anonymous users / anyone can read all your entries. If you are publishing a public directory, this may be okay. If not, create and auth.ldif file with (for example):

dn: olcDatabase={2}hdb,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by dn="cn=ldapadmin,dc=opensolutions,dc=ie" write
  by self read
  by anonymous auth
  by * none
olcAccess: {1}to dn.subtree="dc=opensolutions,dc=ie"
  by dn="cn=ldapadmin,dc=opensolutions,dc=ie" write
  by users read

And apply it with:

ldapmodify -Y EXTERNAL -H ldapi:/// -f auth.ldif

This will:

  • allow access to user password fields for authentication purposes (not for reading);
  • allow any authenticated user to read the corporate directory;
  • allow the ldap admin to make changes;
  • deny all other access to this database (implicit rule).

See OpenLDAP’s Access Control page for more information.

Now, let’s add two sample entries. Create a file people.ldif with:

dn: cn=Barry O'Donovan,ou=people,dc=opensolutions,dc=ie
objectClass: inetOrgPerson
uid: barryo
sn: O'Donovan
givenName: Barry
cn: Barry O'Donovan
cn: barry odonovan
displayName: Barry O'Donovan
userPassword: testpw123
mail: sample-email@opensolutions.ie
mail: sample-email@barryodonovan.com
o: Open Solutions
mobile: +353 86 123 456
title: Chief Packet Pusher
initials: BOD
carlicense: HISCAR 123
ou: Computer Services

dn: Joe Bloggs,ou=people,dc=opensolutions,dc=ie
objectClass: inetOrgPerson
uid: joeb
sn: Bloggs
givenName: Joe
cn: Joe Bloggs
displayName: Joe Bloggs
userPassword: testpw124
mail: joeb@opensolutions.ie
o: Open Solutions
title: Chief Coffee Maker
ou: Kitchen

Add these to the directory using the ldapadmin user:

ldapadd -x -D "cn=ldapadmin,dc=opensolutions,dc=ie" -w Jh90Ckb.c.Tp6 -f people.ldif

You can test this with a couple of searches:

ldapsearch -xLLL -D "cn=Barry O'Donovan,ou=people,dc=opensolutions,dc=ie" -w testpw123 \
    -b dc=opensolutions,dc=ie
ldapsearch -xLLL -D "cn=Barry O'Donovan,ou=people,dc=opensolutions,dc=ie" -w testpw123 \
    -b ou=people,dc=opensolutions,dc=ie mail=sample-email@opensolutions.ie

slapd will listen on all interfaces on the standard port (389) when installed on Ubuntu. So, to test, we turn to Thunderbird:

  1. Open the address book (e.g. Tools -> Address Book)
  2. Add a new directory (File -> New -> LDAP Directory…)
  3. In the General tab (assuming we’re setting up Barry O’Donovan’s Thunderbird), set:

Name: Corporate Directory (whatever you like)
Hostname: 127.0.0.1 (or as appropriate)
Base DN: ou=people,dc=opensolutions,dc=ie
Port number: 389
Bind DN: cn=Barry O’Donovan,ou=people,dc=opensolutions,dc=ie

  1. In the Advanced tab:

Don’t return more than 100 results – change if you wish

Scope: Subtree

Login method: Simple

  1. Click okay to save the settings
  2. Right click on the directory in the left pane and select Properties
  3. Test by going to the Offline tab and click Download Now
  4. Enter your password (and use the password manager) – password is as per the person object above and so in this case: testpw123
  5. Test by typing Joe into the search bar on the top right
  6. Joe Bloggs should appear in the results.

Congratulations! You have a corporate directory.

Next Steps

References

For OpenVPN Fans – Optimising for GE Networks

I came across this article today which discusses OpenVPN optimisations for Gigebit Ethernet networks.

http://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

It also shows the phenomenal improvement that can be made thanks to the AES-NI instruction set on newer Intel and AMD chips.

Git Web Applications (aka GitHub Alternatives)

I’ve been through the mill with a few of these and it’s a growing list of options. I should start by saying that I love GitHub and use it regularly for a large number of projects (including some of my own, some for my company and some for my customers). The only problem I have with GitHub is that I just haven’t made the jump to trust it with our proprietary code – the primary asset of the company. Particularly after a high profile security breech.

Now, nothing I’ve come across yet comes close to GitHub. But two we use daily have good matching features:

  • Gitorious –  whether they intended it or not, it’s a good clone of GitHub but always a little behind on features. It’s also a Ruby on Rails application and the documentation is getting far better. When we started, it was pretty… shite, to be honest. Installing and upgrading was a pain. They have addressed this with http://getgitorious.com/ which includes a virtual appliance for VirtualBox so you can be up and running in minutes.
  • Atlassian Stash – if you’re a small team (<= 10 users) then this could be for you. Installation is pretty easy, it looks great, is very fast (over HTTP/S, but it’s inbuilt SSH client is extremely slow) and is very easy to use. Pull requests are also done really really well. It’s a full Java webapp so it does require some CPU and memory. For <= 10 users, my definite favorite. Thereafter though it gets expensive – e.g. for up to 25 users you’ll fork out $1,800 / annum. That’s not entirely unaffordable but if you’re using Stash, then you may also be using their other products such as Jira, Confluence, Bamboo, Fisheye and Crucible. Now that adds up to a hefty bill!

One’s I have not tested and so cannot speak authoritively on include:

  • GitLab – Looks like a very interesting alternative.
  • RhodeCode – they say it will “change the way you manage your code”.
  • Gitolite – a quick look at the available information for this definitely puts it in the also ran category. There’s no polish or anything nice to entice one to even try it.
  • Gitosis – in their own words: “Manage git repositories, provide access to them over SSH, with tight access control and not needing shell accounts”. The project appears dead.