07 February 2011, 13:05

So I ran into an interesting issue bringing mahara into our environment.  Intermittently  auth from moodle to mahara would fail after trying for 20 to 40 seconds.  Initially I thought that our load balancer was not passing some off the traffic XMLRPC traffic.  This was not the case.

We have been using moodle for about 5 years and have now gotten to about 3000 active users.  We first used mahara about 3 or 4 years ago (with a pre 1.0 version).  It worked alright for the what the faculty wanted to do but their interest languished so we never updated (or looked in on it much).  This last september we cloned our moodle VM and put the two of them behind a cicso load balancer.  This had the effect of cloning our mahara instance as it was on the same server and causing an issue that do to lack of use we did not notice at the time.  

First let me explain how our load balance is setup.  Each VM has 2 interfaces.  One for inbound traffic and one for outbound traffic. The network configuration is necessary for it to pass client IPs to the backend and still receive the outbound traffic which would otherwise be routed directly back to the clients.

Before the load balancer we had a single VM running both moodle and mahara with a single interface.  Now we had 2 vm's running both and everything running through the load balancer.  So when moodle.e.e sent a user to mahara.e.e and mahara.e.e tried to verify the user it would xmlrcp back to moodle.e.e .  In looking at the trace route sometime times there would just be SYN and not ACK (and it would fail) or we would see immediate ACK's for each SYN.  When we pulled one of the VM's out of the load balancer it started failing consistently.  This lead us to the real issue.  The problem was not the load balancer.  

Mahara.e.e on VM1 verifies user on moodle.e.e on VM2 success

Mahara.e.e on VM1 verifies user on moodle.e.e on VM1 failure

The issue was that  VM1 saw traffic go out on interface eth0 but come back on eth1. This behavior indicates two interfaces plugged into each other and the kernel does the proper thing and just drops in on the floor and ignored it.

How we solved it.  On VM1 and VM2 we over road DNS by telling it that moodle.e.e and mahara.e.e were actually one interface.  This meant that mahara <-> moodle traffic never leaves the box but traffic from outside the box still goes though the load balancer.

Since I could not even figure out how to ask about this as I could not even get consistent behavior I thought I should post something.  Yes I realize that running mahara on a separate box would have solved this issue but until out use gets high enough adding an other VM did not make sense.

08 February 2011, 16:31

The issue was that  VM1 saw traffic go out on interface eth0 but come back on eth1. This behavior indicates two interfaces plugged into each other and the kernel does the proper thing and just drops in on the floor and ignored it.

If you are using Linux on your VMs, it may happen that you have 'reverse path filtering' [1] enabled on some or all of the interfaces.

Saludos.
Iñaki. 

[1] See http://www.tolaris.com/2009/07/13/disabling-reverse-path-filtering-in-complex-networks/ for additional details.

08 February 2011, 17:36

It seems like that would also work.  I posted this with two hopes. One was that if someone else ran into this they could find out a what was going on and secondly that someone could point out other solutions than the one we came up with.

Thanks