29 October 2011, 20:27

Hello,

I am looking for some pointers, documentation or experiences on how to best tweak a mahara 1.4 on MYSQL and Debian in a VMWare environment. We are using 1 quad core and 16 GB Ram. Due to VMWare licensing, we presently are unable to assign more cores.

Every month, on a particular day we have an excess of 700 concurrent users updating their portfolios. Most users will update 4 pages, maybe 100 or so will copy a collection at the same time. Staggering this process to help reduce server load will be difficult to implement, if not possible at all.

Although the server never crashes, we see Top at above 130 which is of course not ok and so the site is very slow. Judging from experience with other applications, I don't think adding more hardware will necessarily improve performance much. Bandwith does not seem to be the issue either.

Some things we do on our moodle is to load balance with four webservers, run a separate database server and utilize a apache caching program. The caching is working for mahara as well although we have not tested it under load.

Long story-short: Is there some essential tweaking one could do? is there something in Mahara that would prevent us from using the moodle approach? Would turning off 'Show online users' during heavy traffic help? Is there a program that simulates thousands of users for testing?

Thanks

30 October 2011, 16:15

Hi Dirk,

I can't think of anything specific in Mahara but I'd take a look at your server tuning and configuration.

I'd say look at what you already have as a start and see where you resources are going and what the box is waiting on. Have a look at what programs are actually using the output in top (use the < and > keys to change which column everything is sorted by).

Use something like vmstat to see what's constrained most (e.g. memory, io, etc):

vmstat -S M 1

Have you tuned everything adequately? Are you IO bound (the 'wa' or wait column), memory bound, or CPU bound ('us' or user)?

You say you're using an apache caching program? Do you mean a PHP accelerator like APC? If you're not using one, you should be. I'd go for APC over eAccelerator and I don't think that any of the others are up to date or maintained these days. APC is also available under Debian (php-apc package).

If you are using APC, have you looked at the output of the apc.php file it ships with (although not in Debian) - you'll need to download it from http://svn.php.net/viewvc/pecl/apc/branches/APC_3_0/apc.php?revision=259553&view=co - watch out for the 'Cache full count' being non-zero. If it is, then you've not given your cache enough Shared Memory. When the cache runs out of space, it empties it entirely which makes it pretty much useless. I think the default is only something stupid like 32MB.

On one of our production servers we have about 400MB assigned to APC. If you are increasing the shm_size, you will need to increase the kernel shmmax. The easiest way to do so in Debian is to open /etc/sysctl.conf and add a line for kernel.shmmax and run sysctl -p /etc/sysctl.conf. This is in bytes. You'll need to increase the kernel number before updating your apc configuration. Keep an eye on the hit/miss ratio too.

Similarly, I trust you've tuned MySQL? I can't recall the relevant settings in MySQL off the top of my head, but there's a MySQL log analyser which will tell you how well you're using your cache etc and what you'd need to do to improve things. From memory you want to have a look at the query planner tuning (that's what the log anaylsis will give you information on).

You probably could do with some more cores, and I doubt that you're actually using most of that 16GB RAM you've given it -- we're running a production moodle with over 600 daily unique users, plus a mahara, and a development moodle on a combination of a web server with 8GB RAM, and a database server (postgres) with 2. We've had no performance issues and at present we're barely using 2GB RAM of the available RAM on the frontend. We're also barely stressing any of the cores at present because we have everything tuned well.

It's also worth having a look at your Apache configuration - especially if you discover that you've got lots of clients queueing for apache processes. Check your MaxClients, and MaxRequestPerChild settings. Also check your MaxKeepAliveRequests and KeepAliveTimeout setting.

You may also find that your storage could be part of what's at fault - especially in a virtualised infrastructure, and especially when you're running your database and frontend on the same system. Bit hard to change after the fact, but avoid RAID1. More spindles is better for DB performance. RAID5 is good, RAID10 is far better IMHO. (RAID 0 is dangerous of course).

You say you've got 16GB RAM available to the VM, but are you actually running a 64-bit VM, or at least the Debian bigmem kernel? If not, aptitude install linux-image-686-bigmem and restart so you can actually address that memory. An individual process can still only use 4GB RAM still, but a group of processes can use more.

If VMWare is causing you issues, have you considered using a different virtualisation technology - e.g. VirtualBox, or Xen? That said, you could be using vSphere rather than workstation - I don't know your setup.

On a related note, if your dataroot is on an NFS share or similar storage, and you're seeing issues where users are finding that their session occasionally hangs, then you could be hittingi NFS locking issues. There's a bug in the tracker to address this, but there are workarounds if you're hitting it.

Putting my 'sales and consultancy' hat on, if you're still having issues, have you considered contacting one of the Mahara partners for further support? At LUNS, we've done quite a few specialist installations - our largest included 1,000 moodles with a total of 600,000 users; and 1 mahara (shared) with about 10,000 users if I recall. We've also provided similar consultancy services to a couple of UK universities and can provide references etc.

As a side note, I personally don't recommend MySQL (and not just for Mahara). It's nowhere near as flexible when it comes to tuning. If you're running on a SAN, then there are a number of parameters for Postgres which don't have good MySQL alternatives you can set -- SANs aren't so good for database performance as they introduce far too much latency. It's also not as fast as Postgres when they're both tuned.

Sorry for the long-winded reply, but please let me know how you get on!

Andrew

05 December 2011, 13:52

Hello Andrew and Francois:

First a big thank you for your support. Responses such as yours make it easy to use good open source projects.

We managed to get things working to a point where everyone is happy. To perhaps help others, here is what we did:

Our IT staff adjusted for sessions locked to the maharadatafolder by locking them to the webserver, of which there are four in our setup. We seem to be getting very good results as the other day over 500 concurred users were using the system and I did not get any phone calls after :)

One question that was raised is wether there could be a line in the config file to choose where sessions are held?

Thanks again.

05 December 2011, 17:10

Hopefully we'll be able to get around to fixing this bug soon:

  https://bugs.launchpad.net/mahara/+bug/785469

It would certainly help with setup like yours.

Cheers,

Francois

30 October 2011, 16:23

Oh, and there are quite a few benchmarking tools you can use:

• ab (apache benchmark) which ships with apache under Debian;
• JMeter - Can be used to simulate logins, actions etc. It's quite complicated and takes a bit of getting your head around but it can be very useful

My colleague was playing with another tool recently but I can't recall the name.

Andrew

01 November 2011, 2:39

Hi Dirk,

As you can see from the great list that Andrew made, there are quite a few things to look for.

The key here is to do a lot of measurements because you can spend a lot of time "optimising" your site, but unless you are working on the bottleneck that's making everything slow, then it's not going to help much.

From my experience tuning sites for clients, we usually find that the bottleneck is the database server because there's only one of those (while there are often multiple webservers). DB servers need lots of RAM but very importantly, they need very fast disk I/O. That last point pretty much means that you should avoid virtualising that part of your infrastructure because the overhead tends to kill performance. When shopping for DB servers at Catalyst, we usually go for big physical 64-bit servers.

Cheers,

Francois

01 November 2011, 4:27

Indeed,

I'd never choose to put a database server on any virtualised infrastructure for this very reason, nor would I choose to put the database data on any network storage or SAN. There's too much inherit latency (even if you can get incredibly fast raw read/write speeds).

Andrew

02 November 2011, 21:35

I appreciate the very valuable responses. Its good to know there is support available. Thank you.

As we work though this I will no doubt report back with questions. So far we have discovered that we are using 32bit and not 64 and Fedora and not Debian. We also looked at the Apache confiruration and MaxClients, MaxRequestPerChild, MaxKeepAliveRequests and KeepAliveTimeout settings. Yes, this sounds a little crazy, not knowing what we are running but we are a bit under staffed and have something like 40 +servers to manage.