06 October 2009, 11:49

I'm having performance issues with a secondary school who are new to Mahara, but are mad keen on it. They're currently only using it with 1 year group, and want to role it out to the whole school.

The problem is when the year group (200 users) hits Mahara simultaneously (which happens every tuesday afternoon) the server starts to struggle.

The bottleneck is in the postgres database which becomes CPU bound. The processor is a quad core intel Q6600, so a fair amount of power. The size of the database on the disk is around 60MB so it seems the whole database lives in ram. I believe the postgres database is fairly well optimised according to all the optimisation resources I could find, but it seems to be only handling around 400-700 transactions per second which feels a bit on the low side.

Does anybody else have experience of this level of usage or greater? The mahara performance page talks about 30-40 simultaneous users which we're over by a factor of 5 with only 1 year group.

06 October 2009, 20:44

Hi - with that amount of information, it will be hard to say exactly what the issue is. Performance and scalability are hard problems to address unless you know everything about the system.

So far, I'm not aware of other systems that have come across a load as high as 30-40 concurrent requests a second (that you might be getting from your 200 users). With appropriate hardware, this shouldn't cause too much of a strain. But it requires experience and knowledge to configure servers to handle that kind of load.

There is a possibility that some parts of Mahara might be slower than necessary, either by making more queries than it should or some other way. Generally, we've fought hard to keep the number of queries low - in particular, the number of write queries. Mahara is definitely a lightweight when compared with Moodle, for example! On average, you might see 15 - 20 read queries per page, and no writes. The most expensive pages are Views, but even then they're not too much more.

Still, having said that, I wouldn't rule out that maybe some part of Mahara is partly responsible for the performance. It's hard to know though 

I would be happy to investigate your system and configure/optimise it so it can handle as much load as it should be able to - but that would have to be done by contracting Catalyst IT at our standard rates, as it will require dedicated time and effort. Feel free to send me an e-mail ([email protected]) if you want to persue this option.

Alternatively, if you have any specific questions, I (and others) can probably answer them in the forums. Although that requires knowing the correct questions to ask  

07 October 2009, 11:05

Gosh, so does that mean we're one of the biggest Mahara users in terms of concurrent users?

What I'm trying to work out is whether this performance problem can be solved by tweaking the database to get it to handle more transactions per second, tweaking mahara to keep the load away from the database or whether this can only be solved by more hardware.

 I've got a fair amount of moodle performance experience, and have applied the same techniques to mahara (eaccelerator, lighttpd, various postgres config settings), but I'm wondering if there's something I'm missing.

Does mahara support memcached, and if so does it have a significant effect?

Will email offline about contracting.

07 October 2009, 18:56

You might well be one of the biggest in terms of concurrent users. Although concurrent users isn't a particularly useful benchmark - # of PHP scripts run per second is much more useful 

I've sent you a reply to your e-mail. Looks like there's potentially several options open to you - though it's hard to say what will work/work best unless we can see the system under load.

Mahara doesn't support memcached for any kind of caching, no. 

07 October 2009, 1:37

Some things you can do straight away are:

- enable Mahara's performance collecting, by setting $cfg->perftolog or $cfg->perftofoot to true in config.php

- configure postgres to log slow queries (I usually set the threshold to 200 or 300 ms or so), and set up a script to parse the logs and generate top 10s. I wrote a very ugly perl script to parse the logs about 5 years ago and Nigel (or someone else at Catalyst, can't remember who), rewrote it recently to be much nicer: http://git.catalyst.net.nz/gw?p=pgtools.git;a=blob;f=scripts/pg-log-process.pl;h=83ebe6267a3f85249d4e8e0770ff2724cf341890;hb=refs/heads/pg-log-process-multidb

That might give you some more ideas about where the database is falling over.

Some tips for tuning postgres for Moodle are here:  http://moodle.org/mod/forum/discuss.php?d=68558 which may help as they are similar applications, even though Mahara is lighter, as Nigel says.

07 October 2009, 12:24

I've turned on performance collecting, so I'll see what happens there.

The problem doesn't seem to be slow queries, it seems to be simply the volume of queries, handling 400-700 transactions/sec when under load. From the investigation so far the database is small enough to sit entirely in RAM, and the database seems to be CPU bound (on a quad core CPU).

 I'm trying to understand if 400-700 transaction/sec is a respectable rate for a mahara postgres database, or whether I can reasonable expect to achieve a substantially higher transaction rate.

 I'd already applied most of the tuning tips in the referenced article, but there was a few I hadn't so I've applied those as well.