31 March 2015, 7:29

Hi everyone,

I work at Université de Montréal in Canada and we are about to integrate Mahara as the institutional platform for online Portfolio. The application will be used by an estimate of 20 000 to 30 000 students each semester. In the process, we are doing quality insurance at many levels that have all been successful until we started load testing the application... At best we have been able to get only an average of 25 transactions per second and response time peeks at 10 seconds. Load average of the web server gets pretty bad.

Here's the specs of the tested environment :

• Mahara version 1.10.2
• One web server that is a virtual machine (VMWare) with 4 CPU and 8 GB of ram, and running on Linux Red Hat 6 (64 bits). Apache version 2.2.15 and PHP version 5.4.16 with opcache. All modules are installed as described in the page Mahara requierements page.
• One MySQL 5.5.X database server that is a physical server with 24 CPU and 60 GB of ram (Runs Moodle platform too), running on Linux Red Hat 5 (64 bits)
• Load testing tool is JMeter

Here is the load testing scenario :

• Home page access
• Login with a local account
• Do one if those actions randomly :

1. Editing a page
2. Search for a user and send him a message
3. Send a message in a forum
4. View the page a user (x3)

• Logout

We used a profiler (xhprof) during the test to trace what part of the code is slow and we found two major issues :

• The SQL that retrieves all the data for the dashboard in "lib/view.php" (method View::view_search)
• The "crypt" function that hash the password when the user log in.

But even when temporarily disabling these functions, we still have at best 25 transactions per seconds. This time the profiler shows a lot of equally slow functions (No obvious bottleneck).

I know it is not the same application, but we are also using Moodle in the same installation/configuration and we are able to get more than 100 transactions per second and response time peeks at only 2 seconds with one web server.

Is there anyone else in the community who have also load test their production environment or having about the same amount of users ? If so, what is your feedback ? Are the performances better than what we have ? What kind of tuning were you able to do ?

We still have some tracks we want to explore but we're a bit in a dead end here...

Thx in advance for your help!

Jean-Philippe

31 March 2015, 9:24

Hi Jean-Philippe

We have also noticed via xhprof testing that View::view_search is a bottleneck function with a convoluted sql call.

We are wanting to simplify the sql call but haven't settled on the best way forward.

Some things that we have been doing to improve both this function and performance of Mahara in general can be found in these patches:

https://reviews.mahara.org/#/c/4272/
https://reviews.mahara.org/#/c/4338/

Also you can install the adodb php extension to help speed up sql calls.

Cheers

Robert

01 April 2015, 2:59

Hi Robert,

Thx for the the fast reponse! Like I said to Kristina, we already saw these ameliorations that we are about to test in the next days but it sure helps to see that we might be on the right path!

I'll let you know if any of these ameliorations are improving our performance.

I have tried to find the source code of adodb php extension but can't seem to see anything but a PHP library. Can you help on this ? Problem is that the php adodb extension is not packaged with PHP 5.4 on Red Hat 6 so I have to compile it manually...

Thanks a lot!

Jean-Philippe

01 April 2015, 13:43

I actually asked about the source code for the ADODB C Extension in the ADODB dev github recently, and they said that it's no longer being developed and probably doesn't improve performance. https://github.com/ADOdb/ADOdb/issues/84

But, I did find the source code, via the debian package: http://anonscm.debian.org/cgit/pkg-php/php-adodb.git

I also cloned the source code into my personal github repository: https://github.com/agwells/php-adodb

Testing it casually on my local computer, I didn't see any obvious performance improvement. But I'd be curious to see whether you get different results. There are still hooks for the C extension in the ADODB PHP library, so it's doing something.

Cheers,

Aaron

02 April 2015, 9:56

Thx for the feedback Aaron! I'll check on that very soon and let you what are the results, if I manage to make it work on Red Hat 6.

J-P

12 June 2015, 1:06

Hi J-P. I work in Computing Services at Carleton University and we're in the process of scoping out a similar sized version of Mahara for our Institution. We were wondering if there was any further update on the performance issues you encountered? If a more in-depth off-line conversation would be acceptable, please let me know.

Cheers,

Stephen

[email protected]

============================

16 June 2015, 1:29

Hi Steve,

Sorry for the late response. We finally found that the configuration of our JMeter test wasn't correct. 2 settings were wrong and made us though that the application had performance problems :

• We did not retrieve embedded resources related to the pages (CSS, JS files etc) so the apache accesses did not seem acceptable.
• We used the GUI interface when running the test and it gives us wrong response times because the injector wasn't able to keep up with the load.

Tricky thing was that it was the first time we were using JMeter for load testing (We used Grinder before) so we had to also learn how this complex tool works. Once this was fix, the results were as good as what we expected (Top load : 600 transactions per second and  average 2.5 seconds in response time).

I hope this information will be helpful! Let me know if you have any further questions.

Jean-Philippe

08 September 2015, 0:46

Hi,

Could you give us some info about how you did the test.

I'm trying to do some load testing with JMeter. The problem I have is simulating multiple user login. On Moodle I would create a test course from the site administration menu and create a test plan from the same menu. Moodle would create test users, set their authentication method accordingly and let me download a csv - with all the user,password - that I'd then feed to JMeter.

I don't seem to see such a thing in Mahara.

How did you make the test users, set their authentication method and the csv file (for JMeter), with the username,password info.

Did you also make the test plan by recording your web browser's action?

Any help will be appreciated.

Thanks

Sam

10 September 2015, 1:12

Hi Sam,

we actually write our own program to create all the users and the groups we needed. For the pages we use leap2a extraction to create pages with content. The program output a csv file in the format that we want. The program is a Mahara plugin that use the API of Mahara.

For JMeter, we record the browser actions and made some cleaning in the result. We also add some variables that come from the CSV file.

If you are interested we can share it with you. The plan was to make it public but we have never take the time to do it.

Steve

10 September 2015, 16:50

I'd love to see your data & plugin! Performance improvement is one of our top priorities in Mahara at present.

Cheers,

Aaron