27 May 2015, 23:57

Hi all,

We have some users who are doing some great group work and it's happening on a large scale. Now they're running into serious performance issues.

The issue is with listing group files and uploading of new files. Both are very slow. I think they're both part of the same issue, which is processing a large number of files in the filebrowser code.

At the moment the group has more than 600MB of files, but they are expecting this to increase to many GB.

I tried putting the files into folders and this did help a bit at the browser end, as there are fewer calls to download image icons. But the initial page load is still slow - I timed it at 19.5 seconds.

I've tried to do some performance profiling and I'm looking at the get_my_files_data() function in artefact/file/lib.php

This seems to be a heavy db query and I'm wondering if it can be optimised at all. Or should I be looking at database or PHP caching at the server end?

Any suggestions welcome!

28 May 2015, 1:35

Actually the Pieform, Dwoo and language string functions take longer than the db query but I don't know if it's practical to speed those up?

Mike

28 May 2015, 11:01

Hello Mike,

We are looking into replacing the file browser ideally for 15.10, but will see how involved it will be as it has its fingers in many places in Mahara. We are wanting to replace MochikitJS... One of our developers is currently investigating alternatives. If you happen to know a good open source one, please let us know.

Cheers

Kristina

28 May 2015, 21:20

Hi Kristina,

Once the group files page has loaded, I sometimes get a 'slow script' warning in the browser page, which is for the Mochikit code, probably while it is making the many files drag-and-drop-able.

I worked around that and the loading of all the file icon images by putting the files into folders. However the preparing of the page at the server end is still slow. 

Partly this looks to be down to just retrieving the artefacts from the database - because of the permissions in Mahara, which give it its great flexibility, a lot of checking has to be done against multiple db tables. That's why I was wondering about trying something like memcached. I wonder if any other users have experience of this.

In profiling it looks as though there could also be wins in looking at the templating system and the fetching of language strings. I will try to look at this but unfortunately I don't have much time :(

I can ask my users to split their cohorts into smaller groups, but there is also great energy in a big group, so it would be a shame in this case.

29 May 2015, 8:38

Hello Mike,

Might https://bugs.launchpad.net/mahara/+bug/785469/comments/1 help?

Cheers

Kristina

08 June 2015, 20:27

Hi Kristina,

I'll look into that, thanks.

24 June 2015, 12:31

I've actually moved the code for that memcache patch into the Catalyst Github account. https://github.com/catalyst/mahara/tree/mh1504-memcache

We've been using it in all of our Catalyst-hosted Mahara sites for years now.

Still meaning to upstream it, one of these days. It's just tricky because upstreaming it will mean adding the ability to choose which session handler you want.

Salient to this discussion though, is that we only use memcache as a session handler. So that wouldn't have any effect on the file permissions stuff you're talking about, because we don't store that in the session.

Cheers,

Aaron

28 May 2015, 14:40

Hello Mike,

I am curious: do you see performance issues when you have many smaller or fewer but larger files?

28 May 2015, 21:01

Hi Dirk,

I reproduced the slow performance on my test machine by simply uploading a lot of files (hundreds) to one of my groups. The slowness increased as I added more files, so I'm pretty sure it's due to the number of files. Some of the individual files are quite large, so I should test that aspect, but I don't expect that to be a factor. If it is I will update the thread with more info.