09 September 2015, 15:00

Hm, well there are two main code changes we made to Mahara's database libraries between 1.10 and 15.04:

- https://reviews.mahara.org/#/c/4525/ (Changing column cacheing in insert_record)

- https://bugs.launchpad.net/mahara/+bug/1384473 (upgrading ADODB to 5.19)

You could try reverting one of those and see if it fixes the problem. Of the two, I'd say that the ADODB upgrade is more likely to be the cause of this problem.

If anyone can provide a set of precise instructions to replicate the error, that would also be very helpful in fixing this issue.

Cheers,

Aaron

11 September 2015, 2:08

I'm not sure whether this is only an issue of Mahara 1504: When several students try to copy a view, and do this at the same time, they also get a deadlock-message (we use the most recent version of Mahara 1.10). We were able to reproduce this with 3 users (one was able to copy the view, the other 2 users got the error message.

11 September 2015, 6:57

Hi,

It is interesting to note that some folks are having a problem also with 1.10 not just 15.04.

Can I therefore ask if it is possible that the issue of notes and text boxes may be a contributing factor as I know these changed in 1.10?:

https://mahara.org/interaction/forum/topic.php?id=6500

On our upgrade from 1.9.3 to 15.04 many of of our text boxes seemed to end up as Notes not new basic Text boxes, even if they didn't have advanced use. My experience is that these Notes with attachment options and potential connections with other notes are slower to load because of other checks that are being done associated with the advanced options. Therefore it is likely that copying a collection with many Notes will put more of a load on resources than if the text boxes were the new basic text box.

I may be wrong but my theory is that although other activity at busy times may also lead to errors it is often the groups of students at the same time copying collections and pages with typically many Notes (previously Text boxes) that could be putting extra load on the system which, although not such a problem in 1.9 and before, coupled with the recent code changes to caching and writing to the database causes an error due to this excess load.

Could this be a factor?

15 September 2015, 2:13

unfortunately we have tried this patch and we are still seeing deadlocks

*** (1) TRANSACTION:

TRANSACTION 274999786, ACTIVE 1 sec inserting

mysql tables in use 1, locked 1

LOCK WAIT 9 lock struct(s), heap size 1184, 5 row lock(s), undo log entries 2

MySQL thread id 8431772, OS thread handle 0x7f715db02700, query id 393347638 xxxx.xxx.xxx.ac.uk 128.86.132.68 epxxxtest_xx update

INSERT INTO "ep_view_rows_columns" ("view", "row", "columns") VALUES (48546, 1, 3)

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 0 page no 1245707 n bits 672 index "ep_viewrowscolu_vie_ix" of table "epxxxtest"."ep_view_rows_columns" trx id 274999786 lock_mode X insert intention waiting

Can you suggest anything else?

15 September 2015, 19:16

I created a basic 20-user JMeter test that logs in, views the list of pages and copies one of those pages. 

With MySQL version 5.5.44, I can recreate the problem on Mahara versions 15.04, 1.10 and 1.9.  With enough users (and in many cases only a few) some of the page copies fail due to a deadlock problem in the database.  Deadlock errors similar to those posted earlier appear with all three versions.  These report the 'unrecoverable error' message to the end user.

This seems to be due to the way that MySQL is handling multiple transactions and updates to 'view'.  Probably due to the issues locking the index on 'view' table.

With PostgreSQL version 9.4, I've not managed to recreate the problem on version 15.04.  I can run JMeter tests with over 50 concurrent users copying pages and still can't trigger the problem.  Unfortunately I've not had chance to try this on earlier versions of Mahara on PostgreSQL.

Russell

16 September 2015, 0:21

On my JMter plan, on MySQL,  the error can be triggered with 2 or 3 users.

I've not been able to replicate it on Postgres however some one mentioned Postgres on the forum.

Sam

16 September 2015, 8:25

Hello Russell,

Would you be happy to share your J-Meter test scenarios / scripts? The core team is currently in the hot phase of preparing the release candidate for 15.10. Thus, not having to set up those tests ourselves would be much appreciated as that would speed up the testing process.

Thank you

Kristina

16 September 2015, 22:21

Hello Kristina,

I work with Russell; he suggested I share my test plan.

I couldn't post links to the files on the forum therefore I've sent you a direct message.

Best

Sam Moulem

17 September 2015, 8:08

Kristina,

Thanks for looking at the issue.  I asked Sam to pass along his JMeter test plan as it was a little more sophisticated than mine.  Please let us know if you have any questions.

Using the test plan you've received I've also managed to replicate the deadlock issue on the head of the master Mahara branch with MySQL 5.5.44 with only 5 users.

I've failed to replicate when I use PostgreSQL 9.3 on the same code (even with 100+ simulated users).

Thanks,

Russell

15 September 2015, 2:28

Hi, the following should replicate the error:

1. Log into Mahara

2. Click the 'Portfiolio' tab/button

3. Click the 'Copy a page' button

Choose a page to copy and 

4. click the 'copy page' button.

If a couple of users try to copy the page at the same time, you should get the error.