21 August 2013, 6:04

Hi,

We are running Mahara 1.4.6 at the moment, and I've been experiencing some problems with user notification errors.

It appears that when the site fails to send a notification to a user (for some reason), it tries to send an error message to admin(s). This has seemed to be failing, and then it tries to send an error message about that failure - this process then appears to be looping, and I have been receiving floods of several hundred emails at a time - each time, the initial message will be around 15KB in size, while the later ones can be over 100KB.:

***   ***   ***   ***   ***

You have been sent a notification from e-Portfolio@YSJ. Message follows:

------------------------------------------------------------------------

Subject: User notification error was probably caused by your server

configuration.

exception 'EmailException' with message 'Couldn't send email to Admin User

(admin) with subject e-Portfolio@***: User notification error was probably

caused by your server configuration.. Error from phpmailer was: The following

From address failed: noreply@eportfolio.***.ac.uk<p>SMTP server error: 4.3.1

Insufficient system resources

</p>

' in /www/mahara/htdocs/lib/user.php:457

Stack trace:

#0 /www/mahara/htdocs/notification/email/lib.php(97):

email_user(Object(stdClass), NULL, 'e-Portfolio@YSJ...', 'You have been s...',

NULL, NULL)

...<snip>

***   ***   ***   ***   ***

We have experienced this intermittently for a while now, but today it has been particularly bad (at one point, I was receiving 20+MB of mail in a few minutes). However, other admin users don't appear to be affected to the same degree (one other admin was receiving a smaller number of such messages, two others didn't report receiving any), even though our notification settings appear to be identical  in /account/activity/preferences/.

• Ubuntu server running php 5.3.5 and Apache 2.2.17
• Mahara sends mail via an SMTP relay on another server. I tested this from the server, and was able to access it (and send mail) using telnet on port 25,
• Despite the "Insufficient system resources" message, vmstat showed little processor activity, and over 200MB of free memory when I checked immediately after receiving a batch of these messages.
• However, netstat showed over 300 connections to the relay in "TIME_WAIT" status at this point.

Have any of you seen behaviour like this before? I would be grateful if you could suggest anything that helps us pinpoint the cause, and resolve the problem.

Thanks.

21 August 2013, 7:46

Hi Rob,

We've had this / a similar issue a while ago on a site where an admin user had chosen the email digest as delivery method. Can you please check if one of your admins has that? You can see all admins and their notification settings at /admin/users/notifications.php

Cheers

Kristina

21 August 2013, 21:29

Thanks for the reply, Kristina. I've checked the admin notification settings, and types are set to "Email" for all admins. The only difference I can see is that all admins have an institution set, apart from the the original one called "Admin User":

Wondering whether this could be causing problems.

22 August 2013, 7:49

Hello Rob,

The institution shouldn't matter. I'll see what I can find out.

Cheers

Kristina

22 August 2013, 18:25

Hi Rob,

I think you're correct about the nature of this. Mahara tries to send out a notification, fails for some reason, then its response is to immediately try to send out a notification to the relevant admins about that failure. This can indeed lead to a vicious circle of lots of notifications getting generated.

Would it be possible for you to show us a copy of the full text of one of the initial notifications? In your original post, you've just got the top bit, but since it appends more warning text to the top with each cycle, the actual initial error will be buried at the bottom.

One workaround you could do, is change these admins so that they receive their "System message" notifications to "Inbox" rather than Email or Email Digest. If it's a problem with the email system for whatever reason, that may solve it by just not sending an email.

Another possibility would be to patch Mahara so that it waits before sending out the notification. Currently (in 1.4.6 and later versions) Mahara sends out the "User notification error" message immediately upon the first notification error. However, it is also possible to make Mahara send this after a delay. Doing that might help break up the vicious loop.

You could achieve this by patching the file htdocs/mahara/lib/activity.php . In Mahara 1.4.6 look for the function "notify_user()" in the class "ActivityType". Around line 476 you'll see this line of code:

activity_occurred('maharamessage', $message);

Change it to this:

activity_occurred('maharamessage', $message, null, null, true);

This will cause the notifications to be queued in the database, and not be emailed out until the activity_process_queue cron task runs, which by default is once every 5 minutes.

Or I guess you could just comment that line out entirely.

Cheers,

Aaron