03 January 2011, 21:18

Gang:

After finally getting my "access denied" messages to disappear by finally getting cron to execute properly, they're back. See this thread for my travails. I am now getting out of memory errors as Steve P mentions in that thread. I've upped the maximum memory in php.ini to 512 meg...

memory_limit = 512M

...and still the errors come....

[03-Jan-2011 21:10:17] [DBG] 15 (E:\www\inetwork\lib\cron.php:149) Running core cron rebuild_artefact_parent_cache_complete
[03-Jan-2011 21:11:46] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 6144 bytes) in E:\www\inetwork\lib\dml.php(46) : runtime-created function on line 1

512 meg seems very much larger than any of the googled postings I've found for this setting. I'm not a PHP/Apache guy, so I have no idea how high I can safely go without putting the install at risk. I also have no idea why this rouitine should require so much memory to run. There's obviously something I'm missing. Steve P got his to run with 64 meg.

Mahara 1.3.3 (Mahoodle w/Moodle 1.9.9+) running on Apache 2.2, PHP 5.3.2, MySQL 5.1, WIndows Server 2008 (2 X 2.40Ghz Intel Xeon with 24 gig of memory). Cron is running from Windows Task Scheduler....

"D:\Program Files (x86)\PHP\php-win.exe" -f E:\www\inetwork\lib\cron.php >> E:\logs\mahara-cron.log

Help!

03 January 2011, 22:41

A quick update. Setting the memory_limit to -1 (no limit) cured all access denied problems, but there's no end to warnings out there about that...

06 January 2011, 15:59

Jeffrey,

Removing the memory limit altogether sounds pretty dangerous to me.

As I said on that other thread we should take a look at the rebuild_artefact_parent_cache_complete function soon, and rewrite it.  I suspect it just doesn't handle a large artefact table.

How many rows are there in artefact and artefact_parent_cache?

06 January 2011, 18:03

artefact: "...Showing rows 0 - 29 (~53,6691 total, Query took 0.0023 sec)..."

artefact_parent_cache: "...Showing rows 0 - 29 (~39,1931 total, Query took 0.0004 sec)..."

That's one big honkin' table. We're not code monkeys over here, so if Moodle and Mahara code instances are otherwise playing nice with the memory usage, this one routine running at 12:05 a.m. local shouldn't hurt anyone. But you're right, it's a little scary, but what can I do...

I know there's a bug tracker on this issue (https://bugs.launchpad.net/mahara/+bug/683664), but it just show's "triaged." I'll add my comment to it. Let me know if that covers, that, or if I need to start a new bug item. Wish I has the PHP chops to help!

Thanks for the "come-back" on this...

13 January 2011, 15:56

 Hi Jeffrey,

I got some time to check this out and was able to reproduce the problem with a small table of only 8000 artefacts.  I bet the problem is related to the number of parent relationships rather than the total size of the table.

I still don't know what's eating up all the memory, and I still don't know whether this is a new problem or something that's always been there, but I have managed to fix the problem for my test data by changing that rebuild_artefact_parent_cache_complete function to pull all the parent relationships out of the db in one hit, rather than issuing a separate query for each one.

I won't commit the new function to master until I get a chance to try it out on a larger database, hopefully next week sometime.  If you'd like to try it out, just replace your rebuild_artefact_parent_cache_complete function with the one below (and don't forget to lower your memory limit again).

function rebuild_artefact_parent_cache_complete() {
    db_begin();
    delete_records('artefact_parent_cache');

    $artefacts = get_records_sql_assoc('
        SELECT id, parent, COUNT(aa.artefact) AS attached
        FROM {artefact} a LEFT JOIN {artefact_attachment} aa ON a.id = aa.attachment
        GROUP BY id, parent
        HAVING parent IS NOT NULL OR COUNT(aa.artefact) > 0',
        array()
    );

    if ($artefacts) {

        foreach ($artefacts as &$artefact) {

            // Nothing that can be a parent can be an attachment, so it's good
            // enough to first get everything this artefact is attached to, and
            // then find all its ancestors and the ancestors of everything it's
            // attached to.

            $ancestors = array();
            if ($artefact->attached) {
                $ancestors = get_column('artefact_attachment', 'artefact', 'attachment', $artefact->id);
            }

            $tocheck = $ancestors;
            $tocheck[] = $artefact->id;

            foreach ($tocheck as $id) {
                $p = isset($artefacts[$id]) ? $artefacts[$id]->parent : null;
                while (!empty($p)) {
                    $ancestors[] = $p;
                    $p = isset($artefacts[$p]) ? $artefacts[$p]->parent : null;
                }
            }

            foreach (array_unique($ancestors) as $p) {
                insert_record('artefact_parent_cache', (object) array(
                    'artefact' => $artefact->id,
                    'parent'   => $p,
                    'dirty'    => 0,
                ));
            }
        }
    }
    db_commit();
}

16 January 2011, 17:27

Richard:
       Thanks a bunch for that. I'll stick that on our beta box, set the memory brakes back on, and see if it produces any smoke. We're at the semester transition right now, so it'll be a week or two.

17 January 2011, 22:11

This problem came back to haunt me today, so I replaced the function and it's fixed it straight up. I have 57,000+ records in artefacts and 12,000+ records in artefact_parent_cache.

Thanks Richard.

18 January 2011, 14:21

Great, thanks Steve.  Hopefully later today I'll be in a position to test it out on a  db with 130k artefacts; if that's all good I'll commit the fix.

02 April 2011, 19:26

Richard, was this fix committed?

03 April 2011, 16:18

Yes, it's already in 1.3.4/1.3.5.