29 April 2022, 4:28

I was trying an upgrade from 20.04.5 to 22.04 but it fails without errors. So I tried an incremental upgrade from point releases and it failed at 21.04.4. With a bit of trial and error I discovered that it would fail at this DB upgrade:

if ($oldversion < 2021040802) {
        log_debug('Add "progress" page type');
        ensure_record_exists('view_type',
            (object)array('type' => 'progress'),
            (object)array('type' => 'progress')
        );
        if ($data = check_upgrades('blocktype.verification')) {
            upgrade_plugin($data);
            install_blocktype_extras();
        }
        // Create default progress template
        set_field('usr', 'admin', 1, 'username', 'root');
        install_system_progress_view();
        set_field('usr', 'admin', 0, 'username', 'root');
        // Make sure any existing progress collections now get the 'progress' page
        if ($collections = get_records_sql_array("SELECT c.id FROM {collection} c
                                                  WHERE c.progresscompletion = 1
                                                  AND NOT EXISTS (
                                                      SELECT v.id FROM {collection_view} cv
                                                      JOIN {view} v ON v.id = cv.view
                                                      WHERE v.type = 'progress'
                                                      AND cv.collection = c.id
                                                  )")) {
            require_once(get_config('libroot') . 'collection.php');
            $count = 0;
            $limit = 500;
            $total = count($collections);
            foreach ($collections as $collection) {
                $c = new Collection($collection->id);
                $c->add_progresscompletion_view();
                $count++;
                if (($count % $limit) == 0 || $count == $total) {
                    log_debug("$count/$total");
                    set_time_limit(30);
                }
            }
        }
    }

Removing it I can now upgrade to 22.04.0 without issue. What are the side effects of not having this in the upgrade and is there a workaround?

Is the double entry of (object)array('type' => 'progress'), intentional? 

Thanks a bunch

29 April 2022, 12:25

Hi Patric,

Hmm, it is strange that it failed at that point, what database type/version are you using, postgres or mysql?

Also is there a person in the usr table called 'root'?

 

"What are the side effects of not having this in the upgrade and is there a workaround" - the side effects will be big as you won't be able to use progress completion collections correctly.

There isn't a work around as there are complicated bits in that upgrade block - but you can check if some of the things it's trying to do already exist in your database, the following query should return nothing as the upgrade step adds them.

SELECT * FROM view_type WHERE type = 'progress';

SELECT * FROM blocktype_installed where name ='verification';

SELECT * FROM view where type = 'progress' and template = 2;

 

The only way to work out which bit of that update is failing is to comment in parts of it at a time to find out which part is causing the problems

Cheers

Robert

29 April 2022, 15:22

Thanks Robert,

I am using Postgres 14.2 and there is a root user.

Output from SQL command SELECT * FROM view_type WHERE type = 'progress'; ..

No data returned

Output from SQL command SELECT * FROM blocktype_installed where name ='verification'; ..

name	version	release	active	artefactplugin	quickedit
verification	2020122200	1.0.0	1		0

Output from SQL command SELECT * FROM view where type = 'progress' and template = 2; ..

No data returned

Comparing this with a fresh install I see:

Output from SQL command SELECT * FROM view_type WHERE type = 'progress'; ..

type
progress

Output from SQL command SELECT * FROM view where type = 'progress' and template = 2; ..

id	title	description	owner	ownerformat	group	institution	startdate	stopdate	ctime	mtime	atime	submittedgroup	submittedhost	submittedtime	submittedstatus	numrows	layout	skin	theme	template	retainview	copynewuser	type	visits	allowcomments	approvecomments	locked	urlid	anonymise	lockblocks	instructions	instructionscollapsed	accessibleview	coverimage	locktemplate
5	Portfolio completion	Portfolio completion page for a collection.		4		mahara			2022-02-17 18:54:11	2022-02-17 18:54:11	2022-02-17 18:54:11				0	1				2	0	0	progress	0	1	0	0		0	0		0	0		0

30 April 2022, 9:25

I have narrowed it down to this is what is causing the failure:

if ($data = check_upgrades('blocktype.verification')) {
            upgrade_plugin($data);
            install_blocktype_extras();
        }

I have checked the Blocktype_installed_viewtype table and there seems to be 4 extra fields compared to clean install:

contactinfo	dashboard
contactinfo	grouphomepage
contactinfo	portfolio
contactinfo	profile

 

Not sure if that has anything to do with it but now I see:

Output from SQL command SELECT * FROM view_type WHERE type = 'progress'; ..

type
progress

Output from SQL command SELECT * FROM blocktype_installed where name ='verification'; ..

No data returned

Output from SQL command SELECT * FROM view where type = 'progress' and template = 2; ..

id	title	description	owner	ownerformat	group	institution	startdate	stopdate	ctime	mtime	atime	submittedgroup	submittedhost	submittedtime	layout	theme	template	retainview	copynewuser	type	visits	allowcomments	approvecomments	locked	urlid	numrows	skin	anonymise	submittedstatus	lockblocks	instructions	instructionscollapsed	accessibleview	coverimage	locktemplate
19975	Portfolio completion	Portfolio completion page for a collection.		4		mahara			2022-04-29 13:51:48	2022-04-29 13:51:48	2022-04-29 13:51:48						2	0	0	progress	0	0	0	0		1		0	0	0		0	0		0

 

04 May 2022, 11:57

same issue here, upgrade from 19.04.1 to 20.04.0 works fine.
Then to 22.04.1 fails "silently"; the last entry in log being:
[Tue May 03 22:07:37.578806 2022] [php7:notice] [pid 6772] [client 10.254.20.193:35490] [DBG] 96 (lib/db/upgrade.php:443) Add "progress" page type, referer: LINK_REMOVED?urlsecret=**************************
On:
- Debian GNU/Linux 11 (bullseye)
- mysql  Ver 15.1 Distrib 10.5.15-MariaDB, for debian-linux-gnu (x86_64) using  EditLine wrapper
- PHP 7.4.28 (cli) (built: Feb 17 2022 16:17:19) ( NTS )
- Server version: Apache/2.4.53 (Debian)
- mahara downloaed as tar balls from launchpad .net
There is a  user called root:
<pre>
MariaDB [maharauat]> select * from usr where username='root';
+----+----------+----------+------+----------------+--------+---------+--------+----------------+-----------+---------------+------------+------------------+-------+-------+-----------+----------+-----------+---------------+------------------+-------------+----------------+-----------------+---------------+-----------+-----------+--------------+-------+--------------+------------+--------+-------+-----------+
| id | username | password | salt | passwordchange | active | deleted | expiry | expirymailsent | lastlogin | lastlastlogin | lastaccess | inactivemailsent | staff | admin | firstname | lastname | studentid | preferredname | email            | profileicon | suspendedctime | suspendedreason | suspendedcusr | quota     | quotaused | authinstance | ctime | showhomeinfo | logintries | unread | urlid | probation |
+----+----------+----------+------+----------------+--------+---------+--------+----------------+-----------+---------------+------------+------------------+-------+-------+-----------+----------+-----------+---------------+------------------+-------------+----------------+-----------------+---------------+-----------+-----------+--------------+-------+--------------+------------+--------+-------+-----------+
|  0 | root     | *        | *    |              0 |      1 |       0 | NULL   |              0 | NULL      | NULL          | NULL       |                0 |     0 |     0 | System    | User     | NULL      | NULL          | [email protected] |        NULL | NULL           | NULL            |          NULL | 262144000 |         0 |            1 | NULL  |            1 |          0 |      0 | NULL  |         0 |
+----+----------+----------+------+----------------+--------+---------+--------+----------------+-----------+---------------+------------+------------------+-------+-------+-----------+----------+-----------+---------------+------------------+-------------+----------------+-----------------+---------------+-----------+-----------+--------------+-------+--------------+------------+--------+-------+-----------+
</pre>
And pre upgrade (i.e. at 20.04.0) can confirm the following queries return nothing:
MariaDB [maharauat]> SELECT * FROM view_type WHERE type = 'progress';
Empty set (0.000 sec)
MariaDB [maharauat]> SELECT * FROM blocktype_installed where name ='verification';
Empty set (0.001 sec)
MariaDB [maharauat]> SELECT * FROM view where type = 'progress' and template = 2;
Empty set (0.001 sec)
post "Failed to upgrade" from 20.04.0 to 22.04.0:
MariaDB [maharauat]> SELECT * FROM view_type WHERE type = 'progress';
+----------+
| type     |
+----------+
| progress |
+----------+
1 row in set (0.000 sec)
MariaDB [maharauat]> SELECT * FROM blocktype_installed where name ='verification';
+--------------+------------+---------+--------+----------------+-----------+
| name         | version    | release | active | artefactplugin | quickedit |
+--------------+------------+---------+--------+----------------+-----------+
| verification | 2020122200 | 1.0.0   |      1 | NULL           |         0 |
+--------------+------------+---------+--------+----------------+-----------+
1 row in set (0.000 sec)
MariaDB [maharauat]> SELECT * FROM view where type = 'progress' and template = 2;
+--------+----------------------+---------------------------------------------+-------+-------------+-------+-------------+-----------+----------+---------------------+---------------------+---------------------+----------------+---------------+---------------+-----------------+--------+-------+----------+-------------+----------+--------+---------------+-----------------+------------+--------+-------+---------+------+-----------+------------+--------------+-----------------------+----------------+------------+--------------+
| id     | title                | description                                 | owner | ownerformat | group | institution | startdate | stopdate | ctime               | mtime               | atime               | submittedgroup | submittedhost | submittedtime | submittedstatus | layout | theme | template | copynewuser | type     | visits | allowcomments | approvecomments | retainview | locked | urlid | numrows | skin | anonymise | lockblocks | instructions | instructionscollapsed | accessibleview | coverimage | locktemplate |
+--------+----------------------+---------------------------------------------+-------+-------------+-------+-------------+-----------+----------+---------------------+---------------------+---------------------+----------------+---------------+---------------+-----------------+--------+-------+----------+-------------+----------+--------+---------------+-----------------+------------+--------+-------+---------+------+-----------+------------+--------------+-----------------------+----------------+------------+--------------+
| 139663 | Portfolio completion | Portfolio completion page for a collection. |  NULL | 4           |  NULL | mahara      | NULL      | NULL     | 2022-05-03 23:32:49 | 2022-05-03 23:32:49 | 2022-05-03 23:32:49 |           NULL | NULL          | NULL          |               0 |   NULL | NULL  |        2 |           0 | progress |      0 |             0 |               0 |          0 |      0 | NULL  |       1 | NULL |         0 |          0 | NULL         |                     0 |              0 |       NULL |            0 |
+--------+----------------------+---------------------------------------------+-------+-------------+-------+-------------+-----------+----------+---------------------+---------------------+---------------------+----------------+---------------+---------------+-----------------+--------+-------+----------+-------------+----------+--------+---------------+-----------------+------------+--------+-------+---------+------+-----------+------------+--------------+-----------------------+----------------+------------+--------------+
1 row in set (0.001 sec)

<br>

The error is:
Component or plugin From version To version Information
core 20.04.0 22.04.0 Failed to upgrade.
the 'backtrace' is that there is no error just the last line in the error log above.

PS: as you can probably tell could not find the forum mark up help/doc.

05 May 2022, 8:27

Narrowed it down a little further,  in upgrade.php, function install_blocktype_extras()
it seems to fail for
            install_blocktype_viewtypes_for_plugin(blocktype_single_to_namespaced($bt->name, $bt->artefactplugin));
where the $bt->name is a plugin that is no longer installed (or maintained it seems) "embedly".

[upadate]

and another run and it it seems now it is bombing out in function install_blocktype_viewtypes_for_plugin for plugin embedly in call to safe_require('blocktype', $blocktype);

    log_info('install_blocktype_viewtypes_for_plugin: pre safe_require: ' . $blocktype );
    safe_require('blocktype', $blocktype);
    log_info('  install_blocktype_viewtypes_for_plugin: pre blocktype_namespaced_to_single');

log last line is:
[Wed May 04 21:36:10.512033 2022] [php7:notice] [pid 18061] [client 10.54.120.27:44688] [INF] c4 (lib/upgrade.php:1235) install_blocktype_viewtypes_for_plugin: pre safe_require: embedly, referer: REMOVED_URL_FOR_FORUMS_SAKE?urlsecret=*************************

the second log_info is not reached in the above snippet.

05 May 2022, 9:18

Hi Tom,

The embedly plugin used to be a separate blocktype plugin but became part of core after version 16.04

If your site ever installed the plugin there might be a subdirectory called 'embedly' in your system  - you will need to remove that directory and all it's contents.

This should then make the plugin reference in the database become inactive.

If not then you can check the database:

 SELECT * FROM blocktype_installed WHERE name = 'embedly';

And if line exists:

 UPDATE blocktype_installed SET active = 0 WHERE name = 'embedly';

Cheers

Robert

06 May 2022, 11:45

Hello,

Many thanks for the suggestions and help. It is very much appreciated.

SELECT * FROM blocktype_installed WHERE name = 'embedly';
+---------+------------+---------+--------+----------------+
| name    | version    | release | active | artefactplugin |
+---------+------------+---------+--------+----------------+
| embedly | 2014110200 | 1.1.0   |      0 | NULL           |
+---------+------------+---------+--------+----------------+

So set not to be active.

There was a directory remenant: ./blocktype/externalvideo/embed_services/embedly which is now removed.


I have added some log_info calls to the safe_require function (from mahara.php) as follows and it seems to make it to the throw new SystemException and die

function safe_require($plugintype, $pluginname, $filename='lib.php', $function='require_once', $nonfatal=false, $returnvars = null) {
    $plugintypes = plugin_types();
    log_info('in safe_require');
    log_info('safe_require: pluginname: ' . $pluginname );

<snip>

    if (empty($fullpath)) {
        log_info('safe_require: EEEE: fullpath empty ');
        $fullpath = get_config('docroot') . $plugintype . '/' . $pluginname . '/' . $filename;
    }

    if (!file_exists($fullpath)) {
        log_info('safe_require: EEEE:  ');
        log_info('safe_require: EEEE:  ' . $fullpath);
        if (!empty($nonfatal)) {
            log_info('safe_require: EEEE:  returning false');
            return false;
        }
        log_info('safe_require: EEEE:  pre throw new exception');
        throw new SystemException ("File $fullpath did not exist");
        log_info('safe_require: EEEE:  post throw new exception');
    }



the entries in the log are:
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1666) in safe_require, referer: REMOVED_URL
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1667) safe_require: pluginname: embedly, referer: REMOVED_URL
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1707) safe_require: EEEE: fullpath empty , referer: REMOVED_URL
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1712) safe_require: EEEE:  , referer: REMOVED_URL
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1713) safe_require: EEEE:  /var/www/html/mahara-22.04.0/htdocs/blocktype/embedly/lib.php, referer: REMOVED_URL
[Thu May 05 00:18:54] [php7:not] [pid 19735] [client 10.54.x:y] [INF] 46 (lib/mahara.php:1718) safe_require: EEEE:  pre throw new exception, referer: REMOVED_URL

Not sure why the exception is not causing any output to the httpd log, it just seems to end there. There is the usual output "Faild to upgrade" in the browser. After reseting the database and runing the upgrade from the command line it can be seen that the log calls are made and the last output is the System exception is hit.

[WAR] 3b (lib/mahara.php:1719) File /var/www/html/mahara-22.04.0/htdocs/blocktype/embedly/lib.php did not exist
Call stack (most recent first):
  * safe_require() at /var/www/html/mahara-22.04.0/htdocs/lib/upgrade.php:1236
  * install_blocktype_viewtypes_for_plugin() at /var/www/html/mahara-22.04.0/htdocs/lib/upgrade.php:1294
  * install_blocktype_extras() at /var/www/html/mahara-22.04.0/htdocs/lib/db/upgrade.php:457
  * xmldb_core_upgrade() at /var/www/html/mahara-22.04.0/htdocs/lib/upgrade.php:377
  * upgrade_core() at /var/www/html/mahara-22.04.0/htdocs/lib/mahara.php:281
  * upgrade_mahara() at /var/www/html/mahara-22.04.0/htdocs/admin/cli/upgrade.php:76
A nonrecoverable error occurred. This probably means you have encountered a bug in the system

Thinking out loud here, but could safe_require not be called with nonfatal set to true as opposed to the default value of false in function install_blocktype_viewtypes_for_plugin of upgrade.php ? Or would that cause important operations of the upgrade to be skipped ?

Regards,
Tom.

06 May 2022, 14:22

Hi Tom,

Thanks for your reply.

I've replicated your problem and created a bug report and a patch for it here https://reviews.mahara.org/c/mahara/+/12721

If you make that change to your codebase it should allow you to upgrade.

As for the directory remnant I mentioned, it would have been at ./blocktype/embedly/ and not at ./blocktype/externalvideo/embed_services/embedly - that directory should still stay as it's the new place for the embedly code (as part of the externalvideo blocktype). You will need to put that code back (it can be fetched from https://git.mahara.org/mahara/mahara/-/tree/main/htdocs/blocktype/externalvideo/embed_services if you need to).

Cheers

Robert

07 May 2022, 5:09

Super !

That worked nicely (now bumping into:  Bug 1970680: Only update if lti_assessment table exists )

All good now, many thanks for your help and patch.

Onwards to 22.04.1 :)