01 September 2011, 5:38

Then went sucessfuly to 1.3.7testing (latest on the branch). I didn't clean my schema first, so this would indicate that it didn't get much messier during the life of 1.3:

ALTER TABLE `artefact` MODIFY COLUMN `allowcomments` tinyint(1) NOT NULL DEFAULT '0' AFTER `authorname`;
ALTER TABLE `artefact_resume_educationhistory` MODIFY COLUMN `institutionaddress` text NULL AFTER `institution`;
ALTER TABLE `artefact_resume_employmenthistory` MODIFY COLUMN `employeraddress` text NULL AFTER `employer`;
ALTER TABLE `collection` MODIFY COLUMN `name` text NOT NULL AFTER `mtime`;
ALTER TABLE `collection_view` MODIFY COLUMN `collection` bigint(10) NOT NULL FIRST;
ALTER TABLE `notification_internal_activity` MODIFY COLUMN `urltext` text NULL AFTER `url`;
ALTER TABLE `site_data` MODIFY COLUMN `ctime` datetime NOT NULL FIRST;
ALTER TABLE `usr` MODIFY COLUMN `showhomeinfo` tinyint(1) NOT NULL DEFAULT '1' AFTER `ctime`;
ALTER TABLE `view` MODIFY COLUMN `submittedtime` datetime NULL AFTER `submittedhost`, MODIFY COLUMN `theme` varchar(255) NULL AFTER `layout`, MODIFY COLUMN `visits` bigint(10) NOT NULL DEFAULT '0' AFTER `type`, MODIFY COLUMN `allowcomments` tinyint(1) NOT NULL DEFAULT '1' AFTER `visits`;
ALTER TABLE `view_access` MODIFY COLUMN `startdate` datetime NULL AFTER `visible`, MODIFY COLUMN `stopdate` datetime NULL AFTER `startdate`, MODIFY COLUMN `allowcomments` tinyint(1) NOT NULL DEFAULT '0' AFTER `stopdate`, MODIFY COLUMN `approvecomments` tinyint(1) NOT NULL DEFAULT '1' AFTER `allowcomments`;
ALTER TABLE `view_visit` MODIFY COLUMN `view` bigint(10) NOT NULL FIRST;

I'm increasingly of the opinion that the upgrade is getting killed by hitting some resource limit but the exception handling is failing to spot this resulting in the json upgrader thingy just restarting the core_upgrade. After a lot of hours of banging my head off a wall I can't proove it :(

Just to poor cold water on my own theory the problem seems more pronounced on our production server than on our test server (the production server is an order or magnitude faster). 

01 September 2011, 17:23

Howard, are those diffs really a problem?  I mean, are the types of those columns actually different in the two dbs, or is it just that the columns are in a different order?  If it's just that they're in a different order, I don't think we need to worry about it, it'll have no effect on anything.

I think you're right about the resource limit, unfortunately in php those sorts of things seem to be hard to catch, there's no exception, just silent death.  But if you can get to the point where that 'from' foreign key has been created, and it's dying while updating the data in the table, we can easily rewrite that bit to use less memory.  If it can't create the index in the first place it's going to be harder to deal with.

02 September 2011, 2:37

No they are not a problem. I've just included then while I was there. 

It seems to be a workaround stepping through this in 'small bites', which is why I think it's some resource problem. In addition, I can 'exit' just before the problem area and run the ALTER by hand and it works fine. As they were just constraints I commented them out of the upgrade and it just failed further along. 

I think the only unfortunate thing to come out of this is the way the javascript/json installer operates. This very effectively masks what is actually going on by silently re-running the core upgrade. Plus, the core-upgrade doesn't seem resilient enough when it comes to large jumps. I would say that the latest 1.2 to the latest 1.3 *should* work - regardless of the site size. 

If there's anything else I can do or test that would help I'd be happy to do so. I would make a bug report but I'm not sure how useful it would be - very hard to give instructions to reproduce :)