27 September 2010, 10:47

Hi,

We have problem with new users in mahara. We use mahara with ldap-plugin. Users with special characters like é,ö,... couldn't login. They get a error-page with the message: "Mahara - xyz: Site unavailable. A nonrecoverable error occured. This probably means you have encountered a bug in the system"

In the error-log we get the following error:

[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/errors.php:743) Failed to get a recordset: postgres7 error: [-1: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/errors.php:743) HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".] in adodb_throw(INSERT INTO "usr" ("id", "username", "password", "salt", "passwordchange", "active", "deleted", "expirymailsent", "lastlogin", "inactivemailsent", "staff", "admin", "firstname", "lastname", "studentid", "preferredname", "email", "quota", "quotaused", "authinstance", "ctime", "showhomeinfo") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?), Array), referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/errors.php:743) Command was: INSERT INTO "usr" ("id", "username", "password", "salt", "passwordchange", "active", "deleted", "expirymailsent", "lastlogin", "inactivemailsent", "staff", "admin", "firstname", "lastname", "studentid", "preferredname", "email", "quota", "quotaused", "authinstance", "ctime", "showhomeinfo") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and values was (956,[email protected],,,0,1,0,0,2010-09-27 17:37:32,0,0,0,Martin,K\xfcnzler,,,[email protected],1048576000,0,7,2010-09-27 17:37:32,1), referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] Call stack (most recent first):, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * log_message("Failed to get a recordset: postgres7 error: [-1: E...", 8, true, true) at /srv/mahara/data/lib/errors.php:109, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * log_warn("Failed to get a recordset: postgres7 error: [-1: E...") at /srv/mahara/data/lib/errors.php:743, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * SQLException->__construct("Failed to get a recordset: postgres7 error: [-1: E...") at /srv/mahara/data/lib/dml.php:1069, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * insert_record("usr", object(stdClass), "id", true) at /srv/mahara/data/auth/user.php:358, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * User->commit() at /srv/mahara/data/lib/user.php:1605, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * create_user(object(LiveUser), array(size 0), object(Institution), null) at /srv/mahara/data/auth/lib.php:1229, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * login_submit(object(Pieform), array(size 6)) at Unknown:0, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * call_user_func_array("login_submit", array(size 2)) at /srv/mahara/data/lib/pieforms/pieform.php:512, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * Pieform->__construct(array(size 8)) at /srv/mahara/data/auth/lib.php:474, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * auth_setup() at /srv/mahara/data/init.php:241, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * require("/srv/mahara/data/init.php") at /srv/mahara/data/index.php:32, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] , referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/dml.php:1069) Failed to get a recordset: postgres7 error: [-1: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/dml.php:1069) HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".] in adodb_throw(INSERT INTO "usr" ("id", "username", "password", "salt", "passwordchange", "active", "deleted", "expirymailsent", "lastlogin", "inactivemailsent", "staff", "admin", "firstname", "lastname", "studentid", "preferredname", "email", "quota", "quotaused", "authinstance", "ctime", "showhomeinfo") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?), Array), referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] [WAR] 74 (lib/dml.php:1069) Command was: INSERT INTO "usr" ("id", "username", "password", "salt", "passwordchange", "active", "deleted", "expirymailsent", "lastlogin", "inactivemailsent", "staff", "admin", "firstname", "lastname", "studentid", "preferredname", "email", "quota", "quotaused", "authinstance", "ctime", "showhomeinfo") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and values was (956,[email protected],,,0,1,0,0,2010-09-27 17:37:32,0,0,0,Martin,K\xfcnzler,,,[email protected],1048576000,0,7,2010-09-27 17:37:32,1), referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] Call stack (most recent first):, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * insert_record("usr", object(stdClass), "id", true) at /srv/mahara/data/auth/user.php:358, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * User->commit() at /srv/mahara/data/lib/user.php:1605, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * create_user(object(LiveUser), array(size 0), object(Institution), null) at /srv/mahara/data/auth/lib.php:1229, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * login_submit(object(Pieform), array(size 6)) at Unknown:0, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * call_user_func_array("login_submit", array(size 2)) at /srv/mahara/data/lib/pieforms/pieform.php:512, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * Pieform->__construct(array(size 8)) at /srv/mahara/data/auth/lib.php:474, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * auth_setup() at /srv/mahara/data/init.php:241, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4]   * require("/srv/mahara/data/init.php") at /srv/mahara/data/index.php:32, referer: http://our.mahara.xyz/
[Mon Sep 27 17:37:32 2010] [error] [client 1.2.3.4] , referer: http://our.mahara.xyz/

28 September 2010, 17:10

So I think what's happening is that your LDAP directory is not using UTF8 to encode those special characters, and Mahara is trying to insert these invalid UTF8 characters into its UTF8 database.

We should either try to convert these strings or at least add some UTF8 checking to the LDAP auth plugin to catch the errors before attempting to insert names into the db.

I've filed a bug at https://bugs.launchpad.net/mahara/+bug/650678

29 September 2010, 2:08

It's very weird.

We formerly used a mysql database and changed it to postgresql. We use the same settings as we used with mysql.

Some infos about our infrastructure:

- We use the ldap connection to a MS Active Directory Windows Server 2003 R2.
- Database Postgresql V 8.4.4
- Apache V 2.2.9
- PHP5 V 5.2.6

29 September 2010, 10:33

Postgres is much stricter about almost everything than Mysql :) It could be that Mysql happily inserted non-utf8 characters into a utf8 database for you and never told you.  Can you confirm in LDAP, and if necessary, fix the charset?

05 October 2010, 4:00

I don't know if it is possible to set the encoding on the windows server for LDAP-query... I didn't found any possibility to do that...

We thought if it is possible to import the users with this problem manually to the database until the a conversion is built into the ldap-interface. We switched of the permanent user sync (institution --> edit --> authentication-plugin --> LDAP-authentication --> "Update user info on login" is off). In the error-log I found the failed database-statement. I would pre-import the bad users but the question is, does the ldap-user-import do create anything else than the user?

07 October 2010, 6:45

What I mean is, if somebody already done this.

I want to manually import these problematic users to the database.

I think it should work to import the users with following sql-statement to make them working:

\encoding LATIN1
INSERT INTO usr (id, username, password, salt, passwordchange, active, deleted, expirymailsent, lastlogin, inactivemailsent, staff, admin, firstname, lastname, studentid, preferredname, email, quota, quotaused, authinstance, ctime, showhomeinfo) VALUES (947,'[email protected]','','',0,1,0,0,'2010-10-05 11:02:40',0,0,0,'Martin','Künzler',null,null,'[email protected]',1048576000,0,7,'2010-10-05 11:02:40',1)

Does anybody done these before or does anybody think this will work?

07 October 2010, 15:56

When you create new users you get records in the artefact table as well as the usr table (and sometimes also usr_institution).  So if you can, it would be better to export the users from your ldap directory to a csv file, run that file through something that will fix the character encoding (iconv?), then log in as admin and import the users from the csv.

07 October 2010, 15:58

... and we really need to stop tinymce from creating these divs with a width and height set to 1px!