Planet Mahara

Dan Marsden: Awards Time?

2010-08-22T22:18:49+00:00

With nominations soon closing for the NZ Open Source Awards I’ve just heard about Packt Publishing’s Open Source Awards – It looks like Moodle and Mahara aren’t eligible, but if you’re aware of any open source products that you think should be recognised make sure you nominate them! – Packt are offering cash prizes to the winners and runners-up.

The NZ Open Source Awards are a bit more flexible and a wider range of projects will be eligible but nominations are closing very soon!

François Marier: Cherry-picking a range of git commits

François — 2010-07-21T09:41:15+00:00

The cherry-pick command in git allows you to copy commits from one branch to another, one commit at a time. In order to copy more than one commit at once, you need a different approach.

Cherry-picking a single commit

Say we have the following repository composed of three branches (master, feature1 and stable):

$ git tree --all
* d9484311 (HEAD, master) Delete test file
* 4d4a0da8 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

The "git tree" command is an alias I defined in my ~/.gitconfig:

[alias]
tree = log --oneline --decorate --graph

To copy the license file (commit 5753515c) to the master branch then we simply need to run:

$ git checkout master
$ git cherry-pick 5753515c
Finished one cherry-pick.
[master 08ff7d4] Add a license
1 files changed, 676 insertions(+), 0 deletions(-)
create mode 100644 COPYING

and the repository now looks like this:

$ git tree --all
* 08ff7d4a4 (HEAD, master) Add a license
* d94843113 Delete test file
* 4d4a0da88 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

Cherry-picking a range of commits

In order to only take the third file (commits a7785c10 and 7f545188) from the feature1 branch and add it to the stable branch, I could cherry-pick each commit separately, but there is a faster way if you need to cherry-pick a large range of commits.

First of all, let's create a new branch which ends on the last commit we want to cherry-pick:

$ git branch tempbranch a7785c10
$ git tree --all
* 08ff7d4a (HEAD, master) Add a license
* d9484311 Delete test file
* 4d4a0da8 Add a test file
| * 5753515c (stable) Add a license
| * 4b95278e Add readme file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 (tempbranch) Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

Now we'll rebase that temporary branch on top of the stable branch:

$ git rebase --onto stable 7f545188^ tempbranch
First, rewinding head to replay your work on top of it...
Applying: Add third file
Applying: Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
| * 08ff7d4a (master) Add a license
| * d9484311 Delete test file
| * 4d4a0da8 Add a test file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit

All that's left to do is to make stable point to the top commit of tempbranch and delete the old branch:

$ git checkout stable
Switched to branch 'stable'
$ git reset --hard tempbranch
HEAD is now at ec48867 Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch, stable) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c Add a license
* 4b95278e Add readme file
| * 08ff7d4a (master) Add a license
| * d9484311 Delete test file
| * 4d4a0da8 Add a test file
|/
| * a37658bd (feature1) Add fourth file
| * a7785c10 Add lines to 3rd file
| * 7f545188 Add third file
| * 2bca593b Add line to second file
| * 0c13e436 Add second file
|/
* d3199755 Add a line
* b58d925c Initial commit
$ git branch -d tempbranch
Deleted branch tempbranch (was ec48867).

It would be nice to be able to do it without having to use a temporary branch, but it still beats cherry-picking everything manually.

Another approach

Another way to achieve this is to use the format-patch command to output patches for the commits you are interested in copying to another branch and then using the am command to apply them all to the target branch:

$ git format-patch 7f545188^..a7785c10
0001-Add-third-file.patch
0002-Add-lines-to-3rd-file.patch
$ git am *.patch

Update: looking forward to git 1.7.2

According to a few people who were nice to point this out in a comment, version 1.7.2 of git, which is going to be released soon, will have support for this in cherry-pick:

git cherry-pick 7f545188^..a7785c10

François Marier: Reducing website bandwidth usage

François — 2010-06-20T13:28:55+00:00

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.

HTML, XML, Javascript and CSS files

One of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:

AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript application/javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip

Images

As far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):

gifsicle -O2 -b image.gif
jpegoptim -p --strip-all image.jpg
optipng -o7 -q image.png

(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.

François Marier: Handling email-based events safely

François — 2010-06-14T18:21:40+00:00

Mahara.org users have recently witnessed the unfortunate effects of a bug in the Mahara event handling: the Mahara cron job got stuck in an email loop and kept on sending the same forum post over and over again.

Here is what the affected code used to look like (edited for clarity):

db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  handle_activity($activity);
}
delete_records('activity_queue');
db_commit();

One of the problems with this code was that if the handle_activity() function threw an exception, it would interfere with the processing of the entire queue. So it got fixed in the following way:

db_begin();
$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
}
delete_records('activity_queue');
db_commit();

Much better. However, there was still a problem with the code: the whole queue processing is contained within a transaction. This means that should any of the SQL statements fail at any point, the exception would be caught but the SQL statements would all be rolled back at the end by the database.

Now the idea of a transaction is good: we want activity handling to either succeed entirely or be rolled back. But the fact that some activities cannot be handled should not interfere with other activities. So this has been fixed by moving the transaction to the inside of the loop:

$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  db_begin();
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
  db_commit();
}
delete_records('activity_queue');

So individual activites are allowed to fail and get rolled back, but they will not affect other activites. But there was still one remaining problem: what if we encounter an error we cannot catch? For example, what would happen if PHP were to segfault or run out of memory while the activity queue is being processed?

Well, in that case, it turns out that Mahara will never reach the delete_records() call and the activity queue will never be cleared. Which means that on the next cron run, all of the activities will be handled again, even the ones that were successfully handled already (i.e. emails will be sent over and over again).

The way we fixed this problem was by moving the delete_records() from the end of the function to the beginning of the loop:

$activities = get_records('activity_queue');
foreach ($activities as $activity) {
  delete_records('activity_queue', 'id', $activity->id);

  db_begin();
  try {
      handle_activity($activity);
  }
  catch (MaharaException $e) {
      log_debug($e->getMessage());
  }
  db_commit();
}

Each activity is removed from the queue before it is processed.

Unfortunately, there is a downside to this modification: should an activity handler fail for whatever reason, no further attempts will be made. This means that some notifications could be lost if an unexpected error occurs.

However, given that some of the activity handlers send emails out into the world and that it is not possible to "un-send" them, this is the only way we can guarantee that no duplicate emails will be sent. Of course, if you notice that certain notifications are lost because of a bug in Mahara, let us know and we'll fix it!

Nigel McNie: Removing the 'referer' part of PHP Error Messages from Apache Logs

2010-06-14T03:30:04+00:00

This has always bugged me (scroll right to see the whole thing):

[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first):, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22, referer: http://mahara-test/admin/extensions/test.php?
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] , referer: http://mahara-test/admin/extensions/test.php?

I like using log files to display messages instead of the screen, but somewhere along the way somebody decided that the 'referer' (sic) was needed in the logs too, and that makes them totally unreadable.

Here's my fix, when tailing a log file:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/, referer:.*//'

And, instantly:

[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first):
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22
[Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84]

Much better! But we can do better again:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/\[error\]\s+\[client.*?\]\s+//; s/, referer:.*//'

And we end up with:

[Tue Apr 20 01:06:47 2010] [WAR] 7d (lib/web.php:916) Undefined variable: message
[Tue Apr 20 01:06:47 2010] Call stack (most recent first):
[Tue Apr 20 01:06:47 2010]   * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446
[Tue Apr 20 01:06:47 2010]   * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916
[Tue Apr 20 01:06:47 2010]   * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22
[Tue Apr 20 01:06:47 2010]

Hooray! Readable PHP log files.

You don't want to have to remember to type all of that, so for best results, put it in a script, chmod +x and you have:

nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | phptail

You could even combine it with Monkeytail to make viewing the logs even easier ;).

I've trawled through the PHP codebase looking for where it adds the referer in a vain attempt to change it, or provide a patch to make it a configuration option. Sadly, I never found it. If anyone finds out, please let me know!

Thanks to Martyn for the original idea, and assisting with the arcane perl stuff.

Nigel McNie: What's up in Mahara land recently?

2010-06-14T03:30:04+00:00

Here's a snapshot from my point of view about what's going on across all aspects of Mahara.

Hackfests

Yesterday, Penny, Jordi (both from Mahara partner Liip) and I had another weekend hackfest. Penny has been adding a framework for unit testing, and managed to get the first test working. Jordi was working on swapping Smarty for Dwoo, while I continued hacking on View Themes support. Pics: Switzerland end, NZ end.

Penny has been working on unit testing, as it's something we developers have realised has been lacking in Mahara for some time now. What unit tests do is give us more confidence that when we make a change (whether it's a bug fix or new feature), that the rest of Mahara continues working exactly the way it used to - i.e., we didn't inadvertently add any more bugs. This has the result that we can continue development safe in the knowledge that we didn't break anything that worked before, which will result in better quality. It will also make it easier for enthusiastic hackers to make changes to Mahara and know they haven't broken anything important.

Unit testing isn't the only way we've been trying to integrate testing - Gerald Quimpo from Catalyst has been resurrecting the Selenium tests that come with Mahara, and has done an excellent job in getting them working again, while also setting up a Continuous Integration server (Hudson) so the tests will be run regularly. Soon, we'll have that available publicly so everyone can see which tests are currently passing and failing.

Backtracking a tiny bit - Jordi has been working on swapping Smarty for Dwoo, a change we would like to do for a couple of reasons. Firstly, Smarty is bigger and slower, perhaps largely due to the fact that it needs to support PHP4 (which Mahara doesn't support). Secondly, Smarty doesn't make it easy for us to output variables that are HTML escaped by default - which is currently the leading cause of security vulnerabilities in Mahara. We haven't had many security vulns actually, Mahara is reaping the benefits of a security-oriented development ethic. But of the few we've had, almost all of them have been because someone forgot to put |escape on the end of a smarty variable. So switching to Dwoo will cut down that entire avenue of attack.

Mahara 1.2

We released the first beta of Mahara 1.2 last week. That means we're looking for bug reports now, and encourage people to download it and try out the new features. Here's a short list of the coolest new features in 1.2:

Import and Export - Mahara supports the LEAP2A standard for import/export, and in addition, allows you to export an HTML version of your portfolio (including a nice HTML represntation of an individual View).
"My Files" and file handling improvements - now the My Files section makes it easier to quickly upload files, and degrades to work without javascript. You can now upload files directly into Views, and use a standard directory browser to choose files in Views. For developers, we created a pieform element for uploading/attaching files, which will make it easier than ever to do this in custom plugins.
Theming - we're shipping with 6 themes, to give people an idea of how they can customise their Mahara. We've done a lot of rework of how themes are created, so from 1.2 it'll be even easier to create a custom theme. We even documented the process (finally!).
Usability/Comprehension tweaks - Now groups get a default forum, and blogs get a default blog. If there is only one forum or blog, users are taken straight to it. This should really help in preventing the problem where people create a blog as if it's a blog post, and should make the forums an even more useful tool.
Improved Moodle/Mahara integration - with a custom patch to Moodle 1.9, you'll be able to submit a View to Moodle for assessment as an assignment answer. How useful is that? Now people can keep their data in their portfolio, and mash it together into a View for assessment over on Moodle, where all the useful grading functionality is :)

There's s still a bit of work to go before Mahara 1.2 will be ready though. I need to finish off LEAP2A import - currently resume import is only half finished, and view import is just begun. We'll also need a fancy UI for it, and of course there are some inevitible bugs that will need squashing.

Also, I'd really like to get view themes to a point where it can be merged also. That'll really help people personalise their environment. If I get around to it, maybe we can add a theme chooser for users to change how they see the entire site, not just individual Views.

Other than that though, I think we're looking pretty good!

Code Contribution Policy

We are currently preparing a policy for contributors to the codebase. We aren't going to be wanting copyright assignment, but we are going to make sure all contributions are as legal as we can make them ;). Unlike many open source projects, Mahara is actually in a pretty good position legally, as most members of the team are well aware of the legal issues around open source and licensing, and we take legal issues seriously. Our policy won't be onerous, but will ensure we stay legit now and into the future.

Mahara 1.3

Though 1.2 is not out yet, talk is already shifting in some departments to the plans we have for Mahara 1.3. The big focus there will be usability, and we're going to target that in a couple of main areas - the navigation, and View creation.

For the navigation, our current plan is to adopt a dropdown menu structure, organised more to how users think when they're looking for something. The main categories will be roughly along the lines of "My Stuff", "Friends & Groups" and "Sharing". We'll build a demo of this at some point after 1.2 so the community can see it in action and give feedback, though I've seen it in action elsewhere and it's easy to grasp. Furthermore, the addition of "horizontal navigation" - where when you're at one page you're given suggestions about what else you might be looking for - should help tremendously as well.

We're also going to have a good look at the View creation process. The Drag and Drop and layout concepts won't be changing, but the layout of the editing screens themselves will be improved a lot. We're hoping we can even ditch a step to make things simpler. I think the biggest gains will be seen on the View Access screen, which currently can be a bit of a minefield to navigate. We'll get mockups out some time after 1.2 also.

Other stuff we might do could include Group Views, a User Dashboard (as a View of course!), and overhauling the notification system.

Mahara Growth

And last but not least, here are some interesting statistics about Mahara and its community:

We now have over 3000 members of the mahara.org community, with 17 partners worldwide. The community growth has been steady since we launched the new mahara.org last November. The site receives nearly 25,000 visits a month (for the geeks among you: 57% firefox and 27% IE). Of these visitors, we get the most from the US, followed by the UK, Germany, Spain, Australia and then New Zealand.

Since we added the ability for sites to register on mahara.org in May, we now have over 50 sites doing this every week. These sites together add to some 12,000 users, but this doesn't include mahara.org, or the New Zealand MyPortfolio service - if these numbers were included, the total would rise to nearly 25,000. Again for the geeks - we are seeing a 75/25 split of MySQL to PostgreSQL. Hopefully things will skew more to the PostgreSQL side of that equation as time goes on :). Of the registered sites, the largest seem to be run by Blackpool University, Uni Krems (Austria) and Birmingham City University, all of which have thousands of users.

Nigel McNie: Mahara 1.2 Import/Export Demo

2010-06-14T03:30:04+00:00

OK so my last post was just all text, which is a slog to read through. So here, I made a video about Mahara 1.2's import/export:

I don't think I got the sound levels right and I'd have loved to have added a background track, but video editing on linux is a pain. It's only my second video so be nice. The next one should be better again ;)

Nigel McNie: Themes In Mahara 1.2

2010-06-14T03:30:04+00:00

One of the things we're doing for the upcoming 1.2 release of Mahara is doing a big tidyup of the theming. There are some code level changes, but the most important changes are to do with the themes themselves.

Note that I said "themes". This is because 1.2 won't have just one theme - it'll have six. One is a brushed up version of the default, one is "special" (more on that in a minute), and the other four are all-new themes that will give Mahara users a chance to immediately personalise their installation "out of the box". The new themes are mainly just colour changes, but they're pretty damn good looking colour changes all the same. They'll also give people good examples to work from when creating their own themes.

Now, the special theme. Mahara will now have a theme called 'raw', which is a very stripped down theme. There's very few colours or styles, and most of the theme's job is to position everything correctly and set sensible defaults. All other themes extend from this theme. Mahara has a theme inheritance system, where by you can specify another theme as the 'parent' of your theme, and thus rely on it to provide the templates and most of the CSS. From now, every theme will ultimately extend from the 'raw' theme - including the default theme.

The inheritance can have any number of levels, so for example a theme someone did for Mahara 1.1 might have a parent of the default theme which might be parented by the raw theme. Sadly, because we haven't documented the theme inheritance well, some of the themes out there have copied the entire default/ directory to create their new theme. These people will probably have noticed that when new versions of Mahara come out, sometimes bugs they are supposed to fix are not fixed - this is because the fix was made in a 'default' theme template, which is overridden by their theme's copy. When they upgrade to 1.2, it's highly likely their theme will totally break.

Elsewhere on the backward compatibility front, it should be easier though. If your theme uses theme inheritance, then it will continue to inherit from the (slightly changed) default theme. You may also have to change some CSS selectors in your stylesheet to match the new HTML structure.

Speaking of which, we've cleaned up the HTML immensely, and the CSS along with it. I think the default stylesheet was over 50KB in previous versions of Mahara, at the moment I have the raw theme down to much less than half that, and I don't think it will end up much bigger than 25KB. The default theme's stylesheet will be negligible in size. We're also going to cut the print stylesheet down to size, and if we get the chance we'll add in RTL and browser-specific stylesheets too.

The net result of all of this is a bit of short-term pain while people migrate their themes, but a lot of long-term gain, as the HTML will be cleaner, the CSS smaller and every Mahara faster because of it. Theme development will be much easier from 1.2 onwards, I'll make sure we write documentation so people don't get caught out by the theme inheritance issue again.

Maybe at some point after 1.2 we can look at some fine-tuning and performance of the theming system. I'd love to crank out some CSS sprites and javascript minification, but right now I don't think there's going to be much time other than for 'easy wins', such as expires headers and configuring mod_deflate (which Mahara does already).

Anyway, that's enough from me for now. We have this theming work and finishing off import/export to do, followed by a lot of bug fixing before 1.2 can be released. Look out for another alpha soon, at least.

Nigel McNie: Mahara 1.1 - almost there!

2010-06-14T03:30:04+00:00

There's three bugs and two feature requests left until 1.1 is ready to roll! Finally it looks like we've got on top of the long list of issue and will be able to get a release out later this month.

Here's a quite explanation of how things will progress over the next while:

Once the bugs and feature requests are fixed, we will release the first, and possibly only, release candidate of 1.1. This will be a release that we consider "good enough" for 1.1 - no major crashing bugs, upgrade from 1.0 works, new functionality works.

We'll let that circulate for a few days, to give people a change to have one last go at testing upgrades and looking for major bugs. If there's nothing reported that's too major, we'll do the 1.1 release, otherwise, there will be a second (and potentially third) release candidate.

The 1.1 release will no doubt need some tweaking and minor fixes, which I'm sure will be reported back to us quickly. So there will probably be a series of point releases after 1.1 to iron out the kinks. We'll fully support you if you upgraded from any version of 1.0 to 1.1 though, and hopefully the testing we've done means the upgrade path doesn't break, even if some minor things don't work in 1.1.

Meanwhile, we'll be working hard on the import/export functionality for 1.2. This is the only feature planned for 1.2, and we hope to have it released quite quickly, by mid this year even.

So, look out for Mahara 1.1 RC1 real soon now!

Nigel McNie: Things You Don't Want to See in the Morning

2010-06-14T03:30:04+00:00

maharademo=# select count(*) from activity_queue;
 count
-------
 10043
(1 row)

Aah crap.

So it turns out there's a bug in Mahara's activity queue processing which means that if you make a view, leave objectionable feedback on it then delete the view before the queue is processed, the queue will bomb out every time it gets to processing the feedback and not send any of the other messages. Doh!

I have written a patch for this that'll go to 1.0_STABLE and master. It will probably involve unconditionally deleting any notifications older than two weeks, as the last thing people want is old notifications.

This problem may have affected other Mahara installs out there, but it's not guaranteed to have. For example, it hasn't affected the MyPortfolio sites, nor mahara.org itself. You can find out by doing a SELECT COUNT(*) FROM activity_queue; on your database - large numbers are bad, 0 or small numbers are good.

There will probably be a 1.0.7 soon to fix this.

Nigel McNie: mahara.org finally relaunched!

2010-06-14T03:30:04+00:00

It's taken a while, but finally the new mahara.org has been launched. I think it's a great step up from the old site, and will hopefully serve as a useful place to showcase and document Mahara for a while yet.

DNS is still propogating so it might be a while until everyone can see the new site. In the mean time, we're putting up some content and will tweak the site over the next couple of days.

Our focus will now turn back to the Mahara 1.1 release. There's still some bugs to iron out, but hopefully we can get the release out soon...

Penny Leach: On loving (and hating) software, and how crippling it is to run in high heels

Penny — 2010-05-11T15:07:56+00:00

I have a well established (and well deserved) reputation for hating MySQL and being a Postgres fan. While this is functionally true, I want to examine what is actually behind it.

I tried to find an analogy recently to explain this to my friend Michel, and the best I could come up with was (and actually I think it fits perfectly) the following:

Imagine that you are an athlete. A runner. And you have the most amazing comfortable ergonomic shoes. These shoes fit you perfectly, they are tailored to exactly your body, the way you run. They help you run faster, they enable you to excel at the sport that you have chosen. Then imagine that for whatever reason, you join a running club, and this running club has a policy of running in high heels.

"But why would you do that?!", you exclaim in dismay. "When you could use these other shoes, which are much more comfortable and actually help you run faster".
"Ah well", they reply. "We've always run in high heels and it works fine for us".

That is how I feel whenever I have to touch MySQL.

I remember having seen one of Kathy Sierra's talks about creating passionate users - not sure where, maybe Webstock 2007. 'Passionate users' is something that may seem absurd to many computer users who struggle with daily computer use (I have written about this before), but the reality is that I am a passionate user. There are certain pieces of software that I use every day, and feel passionate about! While writing this blog post, I realised that what I'm really talking about here is tools - things that I use in my daily work as a software developer.

I honestly love working with Postgres. I love working with Git. There are actually times where I need to do something, or solve a problem, and I feel like this software helps me do it. After I solve the problem I am grateful to be able to work with such tools. This is where the random "I love postgres!" or "I love git" tweets come from.

This is a really important point - it's not that I hate MySQL (although I do), or CVS or SVN (also true) - it's that I actually love Postgres and Git, and when I need, for whatever reason, to work with Mysql or CVS or SVN, I really miss the features that make me love their competitors.

Of course, this fails to take into account that different pieces of software have different purposes and features, and this is a good thing, because diversity encourages competition which makes software improve. Interestingly, I don't know if I feel this way about all software - I love Vim, but I don't hate Emacs - I just don't use it. When I have to use Emacs (it happens), I am sometimes frustrated that it's different - but I blame myself for that (not knowing it well enough), rather than blaming the software for being substandard. I'm not sure if Vim is any better or worse than Emacs - Vim just suits me better. In the case of Mysql and Postgres, I actually think that Mysql is substandard, and similarly for Git and CVS or SVN.

There seems to be a scale for my reaction to using different software:

Love: I feel that the software is predictable, elegant and helps me achieve what I want and get my work done. I sometimes feel love for it when it's particularly helpful.
Ambivalence: I use the software, but don't feel particularly strongly about it.
Hatred: Using it actively hinders me working, because it is lacking features I need, or is unpredictable and unreliable. I feel despair when I have to use it.

Obviously, the two extremes are both forms of passion, although I'm sure that the first is what Kathy Sierra was talking about. But I wonder if the software that falls into the third category only exists when there is a competing piece of software that falls into the first category. If I had never used Postgres, would I hate MySQL so much? Probably not, although I might hate it a little bit. I'm sure people that have always run in high heels and never experienced how much easier it is to run in sneakers don't hate their high heels, because they don't know any different.

So the conclusion I must draw, is that I don't actually hate MySQL or CVS or SVN - I hate being forced to use tools that I consider to be substandard than their equivalents, which I actually love. To me this is such a black and white issue I am completely baffled when other people don't see it. Considering the way that most people interact with their computers, if software exists that actually makes people feel love - why on earth would they use anything else?

François Marier: "Abusing" git storage

François — 2010-03-31T14:38:39+00:00

A git repository is primarily intended to store multiple branches of a single program or component. The underlying system however is much more flexible. Here are two ways to add files which are related to the project but outside the normal history.

Storing a single file all by itself

This intermediate trick allows you to store a single file into a git repository without the file being in any of the branches.

First of all, let's create a new file:

echo "Hi" > message.txt

and store it in git:

git hash-object -w message.txt b14df6442ea5a1b382985a6549b85d435376c351

At this stage, the file is stored within the git repository but there is no other way to get to it other than using the hash:

git cat-file blob b14df6442ea5a1b382985a6549b85d435376c351 Hi

A good way to point to the stored contents is to use a tag:

git tag message.txt b14df6442ea5a1b382985a6549b85d435376c351

Now you can access your file this way:

git cat-file blob message.txt Hi

Creating an empty branch

As seen in this git screencast or in the Git Community Book, you can create a branch without a parent or any initial contents:

git symbolic-ref HEAD refs/heads/foo rm .git/index git clean -fdx

To finish it off, add what you were planning on storing there and commit it:

git add myfile git commit -m 'Initial commit'

Dan Marsden: New Mahara book

2010-02-18T10:58:49+00:00

Packt Publishing just sent me a copy of the new Mahara book:
Mahara 1.2 E-Portfolios: Beginner’s Guide

It’s been a while since I’ve sat down and read through a text-book, my eyes tend to glaze over and I find something else to read instead! I’ve also never been very good at sitting down and “studying” – This book surprised me, I was expecting a book detailing the different features in a dry step by step manner, but what I found was quite different. The first paragraph of the content in the book made me feel a little uncomfortable:

So, you’re interested in Mahara? Maybe you are already using it, but you are wondering if you are using it well. Maybe you’ve recently heard of Mahara and you are wondering if this is actually the ePortfolio solution you were looking for? Or, maybe you have been told you have to use it and you just need to get a sense of what Mahara is all about?

The book seemed to be talking to me – that’s not supposed to happen is it? After reading further I began to almost see the writers at the front of a computer lab speaking to me sitting at a PC – this easy to read tone engaged me in the book and the blend of content, hands on activities, and reviews made it an interactive experience – not what I’ve come to expect from the reference manuals I normally have on hand! The book has some great use cases and full detail without overwhelming the reader. It’s split up into logical bite-sized chapters that could be tackled one at a time when the reader has a quiet moment. Packt release this book as an eBook which I found useful – it meant that you could switch over to a web browser and run through the activities on the demo site provided specifically for readers of the book. I have a multi-monitor display which made this easy – some readers will definitely prefer the hard copy. Unfortunately the urls in the eBook weren’t live urls so I had to copy/paste them, but that didn’t take too much effort.

I’d recommend this book to anyone new to ePortfolios or new to Mahara – it gives the reader a really good detailed knowledge of not only the features of Mahara, but how they can be used in a variety of ways for very different groups of people. The book is great for real users – teachers/students/educators – it gives a much “easier to digest”, concise guide than the mahara.org online documentation. It’s not aimed at Administrators or Developers but if you haven’t implemented Mahara yet it gives some good information for new administrators or those considering Mahara’s suitability towards the end of the book. I’d highly recommend that any organisation running Mahara grabs a hard copy of the book to lend someone when they walk into your office and ask “so what’s this Mahara thing?”

It’s available on Amazon, but cheaper from Packt Publishing who sell the eBook as well.

François Marier: Excluding files from git archive exports using gitattributes

François — 2010-02-09T16:57:32+00:00

git archive provides an easy way of producing a tarball directly from a project's git branch.

For example, this is what we use to build the Mahara tarballs:

git archive --format=tar --prefix=mahara-${VERSION}/ ${RELEASETAG} | bzip2 -9 > ${CURRENTDIR}/mahara-${RELEASE}.tar.bz2

If you do this however, you end up with the entire contents of the git branch, including potentially undesirable files like .gitignore.

There is an easy, though not very well-documented, way of specifying files to exclude from such exports: gitattributes.

This is what the Mahara .gitattributes file looks like:

/test export-ignore .gitattributes export-ignore .gitignore export-ignore

With this file in the root directory of our repository, tarballs we generate using git archive no longer contain the selenium tests or the git config files.

If you start playing with this feature however, make sure you commit the .gitattributes file to your repository before running git archive. Otherwise the settings will not be picked up by git archive.

François Marier: Ignoring files in git repositories

François — 2009-12-28T09:56:48+00:00

According to the man page, there are three ways to exclude files from being tracked by git.

Shared list of files to ignore

The most well-known way of preventing files from being part of a git branch is to add such files in .gitignore. (This is analogous to CVS' .cvsignore files.)

Here's an example:

*.generated.html
/config.php

The above ignore list will prevent automatically generated HTML files from being committed by mistake to the repository. Because this is useful to all developers on the project, .gitignore is a good place for this.

The next line prevents the local configuration file from being tracked by git, something else that all developers will want to have.

One thing to note here is the use of a leading slash character with config.php. This is to specifically match the config file in the same directory as the .gitignore file (in this case, the root directory of the repository) but no other. Without this slash, the following files would also be ignored by git:

/app/config.php
/plugins/address/config.php
/module/config.php

Local list (specific to one project)

For those custom files that you don't want version controlled but that others probably don't have or don't want to automatically ignore, git provides a second facility: .git/info/exclude

It works the same way as .gitignore but be aware that this list is only stored locally and only applies to the repository in which it lives.

(I can't think of a good example for when you'd want to use this one because I don't really use it. Feel free to leave a comment if you do use it though, I'm curious to know what others do with it.)

Local list (common to all projects)

Should you wish to automatically ignore file patterns in all of your projects, you will need to use the third gitignore method: core.excludesfile

Put this line in your ~/.gitconfig:

[core]
excludesfile = /home/username/.gitexcludes

(you need to put the absolute path to your home directory, ~/ will not work here unless you use git 1.6.6 or later)

and then put the patterns to ignore in ~/.gitexcludes. For example, this will ignore the automatic backups made by emacs when you save a file:

*~

This is the ideal place to put anything that is generated by your development tools and that doesn't need to appear in your project repositories.

Penny Leach: mahara stand at swiss open expo

Penny — 2009-09-21T08:50:28+00:00

This Wednesday and Thursday I'm going to be at the Swiss Open Expo, "manning" the Mahara stand. Liip are always in heavy presence at the Open Expo, but this time will be the first time we have a Mahara stand, so it's pretty exciting!

If you're in or around Winterthur and interested in Open Source e-portfolios, come and say hi!

Dan Marsden: Moodle Networking bugs

2009-09-11T09:58:06+00:00

I’ve been battling with a couple of issues with Moodle Networking over the past week!

The first was an issue where the configuration of Mnet between a Moodle and Mahara site on the same server worked fine, it obtained the various keys as necessary, but when an Mnet call was made like a Single Sign on call from a Moodle user accessing Mahara, it would fail – with no useable error message anywhere to be found! – I eventually traced it down to a call to xml_parse – turns out that libxml in certain versions of PHP is quite buggy – and this is actually a known issue with other areas of Moodle. MDL-17136 MDL-16759

IMO we should really notify administrators that they are potentially running buggy code so I’ve created MDL-20220 to add such a notification!

The other issue I’ve been battling with is related to Moodle and Mahara not generating any keys at all – this is centered around the installation not being able to locate an openssl.cnf file on the server, and it appears in my testing that adding an environment var on the server didn’t reliably work! – also I needed to get this to work on a XAMPP install or an MoWeS install, so setting an environment var wasn’t really an option! I’ve got an incoming patch for 1.9Stable and Head that fixes this – I also have a patch for Mahara which will go upstream very soon!

François Marier: 3 ways to improve your source control history

François — 2009-07-30T00:52:39+00:00

A few weeks ago, Linus Torvalds wrote a blog post talking about how happy he was with the way that the last Linux merge window was going. Especially given how flexible distributed source control systems are and how long it takes before developers understand how to use them properly.

Most Free Software projects collaborate with external contributors through the use of patches and with project members through shared source control systems. Keeping a clean source control history in this case can:

ease maintenance of the codebase (especially when merging between multiple active branches) and
facilitate collaboration with other developers who need to quickly grasp what your changes are all about.

Here are three things I've put in practice in a few recent projects.

Reduce the noise on the main branch

If you use your source control system in a traditional centralised way, make sure that the main branch has as little noise as possible:

use feature branches when developing new functionality (i.e. use one branch per feature)
rebase into a small number of commits instead of merging directly into the main branch (or use something like git cherry-pick -n on your commits)

It will then be much easier to port these changes across to a different version and to revisit the changes several months later.

Make each commit a logical unit

Ideally, the main branch should be fully usable at any point in its history. For example, if a refactoring commit breaks an existing feature, it should also include any necessary fixes. Progress and fixup commits belong to private branches and should be cleaned up before the code is published on a public repository.

A commit shouldn't be too small, but at the same time, it shouldn't be too large. Before pushing a large commit, ask yourself whether or not it could be broken down into smaller or more meaningful units. Introducing a new feature through a number of smaller self-contained units often makes a branch's history much more readable.

Write meaningful commit messages

A commit message will typically be seen along with its patch. Therefore the ideal commit message will include everything which is not immediatly obvious from the code changes:

A short description of the new feature it introduces (or how to reproduce the bug it fixes)
Links to related commits (e.g. "Fix bug introduced in COMMIT_ID")
Bug numbers related to this commit
The main area of the code (or subsystem) that this commit touches (as part of the short description, so that it's easily grepable)

Here's an example of a good commit message from version 2.6.29 of the Linux kernel:

commit 4869fdea77052c480c54b9e66a927ba036eab29e
Author: Mikulas Patocka 
Date:   Mon Jun 22 10:08:02 2009 +0100

dm mpath: validate table argument count

commit 0e0497c0c017664994819f4602dc07fd95896c52 upstream.

The parser reads the argument count as a number but doesn't check that
sufficient arguments are supplied. This command triggers the bug:

dmsetup create mpath --table "0 `blockdev --getsize /dev/mapper/cr0`
 multipath 0 0 2 1 round-robin 1000 0 1 1 /dev/mapper/cr0
 round-robin 0 1 1 /dev/mapper/cr1 1000"
kernel BUG at drivers/md/dm-mpath.c:530!

Good commits and useful source control histories are great ways of documenting your code. If there are other conventions or practices you've found useful in your use of source control systems, please leave a comment!

François Marier: Writing the perfect patch

François — 2009-07-01T01:10:54+00:00

Other people have written and talked (in Lecture 3) about writing the perfect patch for a Free Software project. The goal there is to increase the likelihood that a patch will be accepted by the project developers.

Integrating and testing patches takes time and so reducing that burden is essential when interacting with busy maintainers. Especially if they're volunteers.

Here's what I try to keep in mind when preparing a patch.

Use the right options to `diff`

These two options should always be part of your call to the diff command:

-u: use the most common patch format, unidiff.
-p: include the name of the function that's being changed.

and this one can be useful if the output seems unnecessarily large:

-d: try hard to find a smaller set of changes.

Minimize the number of changes

You need to draw attention to the changes that you're proposing and remove all other potential distractions:

Follow the coding style of the original file. Your changes must fully blend in or they are likely to be rejected.
Do not re-indent existing code. This will make it look like you modified every line.
Pay attention to whitespace changes. In particular: end-of-line characters, trailing spaces and tab-versus-space differences. Use the dos2unix or unix2dos commands if you need to.
Gratuitous refactoring of existing code. Unless the refactoring makes your change smaller or easier to understand, keep it for another patch.

Of course all of the above would be acceptable patches on their own, just not combined with other types of changes.

Only one logical change at a time

Patches often need to be broken up into a series of logical changes to avoid these two extremes:

the gigantic patch which adds a number of features and fixes a couple of bugs but scares everybody
a series of interdependent patches which all relate to the same change and must all be applied together

It's a bit of a balancing act, but a good rule of thumb is:

to have one patch per feature or bug and
to try to find the smallest (yet meaningful) change which can be applied on its own.

It's not just about the patch

Your patch can be really good, but the email (or the bug tracker update) announcing it should also contain:

a good description of the problem it solves and how it solves it
the output of diffstat to give an idea of the size of the change

François Marier: Troubleshooting Postgres Performance Problems

François — 2009-05-30T18:09:29+00:00

There are great resources to help tune PostgresQL for performance, but say you've got a Postgres database which suddenly becomes very slow. How can you figure out what's going on?

General system state

A slow database server will typically be running out of:

cpu,
memory or
disk.

So the first thing you should do is to get an overview of the system state using common UNIX tools such as:

top (press c to see the full command line): look at CPU and memory usage
iostat -x -m 5: look at i/o wait and service time

If you notice that memory is exhausted, you could play with the following postgresql.conf settings:

shared_buffers
work_mem
effective_cache_size

If you see a lot of I/O, then try adjusting these settings:

checkpoint_segments
autovacuum_naptime
wal_buffers

Finding slow queries

On the database side, start by determining whether:

there is one large query holding up all of the resources or
the number of queries is the main problem.

To do this, make sure that the following setting is turned on in your postgresql.conf:

stats_command_string = on

Then fire up psql dbname and look at the currently executing queries:

SELECT usename, current_query, query_start
FROM pg_stat_activity;

(You can also add the procid column to the query if you want to map a query to the process ID you see in top.)

If you can't get anything useful out of pg_stat_activity, you may want to enable slow query logging by adding this to your postgresql.conf:

log_min_duration_statement = 100

(All queries which take more than 100 ms to execute will be printed to the main Postgres log file.)

Examining a particular query

Once you have identified a slow query, you can time it by running this command before executing the query manually:

\timing

To get an idea of where Postgres spends its time when executing that query, look at the query plan:

EXPLAIN your_query;

The numbers you see there are estimates. To run the query and get actual numbers, use this instead:

EXPLAIN ANALYZE your_query;

If you still can't figure out why a certain query takes so long, have a look at the ratio between live and dead rows in the relevant tables:

ANALYZE VERBOSE tablename;

Having too many dead rows is often a sign of insufficient vacuuming. You might want to turn autovacuum on if it isn't already, or to make it a bit more aggressive by tweaking these settings:

autovacuum_vacuum_scale_factor
autovacuum_analyze_scale_factor

Troubleshooting database performance is a bit of a black art, many thanks to Mark Kirkwood for sharing his Postgres wisdom with me.

If there's anything else you have successfully used to find the cause of your performance woes, please feel free to leave a comment.

François Marier: Handling security bugs in your Free Software project

François — 2009-03-10T22:47:05+00:00

If you are managing a Free Software project, you may eventually be confronted with a security vulnerability. Normally bugs in the Open Source world are discussed in a transparent way on public forums. In the case of security bugs however, there are benefits to temporarily withholding these details from the public.

Some people describe this approach as responsible disclosure. It boils down to this:

if the vulnerability is not publicly known, warn the vendors first and give them some time to fix it before making the details public.

If the vulnerability is already public knowledge, then you can focus on fixing it as soon as possible and maybe let reporters know about a better way to report their findings.

Sample Security Policy

Here's the procedure we now follow in the Mahara project for security bugs we found ourselves or which have been privately disclosed to us through security@mahara.org:

Figure out the extent of the problem: which versions of Mahara are affected by this problem?
Fix the problem on all supported branches of the project in a private source control repository.
Share the vulnerability information with vendor-sec and request a CVE identifier.
Prepare release tarballs and packages.
Draft a security advisory for the Security forum.
Wait until the embargo date to push the release out.

Emails to vendor-sec should include:

a description of the problem
any proof of concept code demonstrating the vulnerability
the list of affected versions
any patches you have prepared
a proposed embargo date

Benefits of properly handling security bugs

First and foremost, it protects end-users by giving them a chance to download and install fixed versions of your software before widespread exploitation of the security flaws.

Secondly, it may increase your project credibility. While you are admitting that your software has flaws, you are also demonstrating that your project is committed to dealing with the most serious ones promptly and in a responsible manner.

Finally, sharing security flaws and having your fixes reviewed by a select group of experts may reveal extra vulnerabilities you missed while preparing your patches. This gives you an opportunity to improve your fixes before they are released and to avoid having to issue yet another security advisory a few days later.

Big thanks to Steffen Joeris for his help in shaping the Mahara policy!

Planet Mahara

Dan Marsden: Awards Time?

François Marier: Cherry-picking a range of git commits

Cherry-picking a single commit

Cherry-picking a range of commits

Another approach

Update: looking forward to git 1.7.2

François Marier: Reducing website bandwidth usage

HTML, XML, Javascript and CSS files

Images

François Marier: Handling email-based events safely

Nigel McNie: Removing the 'referer' part of PHP Error Messages from Apache Logs

Nigel McNie: What's up in Mahara land recently?

Hackfests

Mahara 1.2

Code Contribution Policy

Mahara 1.3

Mahara Growth

Nigel McNie: Mahara 1.2 Import/Export Demo

Nigel McNie: Themes In Mahara 1.2

Nigel McNie: Mahara 1.1 - almost there!

Nigel McNie: Things You Don't Want to See in the Morning

Nigel McNie: mahara.org finally relaunched!

Penny Leach: On loving (and hating) software, and how crippling it is to run in high heels

François Marier: "Abusing" git storage

Storing a single file all by itself

Creating an empty branch

Dan Marsden: New Mahara book

François Marier: Excluding files from git archive exports using gitattributes

François Marier: Ignoring files in git repositories

Shared list of files to ignore

Local list (specific to one project)

Local list (common to all projects)

Penny Leach: mahara stand at swiss open expo

Dan Marsden: Moodle Networking bugs

François Marier: 3 ways to improve your source control history

Reduce the noise on the main branch

Make each commit a logical unit

Write meaningful commit messages

François Marier: Writing the perfect patch

Use the right options to diff

Minimize the number of changes

Only one logical change at a time

It's not just about the patch

François Marier: Troubleshooting Postgres Performance Problems

General system state

Finding slow queries

Examining a particular query

François Marier: Handling security bugs in your Free Software project

Sample Security Policy

Benefits of properly handling security bugs

Use the right options to `diff`