February 09, 2010

François Marier

Excluding files from git archive exports using gitattributes

git archive provides an easy way of producing a tarball directly from a project's git branch.

For example, this is what we use to build the Mahara tarballs:

git archive --format=tar --prefix=mahara-${VERSION}/ ${RELEASETAG} | bzip2 -9 > ${CURRENTDIR}/mahara-${RELEASE}.tar.bz2
If you do this however, you end up with the entire contents of the git branch, including potentially undesirable files like .gitignore.

There is an easy, though not very well-documented, way of specifying files to exclude from such exports: gitattributes.

This is what the Mahara .gitattributes file looks like:
/test export-ignore
.gitattributes export-ignore
.gitignore export-ignore
With this file in the root directory of our repository, tarballs we generate using git archive no longer contain the selenium tests or the git config files.

If you start playing with this feature however, make sure you commit the .gitattributes file to your repository before running git archive. Otherwise the settings will not be picked up by git archive.

by François (fmarier@gmail.com) at February 09, 2010 09:00 AM

December 28, 2009

François Marier

Ignoring files in git repositories

According to the man page, there are three ways to exclude files from being tracked by git.

Shared list of files to ignore

The most well-known way of preventing files from being part of a git branch is to add such files in .gitignore. (This is analogous to CVS' .cvsignore files.)

Here's an example:
*.generated.html
/config.php
The above ignore list will prevent automatically generated HTML files from being committed by mistake to the repository. Because this is useful to all developers on the project, .gitignore is a good place for this.

The next line prevents the local configuration file from being tracked by git, something else that all developers will want to have.

One thing to note here is the use of a leading slash character with config.php. This is to specifically match the config file in the same directory as the .gitignore file (in this case, the root directory of the repository) but no other. Without this slash, the following files would also be ignored by git:
/app/config.php
/plugins/address/config.php
/module/config.php

Local list (specific to one project)

For those custom files that you don't want version controlled but that others probably don't have or don't want to automatically ignore, git provides a second facility: .git/info/exclude

It works the same way as .gitignore but be aware that this list is only stored locally and only applies to the repository in which it lives.

(I can't think of a good example for when you'd want to use this one because I don't really use it. Feel free to leave a comment if you do use it though, I'm curious to know what others do with it.)

Local list (common to all projects)

Should you wish to automatically ignore file patterns in all of your projects, you will need to use the third gitignore method: core.excludesfile

Put this line in your ~/.gitconfig:
[core]
excludesfile = /home/username/.gitexcludes
(you need to put the absolute path to your home directory, ~/ will not work here unless you use git 1.6.6 or later)

and then put the patterns to ignore in ~/.gitexcludes. For example, this will ignore the automatic backups made by emacs when you save a file:
*~
This is the ideal place to put anything that is generated by your development tools and that doesn't need to appear in your project repositories.

by François (fmarier@gmail.com) at December 28, 2009 10:56 AM

October 09, 2009

François Marier

Reducing website bandwidth usage

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.

HTML, XML, Javascript and CSS files

One of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:
AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip

Images

As far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):
  • gifsicle -O2 -b image.gif
  • jpegoptim -p --strip-all image.jpg
  • optipng -o7 -q image.png
(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.

by François (fmarier@gmail.com) at October 09, 2009 07:30 PM

September 21, 2009

Penny Leach

mahara stand at swiss open expo

This Wednesday and Thursday I'm going to be at the Swiss Open Expo, "manning" the Mahara stand. Liip are always in heavy presence at the Open Expo, but this time will be the first time we have a Mahara stand, so it's pretty exciting!

If you're in or around Winterthur and interested in Open Source e-portfolios, come and say hi!

by penny@she.geek.nz (Penny) at September 21, 2009 08:50 AM

September 14, 2009

Penny Leach

LEAP2A support coming soon to a Moodle near you!

Finally, I am very happy to annouce that I will soon be working on adding LEAP2A support to the Moodle Portfolio API that will be in Moodle 2.0.

There's already a rough specification for this work, and it's been on my TODO list for a very long time.

LEAP2A is a very simple and open atom-based e-portfolio standard, to promote interoperability between e-portfolio systems. Interoperability is very important in an e-portfolio system, because it is vital to be able to transport portfolio data around with you, as you move between educational providers, into higher education, and on to professional development. Imagine having in one portfolio system, your entire portfolio of work, starting from your first day at primary school, right up to your continued professional development. Of course, one would use many different portfolio systems over that time, so some sort of open standard to transport data around is imperative.

I was involved with the LEAP standard group last year when I was living in London, adding LEAP2A support to Mahara. I attended the LEAP meetings in the UK, and worked on the export side of the project. Nigel took over when I left Catalyst and continued, adding the import side (which is of course, much harder).

But wait, you ask. Isn't Moodle a learning management system, not an e-portfolio? Why do we care about interoperability with an e-portfolio standard? The answer is of course, that while you're working, you are entering data into Moodle, and at the end of the course, the natural thing to want to do is to export some of that data into your portfolio. At the moment, we have the Portfolio API in Moodle for that, which I worked on for last year for 3 months, during my time at Moodle HQ. However, at the time I wrote it, it wasn't at all clear what portfolio standard we should support, so content is transferred in "raster" format (rendered to HTML or a file like a pdf). Since then though, LEAP has emerged as a clear front runner, and now that Mahara 1.2 (almost released!) fully supports importing and exporting LEAP2A, the time is right for us to take the plunge and add LEAP2A support to Moodle's Portfolio API. This improves the integration between Moodle and Mahara, as well as opening the door for Moodle to integrate better with other e-portfolio systems that implement the LEAP2A standard.

I am therefore very grateful to the State of New Hampshire for providing the funding to Liip for me to do this work. This comes from a grant from the New Hampshire Department of Education, and a collaborate group made up of the following school districts:

Special thanks also to Matt Oquist who has been tirelessly helping me find funding for this work!


by penny@she.geek.nz (Penny) at September 14, 2009 12:25 PM

August 23, 2009

Nigel McNie

What's up in Mahara land recently?

Here's a snapshot from my point of view about what's going on across all aspects of Mahara.

Hackfests

Yesterday, Penny, Jordi (both from Mahara partner Liip) and I had another weekend hackfest. Penny has been adding a framework for unit testing, and managed to get the first test working. Jordi was working on swapping Smarty for Dwoo, while I continued hacking on View Themes support. Pics: Switzerland end, NZ end.

Penny has been working on unit testing, as it's something we developers have realised has been lacking in Mahara for some time now. What unit tests do is give us more confidence that when we make a change (whether it's a bug fix or new feature), that the rest of Mahara continues working exactly the way it used to - i.e., we didn't inadvertently add any more bugs. This has the result that we can continue development safe in the knowledge that we didn't break anything that worked before, which will result in better quality. It will also make it easier for enthusiastic hackers to make changes to Mahara and know they haven't broken anything important.

Unit testing isn't the only way we've been trying to integrate testing - Gerald Quimpo from Catalyst has been resurrecting the Selenium tests that come with Mahara, and has done an excellent job in getting them working again, while also setting up a Continuous Integration server (Hudson) so the tests will be run regularly. Soon, we'll have that available publicly so everyone can see which tests are currently passing and failing.

Backtracking a tiny bit - Jordi has been working on swapping Smarty for Dwoo, a change we would like to do for a couple of reasons. Firstly, Smarty is bigger and slower, perhaps largely due to the fact that it needs to support PHP4 (which Mahara doesn't support). Secondly, Smarty doesn't make it easy for us to output variables that are HTML escaped by default - which is currently the leading cause of security vulnerabilities in Mahara. We haven't had many security vulns actually, Mahara is reaping the benefits of a security-oriented development ethic. But of the few we've had, almost all of them have been because someone forgot to put |escape on the end of a smarty variable. So switching to Dwoo will cut down that entire avenue of attack.

Mahara 1.2

We released the first beta of Mahara 1.2 last week. That means we're looking for bug reports now, and encourage people to download it and try out the new features. Here's a short list of the coolest new features in 1.2:

  • Import and Export - Mahara supports the LEAP2A standard for import/export, and in addition, allows you to export an HTML version of your portfolio (including a nice HTML represntation of an individual View).
  • "My Files" and file handling improvements - now the My Files section makes it easier to quickly upload files, and degrades to work without javascript. You can now upload files directly into Views, and use a standard directory browser to choose files in Views. For developers, we created a pieform element for uploading/attaching files, which will make it easier than ever to do this in custom plugins.
  • Theming - we're shipping with 6 themes, to give people an idea of how they can customise their Mahara. We've done a lot of rework of how themes are created, so from 1.2 it'll be even easier to create a custom theme. We even documented the process (finally!).
  • Usability/Comprehension tweaks - Now groups get a default forum, and blogs get a default blog. If there is only one forum or blog, users are taken straight to it. This should really help in preventing the problem where people create a blog as if it's a blog post, and should make the forums an even more useful tool.
  • Improved Moodle/Mahara integration - with a custom patch to Moodle 1.9, you'll be able to submit a View to Moodle for assessment as an assignment answer. How useful is that? Now people can keep their data in their portfolio, and mash it together into a View for assessment over on Moodle, where all the useful grading functionality is :)

There's s still a bit of work to go before Mahara 1.2 will be ready though. I need to finish off LEAP2A import - currently resume import is only half finished, and view import is just begun. We'll also need a fancy UI for it, and of course there are some inevitible bugs that will need squashing.

Also, I'd really like to get view themes to a point where it can be merged also. That'll really help people personalise their environment. If I get around to it, maybe we can add a theme chooser for users to change how they see the entire site, not just individual Views.

Other than that though, I think we're looking pretty good!

Code Contribution Policy

We are currently preparing a policy for contributors to the codebase. We aren't going to be wanting copyright assignment, but we are going to make sure all contributions are as legal as we can make them ;). Unlike many open source projects, Mahara is actually in a pretty good position legally, as most members of the team are well aware of the legal issues around open source and licensing, and we take legal issues seriously. Our policy won't be onerous, but will ensure we stay legit now and into the future.

Mahara 1.3

Though 1.2 is not out yet, talk is already shifting in some departments to the plans we have for Mahara 1.3. The big focus there will be usability, and we're going to target that in a couple of main areas - the navigation, and View creation.

For the navigation, our current plan is to adopt a dropdown menu structure, organised more to how users think when they're looking for something. The main categories will be roughly along the lines of "My Stuff", "Friends & Groups" and "Sharing". We'll build a demo of this at some point after 1.2 so the community can see it in action and give feedback, though I've seen it in action elsewhere and it's easy to grasp. Furthermore, the addition of "horizontal navigation" - where when you're at one page you're given suggestions about what else you might be looking for - should help tremendously as well.

We're also going to have a good look at the View creation process. The Drag and Drop and layout concepts won't be changing, but the layout of the editing screens themselves will be improved a lot. We're hoping we can even ditch a step to make things simpler. I think the biggest gains will be seen on the View Access screen, which currently can be a bit of a minefield to navigate. We'll get mockups out some time after 1.2 also.

Other stuff we might do could include Group Views, a User Dashboard (as a View of course!), and overhauling the notification system.

Mahara Growth

And last but not least, here are some interesting statistics about Mahara and its community:

We now have over 3000 members of the mahara.org community, with 17 partners worldwide. The community growth has been steady since we launched the new mahara.org last November. The site receives nearly 25,000 visits a month (for the geeks among you: 57% firefox and 27% IE). Of these visitors, we get the most from the US, followed by the UK, Germany, Spain, Australia and then New Zealand.

Since we added the ability for sites to register on mahara.org in May, we now have over 50 sites doing this every week. These sites together add to some 12,000 users, but this doesn't include mahara.org, or the New Zealand MyPortfolio service - if these numbers were included, the total would rise to nearly 25,000. Again for the geeks - we are seeing a 75/25 split of MySQL to PostgreSQL. Hopefully things will skew more to the PostgreSQL side of that equation as time goes on :). Of the registered sites, the largest seem to be run by Blackpool University, Uni Krems (Austria) and Birmingham City University, all of which have thousands of users.

August 23, 2009 03:11 AM

August 02, 2009

Penny Leach

Changing my approach to keysigning

After attending Debconf 9 in Cáceres, Spain, and talking to various people about their different approaches to validating identity, I have decided to change my approach to keysigning.

In order to sign your key, I must have met you before and can recall your face or name. If I have not met you before, I must have had an interesting conversation with you during this meeting, and am relatively sure I will be able to recall you in future meetings.

I will of course still need to look at government issued identification for people I don't know very well who fit into one of the two above criteria.

Part of the reason for this change is that for the second time, someone didn't want to sign my key because my New Zealand passport has the name "Penelope" (along with all other documents like my two driving licenses and Swiss B Permit). While I can understand this, to me it shows that the traditional method of verifying identity using government issued identification is fundamentally flawed. When I think about what comprises my identity, yes, my name is a big part of it, and I consider my name to be Penny. I have never in my life been called Penelope. Not by my family, teachers, colleagues, friends or lovers. It's just not part of my identity, and anyone who knows anything about me, probably knows that. So what are we trying to achieve by keysigning then? The fact that the person you meet is in control of that key, and you've verified their identity. Sorry, but "Penelope" doesn't go a very long way to make up any part of my identity.

Identity is something I've been thinking about a lot lately for various other reasons (largely after reading Simon Grant's book about the use of e-portfolios for personal information, personal development and personal values), and intend to blog about in the future.

At any rate, conferences like Debconf are a great way to meet people and have interesting and thought provoking conversations, and me being difficult and refusing to sign someone's key because I didn't know them, or hadn't had any sort of meaningful conversation with them, led to a few interesting and thought provoking conversations. So there's an added bonus.

by penny@she.geek.nz (Penny) at August 02, 2009 08:27 AM

July 29, 2009

François Marier

3 ways to improve your source control history

A few weeks ago, Linus Torvalds wrote a blog post talking about how happy he was with the way that the last Linux merge window was going. Especially given how flexible distributed source control systems are and how long it takes before developers understand how to use them properly.

Most Free Software projects collaborate with external contributors through the use of patches and with project members through shared source control systems. Keeping a clean source control history in this case can:

  • ease maintenance of the codebase (especially when merging between multiple active branches) and
  • facilitate collaboration with other developers who need to quickly grasp what your changes are all about.
Here are three things I've put in practice in a few recent projects.

Reduce the noise on the main branch

If you use your source control system in a traditional centralised way, make sure that the main branch has as little noise as possible:
  • use feature branches when developing new functionality (i.e. use one branch per feature)
  • rebase into a small number of commits instead of merging directly into the main branch (or use something like git cherry-pick -n on your commits)
It will then be much easier to port these changes across to a different version and to revisit the changes several months later.

Make each commit a logical unit

Ideally, the main branch should be fully usable at any point in its history. For example, if a refactoring commit breaks an existing feature, it should also include any necessary fixes. Progress and fixup commits belong to private branches and should be cleaned up before the code is published on a public repository.

A commit shouldn't be too small, but at the same time, it shouldn't be too large. Before pushing a large commit, ask yourself whether or not it could be broken down into smaller or more meaningful units. Introducing a new feature through a number of smaller self-contained units often makes a branch's history much more readable.

Write meaningful commit messages

A commit message will typically be seen along with its patch. Therefore the ideal commit message will include everything which is not immediatly obvious from the code changes:
  • A short description of the new feature it introduces (or how to reproduce the bug it fixes)
  • Links to related commits (e.g. "Fix bug introduced in COMMIT_ID")
  • Bug numbers related to this commit
  • The main area of the code (or subsystem) that this commit touches (as part of the short description, so that it's easily grepable)
Here's an example of a good commit message from version 2.6.29 of the Linux kernel:
commit 4869fdea77052c480c54b9e66a927ba036eab29e
Author: Mikulas Patocka 
Date:   Mon Jun 22 10:08:02 2009 +0100

dm mpath: validate table argument count

commit 0e0497c0c017664994819f4602dc07fd95896c52 upstream.

The parser reads the argument count as a number but doesn't check that
sufficient arguments are supplied. This command triggers the bug:

dmsetup create mpath --table "0 `blockdev --getsize /dev/mapper/cr0`
 multipath 0 0 2 1 round-robin 1000 0 1 1 /dev/mapper/cr0
 round-robin 0 1 1 /dev/mapper/cr1 1000"
kernel BUG at drivers/md/dm-mpath.c:530!
Good commits and useful source control histories are great ways of documenting your code. If there are other conventions or practices you've found useful in your use of source control systems, please leave a comment!

by François (fmarier@gmail.com) at July 29, 2009 11:52 PM

July 23, 2009

Penny Leach

Best Education Hacker award

It seems I am the recipient of one of this year's Google-O'Reilly Open Source Awards

This is really an amazing thing for me and I'm quite overwhelmed by it. Looking back over the last 5 years I've been working on this Open Source Education stuff, there are so many people who have helped me out, inspired and motivated me, and I'm quite sure that this award is a recognition of all of the work of those people as well. The list is far too long to mention, but some of the people who stand out the most over the years are Andrew McMillan, Martín Langhoff, Richard Wyles, Martin Dougiamas, Eloy Lafuente, Nicolas Connault, Nigel McNie, and Heinz Krettek.

Additionally, I am one of those lucky people who get to work on Open Source during work time - from 2004 to early this year for Catalyst in New Zealand, and now for Liip in Switzerland. I am immensely grateful that I have had the encouragement and support of my employers.

Finally, a pony!

The response to this has been immense - I have had emails, tweets, instant messages from many many people. I cannot respond to everyone individually but I am very grateful for the support from each and every one of you.

by penny@she.geek.nz (Penny) at July 23, 2009 06:31 AM

July 01, 2009

François Marier

Writing the perfect patch

Other people have written and talked (in Lecture 3) about writing the perfect patch for a Free Software project. The goal there is to increase the likelihood that a patch will be accepted by the project developers.

Integrating and testing patches takes time and so reducing that burden is essential when interacting with busy maintainers. Especially if they're volunteers.

Here's what I try to keep in mind when preparing a patch.

Use the right options to diff

These two options should always be part of your call to the diff command:
  • -u: use the most common patch format, unidiff.
  • -p: include the name of the function that's being changed.
and this one can be useful if the output seems unnecessarily large:
  • -d: try hard to find a smaller set of changes.

Minimize the number of changes

You need to draw attention to the changes that you're proposing and remove all other potential distractions:
  • Follow the coding style of the original file. Your changes must fully blend in or they are likely to be rejected.
  • Do not re-indent existing code. This will make it look like you modified every line.
  • Pay attention to whitespace changes. In particular: end-of-line characters, trailing spaces and tab-versus-space differences. Use the dos2unix or unix2dos commands if you need to.
  • Gratuitous refactoring of existing code. Unless the refactoring makes your change smaller or easier to understand, keep it for another patch.
Of course all of the above would be acceptable patches on their own, just not combined with other types of changes.

Only one logical change at a time

Patches often need to be broken up into a series of logical changes to avoid these two extremes:
  • the gigantic patch which adds a number of features and fixes a couple of bugs but scares everybody
  • a series of interdependent patches which all relate to the same change and must all be applied together
It's a bit of a balancing act, but a good rule of thumb is:
  • to have one patch per feature or bug and
  • to try to find the smallest (yet meaningful) change which can be applied on its own.

It's not just about the patch

Your patch can be really good, but the email (or the bug tracker update) announcing it should also contain:
  • a good description of the problem it solves and how it solves it
  • the output of diffstat to give an idea of the size of the change

by François (fmarier@gmail.com) at July 01, 2009 12:10 AM

June 18, 2009

Penny Leach

Mahara hackfest weekend of 26/27/28 June

There's a huge amount of work still to be done for Mahara 1.2 to be released, and Nigel and I have decided to have a formal hackfest rather than just our semi regular Saturday morning CH/Saturday night NZ irc hackfests.

We will be in the irc channel (#mahara-dev on freenode) all weekend (barring normal everyday things like eating dinner) and working on the list of things still to do for Mahara 1.2. We will ourselves mainly be working on the theming changes that Nigel already blogged about, but anyone else interested in getting into Mahara development and helping us get 1.2 out the door would be more than welcome to join in and we have many bugs on the tracker that you can help with!

by penny@she.geek.nz (Penny) at June 18, 2009 06:20 AM

Nigel McNie

Mahara 1.2 Import/Export Demo

OK so my last post was just all text, which is a slog to read through. So here, I made a video about Mahara 1.2's import/export:

I don't think I got the sound levels right and I'd have loved to have added a background track, but video editing on linux is a pain. It's only my second video so be nice. The next one should be better again ;)

June 18, 2009 03:50 AM

June 17, 2009

Nigel McNie

Themes In Mahara 1.2

One of the things we're doing for the upcoming 1.2 release of Mahara is doing a big tidyup of the theming. There are some code level changes, but the most important changes are to do with the themes themselves.

Note that I said "themes". This is because 1.2 won't have just one theme - it'll have six. One is a brushed up version of the default, one is "special" (more on that in a minute), and the other four are all-new themes that will give Mahara users a chance to immediately personalise their installation "out of the box". The new themes are mainly just colour changes, but they're pretty damn good looking colour changes all the same. They'll also give people good examples to work from when creating their own themes.

Now, the special theme. Mahara will now have a theme called 'raw', which is a very stripped down theme. There's very few colours or styles, and most of the theme's job is to position everything correctly and set sensible defaults. All other themes extend from this theme. Mahara has a theme inheritance system, where by you can specify another theme as the 'parent' of your theme, and thus rely on it to provide the templates and most of the CSS. From now, every theme will ultimately extend from the 'raw' theme - including the default theme.

The inheritance can have any number of levels, so for example a theme someone did for Mahara 1.1 might have a parent of the default theme which might be parented by the raw theme. Sadly, because we haven't documented the theme inheritance well, some of the themes out there have copied the entire default/ directory to create their new theme. These people will probably have noticed that when new versions of Mahara come out, sometimes bugs they are supposed to fix are not fixed - this is because the fix was made in a 'default' theme template, which is overridden by their theme's copy. When they upgrade to 1.2, it's highly likely their theme will totally break.

Elsewhere on the backward compatibility front, it should be easier though. If your theme uses theme inheritance, then it will continue to inherit from the (slightly changed) default theme. You may also have to change some CSS selectors in your stylesheet to match the new HTML structure.

Speaking of which, we've cleaned up the HTML immensely, and the CSS along with it. I think the default stylesheet was over 50KB in previous versions of Mahara, at the moment I have the raw theme down to much less than half that, and I don't think it will end up much bigger than 25KB. The default theme's stylesheet will be negligible in size. We're also going to cut the print stylesheet down to size, and if we get the chance we'll add in RTL and browser-specific stylesheets too.

The net result of all of this is a bit of short-term pain while people migrate their themes, but a lot of long-term gain, as the HTML will be cleaner, the CSS smaller and every Mahara faster because of it. Theme development will be much easier from 1.2 onwards, I'll make sure we write documentation so people don't get caught out by the theme inheritance issue again.

Maybe at some point after 1.2 we can look at some fine-tuning and performance of the theming system. I'd love to crank out some CSS sprites and javascript minification, but right now I don't think there's going to be much time other than for 'easy wins', such as expires headers and configuring mod_deflate (which Mahara does already).

Anyway, that's enough from me for now. We have this theming work and finishing off import/export to do, followed by a lot of bug fixing before 1.2 can be released. Look out for another alpha soon, at least.

June 17, 2009 02:21 PM

June 15, 2009

Penny Leach

Announcement: Liip now a Mahara partner!

I am happy to announce that Liip are now a Mahara partner! Mahara is an Open Source ePortfolio system.

In many respects Mahara is a "sister" application to Moodle, providing students with a learning environment that they themselves own, giving them them the ability to showcase their work and collaborate with their peers. However, Mahara is also well suited as a social networking system, running out of the box without Moodle.

Mahara was originally funded by the New Zealand government's Tertiary Education Commission, and has grown into a thriving open source product that is increasingly being adopted worldwide. It makes a lot of sense for Liip to be a partner, both because we're already the official Moodle partner in Switzerland, and Mahara fits very well into our existing list of projects we work with, but also because I have been involved in Mahara since the start, and wanted to continue my involvement since coming to Liip. Liip have been fantastic supporting me in this goal!

by penny@she.geek.nz (Penny) at June 15, 2009 09:21 AM

May 30, 2009

François Marier

Troubleshooting Postgres Performance Problems

There are great resources to help tune PostgresQL for performance, but say you've got a Postgres database which suddenly becomes very slow. How can you figure out what's going on?

General system state

A slow database server will typically be running out of:
  • cpu,
  • memory or
  • disk.
So the first thing you should do is to get an overview of the system state using common UNIX tools such as:
  • top (press c to see the full command line): look at CPU and memory usage
  • iostat -x -m 5: look at i/o wait and service time
If you notice that memory is exhausted, you could play with the following postgresql.conf settings:
shared_buffers
work_mem
effective_cache_size
If you see a lot of I/O, then try adjusting these settings:
checkpoint_segments
autovacuum_naptime
wal_buffers

Finding slow queries

On the database side, start by determining whether:
  • there is one large query holding up all of the resources or
  • the number of queries is the main problem.
To do this, make sure that the following setting is turned on in your postgresql.conf:
stats_command_string = on
Then fire up psql dbname and look at the currently executing queries:
SELECT usename, current_query, query_start
FROM pg_stat_activity;
(You can also add the procid column to the query if you want to map a query to the process ID you see in top.)

If you can't get anything useful out of pg_stat_activity, you may want to enable slow query logging by adding this to your postgresql.conf:
log_min_duration_statement = 100
(All queries which take more than 100 ms to execute will be printed to the main Postgres log file.)

Examining a particular query

Once you have identified a slow query, you can time it by running this command before executing the query manually:
\timing
To get an idea of where Postgres spends its time when executing that query, look at the query plan:
EXPLAIN your_query;
The numbers you see there are estimates. To run the query and get actual numbers, use this instead:
EXPLAIN ANALYZE your_query;
If you still can't figure out why a certain query takes so long, have a look at the ratio between live and dead rows in the relevant tables:
ANALYZE VERBOSE tablename;
Having too many dead rows is often a sign of insufficient vacuuming. You might want to turn autovacuum on if it isn't already, or to make it a bit more aggressive by tweaking these settings:
autovacuum_vacuum_scale_factor
autovacuum_analyze_scale_factor

Troubleshooting database performance is a bit of a black art, many thanks to Mark Kirkwood for sharing his Postgres wisdom with me.

If there's anything else you have successfully used to find the cause of your performance woes, please feel free to leave a comment.

by François (fmarier@gmail.com) at May 30, 2009 05:09 PM

April 07, 2009

Penny Leach

hello, I now live in Switzerland!

Here's my general update of what's happened over the last few months of completely shifting my life around.

I'm living in Fribourg, Switzerland. This is a really cute little town, just 20 minutes train ride from the capital, Bern. I think it's supposed to be officially French-German bilingual but it really isn't. French is the language spoken in all the shops and cafes. I don't have any French yet, so this has been a bit of a challenge - the lowest point was trying to buy the official Rubbish bags at the supermarket.

It took quite awhile to find an apartment - I think I looked at between 20 and 25 altogether before I found a perfect apartment just over a bridge from the fringe of town - I'm living in a renovated apartment in an old house on the river bank and it's absolutely perfect. I can hear the bells from the cathedral and I have a good walk to work, 35 minutes, and there's grass and river and a bus stop outside for the mornings where I take longer than normal to get up. I found a vegetable market just over the bridge on Saturday mornings, that has a whole stall dedicated to mushrooms! This made me disproportionately gleeful. After almost a month of looking at apartments that I didn't want, I visited the apartment, signed the contract, moved in, and bought & assembled most of the furniture within a single weekend, with the help of my amazing boyfriend. Apparently this is almost unheard of in Switzerland.

I'm working at Liip, and it's great. I cannot imagine being thrown into a better bunch of people in a foreign city/country/language. I'm working on Moodle, doing some Open Source projects, working with great people, drinking a lot of beer and throwing the pony around. This is not a euphemism! I got to be a part of the group celebrating Liip doing so well at the Best of Swiss Web awards (more on this in another post), and it was just awesome to watch my new work people excel at what they do.

I went to the German Moodlemoot in Bamberg last month and had a great time (finally getting to try the Moodle Mojitos) and met some more Moodlers for the first time - most notably Heinz, who it was fantastic to meet, and David, who dutifully bought me beer for helping him with git, as well as seeing the familiar faces of Martin, Helen and Petr, which always makes me happy. For the first time I was at a conference and people who weren't me talking about Mahara (although of course I talked about Mahara as well), and that was a pretty big buzz in itself.

I love Swiss trains! Travelling back from Bamberg I changed about 4 times, and the change that made me switch onto a Swiss train made me warm and fuzzy. I also like Spätzle, Lindor chocolate, Cardinal (my local beer), Coop, Olivenbrot, the pizza place near work, Lily's, and Freitag (I am happily telling everyone I'm Swiss now that I have a Freitag wallet, which is complete nonsense). lost.ch makes me happy, having my name on my letterbox makes me happy (this does not happen in New Zealand). My boyfriend makes me happy. Inappropriately appending 'chen' or 'li' on the end of everything makes me gleeful and drives everyone else crazy. So far, I don't think much of Cablecom, Migros, apartment hunting, eating Pferd (pony!) or endless dairy products.

I have seen live: Gazpacho twice (although once was in München) and Explosions in the Sky. I hate smoking in bars, but I loved both bands.

As soon as I have conquered German and French (any day now: yeah right!) I will be completely happy.

by penny@she.geek.nz (Penny) at April 07, 2009 04:57 PM

March 10, 2009

François Marier

Handling security bugs in your Free Software project

If you are managing a Free Software project, you may eventually be confronted with a security vulnerability. Normally bugs in the Open Source world are discussed in a transparent way on public forums. In the case of security bugs however, there are benefits to temporarily withholding these details from the public.

Some people describe this approach as responsible disclosure. It boils down to this:

if the vulnerability is not publicly known, warn the vendors first and give them some time to fix it before making the details public.
If the vulnerability is already public knowledge, then you can focus on fixing it as soon as possible and maybe let reporters know about a better way to report their findings.

Sample Security Policy

Here's the procedure we now follow in the Mahara project for security bugs we found ourselves or which have been privately disclosed to us through security@mahara.org:
  1. Figure out the extent of the problem: which versions of Mahara are affected by this problem?
  2. Fix the problem on all supported branches of the project in a private source control repository.
  3. Share the vulnerability information with vendor-sec and request a CVE identifier.
  4. Prepare release tarballs and packages.
  5. Draft a security advisory for the Security forum.
  6. Wait until the embargo date to push the release out.
Emails to vendor-sec should include:
  • a description of the problem
  • any proof of concept code demonstrating the vulnerability
  • the list of affected versions
  • any patches you have prepared
  • a proposed embargo date

Benefits of properly handling security bugs

First and foremost, it protects end-users by giving them a chance to download and install fixed versions of your software before widespread exploitation of the security flaws.

Secondly, it may increase your project credibility. While you are admitting that your software has flaws, you are also demonstrating that your project is committed to dealing with the most serious ones promptly and in a responsible manner.

Finally, sharing security flaws and having your fixes reviewed by a select group of experts may reveal extra vulnerabilities you missed while preparing your patches. This gives you an opportunity to improve your fixes before they are released and to avoid having to issue yet another security advisory a few days later.

Big thanks to Steffen Joeris for his help in shaping the Mahara policy!

by François (fmarier@gmail.com) at March 10, 2009 10:47 PM

February 13, 2009

Penny Leach

farewell Catalyst

Today is the end of an era. After almost 5 years, this is my last day at Catalyst.

I started working at Catalyst in May 2004. I had about 2 years experience working in the Industry, but I'd never worked on Open Source before. I used Open Source happily, but had no idea that I could be a contributor. Suddenly, that was all to change.

My introduction into the Open Source world was through The NZ Open Source Virtual Learning Environment (NZ(OS)VLE) project, which was a NZ-Government funded project to deliver a Learning Management System that could be deployed through NZ at low cost, to encourage collaboration between all the institutions participating.

And man we did a lot of awesome work on that project. Suddenly there was Moodle being used all through NZ. The Open Polytech deployed it, and at the time it was the largest installation of Moodle in the world. I used to wake up every morning and fix performance issues, I did a huge amount of work for Postgres support, I was pretty much paid to work on Open Source full time.

Eventually NZVLE funding ran out, and we started working on commercial Moodle projects - still managing to keep a happy balance with Open Source. The elearning team, which I was the second member of by about 3 months, grew.

Mid 2006, I started working on another government funded project, to deliver a portfolio system. This was to be what NZVLE was to Learning Management Systems, to portfolios. I started off co-leading this project, that would grow into Mahara, and I fell in love with it.

Eventually, various circumstances dictated that I had to work on other projects for a while, and Mahara fell away from me into other capable hands.

Life continued. Other than the projects I worked on by day, I also collaborated with various people on other projects under the roof of Catalyst - SuperHappyDevHouse the NZ edition was born, Linuxchix stuff happened, Linux installfests, Planet Catalyst was started, at first subversively. I did mentoring, I travelled quite a lot to talk at conferences about the work we were doing, I wrote articles, I helped start the Eight....

Catalyst began to feel like a family to me - there were people there who were like cousins, some like brothers and sisters, again others favourite aunts and uncles. I don't think a single day went by where I didn't learn some new cool computer thing. Traditions were started and continued: beer o'clock, pizza thursday, hot teen vindaloos, special coffee, mjollniraptors, hot cold hot cold, g<tab>, foosball, "where's $person?" emails, so's your face, boozey lunch, thrash metal Friday, bubble wrap, red liquorice bribes, step mania, haiku battles...

But like most young New Zealanders, I wanted to travel. The mjollniraptor was to become the Inter Continental Ballistic Mjollniraptor. After a few months in London, it became clear that Catalyst UK didn't hold much appeal for me, and there were many reasons for me to go to Switzerland. Liip offered me a job and I happily accepted, and look forward to starting with them on Monday.

I'm not sure whether it's completely ridiculous or perfectly appropriate that my last day at Catalyst is Black Friday, but here ends my journey with you for now. Ka kite ano, so long & thanks for all the beer, g<tab> etc.



Edit: The most recent explodingdog is incredibly appropriate:

I am too cold to work

by penny@she.geek.nz (Penny) at February 13, 2009 07:40 AM

February 12, 2009

Nigel McNie

Mahara 1.1 - almost there!

There's three bugs and two feature requests left until 1.1 is ready to roll! Finally it looks like we've got on top of the long list of issue and will be able to get a release out later this month.

Here's a quite explanation of how things will progress over the next while:

Once the bugs and feature requests are fixed, we will release the first, and possibly only, release candidate of 1.1. This will be a release that we consider "good enough" for 1.1 - no major crashing bugs, upgrade from 1.0 works, new functionality works.

We'll let that circulate for a few days, to give people a change to have one last go at testing upgrades and looking for major bugs. If there's nothing reported that's too major, we'll do the 1.1 release, otherwise, there will be a second (and potentially third) release candidate.

The 1.1 release will no doubt need some tweaking and minor fixes, which I'm sure will be reported back to us quickly. So there will probably be a series of point releases after 1.1 to iron out the kinks. We'll fully support you if you upgraded from any version of 1.0 to 1.1 though, and hopefully the testing we've done means the upgrade path doesn't break, even if some minor things don't work in 1.1.

Meanwhile, we'll be working hard on the import/export functionality for 1.2. This is the only feature planned for 1.2, and we hope to have it released quite quickly, by mid this year even.

So, look out for Mahara 1.1 RC1 real soon now!

February 12, 2009 11:11 AM

December 22, 2008

Nigel McNie

Things You Don't Want to See in the Morning

maharademo=# select count(*) from activity_queue;
 count
-------
 10043
(1 row)

Aah crap.

So it turns out there's a bug in Mahara's activity queue processing which means that if you make a view, leave objectionable feedback on it then delete the view before the queue is processed, the queue will bomb out every time it gets to processing the feedback and not send any of the other messages. Doh!

I have written a patch for this that'll go to 1.0_STABLE and master. It will probably involve unconditionally deleting any notifications older than two weeks, as the last thing people want is old notifications.

This problem may have affected other Mahara installs out there, but it's not guaranteed to have. For example, it hasn't affected the MyPortfolio sites, nor mahara.org itself. You can find out by doing a SELECT COUNT(*) FROM activity_queue; on your database - large numbers are bad, 0 or small numbers are good.

There will probably be a 1.0.7 soon to fix this.

December 22, 2008 04:40 AM

November 27, 2008

Penny Leach

mahara.org relaunch

mahara.org has just been relaunched! Nigel and everyone else have done a great job with the project.

Mahara has come a long way since the original mahara.org was launched, and because of this, we're now actually using Mahara to power mahara.org, rather than Drupal, which is what we had been using for the last 2 or so years.

I'm looking forward to seeing some communities building up over there, everything from developers to translators and practitioners. Actually building the community on a Mahara installation will hopefully see some interesting usage patterns, and since they'll all be in a central place it'll be much easier to showcase best practice.

In related news, I was in Birmingham today for another LEAP day - the standard we're going to be adopting for Mahara import/export, as well as interoperability with other portfolio systems. It was really interesting to see what other portfolio vendors are doing. I have a weird mix of pride and disappointment that we're the only open source portfolio represented in the implementation partners.

by penny@she.geek.nz (Penny) at November 27, 2008 08:40 PM

Nigel McNie

mahara.org finally relaunched!

It's taken a while, but finally the new mahara.org has been launched. I think it's a great step up from the old site, and will hopefully serve as a useful place to showcase and document Mahara for a while yet.

DNS is still propogating so it might be a while until everyone can see the new site. In the mean time, we're putting up some content and will tweak the site over the next couple of days.

Our focus will now turn back to the Mahara 1.1 release. There's still some bugs to iron out, but hopefully we can get the release out soon...

November 27, 2008 03:41 AM

Powered by Planet!
Last updated: February 09, 2010 10:30 AM