21 July 2009, 11:12

I've had a few people ask me about all the modifications I've been doing, so I'll give you a rundown of the changes I've made and things I have planned.  Unfortunately, I haven't really been keeping track of the exact code changes I've made and a lot of the changes are very specific to my situation.  However, if there is something listed that you are interested in, I can at least give you a pretty good idea how to implement it on your own site, provided that you know how to code PHP.

To begin with, let me just tell you that my site is social networking and not a site for a school or specific institution. That being said, here are some of the things I have modified or added:

• The index.php page for logged in users is now a view. Users can add blocktypes and move them around. I basically just copied a lot of the code that deals with the profile view and changed it to a "homepage" view. So far, the blocktypes available to the homepage view are: My Groups, My Friends, External Feed, External Video, and My Views. I'll be adding more blocktypes. This was pretty easy to accomplish my duplicating a lot of the code for the profile view (users can have only one homepage view, it can't be deleted, and it doesn't show up in the My Views list, and only the owner can view it).
   
• I'm currently working on a Recent Group Activity blocktype that includes recent forum posts, number of new members, etc.
   
• I added a new search plugin (which required some modifications to the core search code). Users can search by name, keywords (boolean), company (current and former), job title (current & former), school (attended or graduated, as well as start and end years), skills (boolean), location (either by country or by proximity in miles/km based on geocoding info), last visit (within the last day, week, month, or year), and the option to confine the search to current friends only. The name search also searches alternate spellings of the first name, nicknames and related names. This plugin is very specific to my site and all the modifications I've done, so unfortunately, I can't just give you the plugin because it won't work for you at all. But, I can probably help you develop your own. If you want to add postal code info including latitude and longitude data for geocoding, you can get postal code dumps for free at http://download.geonames.org/export/zip/.
   
• I added and changed a lot of artefact types for the profile and resume.
   
• Added different member levels for possible future implementation of premium memberships.
   
• Many of the fields on the profile are links that go to the search page to show users with the same entries (very similar to the way Facebook and MySpace display a lot of their profile fields).
   
• The resume forms include autocomplete fields for schools and companies. This was accomplished adding jQuery.
   
• The profile editing uses jQuery-based tabs instead of fieldsets (http://flowplayer.org/tools/tabs.html). I added a new Pieforms element & renderer for that.  If you want that, I should be able to give them to you the way they are.  I think they should work for you the way they are.
   
• Custom theme that includes jQuery-based dropdown menu (code from: http://www.sohtanaka.com/web-design/examples/drop-down-menu/).  Unlike the demo, mine does the dropdowns when rolled over. The number of unread messages appears next to a "Messages" menu option.
   
• jQuery-based rounded corner divs in some places.

I think that's about all I've modified and added up to this point.

Planned modifications and additions:

• More blocktypes for the homepage
   
• Group categories and subcategories
   
• Overhaul of the notifications/messages including the ability to view messages you've sent. Messages won't be deleted from the database until both the sender and recipent delete them. Very old messages will be automatically deleted unless specifically saved.
   
• A complete photo gallery
   
• Events calendar
   
• Classifieds
   
• Article submissions and ratings
   
• People you may know blocktype for homepage
   
• Logging of some user activities so friends can see when user's update their profiles, for example.
   
• Modal confirmation boxes when deleting or doing other important actions.
   
• The ability to upload a group logo. Group logos will appear in the group search results and in the My Groups blocktype (similar to LinkedIn.com).

I think that's about it.  If you have any suggestions or need any help doing any of these things yourself, let me know.  Again, I probably can't give you exact code for a lot of this, but I can at least tell you a lot of what I did and provide you with some code. 

22 July 2009, 23:46

These sound like amazing changes.  I'm more of a "hacker" than a programmer, but I'd love to know more details.  It sounds like you've implemented many of the changes that I JUST requested in the Feature Tracker.

 I'm especially interested in the homepage changes and the theme modifications.   Thanks for anything you can share!

 Chris : )

 BTW - It would be really nice to get some of these changes into the Mahara core! 

23 July 2009, 6:56

Hi Stephen,

Sounds like you've been pretty busy! Working on the next facebook there? Or something more mundane?

 There is some stuff there that might well be interesting to the Mahara core, if you were willing and able to release the appropriate code. I'll run through the stuff you've done and let you know our thoughts:

Homepage as a view: We have loose plans for doing exactly this - making the homepage into an actionable "dashboard" using a View. It's not just the homepage being a View, it's also a bunch of blocks that give users the ability to see at a glance what is happening and what needs their attention. Sounds like you're a long way down that path already! Would be interesting to see your code for this one

Recent Group Activity: We'd love to make a bunch more use of the inbuilt events system to record a whole lot more activity, so for example we can provide a block that works a bit like facebook's homepage (X became friends of Y etc.). This is a bit of work though. Such blocks would fill the void somewhat in the meantime though, we'd be interested in that

New search plugin: Sounds interesting, unfortunate you can't release it. I personally want to make much more use of postgres' tsearch abilities, and other people using mahara (I should start using 'Maharans'!) are keen on searching by such fields.

Different member levels: Care to elaborate? Is this for groups?

Profile fields as links: We'd be interested in that - but I guess it's dependent on your search plugin?

Autocomplete: Sounds useful and interesting. The addition of jquery is something we'd have to carefully consider first though, as we already bundle mochikit (for better or worse, we are using mochikit because we needed a JS library when we started on Mahara, and jquery didn't exist at that time).

Profile editing tabs: One of our primary concerns is that we don't want to have any functionality depend on javascript to work. I'm not sure how the jquery tabs work - would it work without JS? If so - yes we may well be keen! What's the main reason for you switching to those?

Theme/Dropdown menu: We are planning to switch to a dropdown based menu as part of our planned usability work actually - so would be interested in this. The menu groupings we're going for are most likely to be much different from yours though - we're heading towards a 'My Stuff', 'Other people' and 'Sharing' set of broad categories, under which all the menu items would fit.

Rounded corners: I hate with a vengance! But hey, most people seem to like 'em, in particular the usability people, who say they help simplify the look of the page by breaking up all those boxy lines. We had a theme previously that did rounded corners with gazillions of nested divs in the theme. Hmm, I guess I'm rambling a bit there Whether we wanted this or not would depend on whether we change the theme or not - we might do for the usability work.

 Whew!

A lot of the other stuff you have planned sounds interesting too, and is also in our minds. It sounds to me like you should be in #mahara-dev on freenode, dropping us a line about features you're planning on doing so we can suggest the best way to do them. I'm guessing that you don't want to diverge too far from the Mahara codebase? Otherwise you'll have a maintenance nightmare when we release 1.2/1.3 for example.

Related to that - if you can grok git, then using that would be a great way to track what changes you're making, so you can easily see what stuff is core and what changes you've made. Though I presume you're already using some kind of revision control already. I guess I have a bias towards git beacuse you can do neat things like storing the Mahara upstream code in one branch, doing all your work on another branch and painlessly upgrading the mahara branch then merging it into your branch to grab new stuff

Looking forward to your reply - keep up the hacking!

23 July 2009, 8:42

 Grande Nigel. This is called talking. Big congratulations to Stephen. In addition, look at my views on the position of Richard.

http://mahara.org/interaction/forum/topic.php?id=869#post4178

30 July 2009, 17:30

Haha. No, not the next Facebook. It's actually going to be business social networking.

Homepage as a view
I did this simply by pretty much duplicating the code that handles the profile view and changing "profile' to "homepage". Doing this also ensured that the "homepage" view couldn't be deleted and doesn't show up on the My Views page. In another thread I had mentioned that the blocktype_installed_viewtype table in the database wasn't actually being used to check if a block was available to specific view types.  So, once I modified the code to start checking that table, I was able to specify which blocktypes were available to the "homepage" (namely the External Feeds, My Friends, My Groups, etc.).  I've also started creating a few blocktypes that are available only to the "homepage" view.

Recent Group Activity
Yes, recording a lot more recent activity would be extremely useful.  I'll probably tackle this later on.

New search plugin
I'm using MySQL, so I didn't write anything for PG. It was pretty easy, though. Instead of just a query string being passed to it, it expects an array.  The part that searches for alternate spellings, nicknames, etc. for first names uses a database table that I'm not premitted to distribute.  As for the boolean searching, it's unfortunate that InnoDB tables in MySQL don't support fulltext, so I used a group of functions I got some someone online: http://www.maxlin.ca/boolean-query.zip that I modified a bit.  For the "last active" search, for example, I used the following code:

  // Last Active
  if (!empty($query_array['lastactive'])){
   $lastactive_sql = '';
   switch($query_array['lastactive']){
    case 'day':
     $timeframe = 86400;
     break;
     
    case 'week':
     $timeframe = 86400 * 7;
     break;
     
    case 'month':
     $timeframe = round(86400 * 30.416);
     break;
     
    case 'year':
     $timeframe = 86400 * 365;
     break;
     
    default:
     $timeframe = NULL;
   }
   
   if ($timeframe > 0){
    $lastactive_sql = ' AND UNIX_TIMESTAMP(u.lastaccess) >= (UNIX_TIMESTAMP() - ' . intval($timeframe) . ') ';
   }
  }

I don't remember if Mahara includes a lastaccess field in the usr table, but if not it's easy to add.

Different member levels
This is for the possible addition of premium memberships. With different member levels, I can specify which member levels have access to different features and settings.  For example, the lowest member level might have a file quota of only 15mb, but the highest level may have a quota of 50mb.

Profile fields as links
This is super easy.  You basically just convert the field you want to convert to links into an array with a comma being the delimiter before passing it to Smarty.

Autocomplete
This is extremely useful if you are trying to keep user inputted data as consistent as possible. It really helps with the entry of school and employer names.  For example, one person might enter "Highline Community College" and another might enter "Highline CC".  The autocomplete makes it much more likely that they'll enter the same thing, thus making a search for users from that school bring up both people. I created a database table prefilled with almost all the colleges, universities, and vocational schools in the U.S. (the data came from the U.S. Department of Education web site).

Profile editing tabs
Unfortunately, IE8 doesn't seem to render fieldset styling very well and since Microsoft if pushing people to switch to IE8 via their automatic updates... The tabs also are much more stylable than fieldsets anyway.  Another benefit is that if you click the back button, it goes back to the previous tab rather than the previous page. Do the tabs work at all without JS? Not sure. I'm not really concerned about users not having JS enabled.  The majority of the top 100 sites on the Internet won't function properly without JS enabled and really, only diehard users even know how to disable JS these days anyway. jQuery is an amazing library with many, many, many more plugins available.  I have seen some people try to port a few jQuery plugins to MochiKit, but I have yet to see many that actually work. There seems to be a great lack of development for MochiKit plugins.

Dropdown Menu
Yes, my menu grouping are completely specific to my site. The menu does have limited functionality without JS enabled, though (the main links work, but the dropdowns don't).

Rounded corners
Yeah, either you like them or you hate them.  I'm not going crazy with them or anything.

Git
No git here for me. Don't like it, don't really need it. I'm probably not going to upgrade to 1.2 when it comes out.  I use a program to compare files/directories and I'm fine with it.

Group Categories
I have successfully modified the groups to require the group creator to specify a category for the group. If there are a lot of groups created, it makes it easier to browse them.  For example, in my install I have Alumni Groups, Company Groups, Local Groups, and several others.

Group Logo
Right now I'm struggling with adding the ability to upload a group logo when creating the group. This file would not be included in the group files section.  I'm having a real difficult time doing this, so any tips would be helpful. The logo upload need to be included in the group creation form, as well as the group edit form.

Preferences
I merged all the preferences into one form, so there's not a separate notification preferences page. Eventually, I'll convert the form into tabs to separate sections (password & profile, privacy, and notifications).

Notification/Messages
Next on my list after I get the group logo upload working is to quickly redo the notifications (messages) section. I don't anticipate that taking too long.  The messages from other users will be the user's message inbox. System activity will be seperated.  Users will also be able to view messages they've sent to other users.  This is done by adding a "from" field to the notofication_internal_activity table.  Additionally, messages won't be deleted until both the recipient and sender delete it.  Although, if the message is very old and the recipient deleted it, but the sender hasn't, it will eventually be deleted via a cron job. Other modifications: Mark as unread, forward, a saved messages folder, the ability to send to multiple users, and the ability for those users to opt-out of a message thread (something Facebook doesn't do... very annoying).

So, I'm still hacking away at it.  Once I get a little closer to a public launch, I'll share the link so you guys can check it out.

30 July 2009, 18:38

All sounds very interesting - thanks for that! I daresay we'll implement some of the things you've been doing before too long.

Regarding the group logo: Mahara 1.2 would make this a million times easier, as there is a pieform element for dealing with file attachments there. But in lieu of that, most of the upload code deals with creating artefacts, so if you try and work against that system you're going to have to do more work than you'd otherwise need to.

If you really want them to not be artefacts, then I guess you could create a new directory in dataroot (check_dir_exists() will help there), and make your form submit functions put the images in there. Then you could look for the images there and display them if they're there. Unfortunately if they're not artefacts you won't be able to use artefact/file/download.php, but you could write your own groupimage.php file that looks for the image in the upload directory and serves it. 

30 July 2009, 19:41

I'm sure I'll go the route of making the group logo an artefact. Thanks for the info.

One other thing regarding the search... I really wanted to implement a Levenshtein Distance user-defined function in MySQL, but unfortunately it seems that you can't use UDF's in SQL statements with ADODB.... unless I'm doing something wrong.  I couldn't find any help online anywhere.

Using a Levenshtein Distance (Levenshtein ratio) function makes it easy to return results even when someone mispells a word. It's VERY useful.  I've used the one here on a few projects: http://codejanitor.com/wp/2007/02/10/levenshtein-distance-as-a-mysql-stored-function/

I'm sure there's a PG version out there, too.

30 July 2009, 20:58

I imagine that doing that would get very slow in the end - for my search needs I'd be tempted to farm the job off to a search service like solr. Mahara actually has a solr plugin, but I think it's currently busted with solr 1.3. Might not take too much effort to whip it back into shape though 

At least with solr, you'll get stopwords, stemming, facetting (if desired) and spell checking, and all of those at great speed.

I'm not aware of ADODB putting any limitations on what you can and can't call - so you should be able to call your functions without bother. What are you trying to do and what errors are you getting?

For postgres, I think we'd investigate utilising the tsearch abilities it's had in the core since 8.3. I was playing with them the other day, and it provides stopwords, stemming, boosting and highlighting which isn't a bad set of features for a database-driven search. No spell checking though, and you have to create extra columns on your tables to make it perform well, but that's what happens when you as a database to do a search index's job  

30 July 2009, 22:53

I've used the Levenshtein Ratio function on tables with several million rows and there's been very little difference in speed between using it and leaving it out, at least with MySQL.  It's worth a slight performance hit anyway if users are getting better search results.

I haven't tried using a UDF in a query in Mahara in a few weeks, but from what I remember, I think nothing happened. No results. No error message. Nothing.  I'll have to try it again and check the error log. 

Spell checking is great provided that all the words are in the dictionary.  One of the most common places people mispell words is with names, which typically aren't in any spell checker's dictionary.

Developing a truly great search is pretty difficult, though, no matter what you are trying to do.  Ideally, you'd want the following:

• The ability to recognize possible mispelled words and return results with words that are very close or at least return a "did you mean xxxx?" message.  For example, results with the word "theater" or "theatre" should come up in searches when you search for either of those words.
   
• The ability to return alternate spellings, nicknames, etc. for common first names (I got this one down, but obtaining the names database was a pain).  "Bob" should return "Bob" and "Robert".
   
• The ability to return common synonyms for keywords in search results. "Real Estate Agent" should return "Real Estate Agent", "Realtor", etc.  This I haven't been able to find a perfect solution for yet. Using a thesaurus application on the server is an option, but I think that might have its own set of problems.
   
• Boolean searching: AND, OR, +, -, and exact phrases. This, at least, was easy to accomplish.
   
• The ability to sort results by relevance.  Haven't figured out how to handle this yet, either.  With fulltext indexing in MySQL, you can do this.  But, again, it's not available with InnoDB.

The server I have my site on has only 3 other low-traffic web sites on it.  So, system resources aren't that big of problem for me.  If the site gets enough users and traffic to where the database is starting to strain, I'll just move the database to it's own more powerful server.  That'll also mean the site is doing really, really well. :)

02 August 2009, 0:30

Spell checking doesn't have to be done based on a dictionary. It can include finding similar words that appear more often in the search index - which takes into account names.

The features you're describing for good search come pretty standard with search services like solr - matching "theater" vs. "theatre" is just a case of stemming (both words stem to "theatr" so match the same queries). Alternate spellings can be done via a thesaurus, either manually or automatically generated. Boolean searching is easy, and both solr and postgres' tsearch will do relevance searching (though tsearch is far more primitive). Solr's can even do index vs. query time boosting, which is quite useful (e.g. boost this result at the time it's indexed so it's more important in any future query relative to other results, vs. boost results matching xxx for this query only).

I really wish we had the time to implement a full-blown solr search plugin in Mahara! It's not the advanced search features that would cause issues - more likely, implementing access control would be the hard bit!