01 November 2010, 15:56

Hi Don,

Yes, they are just public pages with a hard-to-guess URL.  But simply emailing a secret url to someone would not get it indexed by google, unless google is doing something very evil like harvesting urls to index out of peoples' private gmail accounts.  If they were caught doing that, I think it'd be big news.  My understanding is that the search engines crawl the public web to find links in the first place, so it shouldn't be a problem unless someone you sent the link to has posted it publicly.

Also, Mahara doesn't link to a secret URL view using the secret URL itself.  Mahara *does* link to every private view using its view id, but generally this is when showing the view to its owner when they're logged in.

It would be interesting to see an example of an indexed secret url view.  My guess is that in these cases, either someone has posted the secret url on the web somewhere, or the view is being made available at a non-secret url as well.  But it's also possible there's a bug in Mahara that is displaying links to secret urls from public pages. If so, we'll fix it pronto!

01 November 2010, 16:09

Hi Richard:

I made one to test earlier today, and opened a Google Alert to it. I'll also try searching for it over the next few weeks.

If Google only opens pages which are linked elsewhere on the site (or on other sites), I guess maybe it shouldn't work. The only place the secret URL appears is the user's access page for that veiw, right? And Google shouldn't have access to that.

Anyway, thanks for listening, and I'll let you know if the experiment leads to anything significant.

Don

02 November 2010, 6:21

The issue we had was in the sharing of the secret URL.

The URL ended up being referenced from elsewhere, from other views, from forums, from email lists.

Mailing lists are a good example, as many people using a list via their mail client do not realise that they are also being dumped to the web.

There will sometimes be multiple recipients of a secret URL, who will not even realise it is supposed to be "secret".

I think we need to accept the fact that these URLs will "leak" into the public domain at some point, at which point, the best we can do is prevent them from indexing. This system is used by school kids, the idea that URL's will be kept safe by their owners is pretty ludicrous when you think about it.

Of course a view being posted into a forum compromises the "secrecy" of the URL, but that is a very different issue to being able to type "Joe Blogs from St Andrews School" into google and finding out where they live.

By adding the no index tags to non public pages you prevent Google and other conforming SE's from indexing the page, and the pages content even if a URL leaks, in which case Googling "Joe Blogs from St Andrews School" will not bring back any results, even though the URL may be compromised.

07 February 2011, 12:59

Hello Don,

What happened to your secret URL experiment? Was it picked up by a Google Alert or something else?

Cheers

Kristina

07 February 2011, 13:39

Hi Kristina:

Not so far, though there has been a bug with Secret URLs  that seems quite persistent - asks for login  50% of the time.

So I'm not sure if the experiment is conclusive...but it makes sense that a crawler would need a link from another public page.

07 February 2011, 18:18

Hello Don,

Oh yeah. The login screen may have prevented showing the content in an alert.

The bug with the login has been fixed: https://bugs.launchpad.net/mahara/+bug/661613

Cheers

Kristina

08 February 2011, 4:47

Hi Kristina:

But Richard Hand from TDM reports in that thread that the bug is still there. I know this because I reported it to him and asked him to verify it after a learner reported it to me.

08 February 2011, 20:48

Hello Don,

Mhh. I don't have a post-fix 1.3.3. It works for me on master.dev. Could you please try there as well?

Thanks

Kristina

09 February 2011, 5:25

Seems to work...it worked twice...maybe you can try this one (or others can) to see if it is consistent:

http://demo.mahara.org/view/view.php?t=h15wBM6tsDRPkyezbgfA

However, here's one back on my 1.3.3 server that doesn't...my crawler test page, actually

http://careerportfolio.mb.ca/view/view.php?t=nUKrCzt2sNkJBT5EbmRV

If indeed the demo server works and mine doesn't, it makes it a bit tougher to figure out...I will bring my lead developer back into this discussion.

Thanks!

09 February 2011, 12:43

Hello Don,

The demo secret URL worked. I got to it straight away from Firefox and Chrome without the login box. For those who want to try it out later, it might not work anymore as the demo site is reset at 7 a.m. UTC each day.

I could not test the link on careerportfolio because the connection timed out. I already had that problem last week when I wanted to check the homepage.

Cheers

Kristina