[Tux3] Robots.txt

[Daniel Phillips]

On Tuesday 16 September 2008 14:15, Maciej Żenczykowski wrote:
> what are you trying to defend against?
> is there anything you're not disallowing?
> You do realize that robots.txt only applies if served from
> http://hostname/robots.txt
> (ie. not in subdirs).

Right.  I was a complete ignoramus re robots.txt this time yesterday,
and today, slightly less so.  Currently we exclude bots from the tux3fs
kernel tree entirely, but I actually want to let them in just to index
the auto-generated part of the file tree that has the tux3 files.  That
is fiddly, so I haven't done it yet.  Sometime.

It's not really very important, we don't need the repo indexed by bots,
as long as those search engines can at least find pointers to the repo,
which they already do very well.

Regards,

Daniel

_______________________________________________
Tux3 mailing list
Tux3 at tux3.org
http://tux3.org/cgi-bin/mailman/listinfo/tux3