[Tux3] Robots.txt

[Maciej Żenczykowski]

what are you trying to defend against?
is there anything you're not disallowing?
You do realize that robots.txt only applies if served from
http://hostname/robots.txt
(ie. not in subdirs).

On Mon, Sep 15, 2008 at 15:50, Daniel Phillips <phillips at phunq.net> wrote:
> Err, sorry:
>
> diff --git a/fs/robots.txt b/fs/robots.txt
> new file mode 100644
> index 0000000..c3a9a3b
> --- /dev/null
> +++ b/fs/robots.txt
> @@ -0,0 +1,138 @@
> +User-agent: *
> +Disallow: 9p
> +Disallow: adfs
> +Disallow: affs
> +Disallow: afs
> +Disallow: aio.c
> +Disallow: anon_inodes.c
> +Disallow: attr.c
> +Disallow: autofs
> +Disallow: autofs4
> +Disallow: bad_inode.c
> +Disallow: befs
> +Disallow: bfs
> +Disallow: binfmt_aout.c
> +Disallow: binfmt_elf.c
> +Disallow: binfmt_elf_fdpic.c
> +Disallow: binfmt_em86.c
> +Disallow: binfmt_flat.c
> +Disallow: binfmt_misc.c
> +Disallow: binfmt_script.c
> +Disallow: binfmt_som.c
> +Disallow: bio.c
> +Disallow: block_dev.c
> +Disallow: buffer.c
> +Disallow: char_dev.c
> +Disallow: cifs
> +Disallow: coda
> +Disallow: compat_binfmt_elf.c
> +Disallow: compat.c
> +Disallow: compat_ioctl.c
> +Disallow: configfs
> +Disallow: cramfs
> +Disallow: dcache.c
> +Disallow: dcookies.c
> +Disallow: debugfs
> +Disallow: devpts
> +Disallow: direct-io.c
> +Disallow: dlm
> +Disallow: dnotify.c
> +Disallow: dquot.c
> +Disallow: drop_caches.c
> +Disallow: ecryptfs
> +Disallow: efs
> +Disallow: eventfd.c
> +Disallow: eventpoll.c
> +Disallow: exec.c
> +Disallow: exportfs
> +Disallow: ext2
> +Disallow: ext3
> +Disallow: ext4
> +Disallow: fat
> +Disallow: fcntl.c
> +Disallow: fifo.c
> +Disallow: file.c
> +Disallow: filesystems.c
> +Disallow: file_table.c
> +Disallow: freevxfs
> +Disallow: fs-writeback.c
> +Disallow: fuse
> +Disallow: generic_acl.c
> +Disallow: gfs2
> +Disallow: hfs
> +Disallow: hfsplus
> +Disallow: hostfs
> +Disallow: hpfs
> +Disallow: hppfs
> +Disallow: hugetlbfs
> +Disallow: inode.c
> +Disallow: inotify.c
> +Disallow: inotify_user.c
> +Disallow: internal.h
> +Disallow: ioctl.c
> +Disallow: ioprio.c
> +Disallow: isofs
> +Disallow: jbd
> +Disallow: jbd2
> +Disallow: jffs2
> +Disallow: jfs
> +Disallow: Kconfig
> +Disallow: Kconfig.binfmt
> +Disallow: libfs.c
> +Disallow: lockd
> +Disallow: locks.c
> +Disallow: Makefile
> +Disallow: mbcache.c
> +Disallow: minix
> +Disallow: mpage.c
> +Disallow: msdos
> +Disallow: namei.c
> +Disallow: namespace.c
> +Disallow: ncpfs
> +Disallow: nfs
> +Disallow: nfs_common
> +Disallow: nfsctl.c
> +Disallow: nfsd
> +Disallow: nls
> +Disallow: no-block.c
> +Disallow: ntfs
> +Disallow: ocfs2
> +Disallow: open.c
> +Disallow: openpromfs
> +Disallow: partitions
> +Disallow: pipe.c
> +Disallow: pnode.c
> +Disallow: pnode.h
> +Disallow: posix_acl.c
> +Disallow: proc
> +Disallow: qnx4
> +Disallow: quota.c
> +Disallow: quota_v1.c
> +Disallow: quota_v2.c
> +Disallow: ramfs
> +Disallow: readdir.c
> +Disallow: read_write.c
> +Disallow: read_write.h
> +Disallow: reiserfs
> +#Disallow: robots.txt
> +Disallow: romfs
> +Disallow: select.c
> +Disallow: seq_file.c
> +Disallow: signalfd.c
> +Disallow: smbfs
> +Disallow: splice.c
> +Disallow: stack.c
> +Disallow: stat.c
> +Disallow: super.c
> +Disallow: sync.c
> +Disallow: sysfs
> +Disallow: sysv
> +Disallow: timerfd.c
> +#Disallow: tux3
> +Disallow: udf
> +Disallow: ufs
> +Disallow: utimes.c
> +Disallow: vfat
> +Disallow: xattr_acl.c
> +Disallow: xattr.c
> +Disallow: xfs
> diff --git a/robots.txt b/robots.txt
> new file mode 100644
> index 0000000..6af2be1
> --- /dev/null
> +++ b/robots.txt
> @@ -0,0 +1,29 @@
> +User-agent: *
> +Disallow: arch
> +Disallow: block
> +Disallow: COPYING
> +Disallow: CREDITS
> +Disallow: crypto
> +Disallow: Documentation
> +Disallow: drivers
> +#Disallow: fs
> +Disallow: include
> +Disallow: init
> +Disallow: ipc
> +Disallow: Kbuild
> +Disallow: kernel
> +Disallow: lib
> +Disallow: MAINTAINERS
> +Disallow: Makefile
> +Disallow: mm
> +Disallow: Module.symvers
> +Disallow: net
> +Disallow: README
> +Disallow: REPORTING-BUGS
> +#Disallow: robots.txt
> +Disallow: samples
> +Disallow: scripts
> +Disallow: security
> +Disallow: sound
> +Disallow: usr
> +Disallow: virt
>
> _______________________________________________
> Tux3 mailing list
> Tux3 at tux3.org
> http://tux3.org/cgi-bin/mailman/listinfo/tux3
>

_______________________________________________
Tux3 mailing list
Tux3 at tux3.org
http://tux3.org/cgi-bin/mailman/listinfo/tux3