Posts visible on WWW

theedge · 31 Jan 2013, 10:25 am

I just found that when I log out, the posts are still visible on the internet.

To check:

Go to uega.net
Go to the links: "View unanswered posts » View active topics"

In addition, google has crawled some of the posts.

Is it possible we can remove the link or lock down the visibility of the posts/link to crawlers? It's a bit of a worry seeing we do want to keep the stuff for members only.

Thanks

rhubarb_king · 31 Jan 2013, 3:10 pm

Yes there is a way - It's called "robots.txt". Shows bots what they are and are not allowed to crawl.

An example of a robots.txt file contents, that blocks bots from everything on a site
# go away
User-agent: *
Disallow: /

rhubarb_king · 31 Jan 2013, 3:12 pm

The robots.txt file should be put in root, so for uega would look like:
http://www.uega.net/robots.txt