What are we cooking today?
How to block Search Engines from indexing site.
Recipe
How to block Search Engines from indexing site.
Recipe
- Add a robot.txt file with instructions to prevent all robots from indexing the page
User-agent: * Disallow: / - I added a tag in the links to prevent from indexing too.
Sample http://www.mylink.com rel="nofollow">My Link
- I added a metatag :
Other details
To exclude all robots from the entire server
User-agent: * Disallow: /To allow all robots complete access
User-agent: * Disallow:
Or create an empty “/robots.txt” file.To exclude all robots from part of the server
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/To exclude a single robot
User-agent: BadBot Disallow: /To allow a single robot
User-agent: WebCrawler Disallow: User-agent: * Disallow: /More on robots --> http://www.robotstxt.org/