About The Author

I am a search marketing geek. Work as APM for one of the leading companies in UK. Interested in socializing and helping others.

Get The Latest News

Sign up to receive latest news

October 13, 2007


A robots.txt is a simple text file placed on the root server of the website. It consists of the information for the various search engine spiders not to crawl or index certain section(s) of the website. It can be used to,

  • Prevent - TOTAL Indexing
  • Or, Certain Areas of Website
  • Or, Certain Search Engines, providing Individual Indexing Instructions
Creating - Robots
Any text editor can be used for the same.


User-Agent : [Spider Name or Bot Name]
Disallow : [Directory or Filename]