Viewstep.com

Robots Generator

Robots.txt generator is a free web tool to create server side robots.txt files for search engines and spider bots for your site.

 

Default -  All Robots are:
Crawl-Delay:
Sitemap: (leave blank for none)
     
Specific Search Robots: Google   googlebot
  MSN Search   msnbot
  Yahoo   yahoo-slurp
  Ask/Teoma   teoma
  Cuil   twiceler
  GigaBlast   gigabot
  Scrub The Web   scrubby
  DMOZ Checker   robozilla
  Nutch   nutch
  Alexa/Wayback   ia_archiver
  Baidu   baiduspider
  Naver   naverbot, yeti
   
Specific Special Bots: Google Image   googlebot-image
  Google Mobile   googlebot-mobile
  Yahoo MM   yahoo-mmcrawler
  MSN PicSearch   psbot
  SingingFish   asterias
  Yahoo Blogs   yahoo-blogs/v3.9
   
Restricted Directories: The path is relative to root and must contain a trailing "/"
 
 
 
 
 
 

 

Now, copy and paste this text into a blank text file called "robots.txt" (don't forget the "s" on the end of "robots") and put it in your root directory. Like all other files on your server, make sure its permissions are set so that visitors (such as search engines) can read


Share and Enjoy


Sponsors