What is the use of robots.txt file in a website?

The robots.txt file is a complete text file used for giving instructions to the search engine spiders as to on which files it has to crawl and which files to avoid.

Create a robots.txt file using any text editor and save with extension .txt. Specify the xml sitemap’s absolute path in live robots.txt file for identifying crawler or spider (boot) fast.

You can give instructions to the search console (Webmaster Tool) by submitting the robots.txt file.

The Structure of robots.txt file


Here user-agent: * indicates it is applicable to all search engine agents and disallows this instruction to block all files inside.

Use of robots.txt file in SEO

Robots.txt is a text file that is uploaded to a website’s root directory (inside public html folder). The robots.txt file contains information as to which page to index or not to index by spiders.

All search engines support these extensions.

To block the complete website from crawler, use the Disallow: option followed by a slash:

User-agent: *
Disallow: /

To allow everything:

User-agent: *
Allow: /

To block specific directories use:

User-agent: *
Disallow: /admin/

You can block individual search engine wise also

User-agent: Googlebot

If the domain is, then the robots.txt URL should be like:

Note: The filename is case sensitive (it should be robots.txt)

  How to use WhatsApp for Business

Benefits of robots.txt file:

  • You can block unrelated pages (Like Admin Access)
  • You can block individual search engine wise
  • simple text file (no html)

Search engine



Baidu General baiduspider
Baidu Images baiduspider-image
Baidu Mobile baiduspider-mobile
Baidu News baiduspider-news
Baidu Video baiduspider-video
Bing General bingbot
Bing General msnbot
Bing Images & Video msnbot-media
Bing Ads adidxbot
Google General Googlebot
Google Images Googlebot-Image
Google Mobile Googlebot-Mobile
Google News Googlebot-News
Google Video Googlebot-Video
Google AdSense Mediapartners-Google
Google AdWords AdsBot-Google
Yahoo! General slurp
Yandex General yandex

Check your robots.txt file at web master tool:

Structure of robots.txt

  Set up live chat on your website

About the author

Akash Tonasalli is a Digital Marketer, Web Developer, SEO Analyst, Consultant and Trainer from Bengalore, Karnataka.