The robots.txt file is a complete text file used for giving instructions to the search engine spiders as to on which files it has to crawl and which files to avoid.
Create a robots.txt file using any text editor and save with extension .txt. Specify the xml sitemap’s absolute path in live robots.txt file for identifying crawler or spider (boot) fast.
You can give instructions to the search console (Webmaster Tool) by submitting the robots.txt file.
The Structure of robots.txt file
User-agent:*
Allow:
Here user-agent: * indicates it is applicable to all search engine agents and disallows this instruction to block all files inside.
Use of robots.txt file in SEO
Robots.txt is a text file that is uploaded to a website’s root directory (inside public html folder). The robots.txt file contains information as to which page to index or not to index by spiders.
All search engines support these extensions.
To block the complete website from crawler, use the Disallow: option followed by a slash:
User-agent: *
Disallow: /
To allow everything:
User-agent: *
Allow: /
To block specific directories use:
User-agent: *
Disallow: /admin/
You can block individual search engine wise also
User-agent: Googlebot
Disallow:
If the domain is digitalakash.in, then the robots.txt URL should be like:
Note: The filename is case sensitive (it should be robots.txt)
https://www.digitalakash.in/robots.txt
Benefits of robots.txt file:
- You can block unrelated pages (Like Admin Access)
- You can block individual search engine wise
- simple text file (no html)
Search engine |
Field |
User-agent |
| Baidu | General | baiduspider |
| Baidu | Images | baiduspider-image |
| Baidu | Mobile | baiduspider-mobile |
| Baidu | News | baiduspider-news |
| Baidu | Video | baiduspider-video |
| Bing | General | bingbot |
| Bing | General | msnbot |
| Bing | Images & Video | msnbot-media |
| Bing | Ads | adidxbot |
| General | Googlebot | |
| Images | Googlebot-Image | |
| Mobile | Googlebot-Mobile | |
| News | Googlebot-News | |
| Video | Googlebot-Video | |
| AdSense | Mediapartners-Google | |
| AdWords | AdsBot-Google | |
| Yahoo! | General | slurp |
| Yandex | General | yandex |
Check your robots.txt file at web master tool:
