Last weekend we have published one blog on robots.txt and explain why robots.txt file is important for SEO. This week we are going to give deeper knowledge on robots.txt file and tell you how to create robots.txt file.
Creating a robots.txt file is very simple. You can create a robots.txt file in any text editor. It should be an ASCII-encoded text file, not an HTML file.
Allow Indexing Of Everything
Block All Web Crawlers From All Web Pages
Block A Specific Web Crawler From A Folder
Block A Specific Web Crawler From A Specific Web Page
- User-Agent: the robot the following rule applies to (e.g. “Googlebot,” etc.)
- Disallow: the pages you want to block the bots from accessing (as many disallow lines as needed)