Deeper Knowledge Of Robots.txt File

Last weekend we have published one blog on robots.txt and explain why robots.txt file is important for SEO. This week we are going to give deeper knowledge on robots.txt file and tell you how to create robots.txt file.

Creating a robots.txt file is very simple. You can create a robots.txt file in any text editor. It should be an ASCII-encoded text file, not an HTML file.

Allow Indexing Of Everything
User-agent: *
Disallow:
or

User-agent: *
Allow: /

Block All Web Crawlers From All Web Pages
User-agent: *
Disallow: /

Block A Specific Web Crawler From A Folder

User-agent: Googlebot
Disallow: /no-google/

Block A Specific Web Crawler From A Specific Web Page

User-agent: Googlebot
Disallow: /no-google/blocked-page.html

Sitemap Parameter

User-agent: *
Disallow:
Sitemap: http://www.example.com/none-standard-location/sitemap.xml
Robots.txt syntax

  • User-Agent: the robot the following rule applies to (e.g. “Googlebot,” etc.)
  • Disallow: the pages you want to block the bots from accessing (as many disallow lines as needed)
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s