Today, Q-vantage is going to start new SEO topic and topic name is “Importance of Robots.txt”. On this blog we have published basic information of robots.txt file and why do we use robots.txt file for SEO?. Wihout wasting our precious time, lets start:
What is Robots.txt File?: When any search engine spider or crawler come to visit your website then it will find out one most important file on your website and that special file name is robots.txt. Your robots.txt file instructs to all search engine spiders and web robots, which webpages of website should be crawled or indexed and which web pages of website should be ignored or not indexed in search engine.
Robots.txt file is simple txt file with no html coding and it must be uploaded in your root directory.
Importance of Robots.txt file:
- If we want search engines to ignore all duplicate pages on our website
- If we don’t want search engines to index our internal search results pages
- If we don’t want search engines to index certain areas of our website or a whole website
- If we don’t want search engines to index certain files of our website (images, PDFs, etc.)
- If we want to tell search engines where our sitemap is placed.