Do you ever tried to customize the robots.txt file in blogger under the search preference tab, If then this topic would help you to apply the strings in robots.txt.
In general words robots.txt file in blog are use to crawl the site by search engine robots, so this includes the secret of making your blog popular with the help of SE bots.
Lets see some examples to understand the concept
Allow All bots to Crawl :
User-agent : *
Disallow :
the above two lines make the bots to completely access your website site. Here "User-agent :" refers to the search engines and next " * " represents all the bots & robots, so in the first line we have included all the robots. and in the second line "Disallow :" make a comment of allowing access or denying access.
Here we haven't included any symbol or text and the field left empty next to "Disallow:" by the default it allows access to the robots to crawl the page
Blocking the Bots to Crawl :
User-agent : *
Disallow : /
You can note that only a slash [/] matters in that. first we are included all bots as a user-agent and in second step by including the slash we are not granting access to bots to craw our page
Allowing particular Robot to crawl :
User-agent : Bing
Disallow :
User-agent : *
Disallow : /
This line only allows the "Bing" search engine to access your site and deny access to all other robots.
So here we comes to know that , how to grant access to all bots or deny all bots and also how to grant access to some particular bots. So make your own decision to make your site popular with your favorite search engine :)
Allowing Google to Access :
If your are working with google adsense then , this code will help your PR .
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Keep tuning for more info..
Like Us on Facebook for more updates
useful post dude
ReplyDelete