Voir le texte source

Sometimes, we possibly may want search-engines to not index certain areas of the site, as well as prohibit other SE from the site completely. This is the place where a simple, little 2 line text file called robots.txt is available in. Once we have a web site up and running, we need to be sure that all visiting search engines can access all the pages we want them to look at. Sometimes, we may want search engines never to list certain elements of the site, as well as exclude other SE from the site completely. To explore additional info, you can have a peep at:  [http://pop-u-lar.com/activity/p/218485/ arc welding aluminum] . That is where a simple, small 2 line text file called robots.txt comes in. Robots.txt lives in your internet sites key directory (on LINUX systems that is your /public_html/ directory), and looks something like the following: User-agent: * Disallow: The initial line controls the bot that'll be visiting your site, the second line controls if they are allowed in, or which elements of the site they're perhaps not allowed to visit Then easy repeat the above mentioned lines, If you like to deal with multiple spiders. So an example: User-agent: googlebot Disallow: User-agent: askjeeves Disallow: / This can enable Goggle (user-agent name GoogleBot) to visit every listing and page, while at the same time banning Ask Jeeves from the website entirely. If you would like to permit every robot to index every page of your site, its still very advisable to put a robots.txt file on your own site to locate a fairly updated list of robot individual names this visit. It will end your error logs filling with records from search engines wanting to access your robots.txt file that doesnt exist. For more information on robots.txt see, the entire set of resources about robots.txt at.