UXDE dot Net Wordpress Themes

Posts Tagged: "robots.txt"

How to edit robots.txt in wordpress

in Wordpress / 9 Comments

Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory. One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents. I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS