Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory.
One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin
Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents.
I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.
Continue reading “How to edit robots.txt in wordpress”