UXDE dot Net Wordpress Themes

How to edit robots.txt in wordpress

in Wordpress / 9 Comments

robots.txt

Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory.

One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin

Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents.

I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.

There is no need to create robots.txt nowadays because while installing WordPress, It automatically creates a virtual robots file with default coding. (Example given below)

How should I verify If I have default/virtual robots.txt file ?

Just you need to verify is by typing this url in your browser:

http://your-website.com/robots.txt

User-agent:  *

Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/

Still there are many threads on WordPress support forums related to editing this virtual robots.txt, In my opinion editing this virtual robots file is difficult.

This is what I have faced with my blog (BlogDum); I tried every possible ways to solve but failed to edit this virtual file, Hence after doing a good research.

I found out two ways to edit robots.txt file one with plugin and other without plugin, both worked for me like a charm.

 Method 1 – Without plugin.

1. Create a text file
2. Add your desired code in this file or you can check mine robots.txt
3. Save file as “robots.txt” – Don’t misspell 
4. Upload this file in your root directory (Same place where you add Google webmasters verification code).
5. Now check www.yourdomainname.com/robots.txt
6. If you can see the file and codes then you have successfully uploaded it. :)

Note: Check if there is already a robots.txt file in your root folder, if yes edit and update it.

Method 2 – Using Plugin

Go to your Dashboard > Plugins > Add New

Type: WP Robots Txt

Or Use this link to download zip file directly: http://wordpress.org/extend/plugins/wp-robots-txt/

Install and activate it.

Go to Dashboard > Settings > Privacy

edit robots.txt wordpress

 

how to edit robots.txt in wordpress

Now you are ready to modify your virtual robots.txt with your desired syntax you wish to add on it.

Add your desired code in the box and update. Check your link (http://your-website.com/robots.txt), If it is updated or not.

You can also view my previous post on : Adding yoast breadcrumbs in every post

 Note: Check if plugin supports your wordpress version

Please share If you like this article :)

 

 

 

Share and Enjoy

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Shasha is an Internet Geek,Engineering Student and Research Analyst who loves wordpress,gadgets,seo etc.View Shasha on Twitter, Facebook, Google+, LinkedIn

  1. Sadek

    Thank you very very much for your help. I installed WP Robots Txt Plugin in my website and it rocks. Added custom instructions in Robots.txt file.

  2. Lifted Websites

    Both worked great and were quick and easy. I preferred option #1 and created a robots.txt file with notepad and uploaded via filezilla. Thanks for the tips.

  3. Sahil jain

    hello admin.
    i am new to wordpress.i have submitted my sitemap to google webmaster tools.But GWT is showing errors as follows:
    Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn’t crawl any pages listed in that file, we postponed our crawl. Your site’s overall robots.txt error rate is 100.0%.

    i am unable to understand all this.please help me.

  4. Praveen

    Many thanks for the article Shasha,
    Details are information which you have provided are really very helpful,..
    Thanks & Keep it Up!

  5. Madeline

    I’m having a hard time understanding what my Robots.txt should say. Currently, it says this:
    User-agent: *
    Disallow: /
    Sitemap: http://ballingonabudget.org/sitemap.xml.gz

    But when I google my website, I get a notice saying: A description for this result is not available because of this site’s robots.txt – learn more.

    I’d like there to be a description of my website when it is googled. What should by robots.txt say to have a description appear? I’m using a standard WordPress theme and have the SEO plugin.

    Any tips will help. Thank you,

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>


+ 2 = 10

Pinterest
Quantcast