A Side Note On WordPress, SEO, sitemap.xml and robots.txt

Last week, I learned that WordPress doesn’t ship with a default robots.txt.

  • this is the default file that search engine crawlers parse to see what resources and URL patterns that it allowed and not allowed to crawl; it’s step 1 in every search engine optimization (SEO) guide.

I guess I just stupidly assumed that it was included in WP. Anyways, I thought it to be fair to tell everyone that if you are using WordPress and you care how your site shows up in search results, you should generate a robots.txt and a sitemap.xml.

Robots.txt?

Know that it’s important for search engines. Read this:

** NOTE: Not all web crawlers are guaranteed to read example.com/robots.txt; it serves as a guideline.

I Feel Dumb…

I feel like an idiot, and I should. The other day I just happened to search for “engfers” on Google, and the result that came back was my site with an indented sub-result that was some error from a file in the WP-Super-Cache plugin. I thought to myself, why is the plugins/ directory being crawled?

Needless to say, I shortly thereafter found Google’s Webmaster Tools to help rectify my situation. It’s a pretty nice web-app that allows you to remove content from Google’s search (which I then used).

I also noticed that the webmaster tools had sections for analyzing your robots.txt and sitemap.xml. Well, I was surprised to find out that this site didn’t have a robots.txt.

Most of you are probably think that I’m an idiot because that’s SEO 101. Well yes, it is; however, I didn’t realize that WordPress doesn’t ship with a default robots.txt! Don’t ask me why I didn’t see that before because I don’t know. Nevertheless, I think WP should ship with a robots.txt that AT LEAST eliminates  plugins/ and wp-include/ from being crawled.

Our Shiny, New robots.txt

There seems to be a billion and one SEO blogs out there; however, I was looking for resources for a robots.txt optimized for WordPress.

I found a couple of articles and examples at askapache.com and an example from the WordPress.org Codex.

The final version of our robots.txt (http://www.engfers.com/robots.txt) was pulled from the WordPress Codex page.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

# Sitemap
Sitemap: http://www.engfers.com/sitemap.xml

**NOTE: This file must to be at the ROOT of your web server!

Final Note: sitemap.xml

The big-daddy search engines like Google, Yahoo, Microsoft, etc use your site’s sitemap.xml (example.com/sitemap.xml) to make it easier crawl your website. It’s also a very important point of SEO; just do a bit of searching on it.

The final line in our robots.txt points to the sitemap:

Sitemap: http://www.engfers.com/sitemap.xml

For WordPress, use a plugin like the Google Sitemap Generator, to have it automacially generate the sitemap for you.

+1 = Moreover, It will automatically regenerate the sitemap.xml when you publish or edit a new article or page. =)

30 thoughts on “A Side Note On WordPress, SEO, sitemap.xml and robots.txt

  1. Great article engfer! Most people and bloggers have never heard about robots.txt files, and that isn’t good for anyone.

    The newer WordPress versions show a default robots.txt “file” by using internal rewrites, which IMHO is not nearly as good as using an actual file. Keep it up..

  2. Finally someone who can write a good blog ! . This is the kind of information that is useful to those want to increase their SERP’s. I loved your post and will be telling others about it. Subscribing to your RSS feed now. Thanks

  3. i have been using WordPress for 2 years but i still dont know how to do SEO using WordPress, is there an SEO pluggin for WordPress?.

  4. Nice looking blog, might I ask you what template you are running and how much it costs? I’ve been using cheap ones but can’t find one that I actually like.

  5. I see a lot of interesting content on your blog.

    You have to spend a lot of time writing, i know how to save you a
    lot of work, there is a tool that creates unique, SEO friendly articles in couple of minutes, just type in google – laranita’s free content source

  6. Follow a healthy diet plan packed with vegetables
    and fruit. Whereas, heavier makeup can be absorbed
    into pores which can cause irritation, acne, and infection.
    And because certain vitamins and minerals react with others, and with medicines,
    it is important to discuss your choices with a doctor, nutritionist, or other healthcare professional to create a program
    tailored to your personal needs.

  7. Hi there! This is kind of off topic but I need some
    guidance from an established blog. Is it difficult to set up your own blog?

    I’m not very techincal but I can figure things out pretty
    fast. I’m thinking about creating my own but
    I’m not sure where to begin. Do you have any points or suggestions?

    Thanks

    My web site: luton emergency locksmith

  8. As you are shopping for your travel golf bag, you will
    need to compare the amenities that you need as opposed to what you want.

    You go on vacation to have fun, but having fun should not mean letting your guard down. TRAVEL for flight tickets, vacation packages, cruise, hotels or car rental.

  9. Also ask your doctor about any facial exercises that you can do and
    have him demonstrate these. If you miss a
    occurrence of your body you can use a cotton bud or alike to fill in the spaces.
    Be sure to spray in 6 inch circles and avoid having a large amount of overspray.

  10. Also ask your doctor about any facial exercises that you can do and have him demonstrate these.
    However, there are many ways of getting a sun tan, and the
    Sun Laboratories Self Tanner Buy enables you
    to explore one of them. The finest self-tanning product manufacturers have a range of tanning creams, lotions, and sprays that you could try to see
    which suit you best.

  11. On the other hand, if you’re going to be stuck in your room for hours
    on end, you might want something a little classier.
    So you have to decide which India Travel package suits
    you the best and where you want to travel for your next
    holiday. Imagine – Books makes it so easy for you
    to design your own personal photobook.

  12. The main aim of such an approach is to plan the poses to make sure that beautiful pictures
    of the couple are clicked while also making sure that the photographs look absolutely random and very natural.
    For example, try taking pictures from hip height or
    even while kneeling down. If the photographer you have selected
    does not require you to sign a contract, choose another one who will.

  13. As of right now, there’s isn’t a goal or story which can really ruin things.
    Minecraft is a 30 megabyte game downloadable in a one to two minutes
    and portable beyond belief. Having found the stone, your wood pickaxe will likely be
    utilized to gather 20 cobble blocks.

  14. Male enhancement pills are exactly what their name implies.
    But remember: when a male enhancement supplement seems too good to be true, it usually is.
    The device is small enough to be worn under loose clothing without
    other people noticing, yet powerful enough to enlarge the penis
    if properly employed.

  15. When selecting memorabilia or general purpose utility items as gifts, avoid items
    such as perfume and religious or superstitious items.
    Struggle no more, Canadian company Geekware have a wide
    range of irresistible gifts that geeks will give their PCs for, and what’s more, they’re all environmentally friendly.
    A customer over-heard the employees talking about the cuts, and got so offended that she decided to
    return everything she just purchased, and left the store.

  16. Sebagai orang majalaya yang sapopoe ngajedog di imah, ngajaitan popoean, dititah sama my mom meuli royco keur nyieun lodeh, dititah mangnyabutkeun cocolok istrikaan, dititah meresan sepre nu pabalatak, dititah mencetan bitisna nepika sare, dititah malikkeun endog na katel bisi tutung, dititah ngomean hpna meh jadi bahasa indonesia, dititah mareuman ledeng bisi mayar PDAM mahal teuing.

  17. Great article engfer! Most people and bloggers have never heard about robots.txt files, and that isn’t good for anyone.

    The newer WordPress versions show a default robots.txt “file” by using internal rewrites, which IMHO is not nearly as good as using an actual file. Keep it up..
    Obat Tradisional Kanker Rahim
    hopefully your site could still exist in providing the information, I am glad of visiting. and we extend our best wishes.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>