Last week, I learned that WordPress doesn’t ship with a default robots.txt.

• this is the default file that search engine crawlers parse to see what resources and URL patterns that it allowed and not allowed to crawl; it’s step 1 in every search engine optimization (SEO) guide.

I guess I just stupidly assumed that it was included in WP. Anyways, I thought it to be fair to tell everyone that if you are using WordPress and you care how your site shows up in search results, you should generate a robots.txt and a sitemap.xml.

Robots.txt?

Know that it’s important for search engines. Read this:

• http://www.robotstxt.org/robotstxt.html

** NOTE: Not all web crawlers are guaranteed to read example.com/robots.txt; it serves as a guideline.

I Feel Dumb…

I feel like an idiot, and I should. The other day I just happened to search for “engfers” on Google, and the result that came back was my site with an indented sub-result that was some error from a file in the WP-Super-Cache plugin. I thought to myself, why is the plugins/ directory being crawled?

Needless to say, I shortly thereafter found Google’s Webmaster Tools to help rectify my situation. It’s a pretty nice web-app that allows you to remove content from Google’s search (which I then used).

I also noticed that the webmaster tools had sections for analyzing your robots.txt and sitemap.xml. Well, I was surprised to find out that this site didn’t have a robots.txt.

Most of you are probably think that I’m an idiot because that’s SEO 101. Well yes, it is; however, I didn’t realize that WordPress doesn’t ship with a default robots.txt! Don’t ask me why I didn’t see that before because I don’t know. Nevertheless, I think WP should ship with a robots.txt that AT LEAST eliminates  plugins/ and wp-include/ from being crawled.

Our Shiny, New robots.txt

There seems to be a billion and one SEO blogs out there; however, I was looking for resources for a robots.txt optimized for WordPress.

I found a couple of articles and examples at askapache.com and an example from the WordPress.org Codex.

The final version of our robots.txt (http://www.engfers.com/robots.txt) was pulled from the WordPress Codex page.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

# Sitemap
Sitemap: http://www.engfers.com/sitemap.xml

**NOTE: This file must to be at the ROOT of your web server!

Final Note: sitemap.xml

The big-daddy search engines like Google, Yahoo, Microsoft, etc use your site’s sitemap.xml (example.com/sitemap.xml) to make it easier crawl your website. It’s also a very important point of SEO; just do a bit of searching on it.

The final line in our robots.txt points to the sitemap:

Sitemap: http://www.engfers.com/sitemap.xml

For WordPress, use a plugin like the Google Sitemap Generator, to have it automacially generate the sitemap for you.

+1 = Moreover, It will automatically regenerate the sitemap.xml when you publish or edit a new article or page. =)

TinyMCE Valid Elements

By default, WordPress’ WYSIWYG editor, TinyMCE, will strip out of your Article and Page HTML code any elements that are not defined as “valid elements”; this can be extremely annoying (especially if you want to include iframes).

This plugin will allow you to extend what TinyMCE defines as “valid elements”. By doing so, TinyMCE will no longer remove, delete, or strip-out the additional elements and attributes that you specify.

Check it out!

The proper way to correct any attributes that TinyMCE strips off of your WordPress posts/pages’ elements is to use the tiny_mce_before_init filter hook. Unfortunately, WordPress doesn’t document this very well, so I will.

UPDATE: 2008-11-06: I have made a WordPress plugin that does this for you!

» TinyMCE Valid Elements

The Wrong Way

I found many examples out there on the webs that say to edit wp-includes/js/tinymce/tiny_mce_config.php and concat to TinyMCE’s init() your extended_valid_elements. However, this is dumb and incorrect because the second another WordPress update comes out, it will overwrite that file and you will lose your changes.

The Right Way

The better thing to do is to change the extended_valid_elements property in the WordPress TinyMCE init array via the tiny_mce_before_init filter hook.

What WordPress doesn’t tell you is what the filter hook expects as parameters and return types, so listen up!

The tiny_mce_before_init filter hook expects:

• Parameters:
  • Associative array of existing TinyMCE init variables
• Returns:
  • The same associative array that you may or may not have modified.

/**
 * Add to extended_valid_elements for TinyMCE
 *
 * @param $init assoc. array of TinyMCE options
 * @return $init the changed assoc. array
 */
function my_change_mce_options( $init ) {
    // Command separated string of extended elements
    $ext = 'pre[id|name|class|style]';

    // Add to extended_valid_elements if it alreay exists
    if ( isset( $init['extended_valid_elements'] ) ) {
        $init['extended_valid_elements'] .= ',' . $ext;
    } else {
        $init['extended_valid_elements'] = $ext;
    }

    // Super important: return $init!
    return $init;
}

add_filter('tiny_mce_before_init', 'my_change_mce_options');

UPDATE: 2008-11-06: I have made a WordPress plugin that does this for you!

» TinyMCE Valid Elements