// you’re reading...

Website Development

robots.txt

Written by ⋅ Post a comment

I recently created a robots.txt file for this site and the yet-to-be-launched Niche Site 1. Graywolf’s video is a nice primer on the problem of search engines crawling your blog, finding too much duplicate content, and lowering your PR. He’s a fan of WordPress and specifically made the video for it.

There are a few nice pages for the nitty gritty details. First is the official Web Robots page. Second is the sample robots.txt file on the SEO page at WordPress. Third is the robots.txt analysis tool under Google’s Webmaster Tools. This is probably the tip of the iceberg but I found these very useful for beginners like us.

47

If you enjoyed this article, make sure you subscribe to our RSS feed! You can also follow us on Twitter here.

Related articles:

  1. Checklist for New WordPress Sites
  2. WordPress Plugins
  3. Using Preview To Modify WordPress Themes Without Breaking Live Sites
  4. Matching Colors for Google AdSense

Print This Post

Discussion

No comments for “robots.txt”

Post a comment

Subscribe

Receive updates by RSS or email.

Mini Updates

No public Twitter messages.

Follow our adventures on Twitter