If you have ever taken a look at one of your website's pages or its competition
within Google's cache, you may wonder why some pages are indexed more frequently
that others. Those sites which are crawled more frequently do better in the
search engines and often appear higher for relevant keywords. Here are just a
few guidelines to ensure that your site is being crawled as often as it should.
While most serious Web professionals already utilize these strategies, beginners
should take this guidance seriously to secure more crawl visits and better site
- Create a sitemap: Most CMS systems provide (or should provide) a
sitemap for your website automatically. There are actually many WordPress
plugins which even help you notify the search engines that your sitemap has been
- Focus on the Clear Structure: Search engines spiders aren't the
brightest tools in the technology toolbox. Oftentimes, Webmasters develop site
architecture which is confusing to spiders, sending them down paths which are
either useless entirely or open up the opportunity to spot duplicate content.
and Flash prevent sites from being fully indexed. If your site is comprised of
these elements and features little pure HTML content, consider starting fresh
with a new site that uses these interactive elements as the sizzle, rather than
- Produce Content Regularly: Content is king - always has been,
always will be. If you're not producing content reguarly, then search engines
make a virtual note to themselves to slow the crawl rate down. Publishing with
some regular frequency ensures that you are on good terms with the search
engines and they'll up the time they spend on your website.