Sometimes it's Hard to Tell, But What Is Clear is That Google Costs Small Biz
So your website's being dropped by Google. Perhaps you had thousands of
pages indexed, and you can see the number plummeting...or perhaps you're already
totally out of the index: search for site:yourdomain.com and Google responds
with "Your search - site:yourdomain.com - did not match any documents."
The first reaction for most site owners is to assume that they have been banned
by Google, that they've done something wrong, something "blackhat,"
that Google has discovered their nefarious deeds and punished them by dropping
them from the index.
But hold on a moment...that's not always true. It's not always clear whether
Google has "banned" a site, or whether something else is going on, and it's
close to impossible to get Google to let you know. What is clear, though, is
that it's costly for the business community, hurting thousands of innocent
business owners, to the tune of tens of millions of dollars each year.
The Nightmare Scenario - Your Site Drops
Let me give you one little example. A one-time consulting client called me in a
panic a few months ago. I had set them on a path, and things had been going
well; more and more pages were getting indexed by Google, and more and more
pages were turning up high in the search results...until one day, things started
to drop. Google appeared to be dropping the company's pages out of its index,
and the few pages that remained weren't doing well in the results.
"Don't worry," I told them, "Yhings will recover. This is all part of the
natural Google cycle. Things go up and things go down." But as is common in
these situations, the company was convinced it must have done something
"wrong." Some change to the site must have displeased the Google Gods, and now
it was being punished. "No," I said, "there's nothing you've shown me that
should get you in trouble with Google. I think it will come back. Give it a
couple of weeks."
This company spent literally thousands of dollars on the problem: executives
wasting valuable time in meetings, discussing what had to be done; Web designers
being paid by the hour to suggest to the client (incorrect) solutions and then
make (irrelevant) changes to the site; money paid to me to sift through dozens
of emails between the client and the Web designers, trying to figure out what
had gone wrong and what, if anything, should be done about it. And what
happened in the end? After running around in circles for a few weeks, the
company eventually saw the site suddenly pop back into the Google index, and the
traffic started flowing again.
Google's Response - Too Little, Too Late
Now, I understand Google can't guarantee anyone a particular position in the
search results. But when this sort of thing happens, it's very expensive -- and
disturbing -- for the businesses involved. We're talking tens of millions of
dollars of expenses incurred to deal with Google fluctuations like this. I also
understand that there are people within Google who know of this problem, and are
concerned by it, which is good. What isn't so good is that Google's response to
the problem has been inadequate and slow; too little, too late, one might say.
Let's look at a scenario. Imagine for a moment that Joe the Website Owner has
seen his site suddenly drop out of the Google index. He must have done something
bad, right? But no, not necessarily. As we've seen, sites drop out and come
back, with no significant changes, and Google itself has publicly stated that
this can happen now and then.
So, what to do? Well, perhaps someone tells Joe about the
Google Site Status
Tool which, according to Google, "lets you check the index status of your
site, and also tells you when your home page was last accessed by Google."
Perfect, Joe thinks, exactly what he needs! So he opens this tool, enters his
site's URL, clicks the Next button, and Google responds with this:
Googlebot has successfully accessed your home page.
Potential indexing problems:
We do not know about all the pages of your site. You can submit a Sitemap to
tell us more about your site.
Well, that's not so helpful; in fact, it's utterly worthless. Joe knows
Google has successfully accessed his home page: it used to be indexed, after
all. He's also well aware that there are indexing problems. (Not just potential
problems!) And Joe has already submitted a sitemap. So this tool is, in
effect, a complete waste of electrons.
So what next? Joe could try logging into his
Webmaster Tools account,
where he submitted his sitemap, and see if he can find any clues. What does he
find? Well, not much. He doesn't see any errors, and although Google sometimes
sends messages to site owners through the Message Center, warning them of
infractions, in Joe's case there's nothing. He does notice one thing, though:
Google is continuing to index the site and to download his site's sitemaps.
Joe's spirits are buoyed. Perhaps this is just a temporary glitch. Maybe the
site will be back soon! In fact, Google might continue downloading sitemaps and
crawling the site every day, even if the site remains out of the index for
Google's Webmaster Guidelines - Part of the Problem
So what is Joe supposed to do? Well, these days Google's preferred process is to
have you read their Webmaster Guidelines, find out what you are doing wrong,
correct the problem, then contact them using the Request Reinclusion form in
your Webmaster Tools account. But there are plenty of problems with this
First, it's not always easy for a non-expert, such as a small-business owner
like Joe--in this case a plumber--to figure out what the guidelines actually
mean. In fact, the guidelines are ambiguous enough for different people even
within the SEO business to hold different interpretations. For example, many
people believe that "cloaking" -- sending one page to Google's crawlers, and a
different page to browsers -- is an absolute no-no that will get your site banned.
This belief is based on Google Webmaster Guideline statements such as, "Don't
deceive your users or present different content to search engines than you
display to users, which is commonly referred to as 'cloaking.'" However, there
are often good reasons for cloaking, and upon a deeper read of Google's
documents you discover that Google doesn't object to cloaking in many
circumstances. They just object if the intention is to mislead them.
Or how about hidden links? Google doesn't like things that are designed purely
for the search engines and not users, and thus is suspicious of links that it
can find but that users cannot. But there are often good reasons to have
"hidden" links, such as in-navigation tools that only appear upon request -- a
pop-up sitemap helping users find their way through the site, for instance. Such
a device would definitely not be against the spirit of the Google
guidelines. But could Google find it and interpret it as a trick?
Thus the Google Webmaster Guideline's ambiguities raise two questions:
1. Has Joe done something in his optimization that is borderline
trickery and that Google objects to?
2. Has Joe done something totally innocent that Google has misconstrued as
Now, put yourself in the Joe's shoes, in the shoes of this non-expert
business owner. (Perhaps you're already in those shoes.) Either Joe has to
decipher the guidelines himself -- and quite frankly, this is not possible for the
average business owner -- or he has to hire someone to decipher them. But who can
Joe hire? The average Web designer or Web-design firm knows next-to-nothing
about SEO (even though they claim otherwise), and as for SEO firms
themselves. Well, too many SEO firms are either outright cons or perhaps
trying their best but only knowing enough about SEO to be dangerous. So, if our
non-expert business owner asks five "experts" to review the site and come up
with suggestions for what's wrong, he's likely to get five different answers,
most of which -- and perhaps all of which -- will be wrong. And then, to cap it all,
a friend tells Joe, "Yep, this happened to me too. My site dropped out of the
index for two months last year, and then it came back. Never did figure out why."
The Human Factor
So has Joe done something wrong? Is this all a Google Glitch? Or perhaps
something has been misinterpreted by a Google employee? Google likes to
have its algorithm do most of the work, automatically ranking sites in a purely
objective manner. (It's objective once the algorithm is released, of course. At some
point subjectivity is employed to write the code.) It also employs people to
check some sites manually, to subjectively rate them. Thus your site may have
been killed by an employee who didn't fairly apply the rules, or who is
inexperienced and doesn't fully understand what he's doing.
If you use Google Adwords, you may have seen this in action with another Google
review process -- the Adwords editorial review -- in which Google employees may too often
misinterpret the rules. I recall one case in which a PPC ad that mentioned the
town of Superior, Colo. was blocked by an editor because the word "superior"
was a superlative. And in another case the publisher of
charitable-donation software had an ad blocked because he wasn't displaying his
charitable tax-exempt status on his website, which of course he neither had
nor needed, because his company wasn't a charity.
Everyone who's done more than a few Adwords campaigns seems to have a "crazy
editor" story, so if Google's process for manually checking websites for spam
has the same level of accuracy as its process for checking ads for compliance
with editorial standards, then too many sites are being unfairly blocked.
Anyway, let's imagine Joe has picked a couple of things that his advisors think
might have caused problem. He's fixed them -- or paid someone to fix them more
likely. Then Joe used the Request Reinclusion form. Then what? Probably
nothing. Joe's chance of getting a response from Google is pretty low.
Sure, he got a boilerplate response:
We'll review the site. If we find that it's no longer in violation of
our Webmaster Guidelines, we'll reconsider our indexing of the site. Please
allow several weeks for the reconsideration request. We do review all
requests, but unfortunately we can't reply individually to each request.
But there's a good chance that's the last Joe will ever hear from Google on
the subject. Of course, Joe will probably try posting a message in the many SEO
forums, including Google's own Webmaster forums, something Google actually
advises. But so much of the advice is conjecture, conflicting, or just plain
confusing (and what Joe doesn't realize, often just plain wrong), that Joe
leaves in disgust.
Joe's Next Step
So now what is Joe going to do? Well, eventually he gives up. He can't find
anyone at Google to talk to, he's changed a few things on his site, and still
he's out of the index. He could start over with a brand new website, of course,
but in this case Joe does something a little different. Yes, he moves his
site, and in fact, he reverts to his original content -- under the theory that
changing his content hadn't fixed the problem anyway- - to a new domain name. But
he also does a 301 redirect from the old site to the new one.
"No!, Joe," I hear some of you shouting, "say it ain't so! That old domain name
has been banned; a 301 redirect will transfer the ban from the old site to the
new, and poor ole Joe's just going to kill the new site, I betcha!" I can also
hear some of you saying, "What does Joe, a plumber, know about 301
redirects?" Not much, I'll admit, but for the sake of illustration I'm assuming
one of the Web designers implemented the redirect for him.
But that's not what happens to Joe's site. In this case (and many similar cases,
in fact), Google begins crawling the new site and adds it to the index. That's
right, sometimes the "banned" domain doesn't transfer its curse to the new
So where does all this leave Joe? First, hundreds, if not thousands, of dollars
in the hole. He's paid Web designers to misinform him, and he's wasted time on this
whole process -- time he could have spent fixing people's bathrooms. He's paid a
designer to change a few things on his site, he's lost business for a couple of
months, and he's paid to move the site.
But in addition, it leaves him confused. Was his original site even "banned"?
Well, it wasn't in the index then, but it is now, and questions are still hanging.
- Why did Google continue crawling the "banned" site?
- Why did it continue downloading its sitemaps?
- Why didn't it send him a message in the Webmaster's console?
- Why didn't it respond to his Request for Reinclusion?
- Did Google ever look at the Request for Reinclusion?
- Why, if the site has been banned, is Google willing to accept the exact
same site when the old domain name is "301 redirected" to a new domain name?
Google, through a combination of great ideas, outstanding innovation, massive
infusions of money, and being at the right place at the right time, finds itself
in an enviable position; it's a virtual monopoly. If you're in business online,
Google has to be part of your marketing strategy. The great majority of Web
searchers query a database managed by Google, either on one of Google's various
national websites, or on one of Google's partner sites. Search at AOL or
Earthlink, for instance, and you're searching Google. Around 75% of all search
results come from this company.
Monopolies and near-monopolies have special responsibilities. All businesses
have responsibilities, of course, but when you're the only game in town, it's
even more important that you act responsibly, because people can't just go
elsewhere. Right now, Google is hurting businesses. Yes, yes, I understand that
Google has to protect the integrity of its index, and there are plenty of people
playing games to trick Google. But that's generally not who's getting hurt,
anyway. The true "black hatters" expect to have sites banned. It's just part of
the cost of doing business. It's often businesses who don't fully understand SEO that are getting hurt, or businesses tripping over Google's ambiguous guidelines,
or businesses using techniques that Google found acceptable yesterday, but not
What, then, could Google do to make life easier for these people? Well, the
first thing Google could do is create a genuine Site Status Tool, one that
provides useful information. A site owner should be able to use a tool, behind
the password-protected wall of the Webmaster Tools, in which he can ask Google
about his site's status. Better still, this information should be a primary
component of the Webmaster Tools, showing you site status when you log in.
The status should include information such as whether your site is in the index
or not. If so, how many pages are indexed. If not, it should give some indication of why, along
with detailed explanations and suggestions, such as the following:
- Googlebot is unable to find any pages on your site.
- Googlebot found pages, but they don't appear to have any indexable
- Googlebot has crawled x pages, but they have not yet been entered into
- Googlebot has found hidden text on some pages; these pages will not be
included in the index.
- Googlebot has crawled x pages, but due to normal fluctuations these
pages are not currently displayed.
- Your site has been automatically dropped from the index because the
Google algorithm found [reason goes here].
- Your site has been manually removed from the index for serious
infractions against the Webmaster guidelines.
Again, telling someone about a problem will rarely do much harm. For example,
telling someone that they shouldn't be using hidden text on a page hardly tips
off the person using black-hat techniques. (These guys are way beyond that!) And
reportedly Google does sometimes inform people, through the Webmaster Tools
Message Center of these types of problems. And Google already tells people
things that they shouldn't be doing in the guidelines, so what's wrong with
telling Joe which of the (sometimes ambiguous) guidelines is the problem?
For instance, Google says that cloaking pages is sometimes okay and sometimes
not. Perhaps I think my use is okay, but Google thinks it's crossed the line. Is
it so unreasonable, when the issue is a fuzzy rule, to expect some feedback? Or
perhaps Google has found some hidden links in my site, links that are clearly
not intended to mislead the search engine but that have been interpreted as
doing so. Is it so unreasonable to expect a tip off that this is the problem?
Innocent Before Proven Guilty
Imagine for a moment that the firm in the first example I gave in this article
had gone into their Webmaster Tools account and seen the message "Googlebot has
crawled x pages, but due to normal fluctuations these pages are not currently
displayed." That one little piece of information would have saved the firm thousands
Here's another thing Google could do: at least tell people when their Request
for Reinclusion has been seen, so you don't just leave them hanging. The current
process of expecting people to drop requests into a black hole reflects badly on
Google, leading people to believe that Google probably doesn't read them, that
Google doesn't care, or even that Google intentionally blocks some sites in
order to boost its AdWords business. (That maybe be a
conspiracy theory, one to which I do not subscribe, but it's a common
Furthermore, if a site has been blocked manually, there should be some kind of
review process, the SEO version of habeas corpus. It shouldn't be possible for a
low-level Google employee to manually "lock up" someone's site without some kind
of recourse, some kind of judicial review that will free the innocent.
Now, one could argue that all this doesn't really matter, that if you're not
playing tricks you won't get banned, that only the bad guys get hurt. But that's
demonstrably false. Clearly many innocent sites are dropped from the index,
sometimes temporarily, sometimes for extended periods. It's similar to the
argument made by Ed Meese, Ronald Reagan's attorney general: "If a person
is innocent of a crime, then he is not a suspect." In fact Google's own Quality
Rater Guidelines contains the phrase, "We prefer that a 'guilty' page remain
unlabeled [as spam] than an 'innocent' page be labeled." Well, come on, then,
Google, step up. You're hurting people, when you have it in your power to help
About the Author: Peter Kent is the author of Search Engine Optimization
for Dummies, The Complete Idiot's Guide to the Internet, and several dozen more
books about the Internet and online business. He consults with businesses large
and small in the areas of SEO and e-commerce. You can find more information at PeterKentConsulting.com.