Article written

  • on 26.10.2010
  • by Gary

Why Would Google De-Index my pages?

Jump To: Google Penalty | Filter | Duplicate Content | Hacked | Google Is Wrong!

You have a website so people can use it, for that to happen it needs to be found, to be found, you really have the best chance being in Google (other search engines do exist).

You spend months, years building up your website then one day you realise that the amount of pages Google has indexed for your site is going down. Your first reaction is to panic and shout at someone, then you might perform a google search about de indexing and why!

There are many reasons why your website may start to be de indexed, below are the most common reasons i have seen and how to fix them.

I need to say to this point that there are several levels of penalty that i refer to, I have been told that ‘google filters’ do not exist and they are simply penalties, however i disagree and below you will see my thinking as to why there is a difference.

Google Penalty

A google penalty is when google have withdrawn your website from its index. The main reasons for this is a direct breach of Google’s guidelines, serious things like, Cloaking, being part of a link farm or generally using any aggressive black hat techniques. If you have been penalised then the chances are its for something you have done and you would normally know what has happened.

How To Detect

Detecting a penalty is easy, simply do a site command and it will show you if you have received a penalty. if you wanted to check this website then simply past the code below into google.

site:seorocks.co.uk

You will see that there are lots of results, if your website is not indexed the you will see something like the following

Your search - site:www.YOURDOMAIN.co.uk -
did not match any documents.

If this is the case then the website has been removed. If this is the case then you need to correct any issues and submit a re inclusion request to google.

Filter

This is the part some SEO’s do not think exists, however in my experience it does and it shouldn’t be refered to as a penalty, although it commonly is.

A filter is when your website is indexed but the pages may not be returned as a result. You will generally notice this when you have a page that has been ranking well and all of a sudden no longer ranks.

The most common reasons for this type of problem would be keyword stuffing,  duplicate content or even over optimisation. I have seen hidden content cause a page to be filtered. Is google really going to ban your whole website because you have over optimised some of your pages? No it’s not. it will simply ‘filter’ the page so it doesn’t perform in the search results, your other pages may still do just fine.

The main reasons i have seen for what i call being filtered are generally naivety. Web masters that have attempted SEO themselves and gone a bit overboard with it, or copied their content from other websites.

How To Detect

Detecting a penalty can sometimes be a little harder, more so is detecting why you have a penalty, if you have done your own SEO and you think you have been filtered then it will be best to ask someone else to take a look, its hard to criticise your own work!

Firstly, do a site command on the page you think is filtered, if it has been penalised then there will be no results, if you have been filtered then the page will still be returned as a result.

If we use a websites homepage as the filtered page, then simply search for the url if you are not position 1 then there may be a problem. There are however a few exceptions to this rule, if you have the same domain as someone else then you may be number two e.g.  search for seorocks (this domain) you see i am number two behind seorocks.com. this is not a concern.

The other issue is if you have a keyword rich domain. If your domain was home-insurance.co.uk then you would need to search ‘home insurance’ then the chances are you wont rank number one.

If you do not rank for your own domain name then search for some content on the page. Again, you should always rank for your own content, if you don’t, again his could be a problem. If another website ranks for your content, then re write your own content, it’s a pain but its the fastest way to get back in the listings.

If you don’t rank for the content and it’s not duplicate then look at the web page, is the content you’re searching for hidden? is it in a scroller which has a small visible area, is the content stuffed with keywords?

With a filter, if you resolve any issues with the website, then the page will return to its original position, you don’t need to ask for a re inclusion request because your page hasn’t been excluded.

Duplicate Content

Duplicate content can be an issue and over and around last Christmas, duplicate content became a problem it shouldn’t have.

Google is good at knowing who has written content, mainly on who had it cached first but i wouldn’t rely solely on that. If someone copies your content, most of the time it wont effect you but if the other person then ranks for that content then google has failed (and remember google is not perfect) Just because you wrote that content yourself 5 years ago, doesn’t mean it’s not the problem now.

Internal duplicate content can be an issue with websites, Google will simply not rank multiple pages with the same content on it. A good example of this is websites that may have 1000′s of products but the products may be very similar

for example,

you sell widgets, your website stocks 100 red widgets and each widget for sale has its own page, good thinking. But! the only difference between the widgets is the thickness, they range from 1mm to 100mm. this means that you have 100 pages and the only difference is the thickness in the description.

On top of this, you may sell the same widget but in yellow, this is another 100 pages that are identical to the red widget pages, apart form the fact the colour is different.

Google will see this as 200 pages with the same content and it simply wont rank or even index most of them.

How To Connect

Simples, copy a paragraph of content from your website and paste it into google, your website should be number one.

If internal duplicate content is your issue, make everything unique, in some rare occasions you may have to look at the way your website works / displays products and change it.

If you have a situation such as the widget example, then instead of having a different widget for each size and colour then you have one widget page, an on that page you can select the colour and size.

Hacked

Your website could be hacked and this would generally cause a filter, the main reason pages get filtered after a hack is because of the hidden content the hack puts on the page, generally 100′s of hidden links that google sees and you don’t, the hack may even use cloaking techniques which would make it hard for you to detect.

How To Detect

View your source code, if there are lots of hidden links in the code then you may have been hacked, these are generally Viagra or porn related links.

If the hacker has been a little more clever, then they may be cloaking, in Firefox, use the user agent switcher plugin to view the page as googlebot, you may find your presented with a different looking website. Do a site command and look at the title and descriptions of your page, if they are wrong and promote a different website then this is a good signal. A good example of this can be seen on my blog post about the Krispy Kreme Hack

Google Is Incorrect With Its Data

As ive said previously, Google is not perfect, the information it gives is good, but not always correct. If you have a good SEO then they will be able to explain why google is wrong and give you evidence to back up what they are saying.

A recent example of this could be a blog, ive seen an example of when a blog has had 1000′s of posts, over a period of time, this count in google kept reducing from almost 2000 indexed to only a few hundred.

How To Detect

its hard to detect if google is displaying wrong data and unless you have data to back up what you think then you cant be sure the data from google is incorrect.

In my example, google displayed very few pages when doing a site command but when we looked into the analytics the traffic going to the blog continued to rise, if you have a blog and 60% of the pages got de indexed then the chances are the traffic to your blog would decrease as well. Again, you need to be sure of the information you are looking at and ensure you understand what google is reporting on.

subscribe to comments RSS

There are 3 comments for this post

  1. Bonafide Marketing says:

    Nice one. Something that has always concerned me is whether using SEO software like SEO Suite is a good thing and whether I would be de-indexed if I were to use software instead of doing everything manually.
    Maybe you could shed some light on this matter for me.

    Thanks
    Jason – Bonafide Marketing

  2. Gary says:

    Ive not come across this before, im a bit cautious of the 'Link Building and Analysis Module'.

    I always feel that tools should be there to aid you in doing your job, but they shouldn't be doing your job,

  3. cdiving says:

    Is there any legitimate reasons why some pages may be de-indexed? Yesterday I had over 700 pages indexed and today I noticed it had falled to 650.

Please, feel free to post your own comment

* these are required fields