Is Your WebSite Guilty By Association?

By Brad S. Konia

Originally Published: June, 2003

My company has been doing search engine optimization since 1999. Several times per week we get calls from people who are looking for help and don't know where else to turn. One of the most frequent complaints we hear is that one or more search engines are completely ignoring their site. The call usually goes something like this:

"Hi, my name is John Smith and I bought WebPosition Gold about three months ago. We've been using it to optimize our pages and submit them to the search engines. We've gotten ourselves listed in Alta Vista, AOL and a few other engines, but Google is completely ignoring us. We've tried everything and we don't know what to do at this point. How can we get listed in Google?"

First, let me make it clear that I'm just using Google as an example here. Sometimes people are able to get listed in Google, but they can't get their site into Alta Vista or Lycos. Why does this happen and what can you do about it?

WebPosition Gold Is Not The Problem

There are many, many reasons why search engines may choose to ignore your site. Contrary to some of the rumors that have floated around from time to time, the reasons have nothing to do with WebPosition Gold. I say this as a professional working in the search engine optimization field every day, not as a representative of FirstPlace Software.

WebPosition Gold is a tool and like any tool it can be misused. For example, if you used WebPosition to generate dozens of pages with duplicate or near-duplicate content, that could cause your site to be banned from search engines. Of course, you could use Microsoft FrontPage to do the same thing.

The search engines have no way of knowing whether you're using WebPosition Gold to create and optimize your pages, or just a common Web page editor. They also have no way of knowing whether you're submitting manually, or using WebPosition Gold's Submitter. WebPosition has many safeguards to help you avoid breaking the rules, but if you override these safeguards and break the rules anyway, your site can be removed or even permanently banned.

I'm not going to get into a detailed discussion of the rules that you must follow. If you use WebPosition's Page Critic and Submitter, it will advise you of the rules and will warn you if it detects that you're breaking one of them. You can also visit the search engine websites to read their official guidelines, though many people consider these "official" guidelines to be overzealous and don't always reflect reality.

Another great resource for learning about the "do's" and "don'ts" of search engine optimization is the Planet Ocean Unfair Advantage book and newsletter that I often read.

Banned "Tainted" IP Number?

What if you're sure that you've followed all the rules, you've established some external links to your site, and the search engines are still not listing you? In that case, it's possible that your Web site is using a "tainted" IP address.

What's an IP address and how could it become tainted? An IP address is like a phone number, but it identifies a Web site connected to the Internet. Every computer connected to the Internet has at least one numerical IP address. When someone types your text-based domain name into his or her web browser, that person's computer quickly looks up the corresponding IP address for the domain name and establishes a connection to that IP address.

This is kind of like looking up someone's name in a phone book to obtain his phone number. The phone book of the Internet is called DNS, which is short for "Domain Name Service." When you type a domain name into your web browser, your computer does a DNS lookup to obtain the IP address for that domain name.

IP addresses consist of four sets of numbers separated by periods. Each number can be between 0 and 255. For example: 192.168.0.1 would be a valid IP address. If you do the math, you'll discover that this yields about four billion possible combinations of addresses. This may sound like a lot, but believe it or not, the Internet is running out of the addresses. Consequently, the groups that regulate the distribution of IP addresses have made it somewhat difficult to obtain a unique IP address. So, instead of having a unique IP address for every website, many hosting companies will use a single IP address for hundreds or even thousands of domains. In fact, the majority of Web sites today utilize shared IPs.

This is possible due to a browser technology known as "host headers." Using host headers allows a web server to display the correct website based on the domain name that the user entered, not the IP address. So, the IP address is used to connect to the correct server and then the domain name is used to determine which website to view on that server.

If all this technical talk is making your head hurt, please bear with me just a bit longer. It's really important that you understand this if you want to prevent your website from becoming banned by the search engines. A recent Harvard Law School study found that 87% of all web sites use a shared IP address. The fact is that unless you've paid extra for a dedicated IP address, your website is probably using a shared IP address.

Now what happens if any ONE of the hundreds or even thousands of web sites that are sharing your IP address breaks the rules of one or more search engines? Yes, the search engines may remove or ban that website domain, but guess what...they may also remove every other website using the same IP address. Since it's easier for search engine spammers to get new domain names than it is to get a new IP, search engines often choose to ban IP addresses for abusive behavior.

This means that your site can become banned even if you follow all their rules to the letter. You're guilty by association! In fact, the April, 2002 issue of this newsletter carried an article about AllTheWeb.Com, an engine that claimed to have blacklisted over 30 million websites based on their IP addresses.

Obtaining A Dedicated IP Address

If you've tried everything and you can't get your site listed in the search engines, my advice would be to get a dedicated IP address. In fact, I would advise that you do this even if you currently have good listings. Otherwise you're just rolling the dice and hoping that no one else using your IP address breaks any rules in the future.

Given the current shortage of IP addresses, your hosting company is required to "justify" any dedicated IP addresses that they release to their customers. The criteria for justification are somewhat vague, but many hosting companies have taken the position that your site must have an SSL (secure commerce) certificate to qualify for a dedicated IP address. If you've placed an order on a Web site and noticed the URL having an https prefix, then you're most likely on an SSL server with a unique IP.

If you're serious about doing business online, then you should have an SSL certificate anyway. This certificate assures your customers that their credit card purchases and other transaction data will be securely transmitted to your company. Depending on where you buy your certificate, it can cost anywhere from $149 to $349 for the first year. In my opinion, many of the $149 certificates are just as good as the $349 certificate from Verisign, so don't pay double the price just to have the Verisign name on your website.

Once you have your SSL certificate, you'll probably have to pay your hosting company a few extra dollars per month for the dedicated IP address, or they may include it free. In any case, it shouldn't be a major price difference.

Another method of obtaining a unique IP is to invest in a dedicated server. Most hosting services charge significantly more for your own server, but if your company is of sufficient size, then this may be an option for you.

As I mentioned at the beginning of this article, there are many reasons why search engines might remove your website, but in my experience, having a tainted IP address is one of the most common culprits. So do yourself a favor and get your website its own IP address. It will be worth it in the long run.

Brad Konia is the CEO of Market My Site, a leading search engine optimization and online marketing firm. Market My Site offers a suite of full-service and self-service solutions designed to increase traffic to your website and convert that traffic into paying customers. Visit Market My Site on the web.

Editor's Note: For more on search engine indexing problems see our article's index at Search Engine Promotion Problems And Solutions and especially Search Engines Indexing Problems, 22 Reasons Why Your Page Did Not Get Indexed

This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition.
FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings.