Resources
Connections Vol. 2 Issue 6 June 2007
Back to Connections Newsletter Menu
Search Engine Spiders
How They Work and Why You Should Care
By: Blaine Hilton
1-877-525-2463
In today's world people don't talk about using the Internet, they talk about using a search engine such as Google, Yahoo or AOL. If you need information, looking for a product to buy, or just researching local service providers, chances are you use Google or another search engine.
Search engines all share a common trait. That is the spider that powers their collection of data. For a web site to come up in a search engine, they must first be known to the engine. The search engines use a process which is commonly referred to as "spidering". The World Wide Web is made up of billions of individual web pages that are connected together through hyperlinks. If you go to Site "A" and they say go to Site "B" for more information, then you have a link. Search engines will send a spider to go through this virtual "web."
Once someone has linked to your web site and it has been picked up by a search engine spider it will "crawl" your site. During this crawling process the search engine will look at all of the text based content on your site and will also look for links. If you have links to other web pages, then the spider will make a note of those links and will crawl through them as well after finishing on the first page. Search engines will only index as often as your web site is updated. If you update daily then your web site may be indexed daily. If you update yearly, you may only get indexed every six (6) months or so.
It is important to note that the spider is not the smartest being. One of the biggest problems they have is that they cannot read images. If a web site contains written material which is contained within a graphic, then that information will not be indexed. For an easy way to see what the search engine sees on your site you can open up your favorite web browser, go to your site, and go to View > Page Source. If you look at your page source and see a lot of jibber-jabbish that you don't understand, chances are the search engine won't be able to understand it either. There are some simple techniques that can be used to hide most of the extra code required to make a site function correctly, while still providing the spider with the information it needs to make you come up high on the search results.
My Web Site Is New. How Does the Search Engine Find Me for the First Time?
By: Blaine Hilton & Joseph Duron
Congratulations! You have just finished your web site and have launched it live on the Internet. Whenever people type in your domain name they are treated to a fabulous destination that will solve all of their ills. The only problem is not everyone on this side of the planet knows your name, or at least not yet. They need to find you through a search engine.
Perhaps you've read our other articles, did some of your own research, or had a firm optimize your web site for the search engines; and now you are just waiting for the flood gates of the Internet to bury you up to your eyeballs in cold, hard, green cash.
In the previous section, we discussed how search engines work. Many people, and indeed many unscrupulous companies, may try to sell you services to get your web site listed on search engines on the Internet. However, the common fact is Google currently has over 48% of the search market, and has been consistently gaining more and more searchers. If you include Yahoo, MSN Search, and AOL Search then you have pretty much the entire world of Internet users. Even if you are listed on a million search engines, unless one of those is Google or Yahoo it most likely will not yield the results you may have hoped for.
In order to have search engines find you, other web sites need to link to yours. The search engines also look at two things when it comes to these links: quality and quantity. The quality and quantity of these links are how the search engines position your search ranking. Having ten high quality links linking to you vs. having 100 that are low quality can give better results. You want to avoid anthing that may cause you to become blacklisted. There are obvious ways to know if your web site has high or low quality incoming links but there are also some non-obvious ways as well. If a web site that is listed in a specific search engine were to link to your web site; then the next time that search engine spider comes along it will follow that link, more than likely picking you up. New web sites can take anywhere from one to six months to get indexed by the search engines, depending on the quantity and quality of the incoming links that have been built up.
Many search engines have what is known as a sand box where new sites are placed for the first couple of months before you are added to the main search engine results.
Points to Ponder
- Have lots of content
- Break content into bite size bits
- Use headers to describe sections of text
- Summarize first, then deep dive into greater depth
- Write for your visitors first, then optimize for the spiders
ABSOLUTELY MUST DO THINGS:
- Have content.
- Have other web sites link to your site
Helpful Hints
The #1 way to be listed at all in the search engine is to have someone link to you.
Asking other web site owners to link to your site is a surefire way to help you in the search engines.
Better Search Engine Placement
Didja' Know?
The difference between "Web Page" and "Web Site". These terms are usually used interchangeably, but a web site is usually thought of as the overall site of an organization, and it is made up of multiple web pages. The web page is simply a single page (or screen) from a web site.
Glossary
Crawl:
The process of a search engine spider going from site to site through use of the hyperlinks between them.
Index:
The process of a search engine spider reading through the content of your site and adding it to their index of web sites.
Spider:
A software program that a search engine company uses to discover and then index web sites and their content.
Blacklisted:
An event that occurs when people try to outsmart the search engines by doing inappropriate things, striving to trick the engine into giving a web site a higher ranking. When blacklisted, your site will not appear in search engine listings at all.
Link:
Short for "hyperlink," it refers to an image or block of text that when clicked on takes you to a new web page. These links take you to web pages within the web site you are currently on (Internal Link) or onto another web site (External Link).