Webmaster Papers




Google
 
Web webhostingpapers.com




/pagead2.googlesyndication.com/pagead/show_ads.js">

Search Engine Robots - How They Work, What They Do (Part II)


If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content. How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page. Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies. The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

*SpiderSpotting - Search Engine Watch http://searchenginewatch.com/webmasters/spiders.html

*Robotstxt.org List of robots and protocols for setting up a robots.txt file. http://www.robotstxt.org/

*Spider-Food Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing. http://spider-food.net/

*Spiderhunter.com Articles and resources about tracking Search Engine spiders. http://www.spiderhunter.com/

*Sim Spider Search Engine Robot Simulator Search Engine World has a spider that simulates what the Search Engine robots read from your website. http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing, a Search Engine Optimization company serving small businesses. She has specialized in Search Engine Promotion since 1998, including three years as the Search Engine Specialist for O'Reilly Media, Inc., a technical book publishing company.

Copyright � 2002-2005 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.

Permission to reprint this article is granted if the article is reproduced in its entirety, without editing, including the bio information. Please include a hyperlink to http://www.searchinnovation.com when using this article in newsletters or online.

RELATED ARTICLES


Optimize Your Site Pt2
6: The hidden benefits to having the right links. Every site online has an index or default file as its home page file. But outside of that rule you can name the other pages whatever you like. So why not cram the names with keywords and/or keyphrases relevant to that page and your site? We've just scored again ;-)
Link Popularity
In today scenario when we talk about Search Engine Optimization, we also talk about one of the most important aspect of SEO, which is Link Building. But there are different types, aspects and limitations of Link Building, which would be discussed now under
Importance of Keywords in Links to Your Website
Search Engine Optimization (SEO) is a very complex process. It is a long-term process that will usually never produce results that you can see in days or weeks. By now you probably heard about the importance of getting other websites to link to yours so that you can get a higher ranking in search engines. You listed your websites in some directories and also received a few other links to your website.
Search Engine Spiders Lost Without Guidance - Post This Sign!
The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them "Bad Bots" or trespassers. We block them by IP exclusion which is another story entirely.
When Google PR Tool Bar is Down!
Where is the need to worry? Who needs a PR Tool Bar Anyway!
What You Did Wrong With Your Domain Names!
Trying to improve search engine rankings is just like a rubics cube. A puzzle that can keep you occupied for hours. How many times have you heard "They keep changing the rules?," and the frustrating part is...they don't even tell you what the rules are in the first place!
Attack Of The Killer Google Zombies!
Don't share this with people of a nervous disposition because I really can't restrain myself any longer.
Google Search Algorithm Patent Application Creates Spring Buzz!
Google applied for a patent on their ranking algorithm as of 15 months ago on December 31, 2003 and that application was posted on March 31st at the US Patent Office. It got the discussion forums buzzing this weekend. Even though I had substantial work to do and was behind on a project, I couldn't resist the temptation to read the very long 14,000 word, 45 page application and see what it could mean to the volatile world of search.
An Ethical Alternative to Doorway Pages
Definition: A doorway page is content created specifically for the purpose of garnering high placements in the search engines.
Link Reputation - How to Improve Search Engine Rankings
One of the most overlooked strategies to improve search engine rankings is building your link reputation. This link popularity strategy is just as important or equal to creating keyword rich content pages.
7 Search Engine Optimization Strategies
Search engine optimization refers to the technique of making your web pages search engine friendly so that search engines are more easy to understand and analyze your website. Consequently, your site has a better chance to gain high search engine ranking.
How To Get Listed in Yahoo!
Getting listed in Yahoo! (The Very Top Search Engine/Index) is a lot like getting a date with the most beautiful girl in high school - often all you have to do is ask PROPERLY. I can make that statement because I have NEVER submitted a site to Yahoo! that wasn't listed. I've read a lot about how difficult it is getting listed, and how frustrating it is - so maybe I am just very lucky. Then again, if you follow the same steps that have ALWAYS worked for me, there is no reason to believe that they won't work for you too. So I wanted to share with you in four short steps how to get listed.
No Cost Search Engine Marketing
As a matter of fact, I recommend NOT wasting money on pay inclusion in most cases because it doesn't offer enough of an advantage (and many times the fees give you absolutely no advantage - the only exception are the few sites that guarantee placement within a specific timeline). Focus your online marketing and gain positive and targeted traffic without paying out for "expedited listings" or "submission software."
Adding Content To Your Site Every Day
If you have been a webmaster for more than a week, you've surely heard the saying "Content Is King". And it really is true! This article will explain why the regular addition of fresh content to your website is so crucial and explain several ways to do it easily and inexpensively (even free).
SEO Quickstart - 3 Things You can Do to Improve Search Engine Rankings Right Now
Increasing your search engine rankings should certainly be one of your main goals if you are looking to increase your targeted traffic. The reason for this is if you improve your search engine rankings, you'll increase the amount of visitors your website receives, and when you increase traffic to your site you are going to increase customers and sales. All of this potential has webmasters spending a lot of time and money getting their pages in the top 10 results on large, popular search engines like Google. There are only 10 spots on the first page of a search and thousands upon thousands of web pages are competing for these top spots. You have to put together an SEO marketing plan and include some of the proven ways to increase your search engine rankings today. Read the following suggestions, implement them in your marketing plan, and watch your traffic increase!
Top Search Engine Ranks, Part 2- Mastering the Secret- Explained
In the first part of this series on ranking at the top of the search engines, we discussed diversifying your Internet marketing efforts. We introduced several methods including RSS feeds, Link Popularity, Article Marketing, Blogs, and physically altering your pages to make them more target-able for select keywords. All of these share the key of great content in order to unlock success.
SEO Blues
SEO, not again!, you may groan. The webmaster world is inundated by articles and "how to's" with regard to SEO (Search Engine Optimization). If you are a rookie webmaster, chances are, like me, you may have embarked on a merry-go-round on the SEO circuit, depending on which article and opinion you were first exposed to.
Owning Your Category Online - Is Our Website Broken?
"Owning your business category online is more than just a few keywords... �Is your website broken?"
Meta Tags Help Optimize Your Web Site Rankings
Are you working on getting your web site up onto the internet?
How To Get Number 1 Spot In Google, Every Time, Guaranteed
Woaah! Wait a minute. Before I exclusively reveal these insider secrets can I ask you question? Well, actually, I want to ask you a series of questions.