Why Your Website Might Not Be Found

How Do Search Engines Work

Hi Folks

I see so many of the newer members ask: Why aren't the search engines finding their new sites because they have not been indexed, or they have been indexed but not ranked?

This blog is my reply to all members who are having the same problem.

A search engine works in the following order:

1. Crawling

2. Deep Crawling Depth-first search (DFS)

3. Fresh Crawling Breadth-first search (BFS)

4. Indexing

5. Searching

Search Engines work by storing information which they retrieve from the Web itself. These pages are retrieved by spiders. These are automated web browsers which follows every link they see (exclusions can be made for individual pages by using the Robots.txt).

The content of each page is then analyzed to determine how it should be indexed, and data about web pages is stored in an index database for use in later queries. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages, whereas some search engines store every word of every page it finds.

This cached pages always holds the actual search text since these are the ones that were actually indexed. This can be very useful when the content of a current page has been updated and the search terms are no longer in it. Increased search relevance makes these cached pages very useful, even beyond the fact that they might contain data that may no longer be available elsewhere.

When someone uses a search engine, typically by giving keywords, the engine looks up the index and provides a listing of the best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text (Meta Description). Most search engines also support the use of stop words such as And, Or, Not for example, to further specify the search query.

The usefulness of a search engine depends on the relevance of the results it gives back. While there may be millions of Web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others.

Most search engines employ methods to rank the results to provide the best results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve.

The majority of search engines are run by private companies using proprietary algorithms and closed databases. The most popular of these being Google, MSN Search, Ask and Yahoo Search.

There you go folks. Now you should have a pretty good idea how search engines work and you should be able to help them via your content and the Meta Tags you use if you want the search engines to find your website and its content.

If you're site has been indexed but not ranked then don't blame the spiders. Your content just wasn't good enough, so go over it again and see where you can make improvements.

Onwards and upwards, and have a great weekend.

Robert Allan