SEARCH ENGINE FACTS

What is an Internet Search Engine?

A search engine consists of three main parts, software that roams the web retrieving and indexing pages, software that stores the index into a searchable database, and software that allows the user to search the database. These three components are different for every search engine. While you only have to learn to use the search software, it is important to evaluate the other two components to determine which search engine best meets your needs.

How Does the Search Engine Work?

Imagine spending a day at your computer, just randomly linking to Internet addresses, reviewing those sites for relevant terms, and putting the information gathered into a database. How many sites do you think you could index in one day? Large search engines use automated software to index between three and ten million sites per day. No two search engines index the exact same web pages, nor do they index the same data from each page. According to an article by Steve Lawrence and C. Lee Giles of the NEC Research Center, there are over 320 million web pages that could be indexed, but no single search engine indexes more than one third of those sites. The authors state that the "... current state of search engines can be compared to a phone book which is updated irregularly, and has most of the pages ripped out." (Science: Searching the World Wide Web, 280, p. 98, April 3, 1998, http://www.neci.nj.nec.com/homepages/lawrence/websize.html) For more information on how search engines work, go to http://www.searchenginewatch.com.

Types of Search Engines

There are three basic types of Internet Search Engines, Web Catalogs, Web Indexes, and Metasearches. The first is a site that has cataloged websites into categories. These sites do not claim to search everything on the web, but they use human beings to review and sort websites, and your search results are broken down by category. The second is a site that has indexed as many websites as it can. Your search sorts through millions of indexed pages for the terms you are looking for, and your results can be sorted by relevancy - the number of times your terms appear in each page. Metasearch pages have sprung up which allow you to enter a search in one location, then uses several major search engines to run your query. Some metasearch engines combine the results into one screen and some open a different screen for each search engine they use. Metasearching can be too broad, with complicated results. In addition, you will find search engines on the web which search only a specific topic or region, but are based on either cataloging the web, indexing the web or searching across search engines.

What is the 'Best' Search Engine?

AltaVista is hands down the largest search engine on the web, indexing over 140 million pages. According to a recent study by the NEC Research Center, AltaVista returns more valid hits than any other search engine. Northern Light was rated second best by this study. (Science: Searching the World Wide Web, 280, p. 98, April 3, 1998, http://www.neci.nj.nec.com/homepages/lawrence/websize.html) PC Magazine recently tested 14 general search sites and recommends Yahoo or Google for broad searching and Northern Light or HotBot for specific requests. (Find it on the Web, PC Magazine, December 1, 1998, http://www.zdnet.com/pcmag/features/websearch98/)

Which Search Engine Should You Use?

The search engine that you choose will depend largely on what type of question you are trying to answer. If you need to locate everything on the web about a particular person or business, you should run searches in two or more web index sites, such as AltaVista (http://www.altavista.com) or Northern Light (http://www.northernlight.com). Do not rely on a catalog of the web for an exhaustive search. If you are looking for the answer to a specific question, such as the population of Chicago, Ask Jeeves (http://www.askjeeves.com) is the best bet. If you are trying to learn about an unfamiliar subject, you should start with a web catalog, such as Yahoo (http://www.yahoo.com), so that your results will be limited and sorted by category. Remember that no two engines will return the same set of results so you should always run your search at more than one search site.

Although search engines are more powerful than catalog sites, the catalog sites are much easier to use. Yahoo (http://www.yahoo.com) is the best known of the catalog sites - humans organize websites into categories and search results are sorted by category. There is a lot of content at Yahoo other than the web cataloging, including news sources and financial information.

AltaVista (http://www.altavista.com) is the only search engine on the web that allows a true proximity connector (through the advanced search only). Proximity allows you to request that two terms appear close together, but not necessarily right next to each other. This is especially helpful when searching for a person's name. In a keyword search you might type "Jane Doe", and you would miss any documents that refer to Jane Q. Doe.

Northern Light (http://www.northernlight.com) categorizies your search results by type of page, type of organization and topic. This can be very helpful to narrow down a search that gets a lot of hits.

Two metasearch websites are very helpful for finding topical search engines. The first is Search.Com (http://www.search.com). Their list of specialty searches helps you to locate more than 100 different ways to search the Web, including telephone directories, financial information and government resources. Go Gettem, (http://www.gogettem.com) lists over 1200 specialty directories by category, plus you can search for any publications or broadcast media website.

GENERAL SEARCH TIPS

DOT / Dot / dot

Some search engines are case sensitive and others are not. Your best bet is to type everything in small caps. When a search engine is case sensitive, it is very literal. DOT stands for the Department of Transportation, Dot is someone's name and dot is what you find on a Dalmatian!

HELP!!!!

Always read the help screens when using a new search engine, and it doesn't hurt to check the help section occasionally to find new features that have been added. Every search engine uses different search connectors, so what works at AltaVista may not work at Northern Light. It only takes a few moments to browse through the help pages to find out what Boolean connectors are available, what field searching is available and how you can organize the results of your search.

Two is better than one

Not all search engines index the same material on the Web. It's always a good idea to try your search in a few different places.

The missing link

Search engines do not index the web more than once a week, and usually won't detect a new website or changes to an old website for at least a month. If you find a link to a site that is no longer there, click on the URL and delete everything after the last slash. If that doesn't work, go back to the slash before that. Once you do pull up a website, look for a link to the information you were looking for. A few months ago I bookmarked the phone list of the 13^th Circuit Court in Tampa, Florida. The URL was http://jud13.flcourts.org/courts/ctel2.html. When I tried to go back to the list this week, all I got was "File Not Found". I changed the location to http://jud13.flcourts.org/courts/ and still got nothing, so I took out even more. The 13^th Circuit Website is at http://jud13.flcourts.org/ and there is a link on the main screen to the phone list I needed.

Finding a needle in a haystack

It never fails. I've run a search and come up with what seems to be the perfect match, but when I link to the resulting page, the text goes on forever with no sign of my search terms in site. Rather than read page after page hoping to run across the terms, I use the Find feature of the browser. Find works just like it does in most word processing programs - it searches the text of the open document (page) for the exact text that you type . Open the Edit Menu and select Find in Page and then type in the search term. Remember that Find is very literal. If you type in john doe it will skip john j. doe.



SEARCH ENGINE LINKS

GUIDE TO ALTA VISTA and NORTHERN LIGHT

Two very powerful engines are described below. Both Northern Light and AltaVista have simple keyword searching available at their home sites. This table addresses more advanced search techniques. Connectors do not have to be in all capital letters.