The purpose of this master's thesis in library and information science is to examine the performance of the three keyword search engines Alta Vista, excite and Lycos, regarding searching in their automatically indexed World Wide Web resources.
Twenty queries have been used and the first twenty hits for each query were evaluated for relevance. Queries were invented by the authors based upon their knowledge in various areas of interest. They were expressed in a general manner in accordance with the search syntax available for each search engine.
Criteria for evaluation of retrieved pages are designed to be as thorough and detailed as possible since evaluations of search engines are based on the inherently subjective notion of relevance. A three-point relevance scale is used, where highly relevant pages are assigned the value 1, partially relevant 0.5 and irrelevant, inactive and duplicate links/pages get a 0. The measure used is precision.
The method of measuring precision is designed to give credit to those search engines that present relevant hits early in their ranked lists. Differences in performance between search engines was not statistically significant, according to the Friedman test.
Excite proved to be the top performing search engine with respect to precision, closely followed by Alta Vista and lastely Lycos. Lycos had the most number of inactive links, in accordance with earlier studies.
We advocate a more user-oriented view when evaluating search engines, since there are so many uncontrollable variables present. These variables include matching function, method of indexing and contents of the database. They all play a role in retrieval performance, but their respective influence is unknown to researchers.
© Pererik Breimark & Per D. Hagman