TWiki > DLibrary > RankingExperiment	TWiki webs: Main | TWiki | Know | Sandbox
	DLibrary . { Changes | Index | Search | Go }

In the RankingExperiment, the cache is first primed using a trace, or perhaps simply from usage over a period. A set of users are then given a list of search queries to perform. These queries can be of two types

1. Do the following exact query (eg "president bush incompetent")
2. Run a query to find pages discussing something (eg. the incompetence of "president" bush)

The query is then performed against both the cache and Google.com, retreiving up to some maximum number of documents each. These results are randomly mixed and merged, and displayed for the user. The url, title, and summary of the document are not displayed, so that the user is not biased by the quality of the summary, title extraction, url cleanup etc, but rather only considers the ranking of the documents.

The user is asked to chose the top five documents in the result set. The user browses all of the results and ranks the top five.

For each of the top five documents chosen by the user, if the documents is one of Google's top five, the google gets a point. If the document is one of the cache's top five, the cache gets a point.

Google's score is then subtracted from the cache's score. A mean and standard deviation is caculated for the difference, over a large set of queries. By gathering rankings on the same query from several users, we can also calculate a standard error for our results.

This experiment can be repeated under different conditions:

• larger/smaller cache size
• more/fewer users (generating requests, not users ranking search queries)
• different traces
• different user groups
  • users in the same research group
  • users in the same institution
  • users in the same library