LIS 601
Final Project: Guidance for those thinking about an IR experiment 

---

The CISI Test Collection:

The CISI Test Collection is up and running at http://slis6000.slis.uwo.ca/601/test.

Introduction:

This page is designed to help those thinking about conducting an IR experiment for their final project.

What should you look for?

The main point of the project is to ask some interesting research question(s) about an IR system (or systems) and then design an experiment (or experiments) to test it/them.

A Possible Research Question:

"Does the ability to use different search styles on a given search engine really help the user?"

A Possible Approach:

The Assumptions:

• Say, search engine X has three search styles:
• simple word,
• boolean,
• advanced (using links, etc.).
• Say, you have drawn up a list of five queries (that you expect should be answerable by the database).
• Say, you believe that precision effectiveness is the most important measure of performance.

First draft of experimental design:

• One could create a very pretty little experiment where one tests the differences in precision using the three different approaches.
• This would give you 5 (queries) X 3 (search styles) = 15 precision measures.

Second draft of experimental design:

• An added, and very informative, twist would be to take your precision measures at two different times:
• a "naive" search where you give the engine only one chance to find the information (like most users do); and,
• a "best case" search where you have used all of your skills to create the most precise retrieved set.
• This would give you  5 (queries) X  3 (search styles)  X  2 (query styles) = 30  precision measures.
• This is a "doable", yet informative, project.

Third draft of experimental design:

• This is the same as Draft Two; however, to control for different searcher skill levels, have each member of your group run the experiment on his/her own.
• N.B. the queries MUST be the same but the search statements will not be necessarily the same.
• Pool your results to arrive at your conclusions.

Comments:

• The questions you could ask are almost endless.
• This guide is intended as a starting point and is not a prescription of what to do.
• Once you have collected your data, comparing and contrasting the results will cause you to:
1. investigate some explanations for the results you achieved; and,
2. allow you answer your research question.
• Drawing up an experimental design chart/table will make your research much easier and enjoyable. For example, look below at the chart/table that represents Draft Two:
• Notice how the process simplifies to filling in the cells with the appropriate data.
• For Draft Three, each experimenter would fill out his/her own chart/table. Then the results could be pooled into a Master chart/table.
• How you calculate your measures is, for the most part, up to you: (e.g. precision measured at first 20 documents in each retrieved set, etc.). Note however:
• one must be consistent
• the measure must be justifiable (i.e. precision at first 3 documents is a tad ridiculous).

 Experimental Design Table (Draft Two):