Experimental design: case study

Two enthusiastic computer science researchers, let's call them Bill Bodgett and Susan Scarper, have devised a new technique for information retrieval from the World-Wide Web. Their technique has a novel user interface, and employs very sophisticated techniques for cataloguing and prioritizing information. They are justly proud of their work, and after a year of refinement they write it up for publication in a prestigious journal.

Sadly the referees are not impressed; they assert that although they ideas sound good in practice, there is no evidence that the technique really improves the speed or ease of end-users' access to information. The work is of technical interest, they say, but not fully evaluated enough to merit publication at this stage.

Dismayed but not downhearted, Bodgett and Scarper round up a group of a half dozen or so of their friends and family, and sit them down in front of a computer. They ask each person to use the new software to carry out some information retrieval task, then ask them to fill in a questionnaire about the experience. The questionnaire asks the users to rate various aspects of their impressions of the system on a scale of 1-10, with `1' being awful and `10' excellent. On the whole the users rate most aspects of the system's performance as 8-10. The average overall score is 8.6.

Thus encouraged, Bodgett and Scarper re-write their paper and re-submit it. To their shock it is rejected again, this time because their experimental methodology was ill-founded. Now thoroughly discouraged, the researchers shelve the project and leave academic work completely. Both get jobs as programmers in a merchant bank and make enough money to retire at 40. But that's a different story.

In this workshop we will be examining ways in which Bodgett and Scarper could have performed their experiment better, and saved themselves from the disgrace of moving to the private sector.