|
|
Experimental design: case study
Two enthusiastic computer science researchers, let's call them Bill Bodgett and
Susan Scarper, have devised a new technique for information retrieval from the
World-Wide Web. Their technique has a novel user interface, and employs very
sophisticated techniques for cataloguing and prioritizing information. They are
justly proud of their work, and after a year of refinement they write it up
for publication in a prestigious journal.
Sadly the referees are not impressed; they assert that although they ideas
sound good in practice, there is no evidence that the technique really improves
the speed or ease of end-users' access to information. The work is of technical
interest, they say, but not fully evaluated enough to merit publication at this
stage.
Dismayed but not downhearted, Bodgett and Scarper round up a group of a half
dozen or so of their friends and family, and sit them down in front of a
computer. They ask each person to use the new software to carry out some
information retrieval task, then ask them to fill in a questionnaire about the
experience. The questionnaire asks the users to rate various aspects of their
impressions of the system on a scale of 1-10, with `1' being awful and `10'
excellent. On the whole the users rate most aspects of the system's performance
as 8-10. The average overall score is 8.6.
Thus encouraged, Bodgett and Scarper re-write their paper and re-submit it. To
their shock it is rejected again, this time because their experimental
methodology was ill-founded. Now thoroughly discouraged, the researchers
shelve the project and leave academic work completely. Both get jobs as
programmers in a merchant bank and make enough money to retire at 40. But
that's a different story.
In this workshop we will be examining ways in which Bodgett and Scarper could
have performed their experiment better, and saved themselves from the disgrace
of moving to the private sector.
|
