Experimental design: comments on generalization of the case study

Generalization issues in the Bodgett and Scarper experiment

The problem with the way Bodgett and Scarper conducted their experiment is not necessarily the size of the experimental group, although this may indeed be a problem, but the population which the sample represents. There is no reason to think that a group of the experimenters' family and friends are representative of anything useful. What population did the experimenters have in mind? Probably they did not think about this at all; if they had they would have had to decide whether the sample should represent experienced computer users, computing professionals, the general public, or something else entirely. In any event, it needs to be obvious to the reader of the paper how far the results can be generalized to cases in which they (the readers) are interested.

Even if a useful experimental group were selected, there could still be problems with bias. People like to please, on the whole, and will tend to return results that they think the experimenters will like. They may not do this consciously, but it is no less disruptive for all that.