Experimental design: the null hypothesis

The null hypothesis is complementary to the experiment's primary hypothis

For example, if Bodgett and Scarper were claiming that their new system led to an improvement in the speed with which users could retrieve information, a compatible null hypothesis might be ``there is no difference between the speed of information retrieval of the new system and other comparable systems''.

`Rejecting' the null hypothesis is in practice an assertion that your hypothesis can be assumed true, at least for the time being

If the above null hypothesis is rejected, this is an indication that there is some difference between the different systems. This is, of course, what Bodgett and Scarper were asserting.

Rejecting the null hypothesis is not technically the same as disproving it. It merely indicates that there is enough evidence to assume it false

We cannot prove a hypothesis to be correct, even in principle. The null hypothesis, however, can, in principle, be proven false. But in practice we are basing our decision whether to reject the null hypothesis on the results of experimental measurements; these are subject to various random effects and errors, so we can't technically prove the null hypothesis false. We can, however, reject it.

The null hypothesis is rejected if there is a low probability that results leading to its rejection could have arisen by random effects in the experiment

There is always the chance that the null hypothesis is true, but there is sufficient random variation in the experimental results to give the erroneous impression that the null hypothesis is false. This would be an error, and we should only reject the null hypothesis if this random variation is unlikely to be large enough to cause a false rejection very often. We therefore calculate the probability that a false rejection will occur.

The actual level of this probability cannot be derived by experiment; it is asserted by the experimenters on the basis of the consequences of rejection of the null hypothesis

In many disciplines it is conventional to reject the null hypothesis when results compatible with it being false will be arrived at by chance less than one time in twenty (i.e., 5% probability). When this happens we say that the experimenal result was `significant at the 5% level'. Rejecting the null hypothesis is tantamount to claiming that your experiment has revealed something potentially important. If rejecting the null hypothesis will have significant and potentially dangerous consequences, it is conventional to set the probability level to be more conservative, e.g., 1%.

We will now turn our attention to the second main problem with the case study: that of the poor generalizability of the results.