|
|
Experimental design: the null hypothesis
The null hypothesis is complementary to the experiment's primary
hypothis
For example, if Bodgett and Scarper were claiming that their new system
led to an improvement in the speed with which users could retrieve information,
a compatible null hypothesis might be ``there is no difference between the speed of
information retrieval of the new system and other comparable systems''.
`Rejecting' the null hypothesis is in practice an assertion that your
hypothesis can be assumed true, at least for the time being
If the above null hypothesis is rejected, this is an indication that there
is some difference between the different systems. This is, of course,
what Bodgett and Scarper were asserting.
Rejecting the null hypothesis is not technically the same as disproving it.
It merely indicates that there is enough evidence to assume it false
We cannot prove a hypothesis to be correct, even in principle. The null
hypothesis, however, can, in principle, be proven false. But in practice we are
basing our decision whether to reject the null hypothesis on the results of
experimental measurements; these are subject to various random effects and
errors, so we can't technically prove the null hypothesis false. We can,
however, reject it.
The null hypothesis is rejected if there is a low probability that results
leading to its rejection could have arisen by random effects in the
experiment
There is always the chance that the null hypothesis is true, but there is
sufficient random variation in the experimental results to give the erroneous
impression that the null hypothesis is false. This would be an error, and we should only reject
the null hypothesis if this random variation is unlikely to be large enough to
cause a false rejection very often. We therefore calculate the probability
that a false rejection will occur.
The actual level of this probability cannot be derived by experiment; it is
asserted by the experimenters on the basis of the consequences of rejection of
the null hypothesis
In many disciplines it is conventional to reject the null hypothesis when
results compatible with it being false will be arrived at by chance less than
one time in twenty (i.e., 5% probability). When this happens we say that the
experimenal result was `significant at the 5% level'. Rejecting the null
hypothesis is tantamount to claiming that your experiment has revealed
something potentially important. If rejecting the null hypothesis
will have significant and potentially dangerous consequences, it is
conventional to set the probability level to be more conservative, e.g., 1%.
We will now turn our attention to the second main problem with the case study:
that of the poor generalizability of the results.
|
