11 – Power analysis

Introduction

The power of a statistical test is the probability that the test will reject the null hypothesis when the alternative hypothesis is true. Most of us are used to thinking that a hypothesis is either right or it is wrong: a Doctor’s diagnosis is correct, the patient has the disease, or the patient does not; an experimental result is objectively true — i.e., true independent of  the observer’s subjectivity — or it is false. As we work through a typical science curriculum, we may even take to heart that, unlike mathematicians, scientists don’t prove scientific ideas no matter how well supported by evidence. Our acceptance of scientific theories is provisional; if new evidence comes along, we revise and if warranted, we abandon the theory in favor of new explanation. However, even this point does not completely reflect the point we are making from statistical thinking. One of the more challenging concepts for new statistics students to understand is that outcomes of a Doctor’s diagnosis or that of an experiment are associated with probability.

The concept of statistical power helps to relate how our ability to confidently conclude one outcome over an another. Statistical power depends on

  • What Type I error rate we set
  • The effect size or difference between affected and unaffected groups
  • The sample size
  • The variability of the subjects

These concepts have all been introduced before, but the idea that even a well designed experiment may lack the capability of detecting “truth” is a new and important topic to add to your growing statistical thinking tool kit.

Chapter 11 contents