Anecdotes of Significance Testing

For the Gaugh quote in the p-value anecdote, briefly explain how the author is misinterpreting the meaning of the p-value, and give a correct interpretation.

"Roughly speaking, the p value is the probability that the observed result happened merely by chance; the complementary value (1-p) is the probability that the source caused a real effect."

• Understand what the components of a hypothesis test are such as: test statistic, alpha level, p-value. Be able to identify these components.
• Understand that the p-value is the probability of getting a sample statistic as extreme or more extreme than what was seen in the sample given that the null hypothesis is true.
• Understand that p-values are strongly related to sample size. Be aware that statistical significance and practical significance are different.
• Beware of searching for significance. Multiple testing may find spuriously significant results.

Gaugh is incorrect in saying that "1- p is the probability that the source caused a real effect." The p-value is calculated assuming that the chance model of the null hypothesis is correct. Therefore, p does not give the chance that the null hypothesis is correct nor does 1 - p give the chance that the alternative is correct as indicated in the quote.

The p-value is the probability, assuming the null hypothesis is true, that the test statistic would take a value as extreme or more extreme than that actually observed in the test.

Assume that the media critic in the Secondhand Smoke anecdote is referring to a hypothesis test done by the EPA. How is he misinterpreting it?

• Understand what the components of a hypothesis test are such as: test statistic, alpha level, p-value. Be able to identify these components.
• Understand that the p-value is the probability of getting a sample statistic as extreme or more extreme than what was seen in the sample given that the null hypothesis is true.
• Understand that p-values are strongly related to sample size. Be aware that statistical significance and practical significance are different.
• Beware of searching for significance. Multiple testing may find spuriously significant results.
  

The media critic that the Associated Press (AP) is referring to is misinterpreting the value 1 - α, where α is the fixed significance level of the test. In a fixed significance level test, the p-value must be as small or smaller than the fixed level α for the null hypothesis to be rejected. When the critic says, "...calling a study significant when it had only a 90 percent chance of accuracy...," he is implying that a significance test with a p-value of (at most) 0.10 is (1 - 0.10) × 100% or 90 percent accurate. This is false. The p-value is calculated assuming that the chance model of the null hypothesis is correct. 1- p does not give the chance that the alternative hypothesis is correct, let alone that the study is accurate. In fact, it is not clear exactly what accurate means, as it is not a standard statistical term.

There is another interpretation of what the media critic was thinking in the Secondhand Smoke anecdote. Perhaps the EPA used the duality of confidence intervals and hypothesis tests to conduct their tests by calculating a confidence interval first. Was the media critic correct in the spirit of his comment? In other words, if he had said, "...calling a study significant when the confidence intervals were only performed at the 90 percent level instead of the usual 95 percent level," would he have had a legitimate complaint?  Explain your position on this.

• Understand what the components of a hypothesis test are such as: test statistic, alpha level, p-value. Be able to identify these components.
• Understand that the p-value is the probability of getting a sample statistic as extreme or more extreme than what was seen in the sample given that the null hypothesis is true.
• Understand that p-values are strongly related to sample size. Be aware that statistical significance and practical significance are different.
• Beware of searching for significance. Multiple testing may find spuriously significant results.
• Understand the relationship between a confidence interval and a hypothesis test.

The key to this interpretation is located in the final paragraph of the excerpt. Dr. Ron said, "In other words, no one says that secondhand smoke is good. So 90 percent was enough to detect either no effect or a bad one, and was the same level EPA used to label radon and dioxin dangerous." This indicates that the EPA was really interested in conducting a one-sided test, not a two-sided test (e.g. H_a > µ as opposed to H_a ≠ µ). Note that the critical point for this one-sided α = 0.05 level test is the same as the upper critical point for the two-sided α = 0.10 level test. If the critic failed to realize the EPA conducted a one-sided test, he or she might be misled into thinking that the EPA conducted a two-sided 0.10 level test based on reported critical values.

The analog of a one-sided hypothesis test is a confidence bound, not a confidence interval. A 90 percent confidence interval will yield the same lower endpoint as that of the 95 percent lower confidence bound. Likewise, the 90 percent confidence interval will yield the same upper endpoint as that of the 95 percent upper confidence bound. Note that the 90 percent confidence interval excludes 5 percent in each tail (both upper and lower). So the EPA simply used the 90 percent interval to get either the upper or lower endpoint on the 95 percent bound. The upshot of all this is that the media critic still would not have a legitimate complaint, but the EPA certainly reported their results in a manner that caused confusion.