Decision Errors

STA35B: Statistical Data Science 2

Akira Horiguchi

Table of contents

• Hypothesis testing turns data into a decision
• Four Possible Outcomes
• Discernibility Level
• What Does \(\alpha\) Control?
• Choosing \(\alpha\)
• One-Sided vs. Two-Sided
• CPR Study
• CPR: Hypotheses
• CPR: Two-Sided p-Value
• CPR: Decision
• Do Not Choose the Tail Afterward
• Power
• Common Misinterpretations

Based on Ch 14 of IMS

Hypothesis testing turns data into a decision

• \(p\)-value small enough \(\rightarrow\) reject \(H_0\)
• But even a careful test can lead to the wrong conclusion.

  • It is possible to commit no mistakes and still lose. That is not a weakness; that is life.

    — Jean Luc Picard

  • It is possible to use the data perfectly and still mispredict. That is not a weakness; that is life’s randomness and uncertainty.

    — Me

• What kinds of mistakes can happen, and how often?

Four Possible Outcomes

Truth	Reject \(H_0\)	Fail to reject \(H_0\)
\(H_0\) true	Type I error	Good decision
\(H_A\) true	Good decision	Type II error

• Type I error: reject \(H_0\) when \(H_0\) is true.
• Type II error: fail to reject \(H_0\) when \(H_A\) is true.

Timing: 3 minutes. This is the central table. Use a courtroom analogy verbally if useful: Type I = convict an innocent person; Type II = fail to convict a guilty person. Emphasize “fail to reject” rather than “accept.”

Example: A spam filter tests whether a message is spam. Let:

\[ H_0: \text{message is legitimate} \]

\[ H_A: \text{message is spam} \]

What are Type I and Type II errors?

Timing: 2 minutes. Expected: Type I = mark a real email as spam. Type II = let spam through. Ask which error students find more costly; this motivates alpha choices.

Discernibility Level

The discernibility level is the cutoff for deciding whether a \(p\)-value is small enough to reject \(H_0\).

• Chosen before the test, not discovered from the data.
• For a chosen discernibility level \(\alpha\) (often chosen to be 0.05):
  • If \(p \le \alpha\), reject \(H_0\).
  • If \(p > \alpha\), fail to reject \(H_0\).

Timing: 1 minute. Keep the distinction crisp: the p-value is data-based; alpha is the pre-set standard for acting on that p-value.

What Does \(\alpha\) Control?

If \(H_0\) is true and we use \(\alpha = 0.05\):

• we will reject \(H_0\) about 5% of the time in repeated testing.
• Those rejections are Type I errors.
• So \(\alpha\) controls the long-run Type I error rate.

Timing: 2 minutes. Important: this is a long-run statement, not “there is a 5% chance this conclusion is wrong.”

Choosing \(\alpha\)

Choose \(\alpha\) based on consequences.

If this error is especially costly	Consider
Type I error	smaller \(\alpha\), e.g. 0.01
Type II error	larger \(\alpha\), e.g. 0.10

• Lowering one kind of error usually increases the other.

Timing: 2 minutes. Use the spam example or courtroom analogy. Smaller alpha makes it harder to reject, reducing false positives but increasing missed detections.

One-Sided vs. Two-Sided

A one-sided alternative asks whether the effect goes in one specified direction:

\[ H_A: p_T - p_C > 0 \]

A two-sided alternative asks whether there is any difference:

\[ H_A: p_T - p_C \ne 0 \]

Timing: 1.5 minutes. The key phrase for two-sided tests is “different,” not “larger” or “smaller.”

Use a two-sided test when:

• either direction would matter,
• the research question is about “any difference,”
• the direction was suggested by the observed data,
• ignoring one direction would reflect confirmation bias.

Timing: 1.5 minutes. One-sided tests are legitimate only when the direction was genuinely specified before seeing the data and the other direction is not scientifically relevant to the claim.

CPR Study

Study question:

Do blood thinners affect 24-hour survival after CPR for a heart attack?

Group	Died	Survived	Total
Control	39	11	50
Treatment	26	14	40
Total	65	25	90

Timing: 1.5 minutes. Source: Bottiger et al. 2001 example in IMS2 Chapter 14. Define treatment: blood thinner. Control: no blood thinner.

• Control survival rate: \(\hat{p}_C = \frac{11}{50} = 0.22\)
• Treatment survival rate: \(\hat{p}_T = \frac{14}{40} = 0.35\)
• Observed difference: \(\hat{p}_T - \hat{p}_C = 0.13\)

Timing: 1.5 minutes. Ask students to compute the two survival rates before revealing. The treatment group has 13 percentage points higher observed survival.

CPR: Hypotheses

Let:

• \(p_C\) = true survival rate without blood thinner
• \(p_T\) = true survival rate with blood thinner

Two-sided hypotheses:

\[ H_0: p_T - p_C = 0 \]

\[ H_A: p_T - p_C \ne 0 \]

Timing: 1.5 minutes. Ask: Why two-sided? Because blood thinners could plausibly help or harm after CPR, and either direction matters.

CPR: Two-Sided p-Value

The observed statistic is \(+0.13\).

For a two-sided test, results at least as extreme include both tails:

\[ \hat{p}_T - \hat{p}_C \le -0.13 \qquad\text{and}\qquad \hat{p}_T - \hat{p}_C \ge 0.13 \]

The two-sided p-value is the shaded area, which is roughly p = 0.262.

CPR: Decision

Using \(\alpha = 0.05\):

\[ 0.262 > 0.05 \]

• Decision: fail to reject \(H_0\).
• Interpretation: the study does not provide convincing evidence that blood thinners changed 24-hour survival.

Timing: 2 minutes. Carefully distinguish “no convincing evidence of an effect” from “proof of no effect.”

Do Not Choose the Tail Afterward

Bad workflow:

• look at data \(\rightarrow\) notice direction \(\rightarrow\) choose one-sided test in that direction
• With \(\alpha = 0.05\), this can make the Type I error rate about 10% instead of 5%.
• Set hypotheses before observing the data.

Timing: 2 minutes. The issue is not one-sided tests themselves; the issue is choosing the side after data inspection.

Power

Power is the probability that a test rejects \(H_0\) when \(H_A\) is true.

\[ \text{Power} = P(\text{reject } H_0 \mid H_A \text{ true}) \]

• High power means a real effect is likely to be detected.
• Power = 1 - probability of Type II error.

Timing: 2 minutes. Power is the complement of the miss rate when the alternative is true. Keep this conceptual; no need for a full power calculation in a 30-minute lecture.

Power tends to increase when:

• the sample size is larger,
• the true effect is larger,
• variability is smaller,
• the discernibility level \(\alpha\) is larger,
• the study design is more efficient.

Timing: 1.5 minutes. Mention the tradeoff: increasing alpha raises power but also raises Type I error. Larger sample size often raises power without changing alpha, but it costs time and resources.

Common Misinterpretations

Avoid saying:

• “Fail to reject \(H_0\) means \(H_0\) is true.”
• “\(\alpha = 0.05\) means there is a 5% chance this conclusion is wrong.”
• “A large p-value proves no effect.”
• “It is okay to choose one-sided after seeing the data.”

Timing: 1.5 minutes. These are the most likely exam/project errors.