Lecture 5: t tests

Probability and

Statistics
Part 2: Statistics

Tests of Significance



Reasoning of significance tests



Stating hypotheses



Test statistics



P-values



Statistical Significance



Test for a population mean



Two-sided significance tests and

confidence intervals



P-values vs. fixed α

Tests of Significance



2 most common forms of statistical
inference



Confidence interval



appropriate when goal is to estimate a
population parameter



Significance test



Appropriate when goal is to assess the
evidence provided by the data in favor of
some claim about the population

Reasoning of significance
tests



Significance test



Formal procedure for comparing observed
data with a hypothesis whose truth we want
to assess.



Hypothesis: statement about the parameters in a
population or model



Results of the test are expressed in
terms of a probability that measures how
well the data and the hypothesis agree.

Stating hypotheses



Null hypothesis



The statement being tested in a test of
significance



The test of significance is designed to
assess the strength of the evidence
against it.



Statement of “no effect” or “no
difference”



Abbreviated as H

0

Stating hypotheses, cont.



Alternative hypothesis



The statement we hope or suspect is true

instead of H

0



Abbreviated as H

a



Hypotheses always refer to some population

or model, not to a particular outcome.



The hypotheses will be stated in terms of

population parameters.



Can have one-sided or two-sided alternative

hypotheses.

Test Statistics



The test is based on a statistic that estimates

the parameter that appears in the

hypotheses. Usually this is the same

estimate we would use in a confidence

interval for the parameter. When H

0

is true,

we expect the estimate to take a value near

the parameter value specified by H

0

.



Values of the estimate far from the parameter

value specified by H

0

give evidence against

H

0

. The alternative hypothesis determines

which directions count against H

0

.

Test Statistics, cont.



Test statistic



Measures compatibility between the
null hypothesis and the data.



Used for the probability calculation
that we need for our test of
significance.



Random variable with a distribution
that we know.

P-values



A test of significance finds the
probability of getting an outcome
as extreme or more extreme than
the actually observed outcome

.



Extreme: far from what we would
expect if the null hypothesis were
true

P-values, cont.



The probability, computed assuming that H

0

is true, that the test statistic would take a

value as extreme or more extreme than that

actually observed is called the P-value of the

test.



The smaller the P-value, the stronger the

evidence against H

0

provided by the data.



Calculate it using the sampling distribution of

the test statistic (for example the standard

normal distribution for the test statistic z).

Statistical Significance



If the P-value is as small or smaller than
α, we say that the data are statistically
significant at level α.



Example:

alpha level of 0.01 means

that we are insisting that we obtain
evidence so strong that it would appear
only 1% of the time if the null
hypothesis is in fact true.



What does a p-value of 0.03 tell you?



Random samples from N(0,1): two
groups of 16 individuals



Collect p-values obtained from any
statistical test



Number of experiments N=25000

Simulation study

N = 1262

Test of Significance Steps



You can assess the significance of the

evidence provided by data against a null

hypothesis by performing these steps:

1.

State the null hypothesis H

0

and the alternative

hypothesis H

a

. The test is designed to assess the

strength of the evidence against H

0

. H

a

is the

statement that we will accept if the evidence

enables us to reject H

0

.

2.

Calculate the value of the test statistic on which

the test will be based. This statistic usually

measures how far the data are from H

0

.

Test of Significance Steps

3.

Find the P-value for the observed data. This is

the probability (calculated assuming that the null

is true) that the statistic will weigh against the

null at least as strongly as it does for these data.

4.

State a conclusion. Choose a significance level α

(how much evidence against the null you regard

as decisive). If the P-value is less than or equal

to α, you conclude that the alternative

hypothesis is true; if it is greater than α, you

conclude that the data do not provide sufficient

evidence to reject the null hypothesis. Your

conclusion is a sentence that summarizes what

you have found by using a test of significance.

Tests for a population
mean



We have an SRS of size n drawn from a

normal population with unknown mean

μ but known variance σ

2

. We want to

test the hypothesis that μ has a

specified value, call it μ

0

.



Null hypothesis: H

0

: μ = μ

0

.



Test statistic:

has the standard normal distribution when

the null hypothesis is true.

n

x

z

/

0









Tests for a population
mean



Alternative hypothesis: If it is one-
sided on the high side, H

a

: μ > μ

0



P-value: probability that a standard
normal random variable Z takes a
value at least as large as the
observed z.



We can similarly reason for other
alternative hypotheses.

Two-sided significance
tests and confidence
intervals



A level α two-sided significance
test rejects a hypothesis H

0

: μ = μ

0

exactly when the value μ

0

falls

outside a level 1- α confidence
interval for μ.

P-values vs. fixed α



The P-value is the smallest level α at

which the data are significant.



Knowing the P-value allows us to assess

significance at any level.



It is more informative than a reject-or-not

finding at a fixed significance level.



A value z* with a specified area to its

right under the standard normal curve is

called a a critical value of the standard

normal distribution.

Use and Abuse of Tests



Choosing a level of significance



Choose a level α in advance if you must

make a decision.



Does not make sense if you wish only to

describe the strength of your evidence.



If you do use a fixed α significance test to

make a decision, choose it by asking how

much evidence is required to reject H

0

.



This depends on how plausible the null

hypothesis is.

Choosing a level of significance



IF H

0

represents an assumption that

everyone in your field has believed
for years, strong evidence (small α)
will be needed to reject it.



Level of evidence required to reject
H

0

depends on the consequences of

such a decision.



Expensive: strong evidence

Choosing a level of significance



Better to report the P-value, which
allows each of us to decide
individually if the evidence is
sufficiently strong.



There is no sharp border between
“significant” and “insignificant,”
only increasingly strong evidence
as the P-value decreases.

What statistical
significance doesn’t mean



Statistical significance is not the same as

practical significance.



Example: hypothesis of no correlation is

rejected



Does not mean strong association, only that there is

strong evidence of some association



A few outliers can produce highly

significant results if you blindly apply

common tests of significance.



Outliers can also destroy the significance

of otherwise convincing data.

Don’t ignore lack of
significance



If a researcher has good reason to
suspect that an effect is present and then
fails to find significant evidence of it, that
may be interesting news---perhaps more
interesting than if evidence in favor of the
effect at the 5% level had been found.



Keeping silent about negative results may
condemn other researchers to repeat the
attempt to find an effect that isn’t there.

Statistical inference is not
valid for all sets of data



Formal statistical inference cannot
correct basic flaws in the design.



Randomization in sampling or
experimentation ensure that the
laws of probability apply to our
tests of significance and
confidence intervals.

Beware of searching for
significance



The reasoning behind statistical
significance works well if you
decide what effect you are
seeking, design an experiment or
sample to search for it, and use a
test of significance to weigh the
evidence you get.

Beware of searching for
significance



Because a successful search for a
new scientific phenomenon often
ends with statistical significance, it
is all too tempting to make
significance itself the object of the
search.

Beware of searching for
significance



Once you have a hypothesis,
design a study to search
specifically for the effect you now
think is there. If the result of this
study is statistically significant,
you have real evidence.

Power and Inference as a
Decision



Examining the usefulness of a
confidence interval



Level of confidence: tells us how
reliable the method is in repeated use



Margin of error: tells us how sensitive
the method is, or how closely the
interval pins down the parameter
being estimated

Power and Inference as a
Decision



Examining the usefulness of fixed

level α significance tests



Significance level: tells us how reliable

the method is in repeated use



Power

: tells us the ability of the test to

detect that the null hypothesis is false



Measured by the probability that the test

will reject the null hypothesis when an

alternative is true.

The higher this

probability is, the more sensitive the test is.

Power



The probability that a fixed level α
significance test will reject H

0

when

a particular alternative value of the
parameter is true is called the
power of the test to detect that
alternative.

Calculating Power



State H

0

, H

a

, the particular

alternative we want to detect,

and the significance level α.



Find the values of that will lead

us to reject H

0

.



Calculate the probability of

observing these values of when

the alternative is true.

x

x

Increasing the Power



Increase α. A 5% test of significance will
have a greater chance of rejecting the
alternative than a 1% test because the
strength of evidence required for rejection
is less.



Consider a particular alternative that is
farther away from μ

0

. Values of μ that are

in H

a

but lie close to the hypothesized

value μ

0

are harder to detect (lower power)

than values of μ that are far from μ

0

.

Increasing the Power,
cont.



Increase the sample size. More data will

provide more information about so

we have a better chance of

distinguishing values of μ.



Decrease σ. This has the same effect as

increasing the sample size: more

information about μ. Improving the

measurement process and restricting

attention to a subpopulation are two

common ways to decrease σ.

x

Two types of error



In significance testing, we must accept one
hypothesis and reject the other.



We hope that our decision will be correct,
but sometimes it will be wrong.



2 types of incorrect decisions



If we reject H

0

(accept H

a

) when in fact H

0

is

true, this is a Type I error (related to p-value).



If we accept H

0

(reject H

a

) when in fact H

a

is

true, this is a Type II error (related to the power).

The common practice of
testing hypotheses



State H

0

and H

a

just as in a test of

significance.



Think of the problem as a decision
problem, so that the probabilities of
Type I and Type II errors are relevant.



Type I errors are more serious. So
choose an α (significance level) and
consider only tests with probability of
Type I error no greater than α.

The common practice of
testing hypotheses



Among these tests, select one that
makes the probability of a Type II error
as small as possible (that is, power as
large as possible). If this probability is
too large, you will have to take a
larger sample to reduce the chance of
an error.

The one-sample one-tail t
test



Suppose that an SRS of size n is drawn

from a population having unknown

mean μ. To test the hypothesis that H

0

:

μ = μ

0

based on an SRS of size n,

compute the one-sample t statistic



In terms of a random variable T having

the t(n-1) distribution, the P-value for a

test of H

0

against H

a

: μ > μ

0

is

or against H

a

: μ < μ

0

is

n

/

s

x

t

0











t

T

P 

)

t

T

(

P



The one-sample one-tail t
test – example 1



Suppose that an SRS of size n is drawn
from a population having unknown mean
μ.



[114, 123.3, 116.7, 129.0, 118, 124.6, 123.1, 117.4, 111,
121.7, 124.5, 130.5]



sample mean = 121.15



sample standard deviation = 5.89



To test the hypothesis that H

0

: μ =

μ

0

=120

based on an SRS of size n,

compute the one-sample t statistic

6764

.

0

12

/

89

.

5

120

15

.

121

n

/

s

x

t

0













The one-sample one-tail t
test – example 1



In terms of a random variable T
having the t(n-1) distribution, the P-
value for a test of H

0

against H

a

: μ >

μ

0

is





t

T

P 

degree of freedom

Critical value at
α=0.95

If t

1

=0.6 then cdf(t

1

)=0.71967 so p

1

=1-

0.71967=0.28033

If t

2

=0.7 then cdf(t

2

)=0.75077 so p

2

=1-

0.75077=0.24923

so p

2

< p < p

1

The one-sample one-tail t
test – example 1

From the exact t-distribution we
get

tcdf(0.6764,11) = 0.743621

p = 1 – 0.743621 =

0.256379

The one-sample one-tail t
test – example 1

The one-sample one-tail t
test – example 1

0.7436

p=0.2564

The one-sample one-tail t
test – example 1



Final conclusion:

we cannot reject the H

0

that

the population mean μ=120

and accept H

a

at p=0.2564

The another example of one-
sample one-tail t test



To test the hypothesis that H

0

: μ = 135

based on an SRS of size n, compute the

one-sample t statistic



In terms of a random variable T having

the t(n-1) distribution, the P-value for a

test of H

0

against

H

a

: μ < μ

0

is

n

/

s

135

x

t









t

T

P 

The one-sample one-tail t
test – example 2

[114, 123.3, 116.7, 129.0, 118, 124.6,

123.1, 117.4, 111, 121.7, 124.5, 130.5]



sample mean = 121.15



sample standard deviation = 5.89



To test the hypothesis that H

0

:

μ = μ

0

=135 compute the one-

sample t statistic

1456

.

8

12

/

89

.

5

135

15

.

121

n

/

s

x

t

0















From the exact t-distribution we
get

P(T ≤ t)

=

tcdf(-8.1456,11) =2.7e-6

p =

0.0000027 < 0.000005

The one-sample one-tail t
test – example 2

The one-sample one-tail t
test – example 2

p=2.7e-6

t = -8.1456

The one-sample one-tail t
test – example 2



Final conclusion:

we reject the H

0

that the population

mean μ=135 and accept H

a

that μ

< μ

0

at p < 0.000005

The one-sample two-tails t
test



To test the hypothesis that H

0

: μ = 115

based on an SRS of size n, compute the

one-sample t statistic



In terms of a random variable T having

the t(n-1) distribution, the P-value for a

test of H

0

against

H

a

: μ ≠ μ

0

is

n

/

s

119

x

t









|

t

|

|

T

|

P



The one-sample two-tails
t test – example 3

[114, 123.3, 116.7, 129.0, 118, 124.6,

123.1, 117.4, 111, 121.7, 124.5, 130.5]



sample mean = 121.15



sample standard deviation = 5.89



To test the hypothesis that H

0

:

μ = μ

0

=115 compute the one-

sample t statistic

6170

.

3

12

/

89

.

5

115

15

.

121

n

/

s

x

t

0













• Since critical t

0.05

=2.2010 and our

observed is bigger than it we reject
the null hypothesis at the α=0.05.

• From the t-distribution we get

P(|T| > |t|)

= 2*(1-

tcdf(3.6170,11)) =

0.0040

So the exact p =

0.0040

The one-sample two-tails
t test – example 3

The one-sample two-tails
t test – example 3

p=0.0040

-t = -3.6170

t = 3.6170

The one-sample two-tails
t test – example 3



Final conclusion:

we reject the H

0

that the

population mean μ=115 and
accept H

a

that μ ≠ μ

0

at p = 0.0040

Matched pairs t
procedures



In a matched pairs study, subjects
are matched in pairs and the
outcomes are compared within
each matched pair.



Example: before and after studies

Key points to remember
concerning matched pairs

•

A matched pair analysis is
needed when there are two
measurements or observations on
each individual and we want to
examine the change from the first
to the second. Typically the
observations are “before” and
“after” measures in some sense.

Key points to remember
concerning matched pairs

•

For each individual, subtract the
“before” measure from the “after”
measure.

•

Analyze the difference using the
one-sample confidence interval
and significance-testing
procedures.

Robustness of the t
procedures



A statistical inference procedure is
called robust if the probability
calculations required are insensitive to
violations of the assumptions made.



The t procedures are quite robust
against nonnormality of the population
except in the case of outliers or strong
skewness.

Robustness of the t
procedures



Larger samples improve the
accuracy of the P-values and critical
values from the t distributions when
the population is not normal.



A normal quantile plot or boxplot to
check for skewness and outliers is
an important preliminary to the use
of t procedures for small samples.

Practical Guidelines for
inference on a single
mean



Sample size less than 15: Use t procedures

if the data are close to normal. If the data

are clearly nonnormal or if outliers are

present, do not use t.



Sample size at least 15: The t procedures

can be used except in the presence of

outliers or strong skewness.



Large samples: The t procedures can be

used even for clearly skewed distributions

when the sample size is large, roughly n >

40.

Comparing Two Means



Two-sample problems



The goal of inference is to compare
the responses in two groups.



Each group is considered to be a
sample from a distinct population.



The responses in each group are
independent of those in the other
group.

Notation



Population



Variable



Mean



Standard Deviation



Sample size



Sample mean



Sample standard deviation

The two-sample z statistic



Natural estimator of the difference μ

1

-

μ

2

is the difference between the sample

means,



To do inference on this statistic, we must
know its sampling distribution.



If the two population distributions are both
normal, then the distribution of
is also normal.

2

1

x

x 

2

1

x

x 



















2

2

2

1

2

1

2

1

,

n

n

N









The two-sample z statistic,
cont.



Suppose that is the mean of an SRS of

size n

1

drawn from a N(μ

1,

σ

1

) population

and that is the mean of an SRS of size

n

2

drawn from a N(μ

2,

σ

2

) population. Then

the two-sample z statistic

has the standard N(0,1) sampling

distribution.

1

x

2

x



 



2

2

2

1

2

1

2

1

2

1

n

n

x

x

z



















The two-sample t
procedure

This test assumes that the variances in

the populations from which the two
samples were taken are identical

The two-sample t
procedures



If the population standard
deviations are not known, then we
estimate them by the sample
standard deviations.



Use the proper formula depending
on the sample sizes n

1

and n

2

.

The two-sample t
procedures

When the sample sizes are unequal and

n

1

or n

2

or both are small (<30) use:

This statistic have a t distribution with

k=n

1

+ n

2

– 2 degrees of freedom.



 













 































2

1

2

1

2

1

2

2

2

2

1

1

2

1

2

1

n

n

n

n

2

n

n

s

)

1

n

(

s

)

1

n

(

x

x

t





The two-sample t
procedures

When n

1

and n

2

are equal (regardless

of size) use:

This statistic have a t distribution

with k=2(n – 1) degrees of freedom.



 



n

s

s

x

x

t

2

2

2

1

2

1

2

1















The two-sample t
procedures

When n

1

and n

2

are unequal but large

(≥30) use:

This statistic have a t distribution with

k=n

1

+ n

2

- 2 degrees of freedom.



 



2

2

2

1

2

1

2

1

2

1

n

s

n

s

x

x

t















Two-sample t significance
test



Suppose that an SRS of size n

1

is

drawn from a normal population
with unknown mean μ

1

and that an

independent SRS of size n

2

is drawn

from another normal population with
unknown mean μ

2.



We assume the standard deviations
for both populations are identical.

Two-sample t significance
test



To test the hypothesis H

0

: μ

1

= μ

2

versus H

a

: μ

1 ≠

μ

2

compute the two-

sample t statistic and use P-values
or critical values for the t(k)
distribution, with proper df.

Two-sample t
significance test -
example



Suppose that an SRS of size n

1

is drawn from a

population having unknown mean μ

1

.



[114, 123.3, 116.7, 129.0, 118, 124.6, 123.1,
117.4, 111, 121.7, 124.5, 130.5]



sample mean = 121.15



sample standard deviation = 5.89



Another SRS of size n

2

is drawn from a

population having unknown mean μ

2

.



[120, 125.5, 126, 125.5, 128.5, 125, 128, 116,
122, 121, 117, 125]



sample mean = 123.29



sample standard deviation = 4.08

Two-sample t
significance test -
example



 



0383

.

1

n

s

s

x

x

t

2

2

2

1

2

1

2

1



















=0 by assumption

p = 0.3104

Conclusion: we cannot reject the hypothesis that
mean values of both populations are the same at
α=0.05

Since n

1

=n

2

=12

Two-sample Confidence
Interval



Suppose that an SRS of size n

1

is drawn

from a normal population with unknown

mean μ

1

and that an independent SRS of

size n

2

is drawn from another normal

population with unknown mean μ

2

. For

large n

1

and n

2

the confidence interval for

μ

1

- μ

2

given by

where t statistics has n

1

+n

2

-2 dfs.









2

2

2

1

2

1

2

1

2

2

2

1

2

1

2

1

*

,

*

n

s

n

s

t

x

x

n

s

n

s

t

x

x













Robustness of the two-
sample procedures



The two-sample t procedures are more

robust than the one-sample t methods.



When the sizes of the two samples are equal

and the distributions of the two populations

being compared have similar shapes, we get

good accuracy.



When the two population distributions have

different shapes, larger samples are needed.



In planning a two-sample study, you should

usually choose equal sample sizes.

Bartlett’s test for
homogeneity of variances



A test of null hypothesis that two
normal populations represented by
two samples have the same
variance.



The alternative hypothesis is that
the two variances are unequal.

Welch’s approximate t-test of
equality of the means of two
samples whose variances are
assumed to be unequal

Welch’s approximate t-test



This test calculates an approximate

t-value, t

’

, for which the critical

value is calculated as a weighted

average of the critical values of t

based on the corresponding degrees

of freedom of the two samples.

Welch’s approximate t-test



The two-sample t

’

statistic is



 



2

2

2

1

2

1

2

1

2

1

n

s

n

s

x

x

't















Welch’s approximate t-test



The critical value of t

α’

for type I

error is computed as:

2

2

2

1

2

1

2

2

2

]

[

1

2

1

]

[

n

s

n

s

n

s

t

n

s

t

't

2

1

















Welch’s approximate t-test



The statistic does NOT have a t
distribution but we can approximate
the distribution of the two-sample t’
statistic by using the t(k)
distribution with an approximation
for the degrees of freedom k.



To calculate p-value k equal to the
smaller of n

1

– 1 and n

2

– 1.

Homework