Lecture 6: Goodness of Fit tests

Probability and

Statistics
Part 2: Statistics

Goodness of Fit tests



These statistical methods test
nominal scale data to determine if
the observed distribution of

counts

(never percentages or ratios) fits a
hypothesized distribution.



The most notable analysis
technique is the chi-square (χ

2

)

test.

Introduction



Assume that a geneticist has
carried out a crossing experiment
between two F

1

hybrids and

obtains an F

2

progeny of 90

offspring, n

1

=80 of which appear

to be wild-type and n

2

=10 of which

are the mutant phenotype.

Introduction



The geneticist assumed dominance
and expected a ratio of 3:1 of the
phenotypes, but the actual ratio is
80/10 = 8:1.



Expected values of p and q are

for wild-type and mutant respectively.

25

.

0

q

ˆ

75

.

0

p

ˆ





and

Introduction



We use ‘hat’ to indicate hypothetical
or expected values of binomial
proportions.



The observed proportions of these
two classes are

respectively.

11

.

0

q

89

.

0

p

90

10

90

80









and

Introduction



Another way of noting the contrast
between observation and
expectation is to state it in counts
(some authors say frequencies).

Introduction



The observed counts are n

1

=80 and

n

2

=10 for the two phenotypes.



The expected counts are

where N refers to the sample size of
offspring from the cross.

5

.

22

N

25

.

0

N

q

ˆ

n

ˆ

5

.

67

90

75

.

0

N

p

ˆ

n

ˆ

2

1





















Introduction



The obvious question is whether
the deviation from 3:1 hypothesis
observed in our sample is of such a
magnitude as to be improbable.



In other words, do the observed
data differ enough from expected
values to cause to reject the null
hypothesis?

Exact distribution
approach



This is a binomial distribution in which p
is the probability of being a wild-type and
q is the probability of being a mutant.



We can work out the probability of
obtaining an outcome of 80 wild-type
phenotypes and 10 mutants, as well as
all ‘worse’ cases in sample of 90 offspring
for

25

.

0

q

ˆ

75

.

0

p

ˆ





and

Exact distribution
approach

which is equal to the cumulative
probability of the tail of the
distribution.





























90

80

k

k

N

k

90

80

k

k

N

k

q

ˆ

p

ˆ

)!

k

N

(

!

k

!

N

q

ˆ

p

ˆ

k

N

Exact distribution
approach



The result is a probability of
0.00084895 for all outcomes as
deviant or more deviant from the
hypothesis.



Note this is a one-tailed test, the
alternative hypothesis being that there
are more wild-type offspring than the
Mendelian hypothesis would postulate.

Exact distribution
approach



The observed sample is a very
unusual outcome, and we conclude
that there is a significant deviation
from expectation.

Confidence limit approach



A less-time consuming approach
based on the same principle is to
look up confidence limits for the
binomial proportions.

Goodness of fit test (GoF
test)



We will develop a third approach to
evaluating the null hypothesis –
done by goodness of fit test.



The table will illustrate how we
might proceed.

Log-likelihood ratio test

Phenoty

pes

Observe

d counts

Observe

d

proportio

ns

Expected

proportio

ns

Expected

counts

Counts

ratio

observed

over

expected

Wild-

type

80

0.89

0.75

67.5

1.185185 13.59192

Mutant

10

0.11

0.25

22.5

0.444444 -8.10930

Sum

90

1.0

1.0

90.0

Ln L =

5.48262













i

i

i

n

ˆ

n

ln

n

Log-likelihood ratio test



The log-likelihood ratio test for
goodness of fit may be developed as
follows:
The probability of observing the
sample result assuming the population
parameter equals to sample proportion
is the same as

1326838

.

0

90

10

90

80

90

80

10

80







































Log-likelihood ratio test

The probability of observing the
sample result assuming proportions
given by Mendelian null hypothesis is
equal to

0005518

.

0

4

1

4

3

90

80

10

80







































Log-likelihood ratio test

If the observed proportions is equal to
proportion postulated under the null
hypothesis, then the two computed
probabilities will be equal, and their ratio, L,
will equal 1.
The greater the difference between
proportions the higher deviation of L from 1.

Log-likelihood ratio test

The ratio of these two probabilities or
likelihoods can be used as a statistics to
measure the degree of agreement
between sampled and expected counts.

A test based on such ratio is called a

likelihood ratio test

.

Log-likelihood ratio test

It has been shown that the distribution
of

G = 2 ln L

can be approximated by the χ

2

distribution with 1 degree of freedom.

Log-likelihood ratio test

In our case

G = 2 ln L = 10.96524

If we compare this observed value with a

χ

2

distribution with 1 degree of

freedom, we find that result is
significant

(p-value = 0.000928 < 0.001)

Chi-squared distribution,
1df

10.96524

Computational formula

2

1

2

1

2

1

n

n

n

n

1

n

n

1

q

ˆ

q

p

ˆ

p

q

ˆ

p

ˆ

n

n

q

p

n

n

L





















































Since

q

ˆ

n

n

ˆ

nq

n

p

ˆ

n

n

ˆ

np

n

2

2

1

1









2

1

n

2

2

n

1

1

n

ˆ

n

n

ˆ

n

L



























then





























2

2

2

1

1

1

n

ˆ

n

ln

n

n

ˆ

n

ln

n

L

ln

and

G test for GoF for more than
two classes



The test for goodness of fit can be applied to
a distribution with more than two classes.



Again we calculate ratios of observed over
expected counts.



The sum yields to ln L, while G = 2 ln L
follows approximately the chi-squared
distribution with a-1 degrees of freedom,
where a stands for the number of classes.

Example 1



Testing the choice of salmon
to select a certain home
stream versus four nearby
streams.

N = 200

fish

Home

stream

Stream

1

Stream

2

Stream

3

Stream

4

Observe

d

counts

135

15

17

10

23

Example 1

Hypotheses:

H

0

: Home stream is chosen 75% of

the time; remaining four streams
25% of the time (6.25% each).

H

a

: not H

o

Example 1



The null hypothesis can be
alternatively stated as
H

0

: the sample came from a

salmon population with a
12:1:1:1:1 ratio of choosing home
and alternate steams.
H

a

: not H

0

.

Example 1

Observed
counts

Expected
counts

Ratio

Home
stream

135

150

0.90

-14.2237

Stream 1

15

12.5

1.20

2.7348

Stream 2

17

12.5

1.36

5.2272

Stream 3

10

12.5

0.80

-2.2314

Stream 4

23

12.5

1.84

14.0246

Sum

200

200

ln L =
5.5315

i

n

ˆ

i

n

i

i

n

ˆ

/

n













i

i

n

ˆ

n

i

ln

n

Example 1

0259

.

0

}

063

.

11

{

P

:

value

p

2

]

4

[









0.05

at

H

reject

we

G

G

since

4877

.

9

063

.

11

L

ln

2

G

0

2

]

4

[

05

.

0

crit

2

]

4

[

05

.

0



















Chi-squared test for GoF



This is an alternative technique,
the traditional approach that is
applied in substantial number of
research publications.



We turn once more to the genetic
cross with 80 wild-type and 10
mutant individuals.

Chi-squared test for GoF



First we calculate the deviations of
observed from expected counts and
square them.



In the next step we make each
deviation relative to the expected
counts.



Finally we find sum of these
quantities.

Chi-squared test for GoF



The obtained statistic is so-called
chi-squared statistic X

2

, but is only

approximately distributed as chi-
square with 1 degree of freedom

.



Some authors have called X

2

the

Pearson statistic.



The chi-squared test is always one-
tailed!!

Chi-squared test for GoF

Phenoty

pes

Observe

d counts

Expected

proportio

ns

Expected

counts

Squared

deviation

s

Relative

squared

deviation

s

Wild-

type

80

0.75

67.5

156.25

2.3148

Mutant

10

0.25

22.5

156.25

6.9444

Sum

90

1.0

90.0

X

2

=

9.2592

Chi-squared test for GoF

0023

.

0

}

2592

.

9

{

P

:

value

p

1

]

1

[









0.05

at

H

reject

we

X

since

8415

.

3

2592

.

9

X

0

2

]

1

[

05

.

0

2

2

]

1

[

05

.

0

2















Chi-squared test for GoF for
more than two classes



The chi-squared test for goodness of fit
can be applied to a distribution with more
than two classes.



Calculate:



X

2

statistic follows approximately the chi-

squared distribution with a-1 degrees of
freedom, where a stands for the number
of classes.











a

i

2

i

i

2

n

ˆ

n

ˆ

n

X

Example 1 – cont.

Observed
counts

Expected
counts

Deviation Relative

deviation
s

Home
stream

135

150

225

1.50

Stream 1

15

12.5

6.25

0.50

Stream 2

17

12.5

20.25

1.62

Stream 3

10

12.5

6.25

0.50

Stream 4

23

12.5

110.25

8.82

Sum

200

200

X

2

=12.94

i

n

ˆ

i

n





2

i

i

n

ˆ

n 

Example 1 – cont.

0116

.

0

}

94

.

12

{

P

:

value

p

2

]

4

[









0.05

at

H

reject

we

X

since

4877

.

9

94

.

12

X

0

2

]

4

[

05

.

0

2

2

]

4

[

05

.

0

2















Subdividing the chi-square
analysis



In our salmon example, the number
of fish returning to Stream 4 seems
to have caused the H

0

rejection.



So we will subdivide it.



Now let’s test H

0

: The sample came

from a salmon population with
12:1:1:1 ratio of choosing home and
alternate streams 1-3.

Example 1 – subdivision

Observed
counts

Expected
counts

Deviation Relative

deviation
s

Home
stream

135

177*12/1

5=141.6

43.56

0.3076

Stream 1

15

177*1/15

=11.8

10.24

0.8678

Stream 2

17

11.8

27.04

2.2915

Stream 3

10

11.8

3.24

0.2746

Sum

177

X

2

=3.741

5

i

n

ˆ

i

n





2

i

i

n

ˆ

n 

Example 1 – subdivision

2908

.

0

}

7415

.

3

{

P

:

value

p

2

]

3

[









0.05

at

H

reject

not

do

we

X

since

8174

.

7

7415

.

3

X

0

2

]

3

[

05

.

0

2

2

]

3

[

05

.

0

2















Continuity corrections



G or X

2

statistic values calculated

based on actual data belong to a
discrete distributions.



However, the theoretical chi-square
distribution is continuous.



With an uncorrected values you
falsely reject H

0

to often (type I error

is higher than intended one).

Continuity corrections



This is a serious problem in case of

2 classes. If N<200 we must apply

the continuity corrections.



G test – Williams’s correction



X

2

test – Yates correction

q

G

G

,

N

2

1

1

q

adj



















2

i

2

i

i

2

adj

n

ˆ

5

.

0

n

ˆ

n

X

Testing again other
distributions



We can apply GoF test to verify the
hypothesis on our data following
other than binomial distributions.



If we estimate the parameters of
the distribution (even the binomial
one) with the use of our data,
we must correctly set the number of
degrees of freedom.

Testing again other
distributions

Distribution

Parameters

estimated

from

sample

Degrees of

freedom

Binomial

p

a-2

Normal

μ,σ

a-3

Poisson

μ

a-2

Kolmogorov-Smirnov test



A nonparametric test that is applicable
to continuous frequency distributions,
where it has greater power than the G-
or chi-square tests for GoF, is the
Kolmogorov-Smirnov (KS) test.



KS test for GoF is especially useful
with small samples, it is inadvisable to
group classes.

Kolmogorov-Smirnov test



This test is based on differences
between two cumulative relative
frequency distributions, between the
observed and expected distribution.



It should be also used for discrete data
in ordered categories.



It is better than standard chi-square test,
because it takes into account the
ordering of the categories.

Kolmogorov-Smirnov test



The test for discrete data looks at the
cumulative observed and expected
counts and takes

the largest

difference

:

This value is compared to a critical
value, where k=number of categories,
N=sample size, α = alpha-level

i

i

i

max

Fˆ

F

max

d

max

d







N

,

k

,

max,

crit

d

d





Example 2 – discrete data



We want to test if insect species
are uniformly distributed along a
light gradient.



Light gradients do not have a
measurable differences, except
that lower values represent darker
light conditions than higher
numbers.

Example 2



H

0

: Insect species uniformly

distributed along a light gradient



H

a

: not H

0



We set α = 0.05

Example 2

N=65

Dark - 1

2

3

4

Light - 5

Observe

d counts

0

7

6

38

14

Expecte

d counts

13

13

13

13

13

Observe

d

cumulati

ve

counts

0

7

13

51

65

Expecte

d

cumulati

ve

counts

13

26

39

52

65

|d

i

|

13

19

26

1

0

Example 2



Test statistic

d

max

= 26



Critical value

d

max,0.05,5,65

= 10 (see

table)



Decision rule

: reject H

0

if d

max

≥10;

otherwise, do not reject.



Since 26>10 (p<0.001), we reject
H

0

.

Example 2



Conclusion:

The observed data do not follow an
uniform distribution across the
ordered light categories (p<0.001).

KS test for GoF – continuous
data



An important fact is that since
expected cumulative frequency
distribution is a continuous function,
the largest difference between
expected and observed is found by
computing differences both before and
after each time observed cumulative
frequency function steps up.