Month: July 2018

Lesson 73 – Learning from “Student”

On the fifteenth day of July 2018, Jenny and Joe discussed the confidence interval for the mean of the population.

The 100(1-\alpha)% confidence interval for the true mean \mu is [\bar{x} - Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{x} + Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}].

\bar{x} is the mean of a random sample of size n. The assumption is that the sample is drawn from a population with a true mean \mu and true standard deviation \sigma.

The end-points [\bar{x} - Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}, \bar{x} + Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}] are called the lower and the upper confidence limits.

While developing the confidence interval of the mean water quality at the beach with a sample of 48, Jenny pointed out that the value for the true standard deviation, \sigma is not known.

Joe collected a much larger sample (all data points available for the Rockaway beach) and computed an estimate for the true standard deviation. Based on the principle of consistency, he suggested that this estimate is close to the truth, so can be used as \sigma.

Jenny rightly pointed out that not always we will have such a large sample. Most often, the data is limited.

Joe’s suggestion was to use sample standard deviation, s, i.e., the estimated standard deviation from the limited sample in place of \sigma. However, Jenny was concerned that this will introduce more error into the estimation of the intervals.

March 1908

This was a concern for W. S. Gosset (aka “Student”) in 1908. He points out that one way of dealing with this difficulty is to run the experiment many times, i.e., collect a large enough sample so that the standard deviation can be computed once for all and used for subsequent similar experiments.

He further points out that there are numerous experiments that cannot easily be repeated often. In the situation where a large sample cannot be obtained, one has to rely on the results from the small sample.

The standard deviation of the population (\sigma) is not known a priori.

The confidence interval Joe and Jenny derived is based on the conjecture that the distribution of the sample mean (\bar{x}) tends to a normal distribution. \bar{x} \sim N(\mu, \frac{\sigma}{\sqrt{n}}), and if the sample size is large enough, it would be reasonable to substitute sample standard deviation s in place of the population standard deviation \sigma. But it is not clear how “large” the sample size should be.

The sample standard deviation can be computed as s=\sqrt{\frac{1}{n-1}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}}. While s is a perfectly good estimate of \sigma, it is not equal to \sigma.

“Student” pointed out that when we substitute s for \sigma, we cannot just assume that \frac{\bar{x}-\mu}{\frac{s}{\sqrt{n}}} will tend to a normal distribution just like Z = \frac{\bar{x}-\mu}{\frac{\sigma}{\sqrt{n}}} \sim N(0,1).

He derived the frequency distribution of \frac{\bar{x}-\mu}{\frac{s}{\sqrt{n}}} in his landmark paper “The Probable Error of a Mean.”

This distribution came to be known as the Student’s t-distribution.

In his paper, he did not use the notation t though. He referred to it as quantity z, obtained by dividing the distance between the mean of a sample and the mean of the population by the standard deviation of the sample.

He derived the distribution of the sample variance and the sample standard deviation, showed that there is no dependence between s and \bar{x} and used this property to derive the joint distribution of \frac{\bar{x}-\mu}{\frac{s}{\sqrt{n}}}.

Today, we will go through the process of deriving the probability distributions for the sample variance s^{2}, sample standard deviation s and the quantity t.

t = \frac{\bar{x}-\mu}{\frac{s}{\sqrt{n}}}

It is important that we know these distributions. They will be a recurring phenomenon from now on and we will be using them in many applications.

I am presenting these derivations using standard techniques. There may be simpler ways to derive them, but I found this step by step thought and derivation process enriching.

During this phase, I will refer back to “Student’s” paper several times. I will also use the explanations given by R.A Fisher in his papers on “Student.”

You can follow along these steps using a pen and paper, or you can just focus on the thought process and skip the derivations, either way, it is fun to learn from “Student.” Trust me.

t = \frac{\bar{x}-\mu}{\frac{s}{\sqrt{n}}}

To know the distribution of t, we should know the distributions of \bar{x} and s.

We already know that \bar{x} tends to a normal distribution; \bar{x} \sim N(\mu, \frac{\sigma}{\sqrt{n}}), but what is the limiting distribution of s, i.e., what is the probability distribution function f(s) of the sample standard deviation?

To know this, we should first know the limiting distribution of the sample variance s^{2}, from which we can derive the distribution of s.

What is the frequency distribution of the sample variance?

Expressing variance as the sum of squares of normal random variables.

Let’s take the equation of the sample variance s^{2} and see if there is a pattern in it.

s^{2} = \frac{1}{n-1} \sum(x_{i}-\bar{x})^{2}

Move the n-1 over to the left-hand side and do some algebra.

(n-1)s^{2} = \sum(x_{i}-\bar{x})^{2}

(n-1)s^{2} = \sum(x_{i} - \mu -\bar{x} + \mu)^{2}

(n-1)s^{2} = \sum((x_{i} - \mu) -(\bar{x} - \mu))^{2}

(n-1)s^{2} = \sum[(x_{i} - \mu)^{2} + (\bar{x} - \mu)^{2} -2(x_{i} - \mu)(\bar{x} - \mu)]

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} + \sum (\bar{x} - \mu)^{2} -2(\bar{x} - \mu)\sum(x_{i} - \mu)

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} + n (\bar{x} - \mu)^{2} -2(\bar{x} - \mu)(\sum x_{i} - \sum \mu)

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} + n (\bar{x} - \mu)^{2} -2(\bar{x} - \mu)(n\bar{x} - n \mu)

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} + n (\bar{x} - \mu)^{2} -2n(\bar{x} - \mu)(\bar{x} - \mu)

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} + n (\bar{x} - \mu)^{2} -2n(\bar{x} - \mu)^{2}

(n-1)s^{2} = \sum(x_{i} - \mu)^{2} - n (\bar{x} - \mu)^{2}

Let’s divide both sides of the equation by \sigma^{2}.

\frac{(n-1)s^{2}}{\sigma^{2}} = \frac{1}{\sigma^{2}}(\sum(x_{i} - \mu)^{2} - n (\bar{x} - \mu)^{2})

\frac{(n-1)s^{2}}{\sigma^{2}} = \sum(\frac{x_{i} - \mu}{\sigma})^{2} - \frac{n}{\sigma^{2}} (\bar{x} - \mu)^{2}

\frac{(n-1)s^{2}}{\sigma^{2}} = \sum(\frac{x_{i} - \mu}{\sigma})^{2} - (\frac{\bar{x} - \mu}{\sigma/\sqrt{n}})^{2}

The right-hand side now looks like the sum of squared standard normal distributions.

\frac{(n-1)s^{2}}{\sigma^{2}} = Z_{1}^{2} + Z_{2}^{2} + Z_{3}^{2} + ... + Z_{n}^{2} - Z^{2}

Sum of squares of (n – 1) standard normal random variables.

Does that ring a bell? Sum of squares is the language of the Chi-square distribution. We learned this in lesson 53.

If there are n standard normal random variables, Z_{1}, Z_{2}, ..., Z_{n}, their sum of squares is a Chi-square distribution with n degrees of freedom. Its probability density function is f(\chi)=\frac{\frac{1}{2}*(\frac{1}{2} \chi)^{\frac{n}{2}-1}*e^{-\frac{1}{2}*\chi}}{(\frac{n}{2}-1)!} for \chi > 0 and 0 otherwise.

Sine we have \frac{(n-1)s^{2}}{\sigma^{2}} = Z_{1}^{2} + Z_{2}^{2} + Z_{3}^{2} + ... + Z_{n}^{2} - Z^{2}

\frac{(n-1)s^{2}}{\sigma^{2}} follows a Chi-square distribution with (n-1) degrees of freedom.

\frac{(n-1)s^{2}}{\sigma^{2}} \sim \chi^{2}_{n-1} with a probability distribution function

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{\frac{1}{2}*(\frac{1}{2} \chi)^{\frac{n-1}{2}-1}*e^{-\frac{1}{2}*\chi}}{(\frac{n-1}{2}-1)!}

⇔

The frequency distribution of the sample variance.

It turns out that, with some modification, this equation is the frequency distribution f(s^{2}) of the sample variance.

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{\frac{1}{2}*(\frac{1}{2} \chi)^{\frac{n-1}{2}-1}*e^{-\frac{1}{2}*\chi}}{(\frac{n-1}{2}-1)!}

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{\frac{1}{2}(\frac{(n-1)s^{2}}{2\sigma^{2}})^{\frac{n-3}{2}}}{(\frac{n-3}{2})!} e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(\frac{n-1}{\sigma^{2}})^{\frac{n-1}{2}}\frac{1}{\frac{n-1}{\sigma^{2}}}\frac{1}{2^{\frac{n-3}{2}}}(s^{2})^{\frac{n-3}{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{1}{\frac{n-1}{\sigma^{2}}}\frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-3}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

f(\frac{(n-1)s^{2}}{\sigma^{2}}) = \frac{1}{\frac{n-1}{\sigma^{2}}}[\frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-3}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}]

The above equation can be viewed as f(aX) = \frac{1}{a}f(X) where X = s^{2} and a = \frac{n-1}{\sigma^{2}}.

These few steps will clarify this further.

Let Y = aX

P(Y \le y) = P(aX \le y)

P(Y \le y) = P(X \le \frac{1}{a}y)

F_{Y}(y) = F_{X}(\frac{1}{a}y)

\frac{d}{dy}(F(y)) = \frac{d}{dy}F_{X}(\frac{1}{a}y)

Applying the fundamental theorem of calculus and chain rule together, we get,

f(y) = \frac{1}{a}f(\frac{1}{a}y)

f(aX) = \frac{1}{a}f(X)

Hence, the frequency distribution of s^{2} is

f(s^{2}) = \frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-3}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

From f(s^{2}), we can derive the probability distribution of s, i.e., f(s).

WHAT IS THE FREQUENCY DISTRIBUTION OF THE SAMPLE Standard Deviation?

Here, I refer you to this elegant approach by “Student.”

“The distribution of s may be found from this (s^{2}), since the frequency of s is equal to that of s^{2} and all that we must do is to compress the base line suitably.”

Let f_{1} = f(s^{2}) be the distribution of s^{2}.

Let f_{2} = f(s) be the distribution of s.

Since the frequency of s^{2} is equal to that of s, we can assume,

f_{1}ds^{2} = f_{2}ds

In other words, the probability of finding s^{2} in an infinitesimal interval ds^{2} is equal to the probability of finding s in an infinitesimal interval ds.

We can reduce this using substitution as

2sf_{1}ds = f_{2}ds

or

f_{2} = 2sf_{1}

The frequency distribution of s is equal to 2s multiplied by the frequency distribution of s^{2}.

Hence, the frequency distribution of s is

f(s) = 2s\frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-3}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

f(s) = \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-2}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

WHAT IS THE FREQUENCY DISTRIBUTION OF t?

We are now ready to derive the frequency distribution of t.

Some of the following explanation can be found in R. A. Fisher’s 1939 paper. I broke it down step by step for our classroom.

t = \frac{\bar{x}-\mu}{s/\sqrt{n}}

\bar{x} - \mu = \frac{st}{\sqrt{n}}

For a given value of s, d\bar{x} = \frac{s}{\sqrt{n}}dt

The frequency distribution of \bar{x} is

\frac{\sqrt{n}}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{n}{2\sigma^{2}}(\bar{x}-\mu)^{2}}

Remember, \bar{x} \sim N(\mu, \frac{\sigma}{\sqrt{n}}).

Substituting \bar{x} - \mu = \frac{st}{\sqrt{n}}

The distribution becomes

\frac{\sqrt{n}}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{n}{2\sigma^{2}}(st/\sqrt{n})^{2}}

or

\frac{\sqrt{n}}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}

For a given value of s, the probability that \bar{x} will be in d\bar{x} is

df = \frac{\sqrt{n}}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}d\bar{x}

Substituting d\bar{x} = \frac{s}{\sqrt{n}}dt, we can get

df = \frac{\sqrt{n}}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}} \frac{s}{\sqrt{n}}dt

df = \frac{1}{\sqrt{2\pi}}\frac{1}{\sigma}e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}} s dt

Fisher points out that for all values of s, we can substitute the frequency distribution of s and integrate it over the interval 0 and \infty. This can be done because \bar{x} and s are independent.

So the joint distribution becomes

df = \frac{1}{\sqrt{2\pi}}\frac{1}{\sigma} s dt e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}} \int_{0}^{\infty}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-2}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

In the next few steps, I will rearrange some terms to convert the integral into a recognizable form.

df = \frac{1}{\sqrt{2\pi}}\frac{1}{\sigma} dt \int_{0}^{\infty}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}s \frac{s^{n-2}}{\sigma^{n-1}}e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

df = \frac{1}{\sqrt{2\pi}}\frac{1}{\sigma} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}} \frac{1}{\sigma^{n-1}}dt \int_{0}^{\infty}s^{n-1} e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

df = \frac{1}{\sqrt{2\pi}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}\frac{(n-1)^{\frac{n}{2}}}{(n-1)^{1/2}}\frac{1}{2^{\frac{n-3}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

df = \frac{1}{\sqrt{2\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{1}{2^{\frac{n-3}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{1}{2^{\frac{n-2}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-\frac{t^{2}}{2\sigma^{2}}s^{2}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}ds

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-\frac{n -1 + t^{2}}{2\sigma^{2}}s^{2}}ds

Hang in there. We will need some more concentration!

Let k = \frac{n - 1 + t^{2}}{2\sigma^{2}}

The equation becomes

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-ks^{2}}ds

Some more substitutions.

Let p = ks^{2}

Then dp = 2ksds

The equation can be reduced as

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \int_{0}^{\infty}s^{n-1} e^{-p}\frac{1}{2ks}dp

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \int_{0}^{\infty}\frac{1}{2k}s^{n-2} e^{-p}dp

Since s = \frac{p^{1/2}}{k^{1/2}}

The equation becomes,

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \frac{1}{2k^{n/2}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

Replacing k = \frac{n - 1 + t^{2}}{2\sigma^{2}}

The equation becomes

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \frac{1}{2(\frac{n - 1 + t^{2}}{2\sigma^{2}})^{n/2}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

Some more reduction …

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \frac{(2\sigma^{2})^{n/2}}{2(n - 1 + t^{2})^{n/2}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

df = \frac{1}{\sqrt{\pi(n-1)}}\frac{1}{\sigma^{n}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}}\frac{2}{2^{\frac{n}{2}}} dt \frac{(2^{n/2}\sigma^{n})}{2(n - 1 + t^{2})^{n/2}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

df = \frac{1}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n}{2}} dt \frac{1}{(n - 1 + t^{2})^{n/2}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

df = \frac{1}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!} dt \frac{1}{(\frac{n-1+t^{2}}{n-1})^{\frac{n}{2}}}\int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

df = \frac{1}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!} dt \frac{1}{(1+\frac{t^{2}}{n-1})^{\frac{n}{2}}} \int_{0}^{\infty}p^{\frac{n-2}{2}} e^{-p}dp

The integral you see on the right is a Gamma function that is equal to (\frac{n-2}{2})! for positive integers.

There we go, the t-distribution emerges

df = \frac{(\frac{n-2}{2})!}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!} \frac{1}{(1+\frac{t^{2}}{n-1})^{\frac{n}{2}}} dt

The probability distribution of t is

f(t) = \frac{(\frac{n-2}{2})!}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!} \frac{1}{(1+\frac{t^{2}}{n-1})^{\frac{n}{2}}}

It is defined as t-distribution with (n-1) degrees of freedom. As you can see, the function only contains n as a parameter.

The probability of t within any limits is fully known if we know n, the sample size of the experiment.

“Student” also derived the moments of this new distribution as

E[T] = 0

V[T] = \frac{n-1}{n-3}

The function is symmetric and resembles the standard normal distribution Z.

The t-distribution has heavier tails, i.e. it has more probability in the tails than the normal distribution. As the sample size increases (i.e., as n \to \infty) the t-distribution approaches Z.

You can look at the equation and check out how it converges to Z in the limit when n \to \infty.

These points are illustrated in this animation.

I am showing the standard Z in red color. The grey color functions are the t-distributions for different values of n. For small sample sizes, the t-distribution has fatter tails — as the sample size increases, there is little difference between T and Z.

😫😫😫

I am pretty sure you do not have the energy to go any further for this week. I don’t too.

So here is where we stand.

f(s^{2}) = \frac{1}{2}\frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-3}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

f(s) = \frac{1}{(\frac{n-3}{2})!}(n-1)^{\frac{n-1}{2}}\frac{1}{2^{\frac{n-3}{2}}}\frac{s^{n-2}}{\sigma^{n-1}}e^{-\frac{1}{2}*\frac{(n-1)s^{2}}{\sigma^{2}}}

and finally

The probability distribution of t is

f(t) = \frac{(\frac{n-2}{2})!}{\sqrt{\pi(n-1)}} \frac{1}{(\frac{n-3}{2})!} \frac{1}{(1+\frac{t^{2}}{n-1})^{\frac{n}{2}}}

See you next week.

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

Lesson 72 – Jenny’s confidence, on the average

Meet Jenny. Jenny is bright and intelligent and is known as “the problem solver” among her friends. She usually goes unnoticed in the crowd due to her calm and composed nature, but by god, she is assertive when it is most needed. Her analytical neurons are razor sharp and you cannot mumbo-jumbo with her. She loves science fiction, history, and occasional Woody Allen. Oh, and she likes swimming, surfing, summer, and beaches.

Her second summer Rockaway beach trip is coming up and she is excited. With all the surfing kit packed, she pulled up the NYC Beach Water Quality website to monitor the status. “The Enterococci Bacteria Count is within limit!” she thought.

She got busy with other things, but the thought of beach samples bothered her. “Where do they take these samples from? They show Rockaway beach, but does the sample represent the whole beach? How many samples do they take? If they use one sample, how do we know what the truth is? I have been to this place several times, I wonder what the true water quality of the beach is?” These questions kept her awake. I told you she is a problem solver.

The next morning she met Joe to discuss the problem. Friends of this classroom know who Joe is. He is now the resident expert on statistical topics.

So, you want to know the true water quality from the sample data. Why don’t you develop the confidence interval of the mean water quality? This will at least give you a range, an interval where the true water quality will be.

 

How can we do that using the sample? Or, maybe I should ask, can you explain what is an interval and how to derive it.

 

Okay. Let’s get some simple/sample data first. NYC Open Data should definitely have data on beaches. Here, they have a link for DOHMH Beach Water Quality Data. The description says that this is the data of water quality sample results collected by the Department of Health and Mental Hygiene from all New York City Beaches. Amazing! Which Rockaway beach are you going to?

 

I usually go to the one on the 95th Street and stay north.

 

We can take the data from ROCKAWAY BEACH 95TH – 116TH then. Let me show you how to write a code in R to extract this subset of the data.

 

 

Hey, I can do that with ease.

 

I forgot to tell you that Jenny is a member of the local girls who code club. She pulls up her Macbook and types a few lines. Meanwhile, Joe does some coding too on the same data.

There are samples that had a result below the detection limit. If I remove those from the ROCKAWAY BEACH 95TH – 116TH data, we are left with 48 data points collected at various times from 2005 to 2018. Look.

 

Wonderful. I am guessing that the blue color triangle in the histogram is the sample mean (\bar{x}).

 

Yes. This is an estimate (good guess) for the average Enterococci Bacteria count for this part of the beach based on 48 samples over several years.

We can describe the uncertainty in this estimate using the confidence intervals. The key link is to realize that this estimate, i.e., the sample mean (\bar{x}) is a random variable. For instance, if we were to take another data sample from different times of the year or over different parts of the beach, we would get a slightly different sample mean. The value of the estimate changes with the change of sample, so we can think of estimate as a range of values or a probability distribution.

 

Yes, that makes perfect sense, but what probability distribution does it follow?

 

The sample mean is given by the equation \bar{x} = \frac{1}{n}{\displaystyle \sum_{i=1}^{n}x_{i}}.
x_{i}s are independent and identically distributed random samples. Look at the equation carefully, it is the summation of random variables. Convolution. We learned in Lesson 48 that for a large enough sample size, this summation will converge to the normal distribution — Central Limit Theorem. As the samples grow (n becomes large), convolution or function multiplications yield a smooth center heavy and thin-tailed bell function — the normal density.

 

Ah, I see. So it is reasonable to assume a normal distribution for the sample mean \bar{x}.

 

 

Yes, \bar{x} follows a normal distribution with an expected value E[\bar{x}] and variance V[\bar{x}].

 

E[\bar{x}]=\mu. The sample mean is an unbiased estimate of the true mean, so the expected value of the sample mean is equal to the truth. This is in Lesson 67.

The variance of the sample mean V[\bar{x}] =\frac{\sigma^{2}}{n}. We derived this in Lesson 68. Variance tells us how widely the estimate is distributed around the center of the distribution.

The standard deviation of the sample mean, or the standard error of the estimate is \frac{\sigma}{\sqrt{n}}.

 

So, \bar{x} \sim N(\mu, \frac{\sigma}{\sqrt{n}})

 

Jenny looks at the equation for a bit. She takes a piece of paper, draws on it and instantly types a few lines of code.

 

Here, I am showing this visually.

 

\bar{x} is a normal distribution. It is centered on \mu with a standard deviation of \frac{\sigma}{\sqrt{n}}. One standard deviation range is \mu \pm \frac{\sigma}{\sqrt{n}} , two standard deviations range is \mu \pm 2\frac{\sigma}{\sqrt{n}} and three standard deviations range is \mu \pm 3\frac{\sigma}{\sqrt{n}} .

 

Yup. The standard normal way of saying this is Z = \frac{\bar{x}-\mu}{\frac{\sigma}{\sqrt{n}}} \sim N(0,1).

 

 

The Z-score!

 

 

Yes, let me ask you something. What is the area under the normal density curve between -1.96 and 1.96?

Looking up from the standard normal tables, we get P(Z \le -1.96) = 0.025, which means the area on the right side of the tail P(Z \ge 1.96) = 0.025. The area between -1.96 and 1.96 is 0.95. 95%.

 

There is a 95% probability that the standard normal variable \frac{\bar{x}-\mu}{\frac{\sigma}{\sqrt{n}}} is between -1.96 and 1.96.

P(-1.96 \le \frac{\bar{x}-\mu}{\frac{\sigma}{\sqrt{n}}} \le 1.96) = 0.95

I will modify the inequality in this equation.

Multiplying throughout by \frac{\sigma}{\sqrt{n}}, we get P(-1.96 \frac{\sigma}{\sqrt{n}} \le \bar{x}-\mu \le 1.96 \frac{\sigma}{\sqrt{n}}) = 0.95

Subtracting \bar{x} and multiplying by -1 throughout, we get

P(\bar{x} - 1.96\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{x} + 1.96 \frac{\sigma}{\sqrt{n}}) = 0.95

We derived that the probability of the true population mean \mu lying between two end-points \bar{x} - 1.96\frac{\sigma}{\sqrt{n}} and \bar{x} + 1.96 \frac{\sigma}{\sqrt{n}} is 0.95.

This interval [\bar{x} - 1.96\frac{\sigma}{\sqrt{n}}, \bar{x} + 1.96 \frac{\sigma}{\sqrt{n}}] is called the 95% confidence interval of the population mean. The interval itself is random since it is derived from \bar{x}. As we dicsused before, a different sample will have a different \bar{x} and hence a different interval or range.

There is a 95% probability that this random interval [\bar{x} - 1.96\frac{\sigma}{\sqrt{n}}, \bar{x} + 1.96 \frac{\sigma}{\sqrt{n}}] contains the true value of \mu.

 

Neat. So to generalize this to any confidence interval, we can replace 1.96 with a Z-critical value.

 

Yes. For instance, if we want a 99% confidence interval, we should find the Z-critical value that gives 99% area under the normal density curve.

 

 

That would be 2.58. So the 99% confidence interval for true mean \mu is \bar{x} - 2.58\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{x} + 2.58 \frac{\sigma}{\sqrt{n}}.

Usually, this % confidence interval is described using a confidence coefficient 1-\alpha where \alpha is between 0 and 1. For a 95% confidence interval, \alpha = 0.05, and for a 99% confidence interval, \alpha = 0.01. The Z-critical value is denoted using Z_{\frac{\alpha}{2}}.

 

Yes yes. For 95% confidence interval, Z_{\frac{\alpha}{2}} = Z_{0.025} = 1.96.

In summary, we can define a 100(1-\alpha)% confidence interval for the true mean \mu as

[\bar{x} - Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{x} + Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}]

 

Correct. \bar{x} is the mean of a random sample of size n. The assumption is that the sample is drawn from a population with a true mean \mu and true standard deviation \sigma. The end-points [\bar{x} - Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}, \bar{x} + Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}] are called the lower and upper confidence limits.

 

Let us develop the confidence intervals of the mean water quality. The sample I have has n = 48 data points. The sample mean is \bar{x}=7.9125 counts per 100 ml.

.
.
.
Wait, we don’t know the value of \sigma, the true standard deviation. There is still an unknown in the equation.

 

Assume it is 9.95 counts per 100 ml.

 

 

 Where did that come from.

 

 

Take my word for now and get the confidence interval.

 

Jenny reluctantly does some calculations on a piece of paper.

The 95% confidence interval for the mean water quality is

\bar{x} - 1.96\frac{\sigma}{\sqrt{n}} \le \mu \le \bar{x} + 1.96\frac{\sigma}{\sqrt{n}}.

7.9125 - 1.96\frac{9.95}{\sqrt{48}} \le \mu \le 7.9125 + 1.96\frac{9.95}{\sqrt{48}}.

7.064 \le \mu \le 8.761

 

There is a 95% probability that the true mean water quality will be between 7.064 and 8.761. Now tell me where the 9.95 came from.

 

Well, strictly speaking, your statement should have been, “there is a 95% probability of selecting a sample for which the confidence interval will contain the true value of \mu.”

 

 

 

Let me explain. 7.064 \le \mu \le 8.761 can be true or can be false depending on the sample we obtained. \bar{x} is a random variable, a probability distribution whose mean is \mu. Our sample mean represents a random draw from this distribution. So, if we got a sample whose mean is close to the truth, the interval [\bar{x} - Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}, \bar{x} + Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}] will contain the truth. If we got a sample who mean is somewhat far away from the truth, its confidence interval may not contain the truth. This graphic should make it more clear. The green interval does not contain the true value. The purple interval does. Both these sample means (\bar{x}) are equally likely; it depends on what sample data we get.

Hmm. So, we should think of this phenomenon as a long-run relative frequency outcome. If we have a lot of different samples, and compute the 95% confidence intervals for these samples, in the long-run, 95% of these intervals will contain the true value of \mu.

 

Exactly. The 95% confidence level is what would happen if a large number of random intervals were constructed; not for any particular interval.

So while you were coding to get the subset of data from the ROCKAWAY BEACH 95TH – 116TH, I downloaded all the data that had ROCKAWAY BEACH in the file. There are eight locations along the beach where these samples are taken. Assuming that these are eight different samples, I developed the 95% confidence intervals for each of these samples. Here is how they look relative to the truth. One of them does not contain the true \mu. Like this, in the long-run, 5% of the samples will not contain the true mean for 95% confidence intervals.

 

How did you get the true value for \mu?

 

 

Based on the principle of consistency.

\displaystyle{\lim_{n\to\infty} P(|T_{n}(\theta)-\theta|>\epsilon)} \to 0.

As n approaches infinity, the sample estimate approaches the true parameter. I took data from the eight beach locations and computed the overall mean and overall standard deviation. While they are not exactly the true values, based on the idea of consistency, we can assume that they are.

\mu = 8.97 and \sigma = 9.95 counts per 100 ml.

 

There you go. I see why you insisted on using \sigma = 9.95.

 

 

You are welcome!

 

 

But not always we will have such a large sample. Most often, the data is limited. How can we know what the true standard deviation is?

 

In that case, you can use the sample standard deviation. In your sample data case it was 6.96 counts per 100 ml, I think.

 

 

Hmm, but that is very different from the true value. Aren’t we inducing more error into the estimation of the intervals?

 

 

Perhaps. Hey, do you want to take a break? Can I buy you a drink? Is Guinness okay?

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

Lesson 71 – How confident are you?

Switch on the television or open your favorite news website; every few weeks you will find people’s opinion about something. Here’s one that came out a few days back. Our best days are ahead – so they say.

If you are a regular reader here, you would have already asked about the survey method and sample size.

Results for this Gallup poll are based on telephone interviews conducted June 18-24, 2018, with a random sample of 1,505 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia. For results based on the total sample of national adults, the margin of sampling error is ±3 percentage points at the 95% confidence level.

The margin of sampling error is ±3% points at the 95% confidence level.

“Margin of sampling error,” “95% confidence level” – Have you ever asked yourself what these are?

In another news, the state of NJ passed a new “multi-millionaires tax,” Airbnb tax and Uber/Lyft tax among others. I hope the residents of NJ were asked about some of these increases, and this is what they wanted.

If they were to be asked, how do we know how many people to survey?

If a subset of people were to represent the population of NJ, how do we know that the resulting opinion is what the entire population actually wants?

Jill is having fun this summer. He is one of the volunteers at the BIG CITY FISHING SUNDAYS on Pier 84 . He is busy collecting water samples from the Hudson River. This week he computed the mean dissolved oxygen level as 4.9 mg/L. If the acceptable water quality level is a mean dissolved oxygen of 5 mg/L, how reliable is the estimate that Jill got? Can it be deemed acceptable?

In another news, Wilfred Quality Jr. is still thinking about his grandfather’s question: “how confident are you that the true value lies within two standard deviations?”

Jill’s boss is a variance control freak. Can we give him an upper bound on how much the dissolved oxygen in the Hudson River can vary? He usually likes to be 99% confident.

In another news, St Patrick’s Day should remind you of all those pints of Guinness, and “Student.”  No, I am not under the influence.

All these questions can be answered with a little knowledge about how to describe uncertainty in estimates.

To take you back to Lesson 62, the objective of inference is to use a ‘sample’ to compute a number which represents a good guess for the true value of the parameter. The true population parameter is unknown, and what we estimate from the sample will enable us to obtain the closest answer in some sense.

The cliched way of saying this is that the mean (\bar{x}) and variance (s^{2}) of the sample data are good guesses (estimates or estimators) of the mean (\mu) and variance (\sigma^{2}) of the population.

A different sample will yield a different estimate for the parameter. So we have to think of the estimate as a range of values or a probability distribution instead of a single value.

For any estimator \hat{\theta}, we can compute the expected value E[\hat{\theta}] as a measure of the central tendency, and V[\hat{\theta}] as a measure of the spread of the distribution giving us a range of values. Like this.

This range is called an interval. Naturally, the interval will be wider for data that has more variability. We can be confident that the truth may be in this interval if we have good representative samples.

How confident we are is a probability statement. For example, if there is a 95% probability that the interval contains the true value, it is equivalent to saying that this interval is the 95% confidence interval.

The interval will have a stated probability of containing the truth, the degree of plausibility specified by the confidence level → 95% interval or 99% interval.

Over the next several lessons, we will dive into the core concepts of confidence intervals, how to construct them for different population parameters and use them for designing experiments.

Meanwhile, do you know about Guinness and Student? Guinness, I am sure, but are you confident that you know Student?

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

Lesson 70 – The quest for truth: Learning estimators in R, part II

T-2 days

Wilfred Jr. was typing away on his Macintosh. Keneth is writing something in his notebook.

“Do you have an estimate yet?” asked Wilfred. “In a minute.”

Wilfred Jr. and Keneth were busy preparing for the meeting in a couple of days. They are systematically analyzing the data they collected over the past few weeks. Their workplace was well lit with a strong beam of sunlight through the window. They both looked out, it was deceivingly pleasant. It is one of those newsworthy summer days. The local weatherman has already given it a rank 2 in his “new normals” show.

The wall behind the big screen computer is a full corkboard that has sticky notes and pins.

“We have an answer for the first two questions,” said Ken. “Last week, we checked this and found that the true proportion of dummies is somewhere around 2% – 0.02024.”

“There you go. Look at this.” Wilfred hit Command + Enter on his keyboard. “I like this version better,” he said.

Keneth looked at the code and wondered if Willy would explain to him what was going on.

“So, we are showing how the estimate for the proportion of dummies \hat{p}=\frac{r}{n} changes as we add more data from the stores. The more the data (i.e., the stores we visit), the closer the estimate goes to the truth. After a while, the change in the value becomes small and the estimate stabilizes and converges to the truth. It is the consistency criterion of the estimates,” said Wilfred as he points out to the blue line on the graph.

“What are the red dots and the boxplot that shows up at the end?” asked Keneth.

“Good questions. Each red dots that appears vertically at the end is the estimated proportion for a given store. So, if a store has 2% dummies, the estimate for that store is 0.02, and a red dot appears at 0.02. Like this, we have 1000 different sample estimates for the proportion of dummies. The boxplot at the end is showing this distribution. The distribution of the sample proportions. Did you notice anything?” asked Wilfred.

Ken looked at the plot again, this time with more attention.

“The blue line meets the big red circle. I can see that the big red circle is the mean of the red dots. E[\hat{p}]. The boxplot also looks asymmetric.” The expected value of the sample proportions and the long-run estimate are the same.”

E[\hat{p}]=p=0.02024

Wilfred nods in acceptance. “We discussed this last week. Since the estimate \frac{r}{n} is an unbiased estimate of the true proportion, the mean of the sample distribution will be the true proportion. The long-run converged estimate is also the same value.”

“Can you share the code with me? I want to take some time and learn how to make these visuals.” Keneth is imagining how cool he would look in front of his friends when he presents his summer project.

“Sure. Here it is. I wrote comments for each line. It should be self-explanatory.”

# Graphic analysis for proportion #

# For each store, compute the sample proportion #
# The expected value of these estimates is the true parameter # 
# With each store visited, we can recompute the sample proportion #
# This will converge and stabilize at the true parameter value #

# Reading the candy data file #
candy_data = read.table("candy_samples.txt",header=F)

nstores = ncol(candy_data) # number of stores 
ncandies = nrow(candy_data) # number of candies in a pack

candies = NULL # an empty vector that will be filled with actual cadies data from each store
dummies = NULL # an empty vector that will be filled with dummy candies data from each store

sample_proportion = NULL # as the sample increases, we re-compute the sample proportion of dummy candies

sample_p = matrix(NA,nrow=nstores,ncol=1) # saving the estimate from each store. The expected value of this vector = truth

plot(0,0,col="white",xlim=c(0,1300),ylim=c(0,0.08),xlab="Stores",ylab="Proportion of Dummies",font=2,font.lab=2)

for (i in 1:nstores)
{
# sample pack of candies and the index of dummies in the pack
candy_pack = candy_data[,i]
dummy_index = which(candy_pack < 1.5)

# separating candies from the dummies in each pack 
sample_dummies = candy_pack[dummy_index]

if(sum(dummy_index)>0){sample_candies = candy_pack[-dummy_index]}else{sample_candies = candy_pack}

# recording the weights on candies and dummies -- this vector will grow as we visit more stores, i.e., collect more samples 
dummies = c(dummies,sample_dummies)
candies = c(candies,sample_candies)

# computing the mean weight of the candies -- this vector will grow .. each incremental value is a better estimate with greater sample size. 
p = length(dummies)/(ncandies*i)
sample_proportion = c(sample_proportion,p)

# storing the sample weight for the store in a matrix 
sample_p[i,1] = length(dummy_index)/ncandies

# plot showing the trace of the sample proportion -- converges to truth
points(i,sample_proportion[i],cex=0.075,col="blue")
points(1000,sample_p[i],cex=0.1,col="red")
Sys.sleep(0.05)
#print(i)
}

Sys.sleep(1)
points(1000,mean(sample_p),pch=19,col="red",cex=2)
Sys.sleep(1)
boxplot(sample_p,add=T,at=1050,boxwex=25,range=0)
Sys.sleep(1)
text(1200,mean(sample_p),expression(E(hat(p))==0.02),col="red",font=2)
What are the population characteristics?

Their attention now shifted to the final question. When the day started, they discussed some ideas and decided they will present the analysis that investigated for the true mean and the true variance of the population.

If the weight of all the Quality Candies is the population, then, it can be represented using the mean (\mu) and the variance (\sigma^{2}) of the population.

Willy and Ken’s task is to find these values. Estimate the parameters.

Here is what they did.

They visited the first shop and found that the estimated proportion of dummy candies in 0.02. As Ken separated the candies from the dummies, he also put them on a weighing scale and measured the weight of each candy.

The average weight of the 98 candies from the first store is 3.00748 grams.

Ken estimated this using the formula \hat{\mu} = \frac{1}{n}{\displaystyle \sum_{i=1}^{n}x_{i}}. This is the maximum likelihood estimator for the true mean.

They record this as the estimated sample mean for the first store.

They went to the second store. Here they did not find any dummies. The estimated sample mean weight of the 100 candies from this store is 3.0073 grams. This is another estimate of the true mean.

Overall, they now have 198 candies (a bigger sample). Ken also computed the sample mean for these 198 candies. This comes out as 3.007389 grams.

Like this, as they visited more and more stores, they re-computed the average weight of the candies using a larger sample. They also computed the individual estimate of the sample mean for that store.

Like in the case of the proportion of dummies, they want to check the convergence of the estimated mean.

“I think you can use the same code as before, but change the lines where you computed the proportions to computing the sample means,” said Ken as he is seen knee-deep into the lines of the code.

“You are correct! All I did was that and here we go. I changed the color of the converging line.”

“I will interpret this now,” said Ken. “The sample mean \frac{1}{n}{\displaystyle \sum_{i=1}^{n}x_{i}} is an unbiased estimate of the population mean \mu . I know it from Lesson 67. The big red circle is the expected value of the sample means, the center of the distribution. So this is E[\frac{1}{n}{\displaystyle \sum_{i=1}^{n}x_{i}}]. This is equal to the true mean \mu. The orange line, the estimate with increasing sample size also converges to this number.”

E[\frac{1}{n}{\displaystyle \sum_{i=1}^{n}x_{i}}] = \mu = 3

“Excellent. What else do you observe? Anything from the boxplot?”

“Hmm, it does look symmetric. Is the distribution of the sample means normally distributed?” asked Ken.

“I will let you mull over that. You do know about the Central Limit Theorem, right?” Wilfred finished writing some comments in his code.

# Graphic analysis for mean #

# For each store, compute the sample mean #
# The expected value of these estimates is the true parameter # 
# With each store visited, we can recompute the sample mean #
# This will converge and stabilize at the true parameter value #

nstores = ncol(candy_data) # number of stores 
ncandies = nrow(candy_data) # number of candies in a pack

candies = NULL # an empty vector that will be filled with actual cadies data from each store
dummies = NULL # an empty vector that will be filled with dummy candies data from each store

sample_meanweight = NULL # as the sample increases, we re-compute the sample mean of the candies

sample_mean = matrix(NA,nrow=nstores,ncol=1) # save the value of sample mean for each store

plot(0,0,xlim=c(0,1200),ylim=c(2.945,3.055),xlab="Stores",ylab="Average Candy Weight",font=2,font.lab=2)

for (i in 1:nstores)
{
# sample pack of candies and the index of dummies in the pack
candy_pack = candy_data[,i]
dummy_index = which(candy_pack < 1.5)

# separating candies from the dummies in each pack 
sample_dummies = candy_pack[dummy_index]

if(sum(dummy_index)>0){sample_candies = candy_pack[-dummy_index]}else{sample_candies = candy_pack}

# recording the weights on candies and dummies -- this vector will grow as we visit more stores, i.e., collect more samples 
dummies = c(dummies,sample_dummies)
candies = c(candies,sample_candies)

# computing the mean weight of the candies -- this vector will grow .. each incremental value is a better estimate with greater sample size. 
xbar = mean(candies)
sample_meanweight = c(sample_meanweight,xbar)

# storing the sample weight for the store in a matrix 
sample_mean[i,1] = mean(sample_candies)

# plot showing the trace of the sample mean -- converges to truth
points(i,sample_meanweight[i],cex=0.15,col="orange")
points(1000,sample_mean[i],cex=0.15,col="red")
Sys.sleep(0.05)
print(i)
}

Sys.sleep(1)
points(1000,mean(sample_mean),pch=19,col="red",cex=2)
Sys.sleep(1)
boxplot(sample_mean,add=T,range=0,boxwex=25,at=1050)
Sys.sleep(1)
text(1150,mean(sample_mean),expression(E(hat(mu))==3),col="red",font=2)
“What about Variance?”

Asked Wilfred.

“Same procedure. In fact, when I computed the sample mean for each store and the overall incremental sample, I also computed the sample variances using its maximum likelihood estimator \hat{\sigma^{2}}=\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\mu)^{2}}. I used this formula for each store and the overall incremental sample.”

“Give me a few minutes, I will code this up and show.” Ken looked excited. He finally got to write his code and look at the outputs. He typed up a few lines, mostly editing the previous versions.

Here is what they found.

Ken looked puzzled. He expected to see the same pattern, however, much to his surprise, he found that the expected value of the sample variance E[\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\mu)^{2}}] and the long-run converged estimate for the variance are not the same.

E[\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\mu)^{2}}] \neq \sigma^{2}

“Is there a bias?” he asked.

Wilfred was expecting this. He had done his homework from Lesson 67!

“Yes, there is.”

“The maximum likelihood estimator \hat{\sigma^{2}}=\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\mu)^{2}} is a biased estimator.”

“Let me show you why.”

“I will rewrite the estimator as \hat{\sigma^{2}}=\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}} by using \bar{x} in place of \mu.”

\hat{\sigma^{2}}=\frac{1}{n}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}

=\frac{1}{n}\sum(x_{i}^{2} + \bar{x}^{2} -2\bar{x}x_{i})

=\frac{1}{n}\sum x_{i}^{2} + \frac{1}{n} \sum \bar{x}^{2} - \frac{2}{n}\sum x_{i}\bar{x}

=\frac{1}{n}\sum x_{i}^{2} + \frac{1}{n} n \bar{x}^{2} - 2 \bar{x} \bar{x}

=\frac{1}{n}\sum x_{i}^{2} + \bar{x}^{2} - 2 \bar{x}^{2}

=\frac{1}{n}\sum x_{i}^{2} - \bar{x}^{2}

\hat{\sigma^{2}}=\frac{1}{n}(\sum x_{i}^{2} - n\bar{x}^{2})

“Now, let’s apply the expectation operator on this formula.”

E[\hat{\sigma^{2}}]=E[\frac{1}{n}(\sum x_{i}^{2} - n\bar{x}^{2})]

=\frac{1}{n}(\sum E[x_{i}^{2}] - n E[\bar{x}^{2}])

=\frac{1}{n}(\sum (V[x_{i}]+(E[x_{i}])^{2}) - n (V[\bar{x}]+(E[\bar{x}])^{2}))

“I am sure you know why I wrote that step. It is from the property of variance operator.”

“Now, since x_{1}, x_{2}, ..., x_{n} are independent and identically distributed random samples from the population with true parameters \mu and \sigma^{2}, they will have the same expected value and variance as the population. E[x_{i}]=\mu and V[x_{i}]=\sigma^{2}. I will use this simple idea to reduce our equation.”

E[\hat{\sigma^{2}}] = \frac{1}{n}(\sum (\sigma^{2}+\mu^{2}) - n (V[\bar{x}]+(E[\bar{x}])^{2}))

“If you go back and read Lesson 68 (Bias joins Variance), you will see that V[\bar{x}]=\frac{\sigma^{2}}{n}. You already know that E[\bar{x}]=\mu. Let’s substitute those here.”

E[\hat{\sigma^{2}}] = \frac{1}{n}(\sum (\sigma^{2}+\mu^{2}) - n (\frac{\sigma^{2}}{n}+\mu^{2}))

E[\hat{\sigma^{2}}] = \frac{1}{n}(n\sigma^{2} + n\mu^{2} - \sigma^{2} -n\mu^{2})

E[\hat{\sigma^{2}}] = \frac{n-1}{n}\sigma^{2}

“So, \hat{\sigma^{2}} is biased by -\frac{\sigma^{2}}{n}.”

“That is why you are seeing that the expected value of the sample variances is not the same as where the overall incremental estimate (population variance) converges. The estimator with n in the denominator underestimates the true variance \sigma^{2} because of the negative bias.”

If you want an unbiased estimator, you should use \frac{1}{n-1} in the denominator instead of n.

The unbiased estimator for variance is \hat{\sigma^{2}}=\frac{1}{n-1}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}

Wilfred added a few lines in the code and showed this plot to Ken.

“Look at this. I have added another boxplot to your figure. This boxplot is the distribution of the bias-adjusted (unbiased) variance estimates. The green dot is now the expected value of this distribution, which matches with the true variance.”

Ken looks at it with a happy face, but something seems to bother him.”

“What is it?” asked Jr.

“Is the distribution of variance asymmetric?” he asked.

“Yes. Is that all?”

“Well, if the variance is bias-adjusted to match the truth, does it also mean the standard deviation, which is \sqrt{\hat{\sigma^{2}}}, will get adjusted?”

“That is a good question.”

No, although \hat{\sigma^{2}}=\frac{1}{n-1}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}} is an unbiased estimator for \sigma^{2}, \hat{\sigma}=\sqrt{\frac{1}{n-1}{\displaystyle \sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}} is not an unbiased estimator for \sigma.

“Here, I re-ran the code with some changes to show this. The adjusted estimate still has some bias.”

“Very interesting. I guess we are ready for the meeting. The true mean weight of the candies is 3 grams and the true standard deviation of the weight of the candies is 0.2 grams. Can we have the code in the archives.” Ken starts packing his bag as he says this. He looks exhausted too.

“Here is the full code.”

T-0 days

Mr. Alfred Quality and Wilfred Quality Sr. were disappointed to know that there is a systematic packing error, although it is only 2%. They deployed their engineering team to fix this immediately.

They were nevertheless very impressed with the quality work of Wilfred Quality Jr. He owned the problem and came up with the answers. All along, he had collected a large sample that can be used for future tests, trained an enthusiastic high-school senior and showed that he is ready to be the next Mr. Quality for Quality Candies.

As he was leaving the boardroom, Mr. Alfred Quality asked him:
“Junior, great work, but how confident are you that the true value lies within two standard deviations?

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

error

Enjoy this blog? Please spread the word :)