Student's T-distribution – dataanalysisclassroom

Lesson 95 – The Two-Sample Hypothesis Test – Part IV

On the Difference in Means

using Welch’s t-Test

$H_{0}: \mu_{1} - \mu_{2} = 0$

$H_{A}: \mu_{1} - \mu_{2} > 0$

$H_{A}: \mu_{1} - \mu_{2} < 0$

$H_{A}: \mu_{1} - \mu_{2} \neq 0$

On the 24th day of January 2021, we examined Tom’s hypothesis on the Mohawk Rivers’ arsenic levels.

After a lengthy expose on the fundamentals behind hypothesis testing on the difference in means using a two-sample t-Test, we concluded that Tom could not reject the null hypothesis $H_{0}: \mu_{1}-\mu_{2}=0$ .

He cannot count on the theory that the factory is illegally dumping their untreated waste into the west branch Mohawk River until he finds more evidence.

However, Tom now has a new theory that the factory is illegally dumping their untreated waste into both the west and the east branches of the Mohawk River. So, he took Ron with him to collect data from Utica River, a tributary of Mohawk that branches off right before the factory.

If his new theory is correct, he should find that the mean arsenic concentration in either west or east branches should be significantly greater than the mean arsenic concentration in the Utica River.

There are now three samples whose concentration in parts per billion are:

West Branch: 3, 7, 25, 10, 15, 6, 12, 25, 15, 7
$n_{1}=10$ | $\bar{x_{1}} = 12.5$ | $s_{1}^{2} = 58.28$

East Branch: 4, 6, 24, 11, 14, 7, 11, 25, 13, 5
$n_{2}=10$ | $\bar{x_{2}} = 12$ | $s_{2}^{2} = 54.89$

Utica River: 4, 4, 6, 4, 5, 7, 8
$n_{3}=7$ | $\bar{x_{3}} = 5.43$ | $s_{3}^{2} = 2.62$

Were you able to help Tom with his new hypothesis?

In his first hypothesis test, since the sample variances were close to each other (58.28 and 54.89), we assumed that the population variances are equal and proceeded with a t-Test.

Under the proposition that the population variance of two random variables $X_{1}$ and $X_{2}$ are equal, i.e., $\sigma_{1}^{2}=\sigma_{2}^{2}=\sigma^{2}$ , the test-statistic is $t_{0}=\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{s^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$ , where $s^{2}=(\frac{n_{1}-1}{n_{1}+n_{2}-2})s_{1}^{2}+(\frac{n_{2}-1}{n_{1}+n_{2}-2})s_{2}^{2}$ is the pooled variance. $t_{0}$ follows a T-distribution with $n_{1}+n_{2}-2$ degrees of freedom.

But, can we make the same assumption in the new case? The Utica River sample has a sample variance $s_{3}^{2}$ of 2.62. It will not be reasonable to assume that the population variances are equal.

How do we proceed when $\sigma_{1}^{2} \neq \sigma_{2}^{2}$ ?

Let’s go back a few steps and outline how we arrived at the test-statistic.

The hypothesis test is on the difference in means: $\mu_{1} - \mu_{2}$

A good estimator of the difference in population means is the difference in sample means: $y = \bar{x_{1}} - \bar{x_{2}}$

The expected value of y, $E[y]$ , is $\mu_{1}-\mu_{2}$ , and the variance of y, $V[y]$ , is $\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}}$ .

Since $y \sim N(\mu_{1}-\mu_{2},\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}})$ , its standardized version $z = \frac{y-(\mu_{1}-\mu_{2})}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}}}}$ is the starting point to deduce the test-statistic.

This statistic reduces to $z = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}}}}$ under the null hypothesis that $\mu_{1}-\mu_{2}=0$ .

Last week, we entertained the idea that we are comparing the difference in the means of two populations whose variance is equal and reasoned that the test-statistic follows a T-distribution with $(n_{1}+n_{2}-2)$ degrees of freedom.

This is because the pooled population variance $\sigma^{2}$ can be replaced by its unbiased estimator $s^{2}$ , which, in turn is related to a Chi-squared distribution with $(n_{1}+n_{2}-2)$ degrees of freedom.

When the population variance is not equal, i.e., when $\sigma_{1}^{2} \neq \sigma_{2}^{2}$ , there is no pooled variance which can be related to the Chi-square distribution.

The best estimate of $V[y]$ is obtained by replacing the individual population variances ( $\sigma_{1}^{2}, \sigma_{2}^{2}$ ) by the sample variances ( $s_{1}^{2}, s_{2}^{2}$ ).

Hence,

$z = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}}$

We should now identify the limiting distribution of $V[y]$

Bernard Lewis Welch, a British statistician, in his works in 1936, 1938, and 1947 explained that, with some adjustments, $V[y]$ can be approximated to a Chi-square distribution, and hence the test-statistic $\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}}$ can be appriximated to a T-distribution.

Let’s digest the salient points of his work

Assume $\lambda_{1} = \frac{1}{n_{1}}$ and $\lambda_{2} = \frac{1}{n_{2}}$

Assume $f_{1} = (n_{1}-1)$ and $f_{2} = (n_{2}-1)$

An estimate for the variance of y is $\lambda_{1}s_{1}^{2}+\lambda_{2}s_{2}^{2}$

Now, in Lesson 73, we learned that $\frac{(n-1)s^{2}}{\sigma^{2}}$ is a Chi-square distribution. Based on this logic, we can write

$s^{2} = \frac{1}{n-1}\sigma^{2}*\chi^{2}$

So, $\lambda_{1}s_{1}^{2}+\lambda_{2}s_{2}^{2}$ is of the form,

$\lambda_{1}\frac{1}{n_{1}-1}\sigma_{1}^{2}\chi_{1}^{2}+\lambda_{2}\frac{1}{n_{2}-1}\sigma_{2}^{2}\chi_{2}^{2}$

or,

$\lambda_{1}s_{1}^{2}+\lambda_{2}s_{2}^{2} = a\chi_{1}^{2}+b\chi_{2}^{2}$

$a = \frac{\lambda_{1}\sigma_{1}^{2}}{f_{1}}$ | $b = \frac{\lambda_{2}\sigma_{2}^{2}}{f_{2}}$

Welch showed that if $z = a\chi_{1}^{2}+b\chi_{2}^{2}$ , then, the distribution of z can be approximated using a Chi-square distribution with a random variable $\chi=\frac{z}{g}$ and f degrees of freedom.

He found the constants f and g by equating the moments of z with the moments of this Chi-square distribution, i.e., the Chi-square distribution where the random variable $\chi$ is $\frac{z}{g}$ .

This is how he finds f and g

The first and the second moments of a general Chi-square distribution are the degrees of freedom (f) and two times the degrees of freedom (2f).

The random variable we considered is $\chi=\frac{z}{g}$ .

So,

$E[\frac{z}{g}] = f$ | $V[\frac{z}{g}] = 2f$

Since g is a constant that needs to be estimated, we can reduce these equations as

$\frac{1}{g}E[z] = f$ | $\frac{1}{g^{2}}V[z] = 2f$

Hence,

$E[z] = gf$ | $V[z] = 2g^{2}f$

Now, let’s take the equation $z = a\chi_{1}^{2}+b\chi_{2}^{2}$ and find the expected value and the variance of z.

$E[z] = aE[\chi_{1}^{2}]+bE[\chi_{2}^{2}]$

$E[z] = af_{1}+bf_{2}$

$V[z] = a^{2}V[\chi_{1}^{2}]+b^{2}V[\chi_{2}^{2}]$

$V[z] = a^{2}2f_{1}+b^{2}2f_{2}=2a^{2}f_{1}+2b^{2}f_{2}$

Now, he equates the moments derived using the equation for z to the moments derived from the Chi-square distribution of $\frac{z}{g}$

Equating the first moments: $gf=af_{1}+bf_{2}$

Equating the second moments: $2g^{2}f = 2a^{2}f_{1}+2b^{2}f_{2}$

The above equation can be written as

$g*(gf) = a^{2}f_{1}+b^{2}f_{2}$

or,

$g*(af_{1}+bf_{2}) = a^{2}f_{1}+b^{2}f_{2}$

From here,

$g = \frac{a^{2}f_{1}+b^{2}f_{2}}{af_{1}+bf_{2}}$

Using this with $gf=af_{1}+bf_{2}$ , we can obtain f.

$f = \frac{(af_{1}+bf_{2})^{2}}{a^{2}f_{1}+b^{2}f_{2}}$

Since we know the terms for a and b, we can say,

$f = \frac{(\lambda_{1}\sigma_{1}^{2}+\lambda_{2}\sigma_{2}^{2})^{2}}{\frac{\lambda_{1}^{2}\sigma_{1}^{4}}{f_{1}}+\frac{\lambda_{2}^{2}\sigma_{2}^{4}}{f_{2}}}$

Since the variance of y follows an approximate Chi-square distribution with f degrees of freedom, we can assume that the test-statistic $\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}}$ follows an approximate T-distribution with f degrees of freedom.

Welch also showed that an unbiased estimate for f is

$f = \frac{(\lambda_{1}s_{1}^{2}+\lambda_{2}s_{2}^{2})^{2}}{\frac{\lambda_{1}^{2}s_{1}^{4}}{f_{1}+2}+\frac{\lambda_{2}^{2}s_{2}^{4}}{f_{2}+2}} - 2$

Essentially, he replaces $s_{1}^{2}$ for $\sigma_{1}^{2}$ , and $s_{2}^{2}$ for $\sigma_{2}^{2}$ , and to correct for the bias, he adds 2 to the degrees of freedom in the denominator, and substracts an overall 2 from this fraction. He argues that this correction produces the best unbiased estimate for f.

Later authors like Franklin E. Satterthwaite showed that the bias correction might not be necessary, and it would suffice to use $s_{1}^{2}$ for $\sigma_{1}^{2}$ , and $s_{2}^{2}$ for $\sigma_{2}^{2}$ in the original equation, as in,

$f = \frac{(\lambda_{1}s_{1}^{2}+\lambda_{2}s_{2}^{2})^{2}}{\frac{\lambda_{1}^{2}s_{1}^{4}}{f_{1}}+\frac{\lambda_{2}^{2}s_{2}^{4}}{f_{2}}}$

Since we know that $\lambda_{1} = \frac{1}{n_{1}}$ , $\lambda_{2} = \frac{1}{n_{2}}$ , $f_{1} = (n_{1}-1)$ , and $f_{2} = (n_{2}-1)$ , we can finally say

When the population variances are not equal, i.e., when $\sigma_{1}^{2} \neq \sigma_{2}^{2}$ , the test-statistic is $t_{0}^{*}=\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}}$ , and it follows an approximate T-distribution with f degrees of freedom.

The degrees of freedom can be estimated as

$f = \frac{(\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}})^{2}}{\frac{(s_{1}^{2}/n_{1})^{2}}{(n_{1} - 1) + 2}+\frac{(s_{2}^{2}/n_{2})^{2}}{(n_{2}-1)+2}} - 2$

or,

$f = \frac{(\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}})^{2}}{\frac{(s_{1}^{2}/n_{1})^{2}}{(n_{1} - 1)}+\frac{(s_{2}^{2}/n_{2})^{2}}{(n_{2}-1)}}$

This is now popularly known as Welch's t-Test.

Let’s now go back to Tom and help him with his new theory.

We will compare the west branch Mohawk River with the Utica River.

West Branch: 3, 7, 25, 10, 15, 6, 12, 25, 15, 7
$n_{1}=10$ | $\bar{x_{1}} = 12.5$ | $s_{1}^{2} = 58.28$

Utica River: 4, 4, 6, 4, 5, 7, 8
$n_{3}=7$ | $\bar{x_{3}} = 5.43$ | $s_{3}^{2} = 2.62$

Since we cannot assume that the population variances are equal, we will use Welch’s t-Test.

We compute the test-statistic and check how likely it is to see such a value in a T-distribution (approximate null distribution) with so many degrees of freedom.

$t_{0}^{*}=\frac{\bar{x_{1}}-\bar{x_{3}}}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{3}^{2}}{n_{3}}}}$

$t_{0}^{*}=\frac{12.5-5.43}{\sqrt{\frac{58.28}{10}+\frac{2.62}{7}}}$

$t_{0}^{*}=2.84$

Let’s compute the bias-corrected degrees of freedom suggested by Welch.

$f = \frac{(\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}})^{2}}{\frac{(s_{1}^{2}/n_{1})^{2}}{(n_{1} - 1) + 2}+\frac{(s_{2}^{2}/n_{2})^{2}}{(n_{2}-1)+2}} - 2$

$f = \frac{(\frac{58.28}{10}+\frac{2.62}{7})^{2}}{\frac{(58.28/10)^{2}}{(10 - 1) + 2}+\frac{(2.62/7)^{2}}{(7-1)+2}} - 2$

$f = 10.38$

We can round it down to 10 degrees of freedom.

The test-statistic is 2.84. Since the alternate hypothesis is that the difference is greater than zero, Tom has to verify how likely it is to see a value greater than 2.84 in the approximate null distribution. Tom has to reject the null hypothesis if this probability (p-value) is smaller than the selected rate of rejection.

Look at this visual.

The distribution is an approximate T-distribution with 10 degrees of freedom. Since he opted for a rejection level of 10%, there is a cutoff on the distribution at 1.37.

1.37 is the quantile on the right tail corresponding to a 10% probability (rate of rejection) for a T-distribution with ten degrees of freedom.

If the test statistic ( $t_{0}^{*}$ ) is greater than $t_{critical}$ , which is 1.37, he will reject the null hypothesis. At that point (i.e., at values greater than 1.37), there would be sufficient confidence to say that the difference is significantly greater than zero.

It is equivalent to rejecting the null hypothesis if $P(T > t_{0}^{*})$ (the p-value) is less than $\alpha$

We can read $t_{critical}$ off the standard T-table, or we can compute $P(T > t_{0}^{*})$ from the distribution.

At ten degrees of freedom (df=10), and $\alpha=0.1$ , $t_{critical}=1.372$ and $P(T>t_{0^{*}})=0.009$ .

Since the test-statistic $t_{0}^{*}$ is in the rejection region, or since $p-value < \alpha$ , Tom can reject the null hypothesis $H_{0}$ that $\mu_{1}-\mu_{2}=0$ .

He now has evidence beyond statistical doubt to claim that the factory is illegally dumping their untreated waste into the west branch Mohawk River.

Is it time for a lawsuit?

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

Lesson 94 – The Two-Sample Hypothesis Test – Part III

On the Difference in Means

using the t-Test

$H_{0}: \mu_{1} - \mu_{2} = 0$

$H_{A}: \mu_{1} - \mu_{2} > 0$

$H_{A}: \mu_{1} - \mu_{2} < 0$

$H_{A}: \mu_{1} - \mu_{2} \neq 0$

Tom grew up in the City of Mohawk, the land of natural springs, known for its pristine water. Tom’s house is alongside the west branch of Mohawk, one such pristine river. Every once in a while, Tom and his family go to the nature park on the banks of Mohawk. It is customary for Tom and his children to take a swim.

Lately, he has been reading in the local newspapers that the rivers’ arsenic levels have increased. Tom starts associating the cause of the alleged arsenic increases to this new factory in his neighborhood just upstream of Mohawk. They could be illegally dumping their untreated waste into the west branch.

He decided to test the waters of the west branch and the east branch of the Mohawk River.

His buddy Ron, an environmental engineer, would help him with the laboratory testing and the likes.

Over the next ten days, Tom and Ron collected water samples from the west and east branches, and Ron got them tested in his lab for arsenic concentration.

In parts per billion, the sample data looked like this.

West Branch: 3, 7, 25, 10, 15, 6, 12, 25, 15, 7

East Branch: 4, 6, 24, 11, 14, 7, 11, 25, 13, 5

If Tom’s theory is correct, they should find the average arsenic concentration in the west branch to be significantly greater than the average arsenic concentration in the east branch.

How can Tom test his theory?

You are right!

He can use the hypothesis testing framework and verify if there is evidence beyond a statistical doubt.

Tom establishes the null and alternate hypotheses. He assumes that the factory does not illegally dump their untreated waste into the west branch, so the average arsenic concentration in the west branch should be equal to the average arsenic concentration in the east branch Mohawk River. Or, the difference in their means is zero.

$H_{0}: \mu_{1} - \mu_{2} = 0$

Against this null hypothesis, he pits his theory that they are indeed illegally dumping their untreated waste. So, the average arsenic concentration in the west branch should be greater than the average arsenic concentration in the east branch Mohawk River — the difference in their means is greater than zero.

$H_{A}: \mu_{1} - \mu_{2} > 0$

The alternate hypothesis is one-sided. A significant positive difference needs to be seen to reject the null hypothesis.

Tom is taking a 10% risk of rejecting the null hypothesis; $\alpha = 0.1$ . His Type I error is 10%.

Suppose the factory does not affect the water quality, but, the ten samples he collected showed a difference in the sample means much greater than zero, he should reject the null hypothesis. So he is committing an error (Type I error) in his decision making.

You must be knowing that there is a certain level of subjectivity in the choice of $\alpha$ .

Tom may want to prove that this factory is the leading cause for the increased arsenic levels in the west branch. So he could have chosen to commit a greater error of rejecting the null hypothesis, i.e., he must be inclined to selecting a larger value of $\alpha$ .

Someone who represents the factory management would be inclined to selecting a smaller value for $\alpha$ as that means less likely to reject the null hypothesis.

So, assuming that the null hypothesis is true, the decision to reject or not to reject is based on the value one chooses for $\alpha$ .

Anyhow, now that the basic testing framework is set up, let’s look at what Tom needs.

He needs a test-statistic to represent the difference in the means of two samples.

He needs the null distribution that this test-statistic converges to. In other words, he needs a probability distribution of the test-statistic to verify his null hypothesis -- how likely it is to see a value as large (or greater) as the test-statistic in the null distribution.

Let’s take Tom on a mathematical excursion

There are two samples represented by random variables $X_{1}$ and $X_{2}$ .

The mean and variance of $X_{1}$ are $\mu_{1}$ and $\sigma_{1}^{2}$ . We have one sample of size $n_{1}$ from this population. Suppose the sample mean and the sample variance are $\bar{x_{1}}$ and $s_{1}^{2}$ .

The mean and variance of $X_{2}$ are $\mu_{2}$ and $\sigma_{2}^{2}$ . We have one sample of size $n_{2}$ from this population. Suppose the sample mean and the sample variance are $\bar{x_{2}}$ and $s_{2}^{2}$ .

The hypothesis test is on the difference in means; $\mu_{1} - \mu_{2}$

Naturally, a good estimator of the difference in population means ( $\mu_{1}-\mu_{2}$ ) is the difference in sample means ( $\bar{x_{1}}-\bar{x_{2}}$ ).

$y = \bar{x_{1}}-\bar{x_{2}}$

If we know the probability distributions of $\bar{x_{1}}$ and $\bar{x_{2}}$ , we could perhaps infer the probability distribution of $y$ .

The sample mean is an unbiased estimate of the true mean, so the expected value of the sample mean is equal to the truth. $E[\bar{x}] = \mu$ . We learned this in Lesson 67.

The variance of the sample mean ( $V[\bar{x}]$ ) is $\frac{\sigma^{2}}{n}$ . It indicates the spread around the center of the distribution. We learned this is Lesson 68.

Putting these two together, and with the central limit theorem, we can say $\bar{x} \sim N(\mu,\frac{\sigma^{2}}{n})$

So, for the two samples,

$\bar{x_{1}} \sim N(\mu_{1},\frac{\sigma_{1}^{2}}{n_{1}})$ | $\bar{x_{2}} \sim N(\mu_{2},\frac{\sigma_{2}^{2}}{n_{2}})$

If $\bar{x_{1}}$ and $\bar{x_{2}}$ are normal distributions, it is reasonable to assume that $y=\bar{x_{1}}-\bar{x_{2}}$ will be a normal distribution.

$y \sim N(E[y], V[y])$

We should see what $E[y]$ and $V[y]$ are.

Expected Value of y

$y = \bar{x_{1}} - \bar{x_{2}}$

$E[y] = E[\bar{x_{1}} - \bar{x_{2}}]$

$E[y] = E[\bar{x_{1}}] - E[\bar{x_{2}}]$

Since the expected value of the sample mean is the true population mean,

$E[y] = \mu_{1} - \mu_{2}$

Variance of y

$y = \bar{x_{1}} - \bar{x_{2}}$

$V[y] = V[\bar{x_{1}} - \bar{x_{2}}]$

$V[y] = V[\bar{x_{1}}] + V[\bar{x_{2}}]$ _{(can you tell why?)}

Since, the variance of the sample mean ( $V[\bar{x}]$ ) is $\frac{\sigma^{2}}{n}$ ,

$V[y] = \frac{\sigma_{1}^{2}}{n_{1}} + \frac{\sigma_{2}^{2}}{n_{2}}$

Using the expected value and the variance of y, we can now say that

$y \sim N((\mu_{1} - \mu_{2}), (\frac{\sigma_{1}^{2}}{n_{1}} + \frac{\sigma_{2}^{2}}{n_{2}}))$

Or, if we standardize it,

$z = \frac{y-(\mu_{1} - \mu_{2})}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}} + \frac{\sigma_{2}^{2}}{n_{2}}}} \sim N(0, 1)$

At this point, it might be tempting to say that z is the test-statistic and the null distribution is the standard normal distribution.

Hold on!

We can certainly say that, under the null hypothesis, $\mu_{1} = \mu_{2}$ . So, z further reduces to

$z = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}} + \frac{\sigma_{2}^{2}}{n_{2}}}} \sim N(0, 1)$

However, there are two unknowns here. The population variance $\sigma_{1}^{2}$ and $\sigma_{2}^{2}$ .

Think about making some assumptions about these unknowns

What could be a reasonable estimate for these unknowns?

You got it. The sample variance $s_{1}^{2}$ and $s_{2}^{2}$ .

Now, let’s take the samples that Tom and Ron collected and compute the sample mean and sample variance for each. The equations for these are anyhow at your fingertips!

West Branch: 3, 7, 25, 10, 15, 6, 12, 25, 15, 7
$\bar{x_{1}}=12.5$ | $s_{1}^{2}=58.28$

East Branch: 4, 6, 24, 11, 14, 7, 11, 25, 13, 5
$\bar{x_{2}}=12$ | $s_{2}^{2}=54.89$

Look at the value of the sample variance. 58.28 and 54.89. They seem close enough. Maybe, just maybe, could the variance of the samples be equal?

I am asking you to entertain the proposition that the population variance of the two random variables $X_{1}$ and $X_{2}$ are equal and that we are comparing the difference in the means of two populations whose variance is equal.

Say, $\sigma_{1}^{2}=\sigma_{2}^{2}=\sigma^{2}$

Our test-statistic will now reduce to

$\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\frac{\sigma^{2}}{n_{1}} + \frac{\sigma^{2}}{n_{2}}}}$

$\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\sigma^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$

There is a common variance $\sigma^{2}$ , and it should suffice to come up with a reasonable estimate for this combined variance.

Let’s say that $s^{2}$ is a good estimate for $\sigma^{2}$ .

We call this the estimate for the pooled variance

If we have a formula for $s^{2}$ and if it is an unbiased estimate for $\sigma^{2}$ , then, we can replace $s^{2}$ for $\sigma^{2}$ .

What is the right equation for $s^{2}$ ?

Since $s^{2}$ is the estimate for the pooled variance, it will be reasonable to assume that it is some weighted average of the individual sample variances.

$s^{2} = w_{1}s_{1}^{2} + w_{2}s_{2}^{2}$

Let’s compute the expected value of $s^{2}$ .

$E[s^{2}] = E[w_{1}s_{1}^{2} + w_{2}s_{2}^{2}]$

$E[s^{2}] = w_{1}E[s_{1}^{2}] + w_{2}E[s_{2}^{2}]$

In Lesson 70, we learned that sample variance $s^{2}=\frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}$ is an unbiased estimate for population variance $\sigma^{2}$ . The factor $(n-1)$ in the denominator is to correct for a bias of $-\frac{\sigma^{2}}{n}$ .

An unbiased estimate means $E[s^{2}]=\sigma^{2}$ . If we apply this understanding to the pooled variance equation, we can see that

$E[s^{2}] = w_{1}\sigma_{1}^{2} + w_{2}\sigma_{2}^{2}$

Since $\sigma_{1}^{2}=\sigma_{2}^{2} = \sigma^{2}$ , we get

$E[s^{2}] = w_{1}\sigma^{2} + w_{2}\sigma^{2}$

$E[s^{2}] = (w_{1} + w_{2})\sigma^{2}$

To get an unbiased estimate for $\sigma^{2}$ , we need the weights to add up to 1. What could those weights be? Could it relate to the sample sizes?

With that idea in mind, let’s take a little detour.

Let me ask you a question.

What is the probability distribution that relates to sample variance?

You might have to go down memory lane to Lesson 73.

$\frac{(n-1)s^{2}}{\sigma^{2}}$ follows a Chi-square distribution with $(n-1)$ degrees of freedom. We learned that the term $\frac{(n-1)s^{2}}{\sigma^{2}}$ is a sum of $(n-1)$ squared standard normal distributions.

So,

$\frac{(n_{1}-1)s_{1}^{2}}{\sigma_{1}^{2}} \sim \chi^{2}_{n_{1}-1}$ | $\frac{(n_{2}-1)s_{2}^{2}}{\sigma_{2}^{2}} \sim \chi^{2}_{n_{2}-1}$

Since the two samples are independent, the sum of these two terms, $\frac{(n_{1}-1)s_{1}^{2}}{\sigma_{1}^{2}}$ and $\frac{(n_{2}-1)s_{2}^{2}}{\sigma_{2}^{2}}$ will follow a Chi-square distribution with $(n_{1}+n_{2}-2)$ degrees of freedom.

Add them and see. $\frac{(n_{1}-1)s_{1}^{2}}{\sigma_{1}^{2}}$ is a sum of $(n_{1}-1)$ squared standard normal distributions, and $\frac{(n_{2}-1)s_{2}^{2}}{\sigma_{2}^{2}}$ is a sum of $(n_{2}-1)$ squared standard normal distributions. Together, they are a sum of $(n_{1}+n_{2}-2)$ squared standard normal distributions.

Since $\frac{(n_{1}-1)s_{1}^{2}}{\sigma_{1}^{2}} + \frac{(n_{2}-1)s_{2}^{2}}{\sigma_{2}^{2}} \sim \chi^{2}_{n{1}+n_{2}-2}$

we can say,

$\frac{(n_{1}+n_{2}-2)s^{2}}{\sigma^{2}} \sim \chi^{2}_{n{1}+n_{2}-2}$

So we can think of developing weights in terms of the degrees of freedom of the Chi-square distribution. The first sample contributes $n_{1}-1$ degrees of freedom, and the second sample contributes $n_{2}-1$ degrees of freedom towards a total of $n_{1}+n_{2}-2$ .

So the weight of the first sample can be $w_{1}=\frac{n_{1}-1}{n_{1}+n_{2}-2}$ and the weight of the second sample can be $w_{2}=\frac{n_{2}-1}{n_{1}+n_{2}-2}$ , and they add up to 1.

This then means that the equation for the estimate of the pooled variance is

$s^{2}=(\frac{n_{1}-1}{n_{1}+n_{2}-2})s_{1}^{2}+(\frac{n_{2}-1}{n_{1}+n_{2}-2})s_{2}^{2}$

Since it is an unbiased estimate of $\sigma^{2}$ , we can use $s^{2}$ in place of $\sigma^{2}$ in our test-statistic, which then looks like this.

$t_{0} = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{s^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$

where, $s^{2}=(\frac{n_{1}-1}{n_{1}+n_{2}-2})s_{1}^{2}+(\frac{n_{2}-1}{n_{1}+n_{2}-2})s_{2}^{2}$ is the estimate for the pooled variance.

Did you notice that I use $t_{0}$ to represent the test-statistic?

Yes, I am getting to $T$ with it.

It is a logical extension of the idea that, for one sample, $t_{0}=\frac{\bar{x}-\mu}{\sqrt{\frac{s^{2}}{n}}}$ follows a T-distribution with $(n-1)$ degrees of freedom.

There, the idea was derived from the fact that when you replace the population variance with sample variance, and the sample variance is related to a Chi-square distribution with $(n-1)$ degrees of freedom, the test-statistic $t_{0}=\frac{\bar{x}-\mu}{\sqrt{\frac{s^{2}}{n}}}$ follows a T-distribution with $(n-1)$ degrees of freedom. Check out Lesson 73 (Learning from “Student”) to refresh your memory.

Here, in the case of the difference in means between two samples ( $\frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{\sigma^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$ ), the pooled population variance $\sigma^{2}$ is replaced by its unbiased estimator $(\frac{n_{1}-1}{n_{1}+n_{2}-2})s_{1}^{2}+(\frac{n_{2}-1}{n_{1}+n_{2}-2})s_{2}^{2}$ , which, in turn, is related to a Chi-square distribution with $(n_{1}+n_{2}-2)$ degrees of freedom.

Hence, under the proposition that the population variance of the two random variables $X_{1}$ and $X_{2}$ are equal, the test-statistic is $t_{0} = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{s^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$ , and it follows a T-distribution with $(n_{1}+n_{2}-2)$ degrees of freedom.

Now let’s evaluate the hypothesis that Tom set up — Finally!!

We can compute the test-statistic and check how likely it is to see such a value in a T-distribution (null distribution) with so many degrees of freedom.

$t_{0} = \frac{\bar{x_{1}}-\bar{x_{2}}}{\sqrt{s^{2}(\frac{1}{n_{1}}+\frac{1}{n_{2}})}}$

where $s^{2}=(\frac{n_{1}-1}{n_{1}+n_{2}-2})s_{1}^{2}+(\frac{n_{2}-1}{n_{1}+n_{2}-2})s_{2}^{2}$

In Tom’s case, both $n_{1}=n_{2}=10$ . So the weights will be equal to 0.5.

$s^{2}=0.5s_{1}^{2}+0.5s_{2}^{2}$

$s^{2}=0.5*58.28+0.5*54.89$

$s^{2}=56.58$

$t_{0} = \frac{12.5-12}{\sqrt{56.58(\frac{1}{10}+\frac{1}{10})}}$

$t_{0} = \frac{0.5}{\sqrt{11.316}}$

$t_{0} = 0.1486$

The test-statistic is 0.1486. Since the alternate hypothesis is that the difference is greater than zeros ( $H_{A}:\mu_{1}-\mu_{2}>0$ ), Tom has to verify how likely it is to see a value greater than 0.1486 in the null distribution. Tom has to reject the null hypothesis if this probability (p-value) is smaller than the selected rate of rejection. A smaller than $\alpha$ probability indicates that the difference is sufficiently large enough that, in a T-distribution with so many degrees of freedom, the likelihood of seeing a value greater than the test-statistic is small. In other words, the difference in the mean is already sufficiently greater than zero and in the region of rejection.

Look at this visual.

The distribution is a T-distribution with 18 degrees of freedom (10 + 10 – 2). Tom had collected ten samples each for this test. Since he opted for a rejection level of 10%, there is a cutoff on the distribution at 1.33.

1.33 is the quantile on the right tail corresponding to a 10% probability (rate of rejection) for a T-distribution with eighteen degrees of freedom.

If the test statistic ( $t_{o}$ ) is greater than $t_{critical}$ , which is 1.33, he will reject the null hypothesis. At that point (i.e., at values greater than 1.33), there would be sufficient confidence to say that the difference is significantly greater than zero.

This decision is equivalent to rejecting the null hypothesis if $P(T>t_{0})$ (the p-value) is less than $\alpha$ .

We can read $t_{critical}$ off the standard T-table, or $P(T>t_{0})$ can be computed from the distribution.

At df = 18, and $\alpha=0.1$ , $t_{critical}=1.33$ and $P(T>t_{0})=0.44$ .

Since the test-statistic $t_{0}$ is not in the rejection region, or since $p-value > \alpha$ , Tom cannot reject the null hypothesis $H_{0}$ that $\mu_{1} - \mu_{2}=0$

He cannot count on the theory that the factory is illegally dumping their untreated waste into the west branch Mohawk River until he finds more evidence.

Tom is not convinced. The endless newspaper stories on arsenic levels are bothering him. He begins to think if the factory is illegally dumping their untreated waste into both the west and east branches of the Mohawk River. That is one reason why he might have seen no significant difference in the concentrations.

Over the next week, he takes Ron with him to Utica River, a tributary of Mohawk that branches off right before Mohawk meets the factory. If his new theory is correct, he should find that the mean arsenic concentration in either the west or east branches should be significantly greater than the mean arsenic concentration in the Utica River.

Ron again helped him with the laboratory testing. In parts per billion, the third sample data looks like this.

Utica River: 4, 4, 6, 4, 5, 7, 8

There are 7 data points. $n_{3}=7$

The sample mean $\bar{x_{3}}=5.43$

The sample variance $s_{3}^{2} = 2.62$

Can you help Tom with his new hypothesis tests? Does the proposition still hold?

To be continued…

If you find this useful, please like, share and subscribe.
You can also follow me on Twitter @realDevineni for updates on new lessons.

Tag: Student's T-distribution

Lesson 95 – The Two-Sample Hypothesis Test – Part IV

On the Difference in Means

using Welch’s t-Test

How do we proceed when $\sigma_{1}^{2} \neq \sigma_{2}^{2}$ ?

We should now identify the limiting distribution of $V[y]$

Let’s now go back to Tom and help him with his new theory.

Lesson 94 – The Two-Sample Hypothesis Test – Part III

On the Difference in Means

using the t-Test

Let’s take Tom on a mathematical excursion

Hold on!

Say, $\sigma_{1}^{2}=\sigma_{2}^{2}=\sigma^{2}$

Now let’s evaluate the hypothesis that Tom set up — Finally!!

Enjoy this blog? Please spread the word :)

On the Difference in Means

using Welch’s t-Test

How do we proceed when ?

We should now identify the limiting distribution of

Let’s now go back to Tom and help him with his new theory.

On the Difference in Means

using the t-Test

Let’s take Tom on a mathematical excursion

Hold on!

Say,

Now let’s evaluate the hypothesis that Tom set up — Finally!!

Enjoy this blog? Please spread the word :)

How do we proceed when $\sigma_{1}^{2} \neq \sigma_{2}^{2}$ ?

We should now identify the limiting distribution of $V[y]$

Say, $\sigma_{1}^{2}=\sigma_{2}^{2}=\sigma^{2}$