Books Probability and Statistical Inference

Random Functions Associated with the Normal Distributions

Theorem 5.7

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}$ be $𝑛$ independent random variables with ${𝑋}_{𝑖}∼𝒩({\mu }_{𝑖},{\sigma }_{𝑖}^{2})$ for $𝑖=1,\ldots ,𝑛$ and $𝑌={\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}$ where ${𝑎}_{1},\ldots ,{𝑎}_{𝑛}\in \mathbb{R}$ are constants. Then,

𝑌∼𝒩\left(\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖},\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}\right).

Remark

Note that the sum of independent normal random variables

𝑌

has mean

{\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖}

and variance

{\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}

. This agrees with the result of Theorem 5.4.

Proof

The MGF of ${𝑋}_{𝑖}$ is given by

{𝑀}_{{𝑋}_{𝑖}}(𝑡)=\exp \left({\mu }_{𝑖}𝑡+\frac{{\sigma }_{𝑖}^{2}{𝑡}^{2}}{2}\right).

By Theorem 5.5, we have

\begin{aligned}{𝑀}_{𝑌}(𝑡)=\prod _{𝑖=1}^{𝑛}{𝑀}_{{𝑋}_{𝑖}}({𝑎}_{𝑖}𝑡) & =\prod _{𝑖=1}^{𝑛}\exp \left({𝑎}_{𝑖}{\mu }_{𝑖}𝑡+\frac{{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}{𝑡}^{2}}{2}\right) \\ & =\exp \left[\left(\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖}\right)𝑡+\frac{({\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}){𝑡}^{2}}{2}\right].\end{aligned}

This is exactly the MGF of a normal distribution with mean ${\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖}$ and variance ${\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}$ . Therefore, $𝑌∼𝒩({\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖},{\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2})$ .

Corollary 5.4

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}∼𝒩(\mu ,{\sigma }^{2})$ be $i.i.d.$ , then

𝑋∼𝒩\left(\mu ,\frac{{\sigma }^{2}}{𝑛}\right).

Theorem 5.8

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}∼𝒩(\mu ,{\sigma }^{2})$ be $i.i.d.$ , then the sample mean

𝑋=\frac{1}{𝑛}\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}

and the sample variance

{𝑆}^{2}=\frac{1}{𝑛-1}\sum _{𝑖=1}^{𝑛}{({𝑋}_{𝑖}-𝑋)}^{2}

are independent. Furthermore,

\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}∼{\chi }^{2}(𝑛-1).

Remark

Note that $𝑍=(𝑋-\mu )/\left(\sigma /\sqrt{𝑛}\right)∼𝒩(0,1)$ and ${𝑍}^{2}∼{\chi }^{2}(1)$ . The idea of proving

\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}∼{\chi }^{2}(𝑛-1)

is inspired by Exercise 5.11. That is, we want to show that

\left[\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}+{𝑍}^{2}\right]∼{\chi }^{2}(𝑛).

Proof

We are not ready to prove that $𝑋$ and ${𝑆}^{2}$ are independent. However, we can show that

\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}∼{\chi }^{2}(𝑛-1).

assuming that $𝑋$ and ${𝑆}^{2}$ are independent.

Let $𝑍=(𝑋-\mu )/\left(\sigma /\sqrt{𝑛}\right)$ . Since $𝑋∼𝒩\left(\mu ,\frac{{\sigma }^{2}}{𝑛}\right)$ , we have $𝑍∼𝒩(0,1)$ . Therefore, ${𝑍}^{2}∼{\chi }^{2}(1)$ .

Since ${𝑆}^{2}$ and $𝑋$ are independent, it follows that $[(𝑛-1){𝑆}^{2}]/{\sigma }^{2}$ and ${𝑍}^{2}$ are also independent. We now consider $[((𝑛-1){𝑆}^{2})/{\sigma }^{2}+{𝑍}^{2}]$ . We have

\begin{aligned}\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}+{𝑍}^{2} & =\frac{1}{{\sigma }^{2}}\sum _{𝑖=1}^{𝑛}{({𝑋}_{𝑖}-𝑋)}^{2}+\frac{{(𝑋-\mu )}^{2}}{{\sigma }^{2}/𝑛} \\ & =\frac{1}{{\sigma }^{2}}\left[\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}^{2}-2𝑋\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}+𝑛{𝑋}^{2}+𝑛({𝑋}^{2}-2\mu 𝑋+{\mu }^{2})\right] \\ & =\frac{1}{{\sigma }^{2}}\left[\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}^{2}-𝑛{𝑋}^{2}+𝑛({𝑋}^{2}-2\mu 𝑋+{\mu }^{2})\right] \\ & =\frac{1}{{\sigma }^{2}}\left(\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}^{2}-2𝑛\mu 𝑋+𝑛{\mu }^{2}\right) \\ & =\frac{1}{{\sigma }^{2}}\left(\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}^{2}-\sum _{𝑖=1}^{𝑛}2\mu {𝑋}_{𝑖}+\sum _{𝑖=1}^{𝑛}{\mu }^{2}\right) \\ & =\frac{1}{{\sigma }^{2}}\left[\sum _{𝑖=1}^{𝑛}{({𝑋}_{𝑖}-\mu )}^{2}\right] \\ & =\sum _{𝑖=1}^{𝑛}\frac{{({𝑋}_{𝑖}-\mu )}^{2}}{{\sigma }^{2}}\end{aligned}

Since ${𝑍}_{𝑖}^{2}={({𝑋}_{𝑖}-\mu )}^{2}/{\sigma }^{2}∼{\chi }^{2}(1)$ and ${𝑍}_{1},\ldots ,{𝑍}_{𝑛}$ are mutually independent, it follows that

\left[\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}+{𝑍}^{2}\right]∼{\chi }^{2}(𝑛).

Therefore, by Exercise 5.11,

\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}∼{\chi }^{2}(𝑛-1).

We now introduce a distribution that forms the foundation of many statistical inferences.

Theorem 5.9Student's t-distribution

Let

𝑇=\frac{𝑍}{\sqrt{𝑈/𝑟}}

(5.11)

where $𝑍∼𝒩(0,1)$ and $𝑈∼{\chi }^{2}(𝑟)$ are independent. Then $𝑇$ has the $𝑡$ -distribution with $𝑟$ degrees of freedom. We write $𝑇∼𝒯(𝑟)$ . Its PDF is given by

𝑓(𝑡)=\frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\sqrt{\pi 𝑟}\Gamma (𝑟/2)}\frac{1}{{(1+{𝑡}^{2}/𝑟)}^{\frac{𝑟+1}{2}}},\quad -\infty <𝑡<\infty .

(5.12)

Remark

The distribution is named for Student, the pen name of William Sealy Gosset (1876–1937). In 1908, while working as a statistician at the Guinness brewery in Dublin, Gosset published his derivation in Biometrika under that pseudonym: his employer discouraged staff from publishing under their own names. He sought a reference law for the sample mean when

\sigma

is unknown and

𝑛

is small; Ronald Fisher later gave a systematic account of the distribution and its degrees of freedom.

Note

At first glance, (5.11) and (5.12) may look intimidating. Notice, however, that

{𝑇}^{2}=\frac{{𝑍}^{2}}{𝑈/𝑟}

is the ratio of two independent chi-squared variables, so

{𝑇}^{2}∼𝐹(1,𝑟)

. To recover the PDF of

𝑇

from that of

{𝑇}^{2}

, we will use the symmetry of

𝑇

and the square-root transformation in Exercise 5.3.

Proof

Let $𝑊={𝑇}^{2}∼𝐹(1,𝑟)$ . Its PDF is

{𝑓}_{𝑊}(𝑤)=\frac{1}{\sqrt{𝑟}}\cdot \frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\Gamma (1/2)\Gamma (𝑟/2)}\cdot \frac{1}{\sqrt{𝑤}}\cdot \frac{1}{{(1+𝑤/𝑟)}^{\frac{𝑟+1}{2}}},\quad 𝑤>0.

Since $\Gamma (1/2)=\sqrt{\pi }$ , this simplifies to

{𝑓}_{𝑊}(𝑤)=\frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\sqrt{\pi 𝑟}\Gamma (𝑟/2)}\cdot \frac{1}{\sqrt{𝑤}}\cdot \frac{1}{{(1+𝑤/𝑟)}^{\frac{𝑟+1}{2}}},\quad 𝑤>0.

(5.13)

The standard normal distribution is symmetric about $0$ , and $𝑈$ is independent of $𝑍$ , so replacing $𝑍$ by $-𝑍$ does not change the distribution of $𝑇=\frac{𝑍}{\sqrt{𝑈/𝑟}}$ . Hence $-𝑇∼𝑇$ and $𝑊={𝑇}^{2}$ has support $(0,\infty )$ , the case $𝑎=\infty$ in Exercise 5.3.

By Exercise 5.3,

{𝑓}_{𝑇}(𝑡)=|𝑡|{𝑓}_{𝑊}({𝑡}^{2}).

Substituting (5.13) with $𝑤={𝑡}^{2}$ ,

\begin{aligned}{𝑓}_{𝑇}(𝑡) & =|𝑡|\cdot \frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\sqrt{\pi 𝑟}\Gamma (𝑟/2)}\cdot \frac{1}{\sqrt{{𝑡}^{2}}}\cdot \frac{1}{{(1+{𝑡}^{2}/𝑟)}^{\frac{𝑟+1}{2}}} \\ & =|𝑡|\cdot \frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\sqrt{\pi 𝑟}\Gamma (𝑟/2)}\cdot \frac{1}{|𝑡|}\cdot \frac{1}{{(1+{𝑡}^{2}/𝑟)}^{\frac{𝑟+1}{2}}} \\ & =\frac{\Gamma \left(\frac{𝑟+1}{2}\right)}{\sqrt{\pi 𝑟}\Gamma (𝑟/2)}\cdot \frac{1}{{(1+{𝑡}^{2}/𝑟)}^{\frac{𝑟+1}{2}}}.\end{aligned}

When $𝑟=1$ , the $𝑡$ -distribution reduces to the Cauchy distribution from Example 5.1. Substituting $𝑟=1$ into (5.12) and using $\Gamma (1)=1$ and $\Gamma (1/2)=\sqrt{\pi }$ ,

𝑓(𝑡)=\frac{\Gamma (1)}{\sqrt{\pi }\Gamma (1/2)}\cdot \frac{1}{1+{𝑡}^{2}}=\frac{1}{\pi (1+{𝑡}^{2})},

which is (5.1). In (5.11), the case $𝑟=1$ is $𝑇=\frac{𝑍}{\sqrt{𝑈}}$ with $𝑈∼{\chi }^{2}(1)$ independent of $𝑍∼𝒩(0,1)$ —the usual construction of a Cauchy ratio.

Graphs of the $𝑡$ -distribution PDF for $𝑟=1$ , $5$ , and $30$ are shown in Figure 5.9. The case $𝑟=1$ is the Cauchy density; as $𝑟$ increases, the tails become lighter and the density approaches the standard normal PDF.

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}$ be a random sample from a normal distribution with mean $\mu$ and variance ${\sigma }^{2}$ . We can use sample mean $𝑋$ and sample variance ${𝑆}^{2}$ to construct a very important $𝑡$ -random variable. Recall that

𝑋∼𝒩(\mu ,{\sigma }^{2}/𝑛)\quad \text{and}\quad \frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}∼{\chi }^{2}(𝑛-1).

Therefore,

𝑇=\frac{\frac{𝑋-\mu }{\sigma /\sqrt{𝑛}}}{\sqrt{\frac{(𝑛-1){𝑆}^{2}}{{\sigma }^{2}}/(𝑛-1)}}

is a $𝑡$ -random variable with $𝑛-1$ degrees of freedom. Simplifying the expression, we get

𝑇=\frac{𝑋-\mu }{𝑆/\sqrt{𝑛}}∼𝒯(𝑛-1).

Exercises

Exercise 5.13

Let ${𝑍}_{1},{𝑍}_{2},{𝑍}_{3}∼𝒩(0,1)$ be $i.i.d.$ .

Find the distribution of
$𝑊=\frac{{𝑍}_{1}}{\sqrt{({𝑍}_{2}^{2}+{𝑍}_{3}^{2})/2}}.$
Show that
$𝑉=\frac{{𝑍}_{1}}{\sqrt{({𝑍}_{1}^{2}+{𝑍}_{2}^{2})/2}}$
has PDF
$𝑓(𝑣)=\frac{1}{\pi \sqrt{2-{𝑣}^{2}}},\quad -\sqrt{2}<𝑣<\sqrt{2}.$
Find the mean of $𝑉$ .
Find the standard deviation of $𝑉$ .
Why $𝑊$ and $𝑉$ are so different?

Solution

(a) Let $𝑈={𝑍}_{2}^{2}+{𝑍}_{3}^{2}$ . We have $𝑈∼{\chi }^{2}(2)$ and $𝑈$ and ${𝑍}_{1}$ are independent. Therefore, by the definition of the $𝑡$ -distribution,

𝑊=\frac{{𝑍}_{1}}{\sqrt{𝑈/2}}∼𝒯(2).

(b) We have

{𝑉}^{2}=\frac{2{𝑍}_{1}^{2}}{{𝑍}_{1}^{2}+{𝑍}_{2}^{2}}=\frac{2}{1+({𝑍}_{2}^{2}/{𝑍}_{1}^{2})}.

Note that

\frac{{𝑍}_{2}^{2}}{{𝑍}_{1}^{2}}∼𝐹(1,1).

And from the result of Exercise 5.10, we know that

𝑋=\frac{1}{1+({𝑍}_{2}^{2}/{𝑍}_{1}^{2})}∼\operatorname{Beta}(1/2,1/2).

Therefore, the PDF of $𝑋$ is given by

{𝑓}_{𝑋}(𝑥)=\frac{\Gamma (1/2+1/2)}{\Gamma (1/2)\Gamma (1/2)}{𝑥}^{1/2-1}{(1-𝑥)}^{1/2-1}=\frac{1}{\pi \sqrt{𝑥(1-𝑥)}},\quad 0<𝑥<1.

Note that $𝑌={𝑉}^{2}=2𝑋$ . Applying the change-of-variable technique, we get

{𝑓}_{𝑌}(𝑦)={𝑓}_{𝑋}(𝑦/2)\cdot \frac{1}{2}=\frac{1}{\pi \sqrt{(𝑦/2)(1-𝑦/2)}}\cdot \frac{1}{2}=\frac{1}{\pi \sqrt{𝑦(2-𝑦)}},\quad 0<𝑦<2.

Finally, $𝑌={𝑉}^{2}$ has support $(0,2)$ and $-𝑉∼𝑉$ , so Exercise 5.3 with $𝑎=2$ gives

{𝑓}_{𝑉}(𝑣)=|𝑣|{𝑓}_{𝑌}({𝑣}^{2})=|𝑣|\cdot \frac{1}{\pi \sqrt{{𝑣}^{2}(2-{𝑣}^{2})}}=\frac{1}{\pi \sqrt{2-{𝑣}^{2}}},\quad -\sqrt{2}<𝑣<\sqrt{2}.

\mu =\mathbb{E}[𝑉]={\int }_{-\sqrt{2}}^{\sqrt{2}}\frac{𝑣}{\pi \sqrt{2-{𝑣}^{2}}}\, \mathrm{d}𝑣.

It can be shown that the improper integral

{\int }_{0}^{\sqrt{2}}\frac{𝑣}{\pi \sqrt{2-{𝑣}^{2}}}\, \mathrm{d}𝑣

converses. It then follows that

\mu =0

because the integrand is an odd function.

(d) The variance of $𝑉$ is given by

{\sigma }^{2}=\operatorname{Var}(𝑉)=\mathbb{E}[{𝑉}^{2}]-{\mu }^{2}=\mathbb{E}[{𝑉}^{2}].

We have

\begin{aligned}\mathbb{E}[{𝑉}^{2}] & ={\int }_{-\sqrt{2}}^{\sqrt{2}}\frac{{𝑣}^{2}}{\pi \sqrt{2-{𝑣}^{2}}}\, \mathrm{d}𝑣 \\ & \text{Substitute}\, 𝑣=\sqrt{2}\sin \theta \, \text{and}\, \mathrm{d}𝑣=\sqrt{2}\cos \theta \, \mathrm{d}\theta \\ & ={\int }_{-\pi /2}^{\pi /2}\frac{2{\sin }^{2}\theta }{\pi \sqrt{2}\cos \theta }\cdot \sqrt{2}\cos \theta \, \mathrm{d}\theta \\ & ={\int }_{-\pi /2}^{\pi /2}\frac{2{\sin }^{2}\theta }{\pi }\, \mathrm{d}\theta \\ & =1-\frac{1}{\pi }{\int }_{-\pi /2}^{\pi /2}\cos 2\theta \, \mathrm{d}\theta \\ & =1.\end{aligned}

Therefore, $\sigma =1$ .