Books Probability and Statistical Inference

The Normal Distribution

Random variable $𝑋$ is said to have a normal distribution with parameters $\mu \in \mathbb{R}$ and ${\sigma }^{2}>0$ if its PDF is given by

𝑓(𝑥)=\frac{1}{\sqrt{2\pi {\sigma }^{2}}}\exp \left[-\frac{{(𝑥-\mu )}^{2}}{2{\sigma }^{2}}\right],\quad 𝑥\in \mathbb{R}.

(3.6)

We write $𝑋∼𝒩(\mu ,{\sigma }^{2})$ . We need to verify that the PDF in (3.6) is well-defined. Clearly, $𝑓(𝑥)\geq 0$ for all $𝑥\in \mathbb{R}$ . It remains to check that the integral of $𝑓$ over $\mathbb{R}$ equals $1$ . This reduces to evaluating

{\int }_{-\infty }^{\infty }{𝑒}^{-{𝑥}^{2}}\, \mathrm{d}𝑥.

Set $𝐼={\int }_{0}^{\infty }{𝑒}^{-{𝑥}^{2}}\, \mathrm{d}𝑥$ . We work on $[0,\infty )$ first because it makes the following polar-coordinate computation shorter. Then

{𝐼}^{2}={\int }_{0}^{\infty }{𝑒}^{-{𝑥}^{2}}\, \mathrm{d}𝑥{\int }_{0}^{\infty }{𝑒}^{-{𝑦}^{2}}\, \mathrm{d}𝑦={\int }_{0}^{\infty }{\int }_{0}^{\infty }{𝑒}^{-({𝑥}^{2}+{𝑦}^{2})}\, \mathrm{d}𝑥\, \mathrm{d}𝑦,

the last integral being over the first quadrant of the plane. In polar coordinates $𝑥=𝑟\cos \theta$ and $𝑦=𝑟\sin \theta$ , with $𝑟\geq 0$ and $\theta \in \left[0,\frac{\pi }{2}\right]$ , the area element satisfies $\mathrm{d}𝑥\, \mathrm{d}𝑦=𝑟\, \mathrm{d}𝑟\, \mathrm{d}\theta$ . Equivalently, the Jacobian is

𝐽=\left|\det \left(\frac{\partial (𝑥,𝑦)}{\partial (𝑟,\theta )}\right)\right|=\left|\det \begin{pmatrix}\cos \theta & -𝑟\sin \theta \\ \sin \theta & 𝑟\cos \theta \end{pmatrix}\right|=𝑟.

Hence

\begin{aligned}{\int }_{0}^{\infty }{\int }_{0}^{\infty }{𝑒}^{-({𝑥}^{2}+{𝑦}^{2})}\, \mathrm{d}𝑥\, \mathrm{d}𝑦 & ={\int }_{0}^{\infty }{\int }_{0}^{\pi /2}{𝑒}^{-{𝑟}^{2}}𝑟\, \mathrm{d}\theta \, \mathrm{d}𝑟 \\ & =\frac{\pi }{2}{\int }_{0}^{\infty }𝑟{𝑒}^{-{𝑟}^{2}}\, \mathrm{d}𝑟 \\ & =\frac{\pi }{2}{\left[-\frac{1}{2}{𝑒}^{-{𝑟}^{2}}\right]}_{0}^{\infty } \\ & =\frac{\pi }{4}.\end{aligned}

Therefore $𝐼=\sqrt{\pi }/2$ and

{\int }_{-\infty }^{\infty }{𝑒}^{-{𝑥}^{2}}\, \mathrm{d}𝑥=\sqrt{\pi }.

(3.7)

To normalize (3.6), substitute $𝑢=\frac{𝑥-\mu }{\sqrt{2{\sigma }^{2}}}$ . Then

{\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi {\sigma }^{2}}}\exp \left[-\frac{{(𝑥-\mu )}^{2}}{2{\sigma }^{2}}\right]\, \mathrm{d}𝑥=\frac{1}{\sqrt{2\pi {\sigma }^{2}}}{\int }_{-\infty }^{\infty }{𝑒}^{-{𝑢}^{2}}\sqrt{2{\sigma }^{2}}\, \mathrm{d}𝑢=1,

(3.8)

where the last step uses (3.7).

Next we compute the MGF of the normal distribution. We have

𝑀(𝑡)=\mathbb{E}[{𝑒}^{𝑡𝑋}]=\frac{1}{\sqrt{2\pi {\sigma }^{2}}}{\int }_{-\infty }^{\infty }\exp \left[-\frac{{(𝑥-\mu )}^{2}}{2{\sigma }^{2}}+𝑡𝑥\right]\, \mathrm{d}𝑥.

(3.9)

The idea is to complete the square in $𝑥$ , writing the exponent as

-\frac{{(𝑥-𝑎)}^{2}}{2{\sigma }^{2}}+𝑏

for constants $𝑎$ and $𝑏$ that do not depend on $𝑥$ .

Note

For any $𝑎\in \mathbb{R}$ ,

{\int }_{-\infty }^{\infty }\exp \left[-\frac{{(𝑥-𝑎)}^{2}}{2{\sigma }^{2}}\right]\, \mathrm{d}𝑥=\sqrt{2\pi {\sigma }^{2}},

by (3.8) with $\mu$ replaced by $𝑎$ .

Completing the square in the exponent of (3.9) gives

-\frac{{(𝑥-\mu )}^{2}}{2{\sigma }^{2}}+𝑡𝑥=-\frac{{[𝑥-(\mu +{\sigma }^{2}𝑡)]}^{2}}{2{\sigma }^{2}}+\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}.

Hence (3.9) becomes

\begin{aligned}𝑀(𝑡) & =\frac{1}{\sqrt{2\pi {\sigma }^{2}}}\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right){\int }_{-\infty }^{\infty }\exp \left[-\frac{{[𝑥-(\mu +{\sigma }^{2}𝑡)]}^{2}}{2{\sigma }^{2}}\right]\, \mathrm{d}𝑥 \\ & =\frac{1}{\sqrt{2\pi {\sigma }^{2}}}\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right)\sqrt{2\pi {\sigma }^{2}} \\ & =\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right).\end{aligned}

Thus the MGF is

𝑀(𝑡)=\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right),\quad 𝑡\in \mathbb{R}.

Differentiating,

\begin{aligned}{𝑀}^{\prime }(𝑡) & =(\mu +{\sigma }^{2}𝑡)\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right) \\ {𝑀}^{\prime\prime }(𝑡) & =[{\sigma }^{2}+{(\mu +{\sigma }^{2}𝑡)}^{2}]\exp \left(\mu 𝑡+\frac{{\sigma }^{2}{𝑡}^{2}}{2}\right).\end{aligned}

At $𝑡=0$ ,

{𝑀}^{\prime }(0)=\mu \quad \text{and}\quad {𝑀}^{\prime\prime }(0)={\sigma }^{2}+{\mu }^{2}.

Therefore $\mathbb{E}[𝑋]=\mu$ and $\operatorname{Var}(𝑋)={𝑀}^{\prime\prime }(0)-{[{𝑀}^{\prime }(0)]}^{2}={\sigma }^{2}$ , in agreement with the usual meaning of $\mu$ and ${\sigma }^{2}$ in $𝒩(\mu ,{\sigma }^{2})$ .

Specially, if $𝑍=𝒩(0,1)$ , then $𝑍$ is said to have a standard normal distribution. Its PDF is given by

𝑓(𝑧)=\frac{1}{\sqrt{2\pi }}{𝑒}^{-\frac{{𝑧}^{2}}{2}},\quad 𝑧\in \mathbb{R}.

And its CDF is often denoted by $\Phi (𝑧)$ :

\Phi (𝑧)=ℙ(𝑍\leq 𝑧)={\int }_{-\infty }^{𝑧}\frac{1}{\sqrt{2\pi }}{𝑒}^{-{𝑡}^{2}/2}\, \mathrm{d}𝑡.

Note

For a general normal distribution $𝑋∼𝒩(\mu ,{\sigma }^{2})$ , the CDF of $𝑋$ can be written as

{𝐹}_{𝑋}(𝑥)=\Phi \left(\frac{𝑥-\mu }{\sigma }\right).

(3.10)

If we can write the CDF of some random variable $𝑌$ as

{𝐹}_{𝑌}(𝑦)=\Phi \left(\frac{𝑦-𝑎}{𝑏}\right)

where $𝑏>0$ , then we may conclude that $𝑌∼𝒩(𝑎,{𝑏}^{2})$ . (The condition $𝑏>0$ is important and it is easy to overlook.)

Finally, the MGF of the standard normal distribution is

𝑀(𝑡)={𝑒}^{{𝑡}^{2}/2},\quad 𝑡\in \mathbb{R}.

Figure 3.5 shows the standard normal PDF.

Figure 3.5 Standard normal PDF $𝑓(𝑧)$ . The shaded area to the left of ${𝑧}_{0}$ equals $\Phi ({𝑧}_{0})$ .

Because the standard normal PDF is symmetric about $𝑧=0$ , intuitively we have $\Phi (-𝑧)=1-\Phi (𝑧)$ . But it is worth an exercise to show this.

Exercise 3.6

Show that

\Phi (-𝑧)=1-\Phi (𝑧).

(3.11)

Proof

We have

\begin{aligned}\Phi (-𝑧) & ={\int }_{-\infty }^{-𝑧}𝑓(𝑡)\, \mathrm{d}𝑡 \\ & \text{Substitute}\, 𝑡=-𝑢 \\ & ={\int }_{\infty }^{𝑧}𝑓(-𝑢)(-1)\, \mathrm{d}𝑢 \\ & ={\int }_{𝑧}^{\infty }𝑓(𝑢)\, \mathrm{d}𝑢 \\ & =1-{\int }_{-\infty }^{𝑧}𝑓(𝑢)\, \mathrm{d}𝑢 \\ & =1-\Phi (𝑧).\end{aligned}

As with the chi-squared distribution, tail probabilities for the standard normal underlie many routine calculations in inference. The upper $\alpha$ critical value ${𝑧}_{\alpha }$ is defined by

ℙ(𝑍\geq {𝑧}_{\alpha })=\alpha .

Equivalently, ${𝑧}_{\alpha }$ is the $100(1-\alpha )$ -th percentile of the standard normal distribution, i.e.

\Phi ({𝑧}_{\alpha })=ℙ(𝑍\leq {𝑧}_{\alpha })=1-\alpha .

Figure 3.6 illustrates the upper tail when $\alpha =0.05$ ; in this case ${𝑧}_{\alpha }\approx 1.6449$ .

Figure 3.6 Standard normal PDF with the upper tail of area $\alpha =0.05$ shaded; the corresponding upper critical value satisfies ${𝑧}_{\alpha }\approx 1.6449$ .

Exercise 3.7

Due to the symmetry of the standard normal distribution, show that

$ℙ(𝑍\leq -{𝑧}_{\alpha })=ℙ(𝑍\geq {𝑧}_{\alpha })=\alpha$ , and
${𝑧}_{1-\alpha }=-{𝑧}_{\alpha }$ .

Proof

To show (a), we apply (3.11) to get

ℙ(𝑍\leq -{𝑧}_{\alpha })=\Phi (-{𝑧}_{\alpha })=1-\Phi ({𝑧}_{\alpha })=1-(1-\alpha )=\alpha =ℙ(𝑍\geq {𝑧}_{\alpha }).

We now show (b). Applying $\Phi$ to ${𝑧}_{1-\alpha }$ , we get

\Phi ({𝑧}_{1-\alpha })=1-\Phi (-{𝑧}_{1-\alpha }).

Using (a), $\Phi (-{𝑧}_{1-\alpha })=1-\alpha$ . Therefore,

\Phi ({𝑧}_{1-\alpha })=1-(1-\alpha )=\alpha .

Meanwhile,

\Phi (-{𝑧}_{\alpha })=1-\Phi ({𝑧}_{\alpha })=1-(1-\alpha )=\alpha .

Therefore,

\Phi ({𝑧}_{1-\alpha })=\Phi (-{𝑧}_{\alpha })=\alpha .

Since $\Phi$ is strictly increasing, we have

{𝑧}_{1-\alpha }=-{𝑧}_{\alpha }.

Theorem 3.1

If $𝑋∼𝒩(\mu ,{\sigma }^{2})$ , then

𝑍=\frac{𝑋-\mu }{\sigma }∼𝒩(0,1).

Proof

We consider the CDF of $𝑍$ . We have

\begin{aligned}{𝐹}_{𝑍}(𝑧)=ℙ(𝑍\leq 𝑧)=ℙ\left(\frac{𝑋-\mu }{\sigma }\leq 𝑧\right) & =ℙ(𝑋\leq \mu +\sigma 𝑧) \\ & ={\int }_{-\infty }^{\mu +\sigma 𝑧}{𝑓}_{𝑋}(𝑥)\, \mathrm{d}𝑥 \\ & \text{Change variable}\, 𝑥=\mu +\sigma 𝑡 \\ & ={\int }_{-\infty }^{𝑧}{𝑓}_{𝑋}(\mu +\sigma 𝑡)\sigma \, \mathrm{d}𝑡 \\ & ={\int }_{-\infty }^{𝑧}\frac{1}{\sqrt{2\pi }}{𝑒}^{-{𝑡}^{2}/2}\, \mathrm{d}𝑡 \\ & =\Phi (𝑧).\end{aligned}

Therefore, $𝑍∼𝒩(0,1)$ .

Remark

If $𝑋$ is any random variable with mean $\mu$ and variance ${\sigma }^{2}$ , then $𝑍=(𝑋-\mu )/\sigma$ is called the standard score associated with $𝑋$ . We have

\mathbb{E}[𝑍]=\mathbb{E}\left[\frac{𝑋-\mu }{\sigma }\right]=\frac{\mathbb{E}[𝑋]-\mu }{\sigma }=0\quad

and

\operatorname{Var}(𝑍)=\mathbb{E}\left[{\left(\frac{𝑋-\mu }{\sigma }\right)}^{2}\right]=\frac{\mathbb{E}[{(𝑋-\mu )}^{2}]}{{\sigma }^{2}}=\frac{{\sigma }^{2}}{{\sigma }^{2}}=1.

That is, the mean and variance of the $𝑍$ are $0$ and $1$ , respectively, no matter what the distribution of $𝑋$ is.

In the case of this theorem, $𝑋$ has a normal distribution and consequently $𝑍$ has a standard normal distribution—of course it has mean $0$ and variance $1$ .

Theorem 3.2

Let $𝑍∼𝒩(0,1)$ , then

{𝑍}^{2}∼{\chi }^{2}(1).

In fact, this is why chi-squared distribution is named as such—it is the square of a random variable.

The proof uses the PDF of ${\chi }^{2}(1)$ , which involves $\Gamma (1/2)$ ; the following exercise evaluates $\Gamma (1/2)$ .

Exercise 3.8

Evaluate

\Gamma (1/2)

Solution

Recall that $\Gamma (𝑡)$ is given by

\Gamma (𝑡)={\int }_{0}^{\infty }{𝑦}^{𝑡-1}{𝑒}^{-𝑦}\, \mathrm{d}𝑦.

At $1/2$ ,

\Gamma \left(\frac{1}{2}\right)={\int }_{0}^{\infty }\frac{1}{\sqrt{𝑦}}{𝑒}^{-𝑦}\, \mathrm{d}𝑦.

Substitute $𝑦={𝑢}^{2}$ , we obtain

\Gamma \left(\frac{1}{2}\right)={\int }_{0}^{\infty }\frac{1}{𝑢}{𝑒}^{-{𝑢}^{2}}2𝑢\, \mathrm{d}𝑢=2{\int }_{0}^{\infty }{𝑒}^{-{𝑢}^{2}}\, \mathrm{d}𝑢=\sqrt{\pi }.

where the last step follows from (3.7). Therefore, $\Gamma (1/2)=\sqrt{\pi }$ .

We now present the proof of Theorem 3.2.

Proof

Substituting $𝑟=1$ into (3.5) and then plugging in the value of $\Gamma (1/2)=\sqrt{\pi }$ , we get the PDF of ${\chi }^{2}(1)$ :

𝑓(𝑥)=\frac{1}{\Gamma (1/2)\cdot \sqrt{2}}{𝑥}^{1/2-1}{𝑒}^{-𝑥/2}=\frac{1}{\sqrt{2\pi }}{𝑥}^{-1/2}{𝑒}^{-𝑥/2}.

(3.12)

Let $𝑋={𝑍}^{2}$ . The CDF of $𝑋$ is given by

\begin{aligned}{𝐹}_{𝑋}(𝑥)=ℙ(𝑋\leq 𝑥)=ℙ({𝑍}^{2}\leq 𝑥) & =ℙ\left(-\sqrt{𝑥}\leq 𝑍\leq \sqrt{𝑥}\right) \\ & =\Phi \left(\sqrt{𝑥}\right)-\Phi \left(-\sqrt{𝑥}\right) \\ & =2\Phi \left(\sqrt{𝑥}\right)-1.\end{aligned}

where the last step uses (3.11). Note that ${𝐹}_{𝑋}$ is differentiable in $(0,\infty )$ . Taking the derivative, we get the PDF of $𝑋$ :

{𝑓}_{𝑋}(𝑥)={𝐹}_{𝑋}\prime (𝑥)=\frac{\mathrm{d}}{\mathrm{d}𝑥}\left(2\Phi \left(\sqrt{𝑥}\right)-1\right)={\Phi }^{\prime }\left(\sqrt{𝑥}\right)\cdot \frac{1}{\sqrt{𝑥}}

Recall that the derivative of $\Phi (𝑥)$ is the PDF of the standard normal distribution. Therefore, we obtain

{𝑓}_{𝑋}(𝑥)=\frac{1}{\sqrt{2\pi }}{𝑒}^{-{\left(\sqrt{𝑥}\right)}^{2}/2}\cdot \frac{1}{\sqrt{𝑥}}=\frac{1}{\sqrt{2\pi }}{𝑥}^{-1/2}{𝑒}^{-𝑥/2}.

(3.13)

Note that (3.13) is the same as (3.12), which shows that $𝑋∼{\chi }^{2}(1)$ .

Exercises

Exercise 3.9

𝑋∼𝒩(\mu ,{\sigma }^{2})

, show that

𝑌=𝑎𝑋+𝑏∼𝒩(𝑎\mu +𝑏,{𝑎}^{2}{\sigma }^{2})

(

𝑎\neq 0

Proof

If $𝑎>0$ , we have

\begin{aligned}{𝐹}_{𝑌}(𝑦)=ℙ(𝑌\leq 𝑦)=ℙ(𝑎𝑋+𝑏\leq 𝑦) & =ℙ\left(𝑋\leq \frac{𝑦-𝑏}{𝑎}\right)={𝐹}_{𝑋}\left(\frac{𝑦-𝑏}{𝑎}\right) \\ & =\Phi \left(\frac{\frac{𝑦-𝑏}{𝑎}-\mu }{\sigma }\right) \\ & =\Phi \left[\frac{𝑦-(𝑎\mu +𝑏)}{𝑎\sigma }\right].\end{aligned}

The denominator $𝑎\sigma >0$ because $𝑎>0$ . By (3.10), we note that the function above is extact the CDF of $𝒩(𝑎\mu +𝑏,{𝑎}^{2}{\sigma }^{2})$ , and hence $𝑌∼𝒩(𝑎\mu +𝑏,{𝑎}^{2}{\sigma }^{2})$ .

If $𝑎<0$ , we have

\begin{aligned}{𝐹}_{𝑌}(𝑦)=ℙ\left(𝑋\geq \frac{𝑦-𝑏}{𝑎}\right) & =1-{𝐹}_{𝑋}\left(\frac{𝑦-𝑏}{𝑎}\right) \\ & =1-\Phi \left(\frac{\frac{𝑦-𝑏}{𝑎}-\mu }{\sigma }\right) \\ & =1-\Phi \left[\frac{𝑦-(𝑎\mu +𝑏)}{𝑎\sigma }\right] \\ & =\Phi \left[\frac{𝑦-(𝑎\mu +𝑏)}{-𝑎\sigma }\right].\end{aligned}

Note that $-𝑎\sigma >0$ . Thus, RHS is indeed the CDF of $𝒩(𝑎\mu +𝑏,{𝑎}^{2}{\sigma }^{2})$ .

Exercise 3.10

Material strength $𝑋$ (in whatever units the data use) is modeled by $𝑋={𝑒}^{𝑌}$ with $𝑌∼𝒩(\mu ,{\sigma }^{2})$ , so $𝑋$ has a lognormal distribution. Then $𝑌=\ln 𝑋$ is the same variability on a log scale and has no unit. One uses this form because strength is positive (so $𝑋={𝑒}^{𝑌}$ rules out the negative values a direct normal model for $𝑋$ could allow) and because variability is often more natural on a relative scale: multiplicative factors in $𝑋$ become additive in $𝑌$ , and a sum of many small perturbations is commonly approximated by a normal law—hence a normal $𝑌$ and a lognormal $𝑋$ . In part (b), the cutoffs $10000$ and $20000$ match the usual order of magnitude for strength or stress quoted in psi (pounds per square inch); in SI, megapascals are more common and the numerals would typically be much smaller.

Find the CDF and PDF of $𝑋$ in terms of $\Phi$ and $\mu$ , ${\sigma }^{2}$ .
For $\mu =10$ and ${\sigma }^{2}=1$ , calculate $ℙ(10000<𝑋<20000)$ .

Solution

(a) We have

{𝐹}_{𝑥}(𝑥)=ℙ(𝑋\leq 𝑥)=ℙ({𝑒}^{𝑌}\leq 𝑥)=ℙ(𝑌\leq \ln 𝑥)=\Phi \left(\frac{\ln 𝑥-\mu }{\sigma }\right).

Differentiating, we get the PDF of $𝑋$ :

\begin{aligned}{𝑓}_{𝑋}(𝑥)={𝐹}_{𝑋}\prime (𝑥)=\frac{\mathrm{d}}{\mathrm{d}𝑥}\Phi \left(\frac{\ln 𝑥-\mu }{\sigma }\right) & =\frac{1}{\sigma 𝑥}{\Phi }^{\prime }\left(\frac{\ln 𝑥-\mu }{\sigma }\right) \\ & =\frac{1}{\sigma 𝑥}\frac{1}{\sqrt{2\pi }}\exp \left[-\frac{{(\ln 𝑥-\mu )}^{2}}{2{\sigma }^{2}}\right] \\ & =\frac{1}{𝑥\sqrt{2\pi {\sigma }^{2}}}\exp \left[-\frac{{(\ln 𝑥-\mu )}^{2}}{2{\sigma }^{2}}\right].\end{aligned}

(b) $ℙ(10000<𝑋<20000)={𝐹}_{𝑋}(20000)-{𝐹}_{𝑋}(10000)\approx 0.2467$ .