Books Probability and Statistical Inference

The Moment-Generating Function Technique

Theorem 5.5

Let ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ be $𝑛$ independent random variables with respective moment-generating functions ${𝑀}_{1}(𝑡),{𝑀}_{2}(𝑡),\ldots ,{𝑀}_{𝑛}(𝑡)$ where each ${𝑀}_{𝑖}(𝑡)$ exists for all $𝑡$ in some open interval $(-{h}_{𝑖},{h}_{𝑖})$ with ${h}_{𝑖}>0$ . If $𝑌={\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}$ , then

{𝑀}_{𝑌}(𝑡)=\prod _{𝑖=1}^{𝑛}{𝑀}_{𝑖}({𝑎}_{𝑖}𝑡),\quad -{h}_{𝑖}<{𝑎}_{𝑖}𝑡<{h}_{𝑖}\forall 𝑖=1,\ldots ,𝑛.

Proof

We have

{𝑀}_{𝑌}(𝑡)=\mathbb{E}[{𝑒}^{𝑡𝑌}]=\mathbb{E}[{𝑒}^{𝑡{\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}}]=\mathbb{E}[{𝑒}^{{𝑎}_{1}𝑡{𝑋}_{1}}\ldots {𝑒}^{{𝑎}_{𝑛}𝑡{𝑋}_{𝑛}}].

Since ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}$ are independent, by Theorem 5.3, we have

{𝑀}_{𝑌}(𝑡)=\mathbb{E}[{𝑒}^{{𝑎}_{1}𝑡{𝑋}_{1}}]\ldots \mathbb{E}[{𝑒}^{{𝑎}_{𝑛}𝑡{𝑋}_{𝑛}}]=\prod _{𝑖=1}^{𝑛}{𝑀}_{𝑖}({𝑎}_{𝑖}𝑡).

Corollary 5.1

Let ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ be $𝑛$ $i.i.d.$ random variables with MGF $𝑀(𝑡)$ where $-h<𝑡<h$ for some $h>0$ . Then the MGF of $𝑌={\sum }_{𝑖=1}^{𝑛}{𝑋}_{𝑖}$ is given by

{𝑀}_{𝑌}(𝑡)={𝑀}^{𝑛}(𝑡),\quad -h<𝑡<h.

The MGF of the sample mean $𝑋$ is given by

{𝑀}_{𝑋}(𝑡)={𝑀}^{𝑛}\left(\frac{𝑡}{𝑛}\right),\quad -h<\frac{𝑡}{𝑛}<h.

Example 5.6

Recall that the MGF of $\operatorname{Ber}(𝑝)$ is

𝑀(𝑡)=𝑝{𝑒}^{𝑡}+(1-𝑝),\quad -\infty <𝑡<\infty .

And the MGF of $\operatorname{Bin}(𝑛,𝑝)$ is

𝑀(𝑡)={[𝑝{𝑒}^{𝑡}+(1-𝑝)]}^{𝑛},\quad -\infty <𝑡<\infty .

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}∼\operatorname{Ber}(𝑝)$ be $i.i.d.$ and $𝑌={𝑋}_{1}+\ldots +{𝑋}_{𝑛}$ . The MGF of ${𝑋}_{𝑖}$ is $𝑀(𝑡)=𝑝{𝑒}^{𝑡}+(1-𝑝)$ , so the MGF of $𝑌$ is

{𝑀}_{𝑌}(𝑡)={[𝑝{𝑒}^{𝑡}+(1-𝑝)]}^{𝑛},\quad -\infty <𝑡<\infty .

This is the MGF of a binomial(n, p) random variable. Therefore, $𝑌∼\operatorname{Bin}(𝑛,𝑝)$ . That is, the sum of $𝑛$ independent Bernoulli random variables is a binomial random variable, which naturally agrees with our development of the binomial distribution by conducting $𝑛$ independent Bernoulli trials.

Example 5.7

Recall that the MGF of $\exp (\theta )$ is

𝑀(𝑡)=\frac{1}{1-\theta 𝑡},\quad -\frac{1}{\theta }<𝑡<\frac{1}{\theta }.

and the MGF of $\operatorname{Gamma}(\alpha ,\theta )$ is

𝑀(𝑡)=\frac{1}{{(1-\theta 𝑡)}^{\alpha }},\quad -\frac{1}{\theta }<𝑡<\frac{1}{\theta }.

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}∼\operatorname{Exp}(\theta )$ be $i.i.d.$ and $𝑌={𝑋}_{1}+\ldots +{𝑋}_{𝑛}$ . The MGF of ${𝑋}_{𝑖}$ is $𝑀(𝑡)=1/(1-\theta 𝑡)$ , so the MGF of $𝑌$ is

{𝑀}_{𝑌}(𝑡)=\frac{1}{{(1-\theta 𝑡)}^{𝑛}},\quad -\frac{1}{\theta }<𝑡<\frac{1}{\theta }.

This implies that $𝑌∼\operatorname{Gamma}(𝑛,\theta )$ .

The MGF of $𝑋$ is given by

{𝑀}_{𝑋}(𝑡)={𝑀}^{𝑛}\left(\frac{𝑡}{𝑛}\right)=\frac{1}{{[1-(\theta /𝑛)𝑡]}^{𝑛}},\quad -\frac{1}{\theta }<\frac{𝑡}{𝑛}<\frac{1}{\theta }.

Therefore, $𝑋∼\operatorname{Gamma}(𝑛,\theta /𝑛)$ .

Theorem 5.6

Let

{𝑋}_{1},\ldots ,{𝑋}_{𝑛}

𝑛

independent chi-squared random variables with

{𝑟}_{1},\ldots ,{𝑟}_{𝑛}

degrees of freedom respectively. Let

𝑌={𝑋}_{1}+\ldots +{𝑋}_{𝑛}

. We have

𝑌∼{\chi }^{2}({𝑟}_{1}+\ldots +{𝑟}_{𝑛})

Proof

The MGF of ${𝑋}_{𝑖}$ is given by

{𝑀}_{𝑖}(𝑡)=\frac{1}{{(1-2𝑡)}^{{𝑟}_{𝑖}/2}},\quad -\frac{1}{2}<𝑡<\frac{1}{2}.

By Theorem 5.5, we have

{𝑀}_{𝑌}(𝑡)=\prod _{𝑖=1}^{𝑛}{𝑀}_{𝑖}(𝑡)=\prod _{𝑖=1}^{𝑛}\frac{1}{{(1-2𝑡)}^{{𝑟}_{𝑖}/2}}=\frac{1}{{(1-2𝑡)}^{({\sum }_{𝑖=1}^{𝑛}{𝑟}_{𝑖})/2}}.

This is the exactly the MGF of ${\chi }^{2}({\sum }_{𝑖=1}^{𝑛}{𝑟}_{𝑖})$ .

Corollary 5.2

Let

{𝑍}_{1},\ldots ,{𝑍}_{𝑛}∼𝒩(0,1)

i.i.d.

and

𝑊={𝑍}_{1}^{2}+\ldots +{𝑍}_{𝑛}^{2}

. Then,

𝑊∼{\chi }^{2}(𝑛)

Proof

By Theorem 3.2, we know that

{𝑍}_{𝑖}^{2}∼{\chi }^{2}(1)

for

𝑖=1,\ldots ,𝑛

. It then follows from Theorem 5.6 that

𝑊∼{\chi }^{2}(𝑛)

Corollary 5.3

If ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}$ are independent and ${𝑋}_{𝑖}∼𝒩({\mu }_{𝑖},{\sigma }_{𝑖}^{2})$ for $𝑖=1,\ldots ,𝑛$ , then

\sum _{𝑖=1}^{𝑛}\frac{{({𝑋}_{𝑖}-{\mu }_{𝑖})}^{2}}{{\sigma }_{𝑖}^{2}}∼{\chi }^{2}(𝑛).

Proof

By Theorem 3.1, we know that

\frac{{𝑋}_{𝑖}-{\mu }_{𝑖}}{{\sigma }_{𝑖}}∼𝒩(0,1).

Then, this corollary follows from Corollary 5.2.

Exercises

Exercise 5.11

Let ${𝑋}_{1}$ and ${𝑋}_{2}$ be two independent random variables. Suppose that ${𝑋}_{1}∼{\chi }^{2}({𝑟}_{1})$ and $𝑌=({𝑋}_{1}+{𝑋}_{2})∼{\chi }^{2}(𝑟)$ with ${𝑟}_{1}<𝑟$ .

Find the MGF of ${𝑋}_{2}$ .
What is the distribution of ${𝑋}_{2}$ ?

Solution

(a) We have

{𝑀}_{{𝑋}_{1}}(𝑡)=\frac{1}{{(1-2𝑡)}^{{𝑟}_{1}/2}}\quad \text{and}\quad {𝑀}_{𝑌}(𝑡)=\frac{1}{{(1-2𝑡)}^{𝑟/2}}.

Meanwhile, since ${𝑋}_{1}$ and ${𝑋}_{2}$ are independent, we have

{𝑀}_{𝑌}(𝑡)={𝑀}_{{𝑋}_{1}}(𝑡){𝑀}_{{𝑋}_{2}}(𝑡).

Therefore,

{𝑀}_{{𝑋}_{2}}(𝑡)=\frac{{𝑀}_{𝑌}(𝑡)}{{𝑀}_{{𝑋}_{1}}(𝑡)}=\frac{1}{{(1-2𝑡)}^{(𝑟-{𝑟}_{1})/2}}.

(b) We note that ${𝑀}_{{𝑋}_{2}}(𝑡)$ is the MGF of a chi-squared distribution with $𝑟-{𝑟}_{1}$ degrees of freedom. Therefore, ${𝑋}_{2}∼{\chi }^{2}(𝑟-{𝑟}_{1})$ .

Exercise 5.12

Let

𝑍∼𝒩(0,1)

and

𝑊={𝑍}^{2}

. Use the MGF technique to show that

𝑊∼{\chi }^{2}(1)

Proof

We compute the MGF of $𝑊$ :

\begin{aligned}{𝑀}_{𝑊}(𝑡)=\mathbb{E}[{𝑒}^{𝑡𝑊}]=\mathbb{E}[{𝑒}^{𝑡{𝑍}^{2}}] & ={\int }_{-\infty }^{\infty }{𝑒}^{𝑡{𝑧}^{2}}\frac{1}{\sqrt{2\pi }}{𝑒}^{-\frac{{𝑧}^{2}}{2}}\, \mathrm{d}𝑧 \\ & =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\exp \left[\left(𝑡-\frac{1}{2}\right){𝑧}^{2}\right]\, \mathrm{d}𝑧 \\ & =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\underbrace{\exp \left(-\frac{{𝑧}^{2}}{2\cdot \frac{1}{1-2𝑡}}\right)}_{\text{Consider}𝒩\left(0,\frac{1}{1-2𝑡}\right)}\, \mathrm{d}𝑧 \\ & =\frac{1}{\sqrt{2\pi }}\sqrt{2\pi \cdot \frac{1}{1-2𝑡}} \\ & =\frac{1}{\sqrt{1-2𝑡}}.\end{aligned}

This is the MGF of a chi-squared distribution with $1$ degree of freedom. Therefore, $𝑊∼{\chi }^{2}(1)$ .