Books Probability and Statistical Inference

Several Random Variables

The following theorem shows that for independent random variables, the expectation of a product of functions factors as the product of the individual expectations.

Theorem 5.3

Let ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ be $𝑛$ independent random variables and

𝑌={𝑢}_{1}({𝑋}_{1}){𝑢}_{2}({𝑋}_{2})\ldots {𝑢}_{𝑛}({𝑋}_{𝑛}).

If $\mathbb{E}[{𝑢}_{𝑖}({𝑋}_{𝑖})]$ exists for $𝑖=1,2,\ldots ,𝑛$ , then

\mathbb{E}[𝑌]=\mathbb{E}[{𝑢}_{1}({𝑋}_{1}){𝑢}_{2}({𝑋}_{2})\ldots {𝑢}_{𝑛}({𝑋}_{𝑛})]=\mathbb{E}[{𝑢}_{1}({𝑋}_{1})]\mathbb{E}[{𝑢}_{2}({𝑋}_{2})]\ldots \mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})].

Remark

We shall prove this assuming that the random variables are discrete. The continuous case is similar in that the summations are replaced by integrals.

Proof

We have

\begin{aligned}\mathbb{E}[𝑌] & =\mathbb{E}[{𝑢}_{1}({𝑋}_{1}){𝑢}_{2}({𝑋}_{2})\ldots {𝑢}_{𝑛}({𝑋}_{𝑛})] \\ & =\sum _{{𝑥}_{1}}\cdots \sum _{{𝑥}_{𝑛}}{𝑢}_{1}({𝑥}_{1}){𝑢}_{2}({𝑥}_{2})\ldots {𝑢}_{𝑛}({𝑥}_{𝑛})𝑓({𝑥}_{1},{𝑥}_{2},\ldots ,{𝑥}_{𝑛}).\end{aligned}

Since the random variables are independent, we have

𝑓({𝑥}_{1},{𝑥}_{2},\ldots ,{𝑥}_{𝑛})={𝑓}_{1}({𝑥}_{1}){𝑓}_{2}({𝑥}_{2})\ldots {𝑓}_{𝑛}({𝑥}_{𝑛}).

Therefore,

\mathbb{E}[𝑌]=\sum _{{𝑥}_{1}}\cdots \sum _{{𝑥}_{𝑛-1}}\sum _{{𝑥}_{𝑛}}[{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})][{𝑢}_{𝑛}({𝑥}_{𝑛}){𝑓}_{𝑛}({𝑥}_{𝑛})].

(5.10)

Now, we focus on the inner most summation:

\sum _{{𝑥}_{𝑛}}\underbrace{[{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})]}_{\text{constant w.r.t.}\, {𝑥}_{𝑛}}[{𝑢}_{𝑛}({𝑥}_{𝑛}){𝑓}_{𝑛}({𝑥}_{𝑛})].

Because the expectation

\mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})]=\sum _{{𝑥}_{𝑛}}{𝑢}_{𝑛}({𝑥}_{𝑛}){𝑓}_{𝑛}({𝑥}_{𝑛}).

exists, we may move the constant term outside the summation:

\begin{aligned} & & & \sum _{{𝑥}_{𝑛}}[{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})][{𝑢}_{𝑛}({𝑥}_{𝑛}){𝑓}_{𝑛}({𝑥}_{𝑛})] \\ & = & & [{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})]\sum _{{𝑥}_{𝑛}}[{𝑢}_{𝑛}({𝑥}_{𝑛}){𝑓}_{𝑛}({𝑥}_{𝑛})] \\ & = & & [{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})]\mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})].\end{aligned}

Reviewing (5.10), we may pull out the term $\mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})]$ from the nested summations to get

\mathbb{E}[𝑌]=\mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})]\sum _{{𝑥}_{1}}\cdots \sum _{{𝑥}_{𝑛-1}}[{𝑢}_{1}({𝑥}_{1}){𝑓}_{1}({𝑥}_{1})]\ldots [{𝑢}_{𝑛-1}({𝑥}_{𝑛-1}){𝑓}_{𝑛-1}({𝑥}_{𝑛-1})].

Repeating this process for the remaining summations, we get

\mathbb{E}[𝑌]=\mathbb{E}[{𝑢}_{1}({𝑋}_{1})]\ldots \mathbb{E}[{𝑢}_{𝑛}({𝑋}_{𝑛})].

Theorem 5.4

Let ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ be $𝑛$ random variables (not necessary independent) with respective means ${\mu }_{1},{\mu }_{2},\ldots ,{\mu }_{𝑛}$ and variances ${\sigma }_{1}^{2},{\sigma }_{2}^{2},\ldots ,{\sigma }_{𝑛}^{2}$ . Let $𝑌={\sum }_{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}$ where ${𝑎}_{1},{𝑎}_{2},\ldots ,{𝑎}_{𝑛}\in \mathbb{R}$ are constants. Then

{\mu }_{𝑌}=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖},\quad \text{and}\quad {\sigma }_{𝑌}^{2}=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}+\sum _{1\leq 𝑖,𝑗\leq 𝑛,𝑖\neq 𝑗}{𝑎}_{𝑖}{𝑎}_{𝑗}\operatorname{Cov}({𝑋}_{𝑖},{𝑋}_{𝑗}).

If ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ are independent, then the formula for ${\sigma }_{𝑌}^{2}$ simplifies to

{\sigma }_{𝑌}^{2}=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}.

Proof

The derivation of the formula for ${\mu }_{𝑌}$ only uses the linearity of the expectation. We have

{\mu }_{𝑌}=\mathbb{E}\left[\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}\right]=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}\mathbb{E}[{𝑋}_{𝑖}]=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖}.

To derive the formula for ${\sigma }_{𝑌}^{2}$ , we start with the definition of variance:

\begin{aligned}{\sigma }_{𝑌}^{2}=\mathbb{E}[{(𝑌-{\mu }_{𝑌})}^{2}] & =\mathbb{E}\left[{\left(\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{𝑋}_{𝑖}-\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}{\mu }_{𝑖}\right)}^{2}\right] \\ & =\mathbb{E}\left[{\left(\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}({𝑋}_{𝑖}-{\mu }_{𝑖})\right)}^{2}\right].\end{aligned}

Expanding the square and splitting the resulting double sum into diagonal terms ( $𝑖=𝑗$ ) and off-diagonal terms ( $𝑖\neq 𝑗$ ), we obtain

\begin{aligned}{\sigma }_{𝑌}^{2} & =\mathbb{E}\left[\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{({𝑋}_{𝑖}-{\mu }_{𝑖})}^{2}+\sum _{𝑖\neq 𝑗}{𝑎}_{𝑖}{𝑎}_{𝑗}({𝑋}_{𝑖}-{\mu }_{𝑖})({𝑋}_{𝑗}-{\mu }_{𝑗})\right] \\ & =\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}\mathbb{E}[{({𝑋}_{𝑖}-{\mu }_{𝑖})}^{2}]+\sum _{𝑖\neq 𝑗}{𝑎}_{𝑖}{𝑎}_{𝑗}\mathbb{E}[({𝑋}_{𝑖}-{\mu }_{𝑖})({𝑋}_{𝑗}-{\mu }_{𝑗})] \\ & =\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}+\sum _{𝑖\neq 𝑗}{𝑎}_{𝑖}{𝑎}_{𝑗}\operatorname{Cov}({𝑋}_{𝑖},{𝑋}_{𝑗}).\end{aligned}

If we assume that ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ are independent, then ${𝑋}_{𝑖}$ and ${𝑋}_{𝑗}$ are independent for $𝑖\neq 𝑗$ , we have

\operatorname{Cov}({𝑋}_{𝑖},{𝑋}_{𝑗})=0.

Therefore,

{\sigma }_{𝑌}^{2}=\sum _{𝑖=1}^{𝑛}{𝑎}_{𝑖}^{2}{\sigma }_{𝑖}^{2}.

A random sample of size $𝑛$ from a population (or probability distribution) is a sequence of independent and identically distributed ( $i.i.d.$ ) random variables ${𝑋}_{1},{𝑋}_{2},\ldots ,{𝑋}_{𝑛}$ .

We now consider the means

𝑋=\frac{1}{𝑛}\sum _{𝑖=1}^{𝑛}{𝑋}_{𝑖}.

of a random sample of size $𝑛$ from some distribution with mean $\mu$ and variance ${\sigma }^{2}$ Note that $𝑋$ is itself a random variable. By Theorem 5.4, its mean and variance are given by

{\mu }_{𝑋}=\mu \quad \text{and}\quad {\sigma }_{𝑋}^{2}=\frac{{\sigma }^{2}}{𝑛}.

Any function of a random sample without any unknown parameters is called a statistic. Clearly, $𝑋$ is a statistic and it is also an estimator of distribution mean $\mu$ . Another important statistic is the sample variance

{𝑆}^{2}=\frac{1}{𝑛-1}\sum _{𝑖=1}^{𝑛}{({𝑋}_{𝑖}-𝑋)}^{2}.

and later we will see that it is an estimator of distribution variance ${\sigma }^{2}$ .