Books / Probability and Statistical Inference

The Central Limit Theorem

Let ${𝑋}_{1},\ldots ,{𝑋}_{𝑛}$ be a random sample of size $𝑛$ from a distribution with mean $\mu$ and variance ${\sigma }^{2}$. By Theorem 5.4, the sample mean $𝑋$ has mean $\mu$ and variance ${\sigma }^{2}/𝑛$. As $𝑛\to \infty$, the variance of $𝑋$ approaches $0$, so $𝑋$ degenerates (or converges in distribution) to a constant random variable with value $\mu$. In a later section, we will introduce the concept of convergence in distribution.

Let

The mean of $𝑊$ is given by

And the variance is

Though the numerator $𝑋-\mu$ of $𝑊$ degenerates to zero, the denominator $\sigma /\sqrt{𝑛}$ spreads out the probability density over a wider range to prevent the degeneration.

We are interested in the distribution of $𝑊$ when $𝑛$ is large. If the random sample is from a normal distribution, then we know that $𝑋∼𝒩(\mu ,{\sigma }^{2}/𝑛)$ and hence $𝑊∼𝒩(0,1)$. But for a general distribution, $𝑊$ needs not to be a standard normal random variable. However, as $𝑛$ increases, the distribution of $𝑊$ approaches the standard normal distribution. That is, the asymptotic behavior of $𝑊$ is like a standard normal random variable. This is the content of the Central Limit Theorem.

In practice, then, for large $𝑛$ we can treat $𝑊$ as approximately $𝒩(0,1)$, or equivalently $𝑋$ as approximately $𝒩(\mu ,{\sigma }^{2}/𝑛)$ and ${\sum }_{𝑖=1}^{𝑛}{𝑋}_{𝑖}$ as approximately $𝒩(𝑛\mu ,𝑛{\sigma }^{2})$. Most often it is the sum that we need, so we usually work with ${\sum }_{𝑖=1}^{𝑛}{𝑋}_{𝑖}$.

Example 5.8

Let ${𝑋}_{1},{𝑋}_{2},{𝑋}_{3},{𝑋}_{4}∼𝒰(0,1)$ be i.i.d. Each ${𝑋}_{𝑖}$ has mean $\mu =1/2$ and variance ${\sigma }^{2}=1/12$, so a sum of $𝑛$ such variables has mean $𝑛\mu$ and variance $𝑛{\sigma }^{2}$. The central limit theorem suggests that for large $𝑛$ the sum is well approximated by $𝒩(𝑛\mu ,𝑛{\sigma }^{2})$. Here we compare the exact PDF of a sum with this normal approximation for $𝑛=2$ and $𝑛=4$.

For $𝑌={𝑋}_{1}+{𝑋}_{2}$ we have $𝑛=2$, so the approximating density is $𝒩(1,1/6)$. From Exercise 5.6, the exact PDF of $𝑌$ is

$${𝑓}_{𝑌}(𝑦)=\begin{cases}𝑦,\quad & 0\leq 𝑦<1 \\ 2-𝑦,\quad & 1\leq 𝑦\leq 2 \\ 0,\quad & \text{otherwise}\end{cases}$$

For $𝑊={𝑋}_{1}+{𝑋}_{2}+{𝑋}_{3}+{𝑋}_{4}$ we have $𝑛=4$, so the approximating density is $𝒩(2,1/3)$. Write ${𝑌}_{1}={𝑋}_{1}+{𝑋}_{2}$ and ${𝑌}_{2}={𝑋}_{3}+{𝑋}_{4}$. Then ${𝑌}_{1}$ and ${𝑌}_{2}$ are independent and each has the same distribution as $𝑌$, and $𝑊={𝑌}_{1}+{𝑌}_{2}$. By Proposition 5.1,

$${𝑓}_{𝑊}(𝑤)=({𝑓}_{𝑌}\ast {𝑓}_{𝑌})(𝑤)=\begin{cases}\frac{{𝑤}^{3}}{6},\quad & 0\leq 𝑤<1 \\ \frac{-3{𝑤}^{3}+12{𝑤}^{2}-12𝑤+4}{6},\quad & 1\leq 𝑤<2 \\ \frac{3{𝑤}^{3}-24{𝑤}^{2}+60𝑤-44}{6},\quad & 2\leq 𝑤<3 \\ \frac{-{𝑤}^{3}+12{𝑤}^{2}-48𝑤+64}{6},\quad & 3\leq 𝑤<4 \\ 0,\quad & \text{otherwise}\end{cases}$$

Figure 5.10 overlays each exact density with its normal approximation. The fit at $𝑛=2$ is only moderate, but at $𝑛=4$ the two curves are already nearly the same, even though the theorem concerns only the limit as $𝑛\to \infty$.