Books / Probability and Statistical Inference

The Exponential, Gamma, and Chi-Squared Distributions

Consider a Poisson process with mean rate $\lambda >0$. Let $𝑊$ denote the waiting time, which is also a random quantity, until the first event occurs. We want to find the distribution of $𝑊$. Consider the probability $ℙ(𝑊\leq 𝑤)$ where $𝑤\geq 0$. (If $𝑤<0$, then it is clear that $ℙ(𝑊\leq 𝑤)=0$.) We have

Let $𝑌$ denote the number of occurrences in the interval $[0,𝑤]$. By the given condition, we know that $𝑌∼\operatorname{Pois}(\lambda 𝑤)$. Then the probability of no occurrences in the interval $[0,𝑤]$ is given by the probability mass function of the Poisson distribution evaluated at 0,

Therefore, we have

This is the CDF of $𝑊$, that is,

When $𝑤>0$, the PDF of $𝑊$ is

We often denote $\lambda =1/\theta$ and say that the random variable $𝑋$ has an exponential distribution if its PDF is given by

where the parameter $\theta >0$. In fact, $\theta$ is precisely the mean of the distribution. To see this, we first find its MGF $𝑀(𝑡)$.

We have

In order for the integral to converge, we require that $𝑡<\frac{1}{\theta }$. Then, we have

The first and second derivatives of $𝑀(𝑡)$ are given by

Evaluating at $𝑡=0$, we get

Therefore, the mean and variance of the exponential distribution are

So, if $\lambda$ is the mean number of occurrences in a unit interval, then $\theta =1/\lambda$ is the mean waiting time until the first occurrence, which agrees with our intuition.

For the exponential random variable $𝑋$ with mean $\theta$, its CDF is

Its median $𝑚$ is the solution to $𝐹(𝑚)=1/2$, that is,

which is slightly less than the mean $\theta$.

The PDF and CDF of the exponential distribution are shown in Figure 3.1.

We can see that the PDF of an exponential distribution is monotonically decreasing. To show this, we need to calcuate the derivative of the PDF and show that it is negative. We have

Thereofre, indeed ${𝑓}^{\prime }(𝑥)<0$ for all $𝑥\geq 0$. An immediate consequence is that the mode of the exponential distribution is $𝑥=0$, at which the maximum value of the PDF is $𝑓(0)=1/\theta$.

Apart from the probability $ℙ(𝑋\leq 𝑥)$ for an exponential random variable $𝑋$ with mean $\theta$ that is given by the CDF, it is also useful to note that

There is an interesting interpretation of the result in Exercise 3.2. Suppose a certain type of electronic component has a lifetime that is exponentially distributed with mean life of $500$ hours. If the component has been in operation for $300$ hours, the conditional probability that it will last for another $600$ hours is

due to the result from Exercise 3.2. Equivalently, the probability that this component survives another $600$ hours equals the probability that a new component survives its first $600$ hours: past operation does not make failure in the next interval more or less likely under this model. This is against our intuition, which means that the exponential distribution is probably not a good model for the lifetime of this component.

In a Poisson process with mean rate $\lambda$, we have seen the waiting time of the first occurrence has an exponential distribution. We now generalize this by considering the waiting time $𝑊$ of the $\alpha$-th occurrence. The CDF of $𝑊$ is given by

The event $\{𝑊>𝑤\}$ means that within the interval $[0,𝑤]$, there are less than $\alpha$ occurrences. Thereofore,

where $𝑌$ denotes the number of occurrences in the interval $[0,𝑤]$ which follows a Poisson distribution with mean $\lambda 𝑤$. Hence, $𝐹(𝑤)$ is given by

When $𝑤>0$, taking the derivative of $𝐹(𝑤)$ with respect to $𝑤$, we get the PDF of $𝑊$ as

A PDF of this form is said to be one of the gamma type, and the random variable $𝑊$ is said to have a gamma distribution . For a general gamma distribution, the parameter $\alpha$ is not necessarily a positive integer like we assumed in the above derivation. But to make this generalization, we need to introduce the concept of the gamma function for which this discussion is named after. The gamma function is defined as

If $𝑡>1$, applying integration by parts, we have

The first term is 0 because ${𝑦}^{𝑡-1}{𝑒}^{-𝑦}\mapsto 0$ as $𝑦\mapsto \infty$. The integral in the second term is $\Gamma (𝑡-1)$ by definition. Therefore, we have

For a positive integer $𝑛$, by applying (3.3) repeatedly, we get

The value of $\Gamma (1)$ is easily calculated as

Therefore, we have

This is the same as the definition of the factorial function for positive integers.

By expressing the denominator of the PDF (3.2) as $(\alpha -1)!=\Gamma (\alpha )$ using gamma function, we get

We see that (3.4) is valid for any positive real number $\alpha$. Similar to exponential distributions, we usually denote $\lambda =\frac{1}{\theta }$.

Let us now formally define the gamma distribution and find its characteristics. A random variable $𝑋$ is said to have a gamma distribution with parameters $\alpha >0$ and $\theta >0$ if its PDF is given by

We need to verify that the integral of the PDF on its support is indeed $1$. We have

Applying change of variable $𝑦=𝑥/\theta$, we get

This shows that the integral of the PDF on its support is indeed $1$.

The MGF of the gamma distribution is given by

Enforcing the condition that $𝑡<\frac{1}{\theta }$ and applying change of variable $𝑦=\left(\frac{1}{\theta }-𝑡\right)𝑥$, we obtain

Therefore, we have

The first and second derivatives of $𝑀(𝑡)$ are given by

Evaluating at $𝑡=0$, we get

Hence, the mean and variance of the gamma distribution are

To see how the parameters $\alpha$ and $\theta$ affect the shape of the PDF, we plot several gamma densities in Figure 3.2. In each panel, one parameter is held fixed while the other varies. Note that for a fixed $\theta$, the probability moves right as $\alpha$ increases. The same is true when $\alpha$ is fixed and $\theta$ increases. We have another interesting observation here:

To study the geometric behavior of the gamma PDF and to locate its modes, we compute ${𝑓}^{\prime }(𝑥)$. We treat $\alpha =1$ and $\alpha \neq 1$ separately. For $\alpha =1$, the distribution is $\operatorname{Exp}(\theta )$, and we have already seen that

For $\alpha \neq 1$, we have

To determine the monotonicity of $𝑓$, we examine the sign of ${𝑓}^{\prime }(𝑥)$ and compare with Figure 3.2. We consider $\alpha <1$, $\alpha =1$, and $\alpha >1$ separately. For $\alpha <1$, we have

Hence $𝑓$ is strictly decreasing on $(0,\infty )$, and from the explicit formula for $𝑓(𝑥)$ the PDF diverges as $𝑥\to {0}^{+}$.

For $\alpha =1$, the distribution is $\operatorname{Exp}(\theta )$, which is strictly decreasing; its maximum occurs at $𝑥=0$.

For $\alpha >1$, we have

Thus $𝑓$ increases on $(0,(\alpha -1)\theta )$, attains its maximum at $𝑥=(\alpha -1)\theta$, and decreases on $((\alpha -1)\theta ,\infty )$.

In summary, the mode of $\Gamma (\alpha ,\theta )$ is

We now consider a special case of the gamma distribution that plays an important role in statistics. Let $𝑋$ have a gamma distribution with parameters $\alpha =\frac{𝑟}{2}$ and $\theta =2$ where $𝑟\in {\mathbb{Z}}^{+}$. Then, its PDF is given by

We say that $𝑋$ has a chi-squared distribution with $𝑟$ degrees of freedom and write $𝑋∼{\chi }^{2}(𝑟)$. The practical meaning of degrees of freedom will be explained later. Recall the formulas for the mean, variance, and MGF of the gamma distribution. By substituting $\alpha =𝑟/2$ and $\theta =2$, we get associated formulas for the chi-squared distribution:

And the MGF is given by

Figure 3.3 shows PDFs of chi-squared distributions for several degrees of freedom $𝑟$. Because ${\chi }^{2}(𝑟)=\Gamma (𝑟/2,2)$, the qualitative shapes match those of the gamma densities in Figure 3.2. For modes, substitute $\alpha =𝑟/2$ and $\theta =2$ into the gamma summary above: if $𝑟=1$, then $\alpha <1$ and there is no finite mode in $(0,\infty )$; if $𝑟=2$, then $\alpha =1$ and the mode is at $𝑥=0$ (cf. ${\chi }^{2}(2)=\operatorname{Exp}(2)$); if $𝑟\geq 3$, then $\alpha >1$ and the unique mode is $𝑥=(\alpha -1)\theta =𝑟-2$ (for instance, $𝑟=4$ gives $𝑥=2$). This follows immediately from the gamma discussion; no further derivation is needed.

Tail probabilities and critical values of the chi-squared distribution are central in statistical inference, especially in hypothesis testing. We write ${\chi }_{\alpha }^{2}(𝑟)$ for the upper $\alpha$ critical value of ${\chi }^{2}(𝑟)$ with $𝑟$ degrees of freedom, defined by

Equivalently, ${\chi }_{\alpha }^{2}(𝑟)$ is the $100(1-\alpha )$-th percentile of ${\chi }^{2}(𝑟)$. Accordingly,

Figure 3.4 illustrates the case $𝑟=5$ and $\alpha =0.1$: each shaded tail has area $\alpha$, with cut points ${\chi }_{1-\alpha =0.9}^{2}(5)$ and ${\chi }_{\alpha =0.1}^{2}(5)$ on the horizontal axis.

Exercises