Books Probability and Statistical Inference

Functions of One Random Variable

Let $𝑋$ be a random variable of continuous type. If we consider a function $𝑢(𝑥)$ defined on the support of $𝑋$ , then we can define a new random variable $𝑌=𝑢(𝑋)$ . If we can find its CDF via

𝐺(𝑦)=ℙ(𝑌\leq 𝑦)=ℙ(𝑢(𝑋)\leq 𝑦),

then its PDF $𝑔(𝑦)$ is given by

𝑔(𝑦)={𝐺}^{\prime }(𝑦).

This method is called distribution function technique.

Example 5.1

A spinner is fixed at $(0,1)$ in the $𝑥𝑦$ -plane and determines a ray from that point. Let $𝑊$ be the angle from the downward vertical (the direction of decreasing $𝑦$ ) to the ray, with $𝑊>0$ when the ray tilts toward the positive $𝑥$ -axis, and assume $𝑊$ is uniformly distributed on $[-\pi /2,\pi /2]$ . Let $𝑋$ be the $𝑥$ -coordinate of the point where the ray meets the $𝑥$ -axis.

We have

𝑋=\tan (𝑊).

The CDF of $𝑊$ is

𝐹(𝑤)=ℙ(𝑊\leq 𝑤)=\begin{cases}0,\quad & 𝑤<-\pi /2 \\ \frac{1}{\pi }\left(𝑤+\frac{\pi }{2}\right),\quad & -\pi /2\leq 𝑤<\pi /2 \\ 1,\quad & 𝑤\geq \pi /2\end{cases}.

We are interested in the distribution of $𝑋$ . The CDF of $𝑋$ is

𝐺(𝑥)=ℙ(𝑋\leq 𝑥)=ℙ(\tan (𝑊)\leq 𝑥)=ℙ(𝑊\leq \operatorname{arctan}𝑥)=𝐹(\operatorname{arctan}𝑥).

Since $-\pi /2\leq \operatorname{arctan}𝑥\leq \pi /2$ , applying the second case of $𝐹(𝑤)$ yields

𝐺(𝑥)=\frac{1}{\pi }\left(\operatorname{arctan}𝑥+\frac{\pi }{2}\right).

Differentiating, we get the PDF of $𝑋$ :

𝑔(𝑥)={𝐺}^{\prime }(𝑥)=\frac{1}{\pi (1+{𝑥}^{2})},\quad 𝑥\in \mathbb{R}.

(5.1)

The spinner geometry and the Cauchy density in (5.1) are illustrated in Figure 5.1.

Figure 5.1 Left: spinner at $(0,1)$ , with angle $𝑊$ measured from the downward vertical to the ray and $𝑥$ -intercept $𝑋=\tan 𝑊$ . Right: Cauchy density $𝑔(𝑥)=\frac{1}{\pi (1+{𝑥}^{2})}$ .

Random variables with PDF of the form of (5.1) are said to have a Cauchy distribution.

Exercise 5.1

Show that the mean

𝐸[𝑋]

of a Cauchy random variable is undefined.

Proof

Consider the improper integral

{\int }_{-\infty }^{\infty }\frac{𝑥}{\pi (1+{𝑥}^{2})}\, \mathrm{d}𝑥.

For it to exist, a necessary condition is that

\lim _{𝑏\to \infty }{\int }_{1}^{𝑏}\frac{𝑥}{\pi (1+{𝑥}^{2})}\, \mathrm{d}𝑥

exists as a finite number. We will show that the integral above diverges by comparison test. When $𝑥$ is large enough (in fact, $𝑥\geq 1$ ), we have

\frac{𝑥}{\pi (1+{𝑥}^{2})}=\frac{1}{\pi \left(𝑥+\frac{1}{𝑥}\right)}\geq \frac{1}{\pi (𝑥+𝑥)}=\frac{1}{2\pi 𝑥}.

Since

{\int }_{1}^{𝑏}\frac{1}{2\pi 𝑥}\, \mathrm{d}𝑥={\left[\frac{\ln 𝑥}{2\pi }\right]}_{1}^{𝑏}=\frac{\ln 𝑏}{2\pi }\to \infty \quad \text{as}\, 𝑏\to \infty ,

we conclude that

\lim _{𝑏\to \infty }{\int }_{1}^{𝑏}\frac{𝑥}{\pi (1+{𝑥}^{2})}\, \mathrm{d}𝑥=\infty .

This means that the improper integral is undefined, and hence the mean of a Cauchy random variable is undefined.

Sometimes when the function $𝑢(𝑥)$ has certain properties, we can shortcut the distribution function technique by using the so-called change of variable technique. Let $𝑋$ be a random variable of continuous type with PDF ${𝑓}_{𝑋}(𝑥)$ on the support ${𝑆}_{𝑋}=({𝑐}_{1},{𝑐}_{2})$ . Let $𝑌=𝑢(𝑋)$ where $𝑢(𝑥)$ is a strictly increasing and differentiable function on ${𝑆}_{𝑋}$ . Then $𝑢$ has an inverse function $𝑣={𝑢}^{-1}$ (that is also strictly increasing and differentiable). The support ${𝑆}_{𝑋}$ is mapped to ${𝑆}_{𝑌}=({𝑑}_{1},{𝑑}_{2})$ by $𝑢$ . Here, ${𝑑}_{1}=𝑢({𝑐}_{1})$ and ${𝑑}_{2}=𝑢({𝑐}_{2})$ . Then, the CDF of $𝑌$ is

{𝐹}_{𝑌}(𝑦)=ℙ(𝑌\leq 𝑦)=ℙ(𝑢(𝑋)\leq 𝑦)=ℙ(𝑋\leq {𝑢}^{-1}(𝑦))={𝐹}_{𝑋}[𝑣(𝑦)]

On the support ${𝑆}_{𝑌}=({𝑑}_{1},{𝑑}_{2})$ , we have

{𝐹}_{𝑌}(𝑦)={\int }_{{𝑐}_{1}}^{𝑣(𝑦)}{𝑓}_{𝑋}(𝑥)\, \mathrm{d}𝑥,\quad {𝑑}_{1}<𝑦<{𝑑}_{2}.

Differentiating, we get the PDF of $𝑌$ :

{𝑓}_{𝑌}(𝑦)={𝐹}_{𝑌}\prime (𝑦)={𝑓}_{𝑋}[𝑣(𝑦)]{𝑣}^{\prime }(𝑦).

An analogous result holds for the case when $𝑢(𝑥)$ is decreasing. In that case, we have

{𝑓}_{𝑌}(𝑦)={𝐹}_{𝑌}\prime (𝑦)={𝑓}_{𝑋}[𝑣(𝑦)][-{𝑣}^{\prime }(𝑦)].

Since ${𝑣}^{\prime }(𝑦)>0$ when $𝑢$ is increasing, and ${𝑣}^{\prime }(𝑦)<0$ when $𝑢$ is decreasing, we can summarize the result for both cases as

{𝑓}_{𝑌}(𝑦)={𝑓}_{𝑋}[𝑣(𝑦)]|{𝑣}^{\prime }(𝑦)|.

(5.2)

(5.2) tells us that we can simply plug $𝑣(𝑦)$ into the PDF of $𝑋$ and multiply by $|{𝑣}^{\prime }(𝑦)|$ to get the PDF of $𝑌$ , bypassing the determination of the CDF of $𝑌$ .

Exercise 5.2

Let $𝑋$ have the PDF

𝑓(𝑥)=3{(1-𝑥)}^{2},\quad 0<𝑥<1.

Find the PDF of $𝑌=𝑢(𝑋)={(1-𝑋)}^{3}$ .

Solution

$𝑢(𝑥)={(1-𝑥)}^{3}$ is strictly decreasing and differentiable in $(0,1)$ . Its inverse is

𝑣(𝑦)=1-{𝑦}^{\frac{1}{3}},\quad 0<𝑦<1.

Differentiate,

{𝑣}^{\prime }(𝑦)=-\frac{1}{3}{𝑦}^{-\frac{2}{3}}.

Therefore, the PDF of $𝑌$ is

𝑔(𝑦)=𝑓[𝑣(𝑦)]|{𝑣}^{\prime }(𝑦)|=1,\quad 0<𝑦<1.

This implies that $𝑌$ has a uniform distribution on $(0,1)$ , i.e., $𝑌∼𝒰(0,1)$ .

Formula (5.2) requires a one-to-one transformation. When $𝑢$ is not injective—for example, $𝑋={𝑌}^{2}$ sends both $𝑦$ and $-𝑦$ to the same value—we must account for every preimage, not just one branch.

Exercise 5.3

Let $𝑎>0$ , where $𝑎$ may be $\infty$ . Let $𝑋$ have support $(0,𝑎)$ and PDF ${𝑓}_{𝑋}$ . Let $𝑌$ have support $\left(-\sqrt{𝑎},\sqrt{𝑎}\right)$ (when $𝑎=\infty$ , read this as $(-\infty ,\infty )$ ). Suppose that $𝑌$ and $-𝑌$ have the same distribution and that $𝑋={𝑌}^{2}$ . Show that

{𝑓}_{𝑌}(𝑦)=|𝑦|{𝑓}_{𝑋}({𝑦}^{2}),\quad -\sqrt{𝑎}<𝑦<\sqrt{𝑎}.

Remark

Here $𝑋=𝑣(𝑌)={𝑌}^{2}$ , so ${𝑣}^{\prime }(𝑦)=2𝑦$ . Applying (5.2) would give

{𝑓}_{𝑌}(𝑦)={𝑓}_{𝑋}[𝑣(𝑦)]|{𝑣}^{\prime }(𝑦)|=2𝑦{𝑓}_{𝑋}({𝑦}^{2}),

which cannot hold for all $𝑦$ when $𝑌$ is symmetric about $0$ . The transformation is two-to-one on $\left(-\sqrt{𝑎},\sqrt{𝑎}\right)$ : each $𝑥\in (0,𝑎)$ arises from both $𝑦=\sqrt{𝑥}$ and $𝑦=-\sqrt{𝑥}$ .

Proof

Symmetry of $𝑌$ gives ${𝑓}_{𝑌}(-𝑦)={𝑓}_{𝑌}(𝑦)$ and ${𝐹}_{𝑌}(-𝑦)=1-{𝐹}_{𝑌}(𝑦)$ for all $𝑦\in \left(-\sqrt{𝑎},\sqrt{𝑎}\right)$ . Because $𝑋={𝑌}^{2}$ , probabilities for ${𝑌}^{2}$ involve values of $𝑌$ on both sides of $0$ . We treat $𝑦<0$ and $𝑦\geq 0$ separately; in each case ${𝑦}^{2}\in (0,𝑎)$ , so ${𝐹}_{𝑋}({𝑦}^{2})$ is defined.

When $𝑦<0$ , we have $𝑌\leq 𝑦$ if and only if $𝑌\geq -𝑦$ , so by symmetry

ℙ(𝑌\leq 𝑦)=ℙ(𝑌\geq -𝑦).

Therefore,

ℙ({𝑌}^{2}\geq {𝑦}^{2})=ℙ(𝑌\leq 𝑦)+ℙ(𝑌\geq -𝑦)=2ℙ(𝑌\leq 𝑦)=2{𝐹}_{𝑌}(𝑦).

It follows that

{𝐹}_{𝑌}(𝑦)=\frac{1}{2}ℙ({𝑌}^{2}\geq {𝑦}^{2})=\frac{1}{2}-\frac{1}{2}ℙ({𝑌}^{2}\leq {𝑦}^{2})=\frac{1}{2}-\frac{1}{2}{𝐹}_{𝑋}({𝑦}^{2}).

Differentiating gives, for $𝑦<0$ ,

{𝑓}_{𝑌}(𝑦)={𝐹}_{𝑌}\prime (𝑦)=-\frac{1}{2}{𝑓}_{𝑋}({𝑦}^{2})\cdot 2𝑦=-𝑦{𝑓}_{𝑋}({𝑦}^{2}).

When $𝑦\geq 0$ , we have

ℙ({𝑌}^{2}\leq {𝑦}^{2})=ℙ(-𝑦\leq 𝑌\leq 𝑦).

By symmetry,

ℙ(-𝑦\leq 𝑌\leq 𝑦)=2ℙ(0\leq 𝑌\leq 𝑦).

Splitting $ℙ(𝑌\leq 𝑦)$ at $-𝑦$ ,

{𝐹}_{𝑌}(𝑦)=ℙ(𝑌\leq 𝑦)=ℙ(𝑌<-𝑦)+ℙ(-𝑦\leq 𝑌\leq 𝑦)=ℙ(𝑌<-𝑦)+ℙ({𝑌}^{2}\leq {𝑦}^{2}).

Since $-𝑦\leq 0$ , the case $𝑦<0$ already gives $ℙ(𝑌<-𝑦)=\frac{1}{2}-\frac{1}{2}{𝐹}_{𝑋}({𝑦}^{2})$ . Hence

\begin{aligned}{𝐹}_{𝑌}(𝑦) & =\frac{1}{2}-\frac{1}{2}{𝐹}_{𝑋}({𝑦}^{2})+\frac{1}{2}ℙ({𝑌}^{2}\leq {𝑦}^{2}) \\ & =\frac{1}{2}-\frac{1}{2}{𝐹}_{𝑋}({𝑦}^{2})+{𝐹}_{𝑋}({𝑦}^{2}) \\ & =\frac{1}{2}+\frac{1}{2}{𝐹}_{𝑋}({𝑦}^{2}).\end{aligned}

Differentiating gives, for $𝑦\geq 0$ ,

{𝑓}_{𝑌}(𝑦)={𝐹}_{𝑌}\prime (𝑦)=\frac{1}{2}{𝑓}_{𝑋}({𝑦}^{2})\cdot 2𝑦=𝑦{𝑓}_{𝑋}({𝑦}^{2}).

Combining the two cases,

{𝑓}_{𝑌}(𝑦)=\begin{cases}-𝑦{𝑓}_{𝑋}({𝑦}^{2}),\quad & 𝑦<0 \\ 𝑦{𝑓}_{𝑋}({𝑦}^{2}),\quad & 𝑦\geq 0\end{cases}=|𝑦|{𝑓}_{𝑋}({𝑦}^{2}).

The following theorem provides a way to generate any continuous type random variable $𝑋$ from a uniform random variable $𝑌∼𝒰(0,1)$ and the CDF of $𝑋$ .

Theorem 5.1

Let

𝑌∼𝒰(0,1)

. Let

𝐹

have the properties of a CDF of the continuous type with

𝐹(𝑎)=0

𝐹(𝑏)=1

, and suppose that

𝐹

is strictly increasing in

(𝑎,𝑏)

where

𝑎

and

𝑏

may be

-\infty

and

\infty

, respectively. Then the random variable

𝑋

defined by

𝑋={𝐹}^{-1}(𝑌)

is a continuous type random variable with the CDF

𝐹

Proof

The CDF of $𝑋$ is given by

{𝐹}_{𝑋}(𝑥)=ℙ(𝑋\leq 𝑥)=ℙ({𝐹}^{-1}(𝑌)\leq 𝑥).

Since the inverse ${𝐹}^{-1}$ is well-defined, the following two events are the same:

\{{𝐹}^{-1}(𝑌)\leq 𝑥\}=\{𝑌\leq 𝐹(𝑥)\}.

Thus,

{𝐹}_{𝑋}(𝑥)=ℙ(𝑌\leq 𝐹(𝑥))=𝐺[𝐹(𝑥)]

where $𝐺(𝑦)$ is the CDF of $𝑌$ , and it is given by

𝐺(𝑦)=\begin{cases}0,\quad & 𝑦<0 \\ 𝑦,\quad & 0\leq 𝑦<1 \\ 1,\quad & 𝑦\geq 1\end{cases}

Since $0\leq 𝐹(𝑦)\leq 1$ , we have

{𝐹}_{𝑋}(𝑥)=𝐺[𝐹(𝑥)]=𝐹(𝑥).

That is, the CDF of $𝑋$ is $𝐹$ .

In practice, simulation via Theorem 5.1 is less straightforward than the theorem makes it look: the recipe ${𝐹}^{-1}(𝑌)$ is simple on paper, but the bottleneck is evaluating the inverse CDF ${𝐹}^{-1}$ . Closed forms are rare, so one usually resorts to numerical methods for quantiles.

Consider drawing a standard normal random variable. Neither $\Phi$ nor ${\Phi }^{-1}$ is available in closed elementary form. Later we use the Box–Muller transformation for that purpose.

To illustrate Theorem 5.1 with a transparent inverse, we take the Cauchy distribution, whose quantile map is an elementary trigonometric function (after a linear change of variables, it is $\tan$ ).

Example 5.2

Let’s return to Example 5.1. Suppose $𝑊∼𝒰\left(-\frac{\pi }{2},\frac{\pi }{2}\right)$ and define

𝑋=\tan (𝑊).

Now suppose $𝑌∼𝒰(0,1)$ . The CDF of $𝑊$ is ${𝐹}_{𝑊}(𝑤)=\frac{𝑤+\frac{\pi }{2}}{\pi }$ for $-\frac{\pi }{2}\leq 𝑤\leq \frac{\pi }{2}$ , so we may write $𝑊=\pi 𝑌-\frac{\pi }{2}$ . Then $𝑋$ as a function of $𝑌$ is

𝑋=\tan \left(\pi 𝑌-\frac{\pi }{2}\right).

(5.3)

In Example 5.1 we showed that $𝑋$ has PDF

𝑓(𝑥)=\frac{1}{\pi (1+{𝑥}^{2})},\quad 𝑥\in \mathbb{R},

and CDF

𝐹(𝑥)={\int }_{-\infty }^{𝑥}\frac{1}{\pi (1+{𝑡}^{2})}\, \mathrm{d}𝑡=\frac{1}{\pi }\operatorname{arctan}𝑥+\frac{1}{2},\quad 𝑥\in \mathbb{R}.

The inverse is

{𝐹}^{-1}(𝑦)=\tan \left(\pi 𝑦-\frac{\pi }{2}\right),\quad 0<𝑦<1.

Therefore, by Theorem 5.1,

𝑋={𝐹}^{-1}(𝑌)=\tan \left(\pi 𝑌-\frac{\pi }{2}\right),

which matches (5.3).

Figure 5.2 illustrates the quantile construction: independent draws $𝑌∼𝒰(0,1)$ are mapped through ${𝐹}^{-1}$ to obtain draws of $𝑋$ , with a histogram of the simulated values compared to the Cauchy density.

Figure 5.2 Histogram of simulated Cauchy draws $𝑋={𝐹}^{-1}(𝑌)$ ( ${10}^{4}$ samples, $𝑌∼𝒰(0,1)$ , $50$ bins).

The following theorem is the converse of Theorem 5.1.

Theorem 5.2

Let

𝑋

be a random variable of continuous type with a strictly increasing CDF

𝐹

defined on support

(𝑎,𝑏)

where

𝑎

and

𝑏

may be

-\infty

and

\infty

, respectively. Define

𝑌=𝐹(𝑋)

. We have

𝑌∼𝒰(0,1)

Proof

Consider the CDF of $𝑌$ . We have

{𝐹}_{𝑌}(𝑦)=ℙ(𝑌\leq 𝑦)=ℙ(𝐹(𝑋)\leq 𝑦),\quad 0<𝑦<1.

Because $𝐹$ is strictly increasing, the event $\{𝐹(𝑋)\leq 𝑦\}$ is equivalent to $\{𝑋\leq {𝐹}^{-1}(𝑦)\}$ . It then follows that

{𝐹}_{𝑌}(𝑦)=ℙ(𝑋\leq {𝐹}^{-1}(𝑦))=𝐹({𝐹}^{-1}(𝑦))=𝑦,\quad 0<𝑦<1.

This is exactly the CDF of $𝒰(0,1)$ . Therefore, $𝑌∼𝒰(0,1)$ .

Exercises

Exercise 5.4

Let

𝑋∼\operatorname{Gamma}(\alpha ,\theta )

. Show that

𝑎𝑋∼\operatorname{Gamma}(\alpha ,𝑎\theta )

for

𝑎>0

Proof

Let $𝑌=𝑎𝑋$ . We have $𝑋=𝑣(𝑌)=𝑌/𝑎$ . Applying the change-of-variable technique, we get the PDF of $𝑌$ :

\begin{aligned}𝑔(𝑦)=𝑓(𝑣(𝑦))|{𝑣}^{\prime }(𝑦)|=𝑓\left(\frac{𝑦}{𝑎}\right)\cdot \frac{1}{𝑎} & =\frac{1}{\Gamma (\alpha ){\theta }^{\alpha }}{\left(\frac{𝑦}{𝑎}\right)}^{\alpha -1}{𝑒}^{-\frac{𝑦}{𝑎\theta }}\cdot \frac{1}{𝑎} \\ & =\frac{1}{\Gamma (\alpha ){\theta }^{\alpha }}\cdot \frac{1}{{𝑎}^{\alpha -1}}\cdot \frac{1}{𝑎}\cdot {𝑦}^{\alpha -1}{𝑒}^{-\frac{𝑦}{𝑎\theta }} \\ & =\frac{1}{\Gamma (\alpha ){(𝑎\theta )}^{\alpha }}{𝑦}^{\alpha -1}{𝑒}^{-\frac{𝑦}{𝑎\theta }}.\end{aligned}

Therefore, $𝑌∼\operatorname{Gamma}(\alpha ,𝑎\theta )$ .