Books / Probability and Statistical Inference

Transformation of Two Random Variables

We have considered the transformation of a single random variable $𝑋$ with PDF $𝑓(𝑥)$. In continuous case, if function $𝑢(𝑥)$ is strictly increasing (or decreasing) and differentiable on the support ${𝑆}_{𝑋}=(𝑎,𝑏)$, then the PDF of $𝑌=𝑢(𝑋)$ is given by

where $𝑣(𝑦)$ is the inverse of $𝑢(𝑥)$ and the support ${𝑆}_{𝑌}=(𝑐,𝑑)$ of $𝑌$ is transformed from ${𝑆}_{𝑋}$ through $𝑢(𝑥)$.

We now consider the transformation of two random variables ${𝑋}_{1}$ and ${𝑋}_{2}$ of continuous type with joint PDF $𝑓({𝑥}_{1},{𝑥}_{2})$ on support ${𝑆}_{𝑋}$. Suppose that the system of functions

maps points in ${𝑆}_{𝑋}$ one-to-one and onto points in ${𝑆}_{𝑌}$. Let random variables ${𝑌}_{1}$ and ${𝑌}_{2}$ be defined by ${𝑌}_{1}={𝑢}_{1}({𝑋}_{1},{𝑋}_{2})$ and ${𝑌}_{2}={𝑢}_{2}({𝑋}_{1},{𝑋}_{2})$. Then the joint PDF of ${𝑌}_{1}$ and ${𝑌}_{2}$ is given by

where $({𝑣}_{1},{𝑣}_{2})$ is the inverse of $({𝑢}_{1},{𝑢}_{2})$ and

is the Jacobian determinant of $({𝑣}_{1},{𝑣}_{2})$ with respect to $({𝑦}_{1},{𝑦}_{2})$.

Example 5.3

Let ${𝑋}_{1}$ and ${𝑋}_{2}$ have the joint PDF

$$𝑓({𝑥}_{1},{𝑥}_{2})=2,\quad 0<{𝑥}_{1}<{𝑥}_{2}<1.$$

Let ${𝑌}_{1}={𝑋}_{1}/{𝑋}_{2}$ and ${𝑌}_{2}={𝑋}_{2}$. That is, ${𝑢}_{1}({𝑥}_{1},{𝑥}_{2})={𝑥}_{1}/{𝑥}_{2}$ and ${𝑢}_{2}({𝑥}_{1},{𝑥}_{2})={𝑥}_{2}$. The support ${𝑆}_{𝑋}=\{({𝑥}_{1},{𝑥}_{2}):0<{𝑥}_{1}<{𝑥}_{2}<1\}$ is transformed to ${𝑆}_{𝑌}=\{({𝑦}_{1},{𝑦}_{2}):0<{𝑦}_{1}<1,0<{𝑦}_{2}<1\}$ through $({𝑢}_{1},{𝑢}_{2})$.

Figure 5.3 shows ${𝑆}_{𝑋}$ as the upper triangle of the unit square (above the diagonal ${𝑥}_{1}={𝑥}_{2}$) and ${𝑆}_{𝑌}$ as the full open unit square. Evidently, ${𝑋}_{1}$ and ${𝑋}_{2}$ are dependent. However, we will show that ${𝑌}_{1}$ and ${𝑌}_{2}$ are independent.

Note

This is not a substitute for a careful argument; it is only a mental draft for guessing ${𝑆}_{𝑌}$ before you check inequalities and bijectivity properly. In your head, trace where each piece of the boundary of ${𝑆}_{𝑋}$ goes under $({𝑢}_{1},{𝑢}_{2})$: segments suggest edges of ${𝑆}_{𝑌}$, and corners suggest corners (sometimes several corners of ${𝑆}_{𝑋}$ collapse to one point of ${𝑆}_{𝑌}$, or conversely a single vertex of ${𝑆}_{𝑋}$ traces out part of an edge of ${𝑆}_{𝑌}$). Then decide which side of that boundary is the plausible interior for a one-to-one map from ${𝑆}_{𝑋}$.

The inverse map is given by

$${𝑣}_{1}({𝑦}_{1},{𝑦}_{2})={𝑦}_{1}{𝑦}_{2},\quad {𝑣}_{2}({𝑦}_{1},{𝑦}_{2})={𝑦}_{2}.$$

Compute the Jacobian:

$$𝐽=\begin{vmatrix}\frac{\partial {𝑣}_{1}}{\partial {𝑦}_{1}} & \frac{\partial {𝑣}_{1}}{\partial {𝑦}_{2}} \\ \frac{\partial {𝑣}_{2}}{\partial {𝑦}_{1}} & \frac{\partial {𝑣}_{2}}{\partial {𝑦}_{2}}\end{vmatrix}=\begin{vmatrix}{𝑦}_{2} & {𝑦}_{1} \\ 0 & 1\end{vmatrix}={𝑦}_{2}.$$

Thus, the joint PDF of ${𝑌}_{1}$ and ${𝑌}_{2}$ is given by

$$𝑔({𝑦}_{1},{𝑦}_{2})=𝑓({𝑣}_{1}({𝑦}_{1},{𝑦}_{2}),{𝑣}_{2}({𝑦}_{1},{𝑦}_{2}))|𝐽|=2{𝑦}_{2},\quad 0<{𝑦}_{1}<1,0<{𝑦}_{2}<1.$$

We then compute the marginal PDFs of ${𝑌}_{1}$ and ${𝑌}_{2}$. For ${𝑌}_{1}$, we have

$${𝑔}_{{𝑌}_{1}}({𝑦}_{1})={\int }_{0}^{1}𝑔({𝑦}_{1},{𝑦}_{2})\, \mathrm{d}{𝑦}_{2}={\int }_{0}^{1}2{𝑦}_{2}\, \mathrm{d}{𝑦}_{2}={[{𝑦}_{2}^{2}]}_{0}^{1}=1,\quad 0<{𝑦}_{1}<1.$$

And for ${𝑌}_{2}$, we have

$${𝑔}_{{𝑌}_{2}}({𝑦}_{2})={\int }_{0}^{1}𝑔({𝑦}_{1},{𝑦}_{2})\, \mathrm{d}{𝑦}_{1}={\int }_{0}^{1}2{𝑦}_{2}\, \mathrm{d}{𝑦}_{1}=2{𝑦}_{2},\quad 0<{𝑦}_{2}<1.$$

Note that

$$𝑔({𝑦}_{1},{𝑦}_{2})=2{𝑦}_{2}={𝑔}_{{𝑌}_{1}}({𝑦}_{1}){𝑔}_{{𝑌}_{2}}({𝑦}_{2}).$$

This means that ${𝑌}_{1}$ and ${𝑌}_{2}$ are independent.

With the techniques of transformation of two random variables, we will develop two important distributions in the following examples.

Formally, if a random variable $𝑋$ has a PDF with the form

then we say $𝑋$ has a beta distribution with parameters $\alpha$ and $\beta$.

We may also develop the beta distribution by considering the function of the form

where $𝑐$ is a normalizing constant to ensure the $𝑓$ integrates to $1$ so that $𝑓$ may be a valid PDF. The beta function is defined as

Thus, the beta PDF can also be written as

Figure 5.4 shows several beta PDFs for representative parameter pairs $(\alpha ,\beta )$.

Read the picture loosely: the factor ${𝑥}^{\alpha -1}$ pulls probability toward the left endpoint $𝑥=0$ when $\alpha$ is small (the curve may spike there) and pushes it away from $0$ when $\alpha$ is large, leaving room for a hump in the interior. The factor ${(1-𝑥)}^{\beta -1}$ does the analogous job at $𝑥=1$.

When $\alpha =\beta$, the two sides balance, so the density is symmetric about $0.5$, as with $(2,2)$ and $(5,5)$; raising both parameters together keeps that balance but concentrates more mass around the middle—a bell-like profile where middling proportions are typical.

When $\beta >\alpha$, the right endpoint wins the tug-of-war, so the hump slides toward $1$ (compare $(2,5)$ with $(5,2)$).

When both parameters are below $1$, both endpoints win at once: the middle thins out and the mass sits near $0$ and near $1$, giving a U-shaped curve that favors very small or very large values over middling ones.

When only one parameter is below $1$, that side of the interval dominates on its own—for instance, $(2,0.5)$ hugs the right edge $𝑥=1$.

In the gamma-ratio story in Example 5.4, $\alpha$ and $\beta$ came from the two underlying gamma shapes; qualitatively, whichever shape parameter is larger tilts ${𝑌}_{1}={𝑋}_{1}/({𝑋}_{1}+{𝑋}_{2})$ toward the corresponding component.

Example 5.5

Let $𝑈$ and $𝑉$ be two independent chi-squared random variables with ${𝑟}_{1}$ and ${𝑟}_{2}$ degrees of freedom, respectively. We are interested in the distribution of

$$𝐹=\frac{𝑈/{𝑟}_{1}}{𝑉/{𝑟}_{2}}.$$

To prevent the confusion of the random variable $𝐹$ and the notation of CDF $𝐹$, we will use the letter $𝑊$ to denote the random variable in the rest of this example. That is,

$$𝑊=\frac{𝑈/{𝑟}_{1}}{𝑉/{𝑟}_{2}}.$$

Recall the formula for the PDF of ${\chi }^{2}(𝑟)$:

$$𝑓(𝑥)=\frac{1}{\Gamma (𝑟/2){2}^{𝑟/2}}{𝑥}^{𝑟/2-1}{𝑒}^{-𝑥/2},\quad 𝑥>0.$$

Because $𝑈$ and $𝑉$ are independent, their joint PDF is given by

$$𝑔(𝑢,𝑣)=\frac{1}{\Gamma ({𝑟}_{1}/2){2}^{{𝑟}_{1}/2}}{𝑢}^{{𝑟}_{1}/2-1}{𝑒}^{-𝑢/2}\cdot \frac{1}{\Gamma ({𝑟}_{2}/2){2}^{{𝑟}_{2}/2}}{𝑣}^{{𝑟}_{2}/2-1}{𝑒}^{-𝑣/2},\quad 𝑢>0,𝑣>0.$$

We want to find the CDF $𝐹(𝑤)$ of $𝑊$. We have

$$𝐹(𝑤)=ℙ(𝑊\leq 𝑤)=ℙ\left(\frac{𝑈/{𝑟}_{1}}{𝑉/{𝑟}_{2}}\leq 𝑤\right)=ℙ(𝑈\leq ({𝑟}_{1}/{𝑟}_{2})𝑤𝑉).$$

The event $\{𝑈\leq ({𝑟}_{1}/{𝑟}_{2})𝑤𝑉\}$ yields a region in the $𝑢𝑣$-plane, displayed in Figure 5.5, that we should integrate over to find the CDF.

Thus,

$$𝐹(𝑤)={\int }_{0}^{\infty }{\int }_{0}^{({𝑟}_{1}/{𝑟}_{2})𝑤𝑣}𝑔(𝑢,𝑣)\, \mathrm{d}𝑢\, \mathrm{d}𝑣.$$

Differentiate with respect to $𝑤$:

$$\begin{aligned}𝑓(𝑤)={𝐹}^{\prime }(𝑤) & =\frac{\mathrm{d}}{\mathrm{d}𝑤}{\int }_{0}^{\infty }{\int }_{0}^{({𝑟}_{1}/{𝑟}_{2})𝑤𝑣}𝑔(𝑢,𝑣)\, \mathrm{d}𝑢\, \mathrm{d}𝑣 \\ & ={\int }_{0}^{\infty }\frac{\mathrm{d}}{\mathrm{d}𝑤}{\int }_{0}^{({𝑟}_{1}/{𝑟}_{2})𝑤𝑣}𝑔(𝑢,𝑣)\, \mathrm{d}𝑢\, \mathrm{d}𝑣 \\ & ={\int }_{0}^{\infty }({𝑟}_{1}/{𝑟}_{2})𝑣\cdot 𝑔(({𝑟}_{1}/{𝑟}_{2})𝑤𝑣,𝑣)\, \mathrm{d}𝑣\end{aligned}$$

(5.8)

Moving terms independent of $𝑣$ out of the integral, we get

$$\begin{aligned}𝑓(𝑤)= & {({𝑟}_{1}/{𝑟}_{2})}^{{𝑟}_{1}/2}\frac{{𝑤}^{{𝑟}_{1}/2-1}}{\Gamma ({𝑟}_{1}/2)\Gamma ({𝑟}_{2}/2){2}^{({𝑟}_{1}+{𝑟}_{2})/2}} \\ & \cdot \underbrace{{\int }_{0}^{\infty }{𝑣}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}-1}\exp \left[-\frac{[({𝑟}_{1}/{𝑟}_{2})𝑤+1]𝑣}{2}\right]\, \mathrm{d}𝑣}_{=𝐼}.\end{aligned}$$

The rest of the work is to evaluate the integral $𝐼$. We note that the integrand of $𝐼$ is similar to the form of a chi-squared PDF. Substituting $𝑣=𝑡/[({𝑟}_{1}/{𝑟}_{2})𝑤+1]$ into $𝐼$, we get

$$\begin{aligned}𝐼 & =\frac{1}{{[({𝑟}_{1}/{𝑟}_{2})𝑤+1]}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}}}{\int }_{0}^{\infty }\underbrace{{𝑡}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}-1}{𝑒}^{-𝑡/2}}_{\text{Compare to the PDF of}\, {\chi }^{2}({𝑟}_{1}+{𝑟}_{2})}\, \mathrm{d}𝑡 \\ & =\frac{1}{{[({𝑟}_{1}/{𝑟}_{2})𝑤+1]}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}}}\Gamma \left(\frac{{𝑟}_{1}+{𝑟}_{2}}{2}\right){2}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}}.\end{aligned}$$

Plugging $𝐼$ back into (5.8) yields

$$𝑓(𝑤)={({𝑟}_{1}/{𝑟}_{2})}^{{𝑟}_{1}/2}\frac{\Gamma \left(\frac{{𝑟}_{1}+{𝑟}_{2}}{2}\right)}{\Gamma ({𝑟}_{1}/2)\Gamma ({𝑟}_{2}/2)}\cdot \frac{{𝑤}^{{𝑟}_{1}/2-1}}{{[({𝑟}_{1}/{𝑟}_{2})𝑤+1]}^{\frac{{𝑟}_{1}+{𝑟}_{2}}{2}}},\quad 𝑤>0.$$

(5.9)

Suppose that ${𝑟}_{1},{𝑟}_{2}\in {\mathbb{Z}}^{+}$. We say random variable $𝑊$ has a $𝐹$-distribution with ${𝑟}_{1}$ and ${𝑟}_{2}$ degrees of freedom if its PDF is given by (5.9).

Graphs of the $𝐹$-distribution PDFs for representative parameter pairs $({𝑟}_{1},{𝑟}_{2})$ are shown in Figure 5.6.

When ${𝑟}_{1}=2$, every PDF in Figure 5.6 decreases from $(0,1)$, regardless of ${𝑟}_{2}$. We can prove this as an exercise.

Recall the construction of an $𝐹$ random variable:

where $𝑈$ and $𝑉$ are independent chi-squared random variables with ${𝑟}_{1}$ and ${𝑟}_{2}$ degrees of freedom, respectively. Note that if we take the reciprocal of $𝐹$, we get

which also follows an $𝐹$-distribution but with ${𝑟}_{1}$ and ${𝑟}_{2}$ swapped, i.e. $𝐹({𝑟}_{2},{𝑟}_{1})$.

For a right-tail probability of $\alpha$, we write

Of course, for a left-tail probability of $\alpha$, we have

For the random variable $1/𝑊$, we have

On the other hand, Since $1/𝑊∼𝐹({𝑟}_{2},{𝑟}_{1})$,

It then follows that

Next we introduce an important simulation tool, the Box-Muller transformation, whose main role is to turn independent uniforms into independent standard normal random variables (Gaussian noise, in applications).

Let ${𝑈}_{1}$ and ${𝑈}_{2}$ be independent random variables, each uniformly distributed on $(0,1)$. Define

Geometrically, $\sqrt{-2\ln {𝑈}_{1}}$, which ranges over $(0,\infty )$, is the Euclidean radius of $({𝑍}_{1},{𝑍}_{2})$ in the $({𝑧}_{1},{𝑧}_{2})$-plane, while $\cos (2\pi {𝑈}_{2})$ and $\sin (2\pi {𝑈}_{2})$ fix the direction. Note that the polar angle $2\pi {𝑈}_{2}$ lies in $(0,2\pi )$. Intuitively, the open unit square—the support of $({𝑈}_{1},{𝑈}_{2})$—is mapped onto the whole $({𝑧}_{1},{𝑧}_{2})$-plane with the origin and the positive ${𝑧}_{1}$-axis removed. We have

and, whenever ${𝑈}_{2}$ is neither $\pi /2$ nor $3\pi /2$,

We now consider the inverse transformation. Clearly

To obtain a formula for ${𝑈}_{2}$, we must be careful with $\operatorname{arctan}$ and quadrant ambiguities, so we treat the cases by quadrant. We have

The Jacobian is given by

Since the joint PDF of ${𝑈}_{1}$ and ${𝑈}_{2}$ is

the change-of-variables formula gives the joint PDF of ${𝑍}_{1}$ and ${𝑍}_{2}$ as

The same expression holds on ${\mathbb{R}}^{2}$ with the half-line $\{({𝑧}_{1},{𝑧}_{2}):{𝑧}_{1}\geq 0,{𝑧}_{2}=0\}$ removed, where the polar-angle map is not differentiable; that set has Lebesgue measure zero, so it does not affect the distribution. Thus ${𝑍}_{1}$ and ${𝑍}_{2}$ are independent, and each is standard normal.

As noted earlier, inverse-transform sampling via Theorem 5.1 is awkward for standard normals because ${\Phi }^{-1}$ has no convenient closed form. The Box-Muller transformation avoids that obstacle. Let ${𝑈}_{1}$ and ${𝑈}_{2}$ be independent $𝒰(0,1)$ random variables and set

or

Then $𝑍∼𝒩(0,1)$, as we showed above. Figure 5.7 gives a histogram of simulated draws of $𝑍=\sqrt{-2\ln {𝑈}_{1}}\cos (2\pi {𝑈}_{2})$, which tracks the $𝒩(0,1)$ density closely.

A particularly useful application is finding the PDF of the sum of two independent random variables, a problem that arises constantly in practice. The answer turns out to be remarkably compact: it is simply the convolution of the two PDFs.

For the proof, we use the constructive approach: define $𝑊=𝑋$ alongside $𝑍=𝑋+𝑌$. This reframes the problem as a bivariate transformation $(𝑋,𝑌)\to (𝑍,𝑊)$, so the Jacobian method of this section applies directly.

Solution

Since ${𝑋}_{1}$ and ${𝑋}_{2}$ are i.i.d., each has PDF

$$𝑓(𝑥)=\begin{cases}1,\quad & 0<𝑥<1 \\ 0,\quad & \text{otherwise}\end{cases}$$

By Proposition 5.1, the PDF of $𝑌$ is the convolution

$$𝑔(𝑦)=(𝑓\ast 𝑓)(𝑦)={\int }_{-\infty }^{\infty }𝑓(𝑤)𝑓(𝑦-𝑤)\, \mathrm{d}𝑤.$$

Note

A useful picture for evaluating a convolution is to fix one function and slide the other across the $𝑤$-axis. Here $𝑓(𝑤)$ stays fixed on $(0,1)$ while $𝑓(𝑦-𝑤)$ acts as the sliding window; as $𝑦$ varies, the window shifts. Only values of $𝑤$ where both factors are nonzero contribute to the integral.

Figure 5.8 illustrates the sliding-window picture for the two nontrivial overlap cases.

If $𝑦<0$, the supports of $𝑓(𝑤)$ and $𝑓(𝑦-𝑤)$ do not overlap, so $𝑔(𝑦)=0$.

If $0\leq 𝑦<1$, both factors equal $1$ precisely when $0<𝑤<𝑦$, so

$$\begin{aligned}𝑔(𝑦) & ={\int }_{-\infty }^{\infty }𝑓(𝑤)𝑓(𝑦-𝑤)\, \mathrm{d}𝑤 \\ & ={\int }_{0}^{𝑦}𝑓(𝑤)𝑓(𝑦-𝑤)\, \mathrm{d}𝑤 \\ & ={\int }_{0}^{𝑦}1\cdot 1\, \mathrm{d}𝑤 \\ & =𝑦.\end{aligned}$$

If $1\leq 𝑦\leq 2$, overlap occurs when $𝑦-1<𝑤<1$, so

$$\begin{aligned}𝑔(𝑦) & ={\int }_{-\infty }^{\infty }𝑓(𝑤)𝑓(𝑦-𝑤)\, \mathrm{d}𝑤 \\ & ={\int }_{𝑦-1}^{1}𝑓(𝑤)𝑓(𝑦-𝑤)\, \mathrm{d}𝑤 \\ & ={\int }_{𝑦-1}^{1}1\cdot 1\, \mathrm{d}𝑤 \\ & =2-𝑦.\end{aligned}$$

If $𝑦>2$, the supports again do not overlap, so $𝑔(𝑦)=0$.

Hence the PDF of $𝑌={𝑋}_{1}+{𝑋}_{2}$ is

$$𝑔(𝑦)=\begin{cases}𝑦,\quad & 0\leq 𝑦<1 \\ 2-𝑦,\quad & 1\leq 𝑦\leq 2 \\ 0,\quad & \text{otherwise}\end{cases}$$

Exercises