Books Probability and Statistical Inference

The Correlation Coefficient

Putting $𝑢(𝑋,𝑌)=(𝑋-{\mu }_{𝑋})(𝑌-{\mu }_{𝑌})$ in (4.1), we obtain a special mathematical expectation:

\mathbb{E}[(𝑋-{\mu }_{𝑋})(𝑌-{\mu }_{𝑌})]=\sum _{(𝑥,𝑦)\in 𝑆}(𝑥-{\mu }_{𝑋})(𝑦-{\mu }_{𝑌})𝑓(𝑥,𝑦)≕{\sigma }_{𝑋𝑌}≕\operatorname{Cov}(𝑋,𝑌).

(4.2)

This is called the covariance of $𝑋$ and $𝑌$ .

The correlation coefficient of $𝑋$ and $𝑌$ is defined as

\rho =\frac{{\sigma }_{𝑋𝑌}}{{\sigma }_{𝑋}{\sigma }_{𝑌}}.

(4.3)

By playing with (4.2) and (4.3), we can obtain some simple facts about the correlation coefficient:

$\operatorname{Cov}(𝑋,𝑌)=\mathbb{E}[𝑋𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}$ .
$\mathbb{E}[𝑋𝑌]={\mu }_{𝑋}{\mu }_{𝑌}+\rho {\sigma }_{𝑋}{\sigma }_{𝑌}$ . An interpretation of this is that the expected value of the product of $𝑋$ and $𝑌$ is roughly ${\mu }_{𝑋}{\mu }_{𝑌}$ —the product of the often assumed values of $𝑋$ and $𝑌$ plus a term that is proportional to the correlation coefficient $\rho$ .

Exercise 4.3

Let $𝑋$ and $𝑌$ have a trinomial distribution with parameters $𝑛$ , ${𝑝}_{𝑋}$ , and ${𝑝}_{𝑌}$ . Show that

\mathbb{E}[𝑋𝑌]=𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌}

and consequently,

\operatorname{Cov}(𝑋,𝑌)=-𝑛{𝑝}_{𝑋}{𝑝}_{𝑌}\quad \text{and}\quad \rho =-\sqrt{\frac{{𝑝}_{𝑋}{𝑝}_{𝑌}}{(1-{𝑝}_{𝑋})(1-{𝑝}_{𝑌})}}.

We first prove this via brute force calculation and we will provide another more interesting and neater proof later.

Proof

By definition and the formula of the trinomial PMF,

\begin{aligned}\mathbb{E}[𝑋𝑌] & =\sum _{𝑥=0}^{𝑛}\sum _{𝑦=0}^{𝑛-𝑥}𝑥𝑦\frac{𝑛!}{𝑥!𝑦!(𝑛-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥}{𝑝}_{𝑌}^{𝑦}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-𝑥-𝑦} \\ & \text{Drop the terms where either}\, 𝑥\, \text{or}\, 𝑦\, \text{is}\, 0 \\ & =\sum _{𝑥=1}^{𝑛-1}\sum _{𝑦=1}^{𝑛-1-𝑥}𝑥𝑦\frac{𝑛!}{𝑥!𝑦!(𝑛-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥}{𝑝}_{𝑌}^{𝑦}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-𝑥-𝑦} \\ & =\sum _{𝑥=1}^{𝑛-1}\sum _{𝑦=1}^{𝑛-1-𝑥}\frac{𝑛!}{(𝑥-1)!(𝑦-1)!(𝑛-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥}{𝑝}_{𝑌}^{𝑦}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-𝑥-𝑦} \\ & \text{Reindex}\, 𝑥←𝑥+1\, \text{and}\, 𝑦←𝑦+1 \\ & =\sum _{𝑥=0}^{𝑛-2}\sum _{𝑦=0}^{𝑛-2-𝑥}\frac{𝑛!}{𝑥!𝑦!(𝑛-2-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥+1}{𝑝}_{𝑌}^{𝑦+1}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-2-𝑥-𝑦} \\ & \text{Pull out}\, 𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌} \\ & =𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌}\underbrace{\sum _{𝑥=0}^{𝑛-2}\sum _{𝑦=0}^{𝑛-2-𝑥}\frac{(𝑛-2)!}{𝑥!𝑦!(𝑛-2-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥}{𝑝}_{𝑌}^{𝑦}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-2-𝑥-𝑦}}_{={[{𝑝}_{𝑋}+{𝑝}_{𝑌}+(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})]}^{𝑛-2}={1}^{𝑛-2}=1} \\ & =𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌}.\end{aligned}

Recall that $𝑋∼\operatorname{Bin}(𝑛,{𝑝}_{𝑋})$ and $𝑌∼\operatorname{Bin}(𝑛,{𝑝}_{𝑌})$ . We have

{\mu }_{𝑋}=𝑛{𝑝}_{𝑋},\quad {\sigma }_{𝑋}^{2}=𝑛{𝑝}_{𝑋}(1-{𝑝}_{𝑋}),\quad {\mu }_{𝑌}=𝑛{𝑝}_{𝑌},\quad {\sigma }_{𝑌}^{2}=𝑛{𝑝}_{𝑌}(1-{𝑝}_{𝑌}).

It then follows that

\operatorname{Cov}(𝑋,𝑌)=\mathbb{E}[𝑋𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}=𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌}-{𝑛}^{2}{𝑝}_{𝑋}{𝑝}_{𝑌}=-𝑛{𝑝}_{𝑋}{𝑝}_{𝑌}

and

\rho =\frac{\operatorname{Cov}(𝑋,𝑌)}{{\sigma }_{𝑋}{\sigma }_{𝑌}}=\frac{-𝑛{𝑝}_{𝑋}{𝑝}_{𝑌}}{\sqrt{𝑛{𝑝}_{𝑋}(1-{𝑝}_{𝑋})}\sqrt{𝑛{𝑝}_{𝑌}(1-{𝑝}_{𝑌})}}=-\sqrt{\frac{{𝑝}_{𝑋}{𝑝}_{𝑌}}{(1-{𝑝}_{𝑋})(1-{𝑝}_{𝑌})}}.

We now give the alternative proof alluded to above, trading lengthy manipulation of the trinomial PMF for a more structural point of view. The central move—expressing each count as a sum of i.i.d. Bernoulli (indicator) random variables, one for each of the $𝑛$ independent trials—makes the calculation of $\mathbb{E}[𝑋𝑌]$ a matter of bookkeeping over independent trials; the same random variable as a sum of $i.i.d.$ pieces device is a standard and widely reusable first step in many problems.

Proof

Consider $𝑛$ independent draws with replacement from an urn with three colors ${𝐶}_{1}$ , ${𝐶}_{2}$ , and ${𝐶}_{3}$ , with probabilities ${𝑝}_{𝑋}$ , ${𝑝}_{𝑌}$ , and $1-{𝑝}_{𝑋}-{𝑝}_{𝑌}$ on each draw. For $𝑖=1,\ldots ,𝑛$ , let ${𝐼}_{𝑖}^{(1)}$ indicate whether the $𝑖$ -th draw is ${𝐶}_{1}$ , and let ${𝐼}_{𝑖}^{(2)}$ indicate whether the $𝑖$ -th draw is ${𝐶}_{2}$ . Then ${𝐼}_{𝑖}^{(1)}∼\operatorname{Ber}({𝑝}_{𝑋})$ and ${𝐼}_{𝑖}^{(2)}∼\operatorname{Ber}({𝑝}_{𝑌})$ , and the $𝑛$ trials are mutually independent (for a given $𝑖$ , of course, ${𝐼}_{𝑖}^{(1)}{𝐼}_{𝑖}^{(2)}=0$ ).

The trinomial counts are

𝑋=\sum _{𝑖=1}^{𝑛}{𝐼}_{𝑖}^{(1)}\quad \text{and}\quad 𝑌=\sum _{𝑗=1}^{𝑛}{𝐼}_{𝑗}^{(2)}.

Thus

𝑋𝑌=\sum _{𝑖=1}^{𝑛}\sum _{𝑗=1}^{𝑛}{𝐼}_{𝑖}^{(1)}{𝐼}_{𝑗}^{(2)}.

Split the double sum according to whether $𝑖=𝑗$ or $𝑖\neq 𝑗$ :

𝑋𝑌=\sum _{𝑖\neq 𝑗}{𝐼}_{𝑖}^{(1)}{𝐼}_{𝑗}^{(2)}+\sum _{𝑖=𝑗}{𝐼}_{𝑖}^{(1)}{𝐼}_{𝑗}^{(2)}.

The second sum is zero, because if $𝑖=𝑗$ then ${𝐼}_{𝑖}^{(1)}{𝐼}_{𝑖}^{(2)}=0$ (the $𝑖$ -th draw cannot be both ${𝐶}_{1}$ and ${𝐶}_{2}$ ). Hence

𝑋𝑌=\sum _{𝑖\neq 𝑗}{𝐼}_{𝑖}^{(1)}{𝐼}_{𝑗}^{(2)}.

Whenever $𝑖\neq 𝑗$ , the indicators ${𝐼}_{𝑖}^{(1)}$ and ${𝐼}_{𝑗}^{(2)}$ refer to different trials, so they are independent. Taking expectations,

\begin{aligned}\mathbb{E}[𝑋𝑌] & =\sum _{𝑖\neq 𝑗}\mathbb{E}[{𝐼}_{𝑖}^{(1)}{𝐼}_{𝑗}^{(2)}] \\ & \text{Due to independence} \\ & =\sum _{𝑖\neq 𝑗}\mathbb{E}[{𝐼}_{𝑖}^{(1)}]\mathbb{E}[{𝐼}_{𝑗}^{(2)}] \\ & =\sum _{𝑖\neq 𝑗}{𝑝}_{𝑋}{𝑝}_{𝑌} \\ & =𝑛(𝑛-1){𝑝}_{𝑋}{𝑝}_{𝑌}.\end{aligned}

And yet we will have another derivation of the covariance of trinomial distribution when we introduce the conditional distributions later in Example 4.4.

Consider the $2$ -dimensional space of the joint support $𝑆$ of random variables $𝑋$ and $𝑌$ and a line, with finite slope $𝑏$ , passing through the point $({\mu }_{𝑋},{\mu }_{𝑌})$ . (If we sample the points $(𝑥,𝑦)$ randomly following the joint distribution and depict a scatter plot, then we will see that $({\mu }_{𝑋},{\mu }_{𝑌})$ is roughly the centroid of this cluster.) The equation of such a line is $𝑦=𝑏(𝑥-{\mu }_{𝑋})+{\mu }_{𝑌}$ . Randomly sample a point $(𝑥,𝑦)$ , the vertical distance between the point and the line is given by $|𝑦-𝑏(𝑥-{\mu }_{𝑋})-{\mu }_{𝑌}|$ . We are now interested in the expected value of the square of this distance, i.e.,

𝐿(𝑏)=\mathbb{E}[{|𝑌-𝑏(𝑋-{\mu }_{𝑋})-{\mu }_{𝑌}|}^{2}].

(4.4)

As (4.4), this expectation can be regarded as a function $𝐿(𝑏)$ of $𝑏$ . The problem now is to find the value of $𝑏$ that minimizes $𝐿(𝑏)$ . Rearranging the terms and applying the linearity of expectations, we have

𝐿(𝑏)={\sigma }_{𝑌}^{2}+{𝑏}^{2}{\sigma }_{𝑋}^{2}-2𝑏{\sigma }_{𝑋𝑌}.

Differentiating,

{𝐿}^{\prime }(𝑏)=2𝑏{\sigma }_{𝑋}^{2}-2{\sigma }_{𝑋𝑌}.

Setting to zero, we solve for $𝑏$ to get

{𝑏}^{\ast }=\frac{{\sigma }_{𝑋𝑌}}{{\sigma }_{𝑋}^{2}}=\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}.

Therefore, the equation of the optimal line is

𝑦=\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}(𝑥-{\mu }_{𝑋})+{\mu }_{𝑌}.

Evaluated at $\rho {\sigma }_{𝑌}/{\sigma }_{𝑋}$ , $𝐿(𝑏)$ , or the minimum value of the expected value of the square of the vertical distance, is

𝐿\left(\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}\right)={\sigma }_{𝑌}^{2}(1-{\rho }^{2}).

(4.5)

We can gain a lot of insights from (4.5). First, since the expected value of a square is always non-negative, we clearly have ${\sigma }_{𝑌}^{2}(1-{\rho }^{2})\geq 0$ , which yields

-1\leq \rho \leq 1.

That is, the value of correlation coefficient is bounded by $1$ .

When $|\rho |=1$ , that is, $𝑋$ is strongly correlated with $𝑌$ either in the positive or negative direction, we get $𝐿({𝑏}^{\ast })=0$ , which means that all the sampled points $(𝑥,𝑦)$ will lie on the line $𝑦={𝑏}^{\ast }𝑥+{\mu }_{𝑌}$ .

When $\rho =0$ , we have $𝐿({𝑏}^{\ast })={\sigma }_{𝑌}^{2}$ . The same quantity is $\mathbb{E}[{(𝑌-{\mu }_{𝑌})}^{2}]$ , the mean squared error from predicting $𝑌$ by the constant ${\mu }_{𝑌}$ without using $𝑋$ . Intuitively, pushing each sample $(𝑥,𝑦)$ onto the $𝑦$ -axis discards $𝑥$ , yet nothing is gained by retaining it in mean squared error; that is the geometric echo of $𝑋$ and $𝑌$ being uncorrelated.

Suppose that $𝑋$ and $𝑌$ are independent. Then they must be uncorrelated, i.e., $\rho =0$ or $\operatorname{Cov}(𝑋,𝑌)=0$ . Indeed, we have

\operatorname{Cov}(𝑋,𝑌)=\mathbb{E}[𝑋𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}=\mathbb{E}[𝑋]\mathbb{E}[𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}={\mu }_{𝑋}{\mu }_{𝑌}-{\mu }_{𝑋}{\mu }_{𝑌}=0.

The condition of independence is applied when we write $\mathbb{E}[𝑋𝑌]=\mathbb{E}[𝑋]\mathbb{E}[𝑌]$ . However, the converse is not true. That is, if two random variables are uncorrelated, they are not, in general, independent.

Note

The derivation does not require

𝑋

and

𝑌

to be discrete. It uses only the algebra of expectations, not mass functions and sums over the support, so the same results apply to continuous (or more general) joint distributions, with integrals in place of sums when moments are defined in the usual way.

In fact, it was not necessary to parametrize the line as passing through $({\mu }_{𝑋},{\mu }_{𝑌})$ in order to minimize the expected squared vertical error. Exercise 4.4 instead uses the general form $𝑦=𝑎+𝑏𝑥$ and optimizes over $𝑎$ and $𝑏$ ; the same best line is obtained, and it still passes through $({\mu }_{𝑋},{\mu }_{𝑌})$ .

Example 4.3

Let $𝑋$ and $𝑌$ have the joint PMF

𝑓(𝑥,𝑦)=\frac{1}{3},\quad (𝑥,𝑦)\in \{(0,1),(1,0),(2,1)\}.

Figure 4.2 shows this joint PMF. From the plot, we can easily find that ${\mu }_{𝑋}=1$ , ${\mu }_{𝑌}=2/3$ and

\mathbb{E}[𝑋𝑌]=0\cdot 1\cdot \frac{1}{3}+1\cdot 0\cdot \frac{1}{3}+2\cdot 1\cdot \frac{1}{3}=\frac{2}{3}.

Therefore, $\operatorname{Cov}(𝑋,𝑌)=\mathbb{E}[𝑋𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}=0$ , which meas that $𝑋$ and $𝑌$ are uncorrelated. But clearly they are not independent since their support is not rectangular.

Figure 4.2 Joint PMF on the three-point support $\{(0,1),(1,0),(2,1)\}$ , uniform with mass $\frac{1}{3}$ at each point. Here $𝑋$ and $𝑌$ are uncorrelated yet dependent.

Exercises

Exercise 4.4

Let $𝑋$ and $𝑌$ be random variables with respective means ${\mu }_{𝑋}$ and ${\mu }_{𝑌}$ , respective variances ${\sigma }_{𝑋}^{2}$ and ${\sigma }_{𝑌}^{2}$ and correlation coefficient $\rho$ . Fit the line $𝑦=𝑎+𝑏𝑥$ by the method of least squares to the probability distribution by minimizing the expectation

𝐿(𝑎,𝑏)=\mathbb{E}[{(𝑌-(𝑎+𝑏𝑋))}^{2}]

with respect to $𝑎$ and $𝑏$ .

Solution

Expand the expression inside the expectation,

𝐿(𝑎,𝑏)={𝑎}^{2}+\mathbb{E}[{𝑋}^{2}]{𝑏}^{2}+2{\mu }_{𝑋}𝑎𝑏-2{\mu }_{𝑌}𝑎-2\mathbb{E}[𝑋𝑌]𝑏+\mathbb{E}[{𝑌}^{2}].

Differentiate with respect to $𝑎$ and $𝑏$ ,

\begin{aligned}\frac{\partial }{\partial 𝑎}𝐿(𝑎,𝑏) & =2𝑎+2{\mu }_{𝑋}𝑏-2{\mu }_{𝑌} \\ \frac{\partial }{\partial 𝑏}𝐿(𝑎,𝑏) & =2{\mu }_{𝑋}𝑎+2\mathbb{E}[{𝑋}^{2}]𝑏-2\mathbb{E}[𝑋𝑌].\end{aligned}

Setting to zero, we get the system of equations

\begin{cases}𝑎+{\mu }_{𝑋}𝑏-{\mu }_{𝑌} & =0 \\ {\mu }_{𝑋}𝑎+\mathbb{E}[{𝑋}^{2}]𝑏-\mathbb{E}[𝑋𝑌] & =0\end{cases}

We solve $𝑏$ by multiplying the first equation by ${\mu }_{𝑋}$ and then subtract it from the second equation:

{𝑏}^{\ast }=\frac{{\sigma }_{𝑋𝑌}}{{\sigma }_{𝑋}^{2}}=\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}.

Then plugging ${𝑏}^{\ast }$ back into the first equation, we get

{𝑎}^{\ast }={\mu }_{𝑌}-\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}\cdot {\mu }_{𝑋}.

Therefore, the line of the best fit is

𝑦={\mu }_{𝑌}-\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}\cdot {\mu }_{𝑋}+\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}𝑥={\mu }_{𝑌}+\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}(𝑥-{\mu }_{𝑋}).

Note that it indeed passes through the point $({\mu }_{𝑋},{\mu }_{𝑌})$ .