Books Probability and Statistical Inference

Conditional Distributions

Definition 4.3

The conditional probability mass function of $𝑋$ given $𝑌=𝑦$ is defined by

𝑔(𝑥|𝑦)=\frac{𝑓(𝑥,𝑦)}{{𝑓}_{𝑌}(𝑦)},\quad \text{provided that}\, {𝑓}_{𝑌}(𝑦)>0.

Similarly, the conditional probability mass function of $𝑌$ given $𝑋=𝑥$ is defined by

h(𝑦|𝑥)=\frac{𝑓(𝑥,𝑦)}{{𝑓}_{𝑋}(𝑥)},\quad \text{provided that}\, {𝑓}_{𝑋}(𝑥)>0.

Note that $h(𝑦|𝑥)\geq 0$ , and summing it over $𝑦$ gives

\sum _{𝑦}h(𝑦|𝑥)=\sum _{𝑦}\frac{𝑓(𝑥,𝑦)}{{𝑓}_{𝑋}(𝑥)}=\frac{1}{{𝑓}_{𝑋}(𝑥)}\sum _{𝑦}𝑓(𝑥,𝑦)=\frac{1}{{𝑓}_{𝑋}(𝑥)}\cdot {𝑓}_{𝑋}(𝑥)=1.

This shows that $h(𝑦|𝑥)$ is a valid probability mass function. We may compute probaility of event like $\{𝑎\leq 𝑌\leq 𝑏\}$ given $𝑋=𝑥$ as

ℙ(𝑎\leq 𝑌\leq 𝑏|𝑋=𝑥)=\sum _{𝑎\leq 𝑦\leq 𝑏}h(𝑦|𝑥)

and compute conditional expectation of a function $𝑢(𝑌)$ given $𝑋=𝑥$ as

\mathbb{E}[𝑢(𝑌)|𝑋=𝑥]=\sum _{𝑦}𝑢(𝑦)h(𝑦|𝑥).

Note

Note that the conditional expectation of

𝑌

given

𝑋=𝑥

is a function of

𝑥

Two special conditional expectations are conditional mean

{\mu }_{𝑌|𝑥}=\mathbb{E}[𝑌|𝑋=𝑥]=\sum _{𝑦}𝑦h(𝑦|𝑥),

and conditional variance

{\sigma }_{𝑌|𝑥}^{2}=\mathbb{E}[{(𝑌-{\mu }_{𝑌|𝑥})}^{2}|𝑋=𝑥]=\sum _{𝑦}{(𝑦-{\mu }_{𝑌}|𝑋)}^{2}h(𝑦|𝑥).

One can show that

{\sigma }_{𝑌|𝑥}^{2}=\mathbb{E}[{𝑌}^{2}|𝑋=𝑥]-{(\mathbb{E}[𝑌|𝑋=𝑥])}^{2}=\mathbb{E}[{𝑌}^{2}|𝑋=𝑥]-{\mu }_{𝑌|𝑥}^{2}.

Exercise 4.5

Let $𝑋$ and $𝑌$ be random variables of discrete type. Show that if ${\mu }_{𝑌|𝑥}$ is linear in $𝑥$ (we have already noted that it is a function of $𝑥$ ), i.e., ${\mu }_{𝑌|𝑥}=𝑎+𝑏𝑥$ , then it is given by the formula

{\mu }_{𝑌|𝑥}={\mu }_{𝑌}+\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}(𝑥-{\mu }_{𝑋}).

(4.6)

Similarly, if ${\mu }_{𝑋|𝑦}$ is linear in $𝑦$ , then

{\mu }_{𝑋|𝑦}={\mu }_{𝑋}+\rho \frac{{\sigma }_{𝑋}}{{\sigma }_{𝑌}}(𝑦-{\mu }_{𝑌}).

(4.7)

Remark

The formula for

{\mu }_{𝑌|𝑥}

is the same as the equation for the best-fitting line (least squares regression line) in Exercise 4.4.

Proof

We have

{\mu }_{𝑌}=\mathbb{E}[{\mu }_{𝑌|𝑋}]=\mathbb{E}[𝑎+𝑏𝑋]=𝑎+𝑏{\mu }_{𝑋}.

Then, we study $\mathbb{E}[𝑋𝑌]$ . We have

\begin{aligned}\mathbb{E}[𝑋𝑌] & =\sum _{𝑥}\sum _{𝑦}𝑥𝑦𝑓(𝑥,𝑦) \\ & =\sum _{𝑥}\sum _{𝑦}𝑥𝑦h(𝑦|𝑥){𝑓}_{𝑋}(𝑥) \\ & =\sum _{𝑥}𝑥{𝑓}_{𝑋}(𝑥)\sum _{𝑦}𝑦h(𝑦|𝑥) \\ & =\sum _{𝑥}𝑥{𝑓}_{𝑋}(𝑥){\mu }_{𝑌|𝑥} \\ & \text{Plug in the formula for}\, {\mu }_{𝑌|𝑥} \\ & =\sum _{𝑥}𝑥(𝑎+𝑏𝑥){𝑓}_{𝑋}(𝑥) \\ & =𝑎{\mu }_{𝑋}+𝑏({\sigma }_{𝑋}^{2}+{\mu }_{𝑋}^{2}).\end{aligned}

Combined with the equation $\operatorname{Cov}(𝑋,𝑌)=\mathbb{E}[𝑋𝑌]-{\mu }_{𝑋}{\mu }_{𝑌}$ , we get a system of equations

\begin{cases}\mathbb{E}[𝑋𝑌] & =𝑎{\mu }_{𝑋}+𝑏({\sigma }_{𝑋}^{2}+{\mu }_{𝑋}^{2}) \\ \mathbb{E}[𝑋𝑌] & ={\mu }_{𝑋}{\mu }_{𝑌}+\rho {\sigma }_{𝑋}{\sigma }_{𝑌}\end{cases}

Equate the RHS of the two equations,

𝑎{\mu }_{𝑋}+𝑏({\sigma }_{𝑋}^{2}+{\mu }_{𝑋}^{2})={\mu }_{𝑋}{\mu }_{𝑌}+\rho {\sigma }_{𝑋}{\sigma }_{𝑌}.

Substitute ${\mu }_{𝑌}=𝑎+𝑏{\mu }_{𝑋}$ ,

𝑎{\mu }_{𝑋}+𝑏({\sigma }_{𝑋}^{2}+{\mu }_{𝑋}^{2})=𝑎{\mu }_{𝑋}+𝑏{\mu }_{𝑋}^{2}+\rho {\sigma }_{𝑋}{\sigma }_{𝑌}.

Simplify,

𝑏=\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}.

Then, the value of $𝑎$ is given by

𝑎={\mu }_{𝑌}-\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}\cdot {\mu }_{𝑋}.

Therefore, we obtain the formula for ${\mu }_{𝑌|𝑥}$

{\mu }_{𝑌|𝑥}={\mu }_{𝑌}+\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}(𝑥-{\mu }_{𝑋}).

Similarly, by swapping the roles of $𝑋$ and $𝑌$ , we can show that

{\mu }_{𝑋|𝑦}={\mu }_{𝑋}+\rho \frac{{\sigma }_{𝑋}}{{\sigma }_{𝑌}}(𝑦-{\mu }_{𝑌}).

As a quick follow-up, we note that if both ${\mu }_{𝑌|𝑥}$ and ${\mu }_{𝑋|𝑦}$ are linear in $𝑥$ and $𝑦$ , respectively, then the product of the slopes of the two lines is ${\rho }^{2}$ and the quotient is ${\sigma }_{𝑌}^{2}/{\sigma }_{𝑋}^{2}$ (slope of line ${\mu }_{𝑌|𝑥}$ divided by the slope of line ${\mu }_{𝑋|𝑦}$ ).

Example 4.4

Let us yet again revisit the example of the trinomial distribution. Let $𝑋$ and $𝑌$ have the trinomial distribution with parameters $𝑛$ , ${𝑝}_{𝑋}>0$ , and ${𝑝}_{𝑌}>0$ . The joint PMF is given by

𝑓(𝑥,𝑦)=\frac{𝑛!}{𝑥!𝑦!(𝑛-𝑥-𝑦)!}{𝑝}_{𝑋}^{𝑥}{𝑝}_{𝑌}^{𝑦}{(1-{𝑝}_{𝑋}-{𝑝}_{𝑌})}^{𝑛-𝑥-𝑦}.

As noted before, $𝑋∼\operatorname{Bin}(𝑛,{𝑝}_{𝑋})$ . Thus, the marginal PMF of $𝑋$ is

{𝑓}_{𝑋}(𝑥)=\frac{𝑛!}{𝑥!(𝑛-𝑥)!}{𝑝}_{𝑋}^{𝑥}{(1-{𝑝}_{𝑋})}^{𝑛-𝑥}.

We are now interested in the conditional of $𝑌$ given $𝑋=𝑥$ . By definition, the conditional PMF of $𝑌$ given $𝑋=𝑥$ is given by

h(𝑦|𝑥)=\frac{𝑓(𝑥,𝑦)}{{𝑓}_{𝑋}(𝑥)}=\frac{(𝑛-𝑥)!}{𝑦!(𝑛-𝑥-𝑦)!}\cdot {\left(\frac{{𝑝}_{𝑌}}{1-{𝑝}_{𝑋}}\right)}^{𝑦}{\left(\frac{1-{𝑝}_{𝑋}-{𝑝}_{𝑌}}{1-{𝑝}_{𝑋}}\right)}^{𝑛-𝑥-𝑦}.

By comparing with the binomial PMF, we see that $h(𝑦|𝑥)$ is the PMF of

\operatorname{Bin}\left(𝑛-𝑥,\frac{{𝑝}_{𝑌}}{1-{𝑝}_{𝑋}}\right).

The conditional mean of $𝑌$ given $𝑋=𝑥$ is simply the mean of the above binomial distribution, i.e.,

{\mu }_{𝑌|𝑥}=\frac{{𝑝}_{𝑌}}{1-{𝑝}_{𝑋}}(𝑛-𝑥).

Note that this is a linear function of $𝑥$ . Therefore, by Exercise 4.5, we have

{\mu }_{𝑌|𝑥}={\mu }_{𝑌}+\rho \frac{{\sigma }_{𝑌}}{{\sigma }_{𝑋}}(𝑥-{\mu }_{𝑋}).

Similarly, by swapping the roles of $𝑋$ and $𝑌$ , the conditional distribution of $𝑋$ given $𝑌=𝑦$ is $\operatorname{Bin}(𝑛-𝑦,{𝑝}_{𝑋}/(1-{𝑝}_{𝑌}))$ and the conditional mean of $𝑋$ given $𝑌=𝑦$ is

{\mu }_{𝑋|𝑦}=\frac{{𝑝}_{𝑋}}{1-{𝑝}_{𝑌}}(𝑛-𝑦).

As noted before, the product of the coefficients of the variables of the two conditional means is ${\rho }^{2}$ . Therefore, we have

{\rho }^{2}=\left(-\frac{{𝑝}_{𝑌}}{1-{𝑝}_{𝑋}}\right)\left(-\frac{{𝑝}_{𝑋}}{1-{𝑝}_{𝑌}}\right)=\frac{{𝑝}_{𝑋}{𝑝}_{𝑌}}{(1-{𝑝}_{𝑋})(1-{𝑝}_{𝑌})}.

And becuase the correlation coefficient of the trinomial distribution is negative, we have

\rho =-\sqrt{\frac{{𝑝}_{𝑋}{𝑝}_{𝑌}}{(1-{𝑝}_{𝑋})(1-{𝑝}_{𝑌})}}.