Monday, September 22, 2014

Change of variables in expectation

Expectation

For discrete variables it's defined as

\begin{align} E(x) = \sum_x x p(x) \end{align}

Expectation of a function

Substitute $x$ with $f(x)$ in the above formula:

\begin{align*} E(f(x)) &= \sum_{f(x)} f(x) p(f(x)) \\ &= \sum_{f(x)} f(x) \sum_{x_0 | f(x_0) = f(x)} p(x_0) \\ &= \sum_x f(x) p(x) \end{align*}

But let's go an extra mile:

Expectation of a function of a function

\begin{align} E(f(g(x))) &= \sum_{f(g(x))} f(g(x)) p(f(g(x))) \\ &= \sum_{f(g(x))} f(g(x)) \sum_{D_1} p(g(x)) \\ &= \sum_{f(g(x))} f(g(x)) \sum_{D_1} \sum_{D_2} p(x) \\ &= \sum_x f(g(x)) p(x) \end{align}

If you let $y = g(x)$, then from (3) we can have:

\begin{align} &\sum_{f(g(x))} f(g(x)) \sum_{D_1} p(g(x)) \\ &= \sum_{f(y)} f(y) \sum_{D_1} p(y) \\ &= \sum_{y} f(y) p(y) \\ \end{align}

Connecting (5) and (8) we get a beautiful transformation:

\begin{align} \sum_{y} f(y) p(y) = \sum_x f(g(x)) p(x) \end{align}

This can go on and on, of course:

\begin{align} \sum_{y} f(y) p(y) &= \sum_x f(g(x)) p(x) \\ &= \sum_z f(g(h(z))) p(z) \\ &= \cdots \end{align}