E(min(a,b)), model selection by validation

A simple result from probability theory:

\begin{align*} min(a, b) &\le a \\ E(min(a, b)) &\le E(a) \end{align*}

This can be applied in validation-model-selection in machine learning, i.e. the estimate of out-of-sample error by validation is optimistically biased.

This illustrated in the following figure. But as the validation set size increases, the bias gets smaller and smaller: