Function increases in the direction of the gradient vector

Function increases in the direction of the gradient vector

Let \begin{align*} g(x, y) = f(x_0, y_0) + f_x(x_0, y_0)(x - x_0) + f_y(x_0, y_0)(y - y_0) \end{align*} be the Taylor expansion for f around $(x_0, y_0)$. In this context the partial derivative $\partial f / \partial x$ means the change in $g$ when $x$ increase by one and $y$ remains the same , and $\partial f / \partial y$ means something similar. Now it's easy to see that the gradient vector \begin{pmatrix} \partial f / \partial x \\ \partial f / \partial y \\ \end{pmatrix} points to the direction where $f$ increases. When $f$ increases with $x$, it points to the right, otherwise to the left. Similar conclusion can be made with respect to $y$.

For a more algebraic and rigorous proof, let's use Taylor expansion again:

\begin{align*} f(x, y) = f(x_0, y_0) + f_x(x_0, y_0)(x - x_0) + f_y(x_0, y_0)(y - y_0) \end{align*}

and let \begin{align*} x &= x_0 + f_x(x_0, y_0) \\ y &= y_0 + f_y(x_0, y_0) \\ \end{align*}

then

\begin{align*} f(x, y) = f(x_0, y_0) + f_x(x_0, y_0)^2 + f_y(x_0, y_0)^2 \end{align*}

if the partial derivatives are non-zero, then $f(x, y)$ will always be greater than $f(x_0, y_0)$. (i.e. f increases in the direction of the gradient vector!)