Gradient Descent - Solution of A Non-linear System

Solution of A Non-linear System

Gradient descent can also be used to solve a system of nonlinear equations. Below is an example that shows how to use the gradient descent to solve for three unknown variables, x1, x2, and x3. This example shows one iteration of the gradient descent.

Consider a nonlinear system of equations:

\begin{cases} 3x_1-\cos(x_2x_3)-\tfrac{3}{2}=0 \\ 4x_1^2-625x_2^2+2x_2-1=0 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3}=0 \end{cases}

suppose we have the function

G(\mathbf{x}) = \begin{bmatrix} 3x_1-\cos(x_2x_3)-\tfrac{3}{2} \\ 4x_1^2-625x_2^2+2x_2-1 \\ \exp(-x_1x_2)+20x_3+\tfrac{10\pi-3}{3} \\ \end{bmatrix}

where

\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix}

and the objective function

With initial guess

\mathbf{x}^{(0)}=\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} =\begin{bmatrix} 0 \\ 0 \\ 0 \\ \end{bmatrix}

We know that

where

The Jacobian matrix

J_G = \begin{bmatrix} 3 & \sin(x_2x_3)x_3 & \sin(x_2x_3)x_2 \\ 8x_1 & -1250x_2+2 & 0 \\ -x_2\exp{(-x_1x_2)} & -x_1\exp(-x_1x_2) & 20\\ \end{bmatrix}

Then evaluating these terms at

J_G \left(\mathbf{x}^{(0)}\right) = \begin{bmatrix} 3 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 20 \end{bmatrix}

and

G(\mathbf{x}^{(0)}) = \begin{bmatrix} -2.5\\ -1\\ 10.472 \end{bmatrix}

So that

\mathbf{x}^{(1)}=0-\gamma_0 \begin{bmatrix} -7.5\\ -2\\ 209.44 \end{bmatrix}.

and

F \left(\mathbf{x}^{(0)}\right) = 0.5((-2.5)^2 + (-1)^2 + (10.472)^2) = 58.456

Now a suitable must be found such that . This can be done with any of a variety of line search algorithms. One might also simply guess which gives

\mathbf{x}^{(1)}=\begin{bmatrix} 0.0075 \\ 0.002 \\ -0.20944 \\ \end{bmatrix}

evaluating at this value,

F \left(\mathbf{x}^{(1)}\right) = 0.5((-2.48)^2 + (-1.00)^2 + (6.28)^2) = 23.306

The decrease from to the next step's value of is a sizable decrease in the objective function. Further steps would reduce its value until a solution to the system was found.

Read more about this topic:  Gradient Descent

Famous quotes containing the words solution of, solution and/or system:

    “The truth of the thoughts that are here set forth seems to me unassailable and definitive. I therefore believe myself to have found, on all essential points, the final solution of the problems. And if I am not mistaken in this belief, then the second thing in which the value of this work consists is that it shows how little is achieved when these problems are solved.”
    —Ludwig Wittgenstein (1889–1951)

    “There is a lot of talk now about metal detectors and gun control. Both are good things. But they are no more a solution than forks and spoons are a solution to world hunger.”
    —Anna Quindlen (b. 1953)

    “My advice to people today is as follows: If you take the game of life seriously, if you take your nervous system seriously, if you take your sense organs seriously, if you take the energy process seriously, you must turn on, tune in, and drop out.”
    —Timothy Leary (b. 1920)