Solution of A Linear System
Gradient descent can be used to solve a system of linear equations, reformulated as a quadratic minimization problem, e.g., using linear least squares. Solution of
in the sense of linear least squares is defined as minimizing the function
In traditional linear least squares for real and the Euclidean norm is used, in which case
In this case, the line search minimization, finding the locally optimal step size on every iteration, can be performed analytically, and explicit formulas for the locally optimal are known.
For solving linear equations, gradient descent is rarely used, with the conjugate gradient method being one of the most popular alternatives. The speed of convergence of gradient descent depends on the maximal and minimal eigenvalues of, while the speed of convergence of conjugate gradients has a more complex dependence on the eigenvalues, and can benefit from preconditioning. Gradient descent also benefits from preconditioning, but this is not done as commonly.
Read more about this topic: Gradient Descent
Famous quotes containing the words solution of, solution and/or system:
“Give a scientist a problem and he will probably provide a solution; historians and sociologists, by contrast, can offer only opinions. Ask a dozen chemists the composition of an organic compound such as methane, and within a short time all twelve will have come up with the same solution of CH4. Ask, however, a dozen economists or sociologists to provide policies to reduce unemployment or the level of crime and twelve widely differing opinions are likely to be offered.”
—Derek Gjertsen, British scientist, author. Science and Philosophy: Past and Present, ch. 3, Penguin (1989)
“Let us begin to understand the argument.
There is a solution to everything: Science.”
—Allen Tate (1899–1979)
“We are now going through a period of demolition. In morals, in social life, in politics, in medicine, and in religion there is a universal upturning of foundations. But the day of reconstruction seems to be looming, and now the grand question is: Are there any sure and universal principles that will evolve a harmonious system in which we shall all agree?”
—Catherine E. Beecher (1800–1878)