Solution of A Linear System
Gradient descent can be used to solve a system of linear equations, reformulated as a quadratic minimization problem, e.g., using linear least squares. Solution of
in the sense of linear least squares is defined as minimizing the function
In traditional linear least squares for real and the Euclidean norm is used, in which case
In this case, the line search minimization, finding the locally optimal step size on every iteration, can be performed analytically, and explicit formulas for the locally optimal are known.
For solving linear equations, gradient descent is rarely used, with the conjugate gradient method being one of the most popular alternatives. The speed of convergence of gradient descent depends on the maximal and minimal eigenvalues of, while the speed of convergence of conjugate gradients has a more complex dependence on the eigenvalues, and can benefit from preconditioning. Gradient descent also benefits from preconditioning, but this is not done as commonly.
Read more about this topic: Gradient Descent
Famous quotes containing the words solution of, solution and/or system:
“What is history? Its beginning is that of the centuries of systematic work devoted to the solution of the enigma of death, so that death itself may eventually be overcome. That is why people write symphonies, and why they discover mathematical infinity and electromagnetic waves.”
—Boris Pasternak (1890–1960)
“To the questions of the officiously meddling police Falter replied absently and tersely; but, when he finally grew tired of this pestering, he pointed out that, having accidentally solved “the riddle of the universe,” he had yielded to artful exhortation and shared that solution with his inquisitive interlocutor, whereupon the latter had died of astonishment.”
—Vladimir Nabokov (1899–1977)
“Those words freedom and opportunity do not mean a license to climb upwards by pushing other people down. Any paternalistic system that tries to provide for security for everyone from above only calls for an impossible task and a regimentation utterly uncongenial to the spirit of our people.”
—Franklin D. Roosevelt (1882–1945)