Mathematics

This paper explores some optimization methods such as the gradient descent method, Gauss-Newton method, and stochastic gradient method. Some examples of minimizing objective functions are given to validate the theories. Then we introduce a simple example of artificial neural networks, define its structure, and apply the optimization methods to it. Finally we use an image classification example to end this expository paper.