The optimization process is a vital tool for the learning and the development of algorithms. Machine Learning heavily relies on optimization, by which accurate models can be achieved by tuning and optimizating some parameters. In general, when data is available, a good model is obtained by minimization the error between a model's predictions and the ground truth targets.

The problem of determining the smallest or largest value of a function can take is a centuries-old pursuit that has numerous applications throughout the sciences and engineering. Zero-order (or derivative-free) optimization techniques (Random Search, Coordinate Search, and Zero-Order Coordinate Descent) are the simplest tools. This will allow to lay a range of crucial concepts that will follow in more complex settings.