Published Paper Details:

The fields of machine learning and mathematical programming are increasingly intertwined. Optimization of machine learning algorithms plays a significant role in training the models efficiently and applying them effectively. Machine learning algorithms can take a lot of time to train without the use of optimization techniques. Optimization algorithms help us tominimize (or maximize) an objective function which is simply a mathematical function. Several researchers have identified the flaws of previously used models to come up with more optimized or better algorithms over the period. A combined survey of various optimization techniques will help to identify a suitable algorithm for different kinds of problems. In this paper, we have compared different optimization techniques and identified limitations and advantages of different algorithms. We observed the impact of each algorithm on the efficiency of the model and its impact on training time. We have studied the impact of these algorithms on different types and sizes of data sets. Next, we summarize the applications and developments of optimization methods in machine learning fields. We have displayed the need to move to the more recently developed algorithms which have advantage over the precedent algorithms.