Practical Mathematical Optimization Pdf

[Patrizia Leones]

Thistextbook presents a wide range of tools for a course in mathematical optimization for upper undergraduate and graduate students in mathematics, engineering, computer science, and other applied sciences. Basic optimization principles are presented with emphasis on gradient-based numerical optimization strategies and algorithms for solving both smooth and noisy discontinuous optimization problems. Attention is also paid to the difficulties of expense of function evaluations and the existence of multiple minima that often unnecessarily inhibit the use of gradient-based methods. This second edition addresses further advancements of gradient-only optimization strategies to handle discontinuities in objective functions. New chapters discuss the construction of surrogate models as well as new gradient-only solution strategies and numerical optimization using Python. A special Python module is electronically available (via springerlink) that makes the new algorithms featured in the text easily accessible and directly applicable. Numerical examples and exercises are included to encourage senior- to graduate-level students to plan, execute, and reflect on numerical investigations. By gaining a deep understanding of the conceptual material presented, students, scientists, and engineers will be able to develop systematic and scientific numerical investigative skills.

Jan A. Snyman currently holds the position of emeritus professor in the Department of Mechanical and Aeronautical Engineering of the University of Pretoria, having retired as full professor in 2005. He has taught physics, mathematics and engineering mechanics to science and engineering students at undergraduate and postgraduate level, and has supervised the theses of 26 Masters and 8 PhD students. His research mainly concerns the development of gradient-based trajectory optimization algorithms for solving noisy and multi-modal problems, and their application in approximation methodologies for the optimal design of engineering systems. He has authored or co-authored 89 research articles in peer-reviewed journals as well as numerous papers in international conference proceedings.

Daniel N. Wilke is a senior lecturer in the Department of Mechanical and Aeronautical Engineering of the University of Pretoria. He teaches computer programming, mathematicalprogramming and computational mechanics to science and engineering students at undergraduate and postgraduate level. His current research focuses on the development of interactive design optimization technologies, and enabling statistical learning (artificial intelligence) application layers, for industrial processes and engineering design. He has co-authored over 50 peer-reviewed journal articles and full length conference papers.

Have you ever wondered what it takes to build a non-convex solver? And how different approaches impact the solution that is outputted? Go on a practical tour through non-convex optimization, hands-on code examples are used to:

Jaromił Najman obtained his M.Sc. in Mathematics at RWTH Aachen University. He did his Ph.D. at the Institute for Process Systems Engineering (AVT.SVT) in Aachen in nonlinear programming, in particular deterministic global optimization.

Kostja holds a Ph.D. in Operations Research from the University of Paderborn (Germany). He joined Gurobi in 2015 after many years of experience in the development and design of decision support systems using mathematical optimization. Before joining Gurobi he worked at Daimler Research & Development and as a lecturer at the University of Paderborn. Since 1998, before focusing on optimization and during his studies he continuously worked as system administrator, software developer and support engineer for an IT service company.

Dr. Richard Oberdieck is a Technical Account Manager at Gurobi Optimization. He obtained his BSc and MSc degrees from ETH Zurich in Switzerland (2009-2013), before pursuing a Ph.D. in Chemical Engineering at Imperial College London, UK, which he completed in 2017. In his PhD, he worked on a special class of optimization problems called multi-parametric programming problems, where he discovered fundamental new properties related to these problems, as well as implemented them in a software toolbox.

Scheduled for release by Cambridge University Press as an inexpensive textbook in 2024, our book, also available under a Green Open Access license, is designed to empower learners with practical insights. The latest prepublication version is accessible at -insights/insights/math-optimization-python. Realistic examples accompany each topic, and to further enhance the learning experience, we provide an online companion. This resource comprises 50+ fully functional notebooks readily deployable in Google Colab, serving as practical starting points to address diverse real-life challenges you may encounter.

During this webinar, I will demonstrate representative content of the book in action and will also showcase how ChatGPT serves as a valuable collaborator in my professional endeavors, offering insights into its role in my workflow.

Visit the website and register as a member for free, to get the regular updates on all activities: EPF Member registration page. The recordings and details from previous webinars are also available on this website.

Mathematical optimization deals with theproblem of finding numerically minimums (or maximums or zeros) ofa function. In this context, the function is called cost function, orobjective function, or energy.

Here, we are interested in using scipy.optimize for black-boxoptimization: we do not rely on the mathematical expression of thefunction that we are optimizing. Note that this expression can often beused for more efficient, non black-box, optimization.

Optimizing smooth functions is easier(true in the context of black-box optimization, otherwiseLinear Programmingis an example of methods which deal very efficiently withpiece-wise linear functions).

Many optimization methods rely on gradients of the objective function.If the gradient function is not given, they are computed numerically,which induces errors. In such situation, even if the objectivefunction is not noisy, a gradient-based optimization may be a noisyoptimization.

As can be seen from the above experiments, one of the problems of thesimple gradient descent algorithms, is that it tends to oscillate acrossa valley, each time following the direction of the gradient, that makesit cross the valley. The conjugate gradient solves this problem by addinga friction term: each step depends on the two last values of thegradient and sharp turns are reduced.

In scipy, you can use the Newton method by setting method to Newton-CG inscipy.optimize.minimize(). Here, CG refers to the fact that an internalinversion of the Hessian is performed by conjugate gradient

L-BFGS: Limited-memory BFGS Sits between BFGS and conjugate gradient:in very high dimensions (> 250) the Hessian matrix is too costly tocompute and invert. L-BFGS keeps a low-rank version. In addition, box boundsare also supported by L-BFGS-B:

The Nelder-Mead algorithms is a generalization of dichotomy approaches tohigh-dimensional spaces. The algorithm works by refining a simplex, the generalization of intervalsand triangles to high-dimensional spaces, to bracket the minimum.

Strong points: it is robust to noise, as it does not rely oncomputing gradients. Thus it can work on functions that are not locallysmooth such as experimental data points, as long as they display alarge-scale bell-shape behavior. However it is slower than gradient-basedmethods on smooth, non-noisy functions.

If your problem does not admit a unique local minimum (which can be hardto test unless the function is convex), and you do not have priorinformation to initialize the optimization close to the solution, youmay need a global optimizer.

scipy.optimize.brute() evaluates the function on a given grid ofparameters and returns the parameters corresponding to the minimumvalue. The parameters are specified with ranges given tonumpy.mgrid. By default, 20 steps are taken in each direction:

A very common source of optimization not converging well is humanerror in the computation of the gradient. You can usescipy.optimize.check_grad() to check that your gradient iscorrect. It returns the norm of the different between the gradientgiven, and a gradient computed numerically:

Least square problems occur often when fitting a non-linear to data.While it is possible to construct our optimization problem ourselves,scipy provides a helper function for this purpose:scipy.optimize.curve_fit():

Box bounds correspond to limiting each of the individual parameters ofthe optimization. Note that some problems that are not originally writtenas box bounds can be rewritten as such via change of variables. Bothscipy.optimize.minimize_scalar() and scipy.optimize.minimize()support bound constraints with the parameter bounds:

Several optimization modelling textbooks are available online. Some are digital versions of hard-copy textbooks, while others are solely in web format. Content varies from focussing on theoretical aspects of optimization mathematics, through to practical applications, and developing models in specific programming languages.

This book provides a broad introduction to optimization with a focus on practical algorithms for the design of engineering systems. The text is intended for advanced undergraduates and graduate students as well as professionals. The examples are implemented in the Julia programming language.

This book is about convex optimization, a special class of mathematical optimization problems, which includes least-squares and linear programming problems. It is well known that least-squares and linear programming problems have a fairly complete theory, arise in a variety of applications, and can be solved numerically very efficiently. The basic point of this book is that the same can be said for the larger class of convex optimization problems.

3a8082e126