Numerical Optimisation Python

[Annemie]

Thischapter gives an introduction to the basics of numerical optimization and will help build the tools required for our in-depth understanding in the later chapters. Some fundamental linear algebra concepts will be touched which will be required for further studies in optimization along with introduction to simple Python codes.

In case of a minimization problem, the optimum vector \(\mathbfx^*\) is referred to as the global minimizer of \(f\), and \(f\) attains the least possible value at \(\mathbfx^*\). To design an algorithm that finds out the global minimizer for a function is quite difficult, as in most cases we do not have the idea of the overall shape of \(f\). Mostly our knowledge is restricted to a local portion of \(f\).

Definition 1.5 For an objective function \(f(\mathbfx)\) where, \(\mathbfx \in \mathbbR^2\), a point \(\mathbfx^s=\beginbmatrix x_1^s \\ x_2^s \endbmatrix\) is called a saddle point if \(\forall\ \mathbfx\), there exists an \(\epsilon>0\), such that the following conditions are satisfied:

\[\beginalign&\!\min_\mathbfx \in \mathbbR^n &\qquad& f(\mathbfx)=\mathbfc^T\mathbfx,\\ &\textsubject to & & \mathbfA\mathbfx \leq \mathbfb,\\& & & \mathbfx \geq \mathbf0,\\& & & \mathbfc \in \mathbbR^n, \mathbfb \in \mathbbR^m, \mathbfA\in \mathbbR^m \times n. \tag1.5\endalign\]

While formulating optimization problems, it must be guaranteed that the scale of the decision variables are approximately of the same order. If this is not taken care of, some optimization algorithms that are sensitive to scaling will perform poorly and will flounder to converge to the solution. Two of the fundamental fields that get disturbed due to poor scaling are computing the optimized step lengths and the numerical gradients. One of the widely accepted best practices is to make the decision variables dimensionless and vary them approximately between 0 and 1. One should always prefer optimization algorithms that are not sensitive to scaling.

Definition 1.9 For a differentiable objective function \(f(\mathbfx): \mathbbR^n \rightarrow \mathbbR\), its gradient vector given by \(\nabla f(\mathbfx): \mathbbR^n \rightarrow \mathbbR^n\), is defined at the point \(\mathbfx\) in the \(n\)-dimensional space as the vector of first order partial derivatives:

We will try to work out the same example with Python scripting now. For that we need an extra package called autograd, besides the numpy package. The autograd package is used for automatically differentiating native Python and Numpy code. Fundamentally autograd is used in gradient-based optimization. First pip install the autograd package

The above proof can be extended to positive semi-definite matrix too in which case the eigenvalues are non-negative, i.e, either 0 or positive and we will exploit these properties in our python script to check for positive definiteness or positive semi-definiteness of a given matrix.

The book by Nocedal and Wright [Nocedal, Jorge, and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.] states that most of the optimization strategies make use of either the objective function \(f(\mathbfx)\), the constraint functions \(g(\mathbfx)\) and \(h(\mathbfx)\), the first or second derivatives of these said functions, information collected during previous iterations and/or local information gathered at the present point. As Nocedal and Wright mentions, a good optimization algorithm should have the following fundamental properties:

I am a beginning PhD student in math, and I would like to focus on optimization. I am learning programming for the first time, and I have written out some rudimentary optimization algorithms in both Python and MATLAB. From my very limited experience, MATLAB was a whole lot easier to use than Python. Being able to do matrix and vector operations directly in MATLAB (rather than having to go through NumPy in Python) was extremely nice.

I saw this question, and it appears that Python is extremely popular in industry, at least in optimization. I was wondering if, to maximize my job prospects, I should learn Python or MATLAB? I would be interested in answers from any field, not only optimization.

Nikos Kazazakis and EhsanK have given you great reasons for using Python. I will focus on the point from you about needing to use an additional package/library in Python for matrix and vector operations.In industry projects you will encounter many challenges and things you want to do, but you don't want to (and shouldn't) implement. Currently I work on a route planning software, most of what I do is sadly not core optimization work, but everything needed to making it useful for customers. Make imports from various data sources, integrating the data into objects (doing this in a non object oriented language would be annoying), make your product available via an API, and so on. This is to say I use a lot of things that come not out of the box. Adding numpy to that list of imports for the optimization work won't break the camel's back.

Many bluechip and non B.C companies already have Python in heavy use.If I were you, I would concentrate on becoming knowledgeable with Python, not just for optimization purposes, but it's very useful in other areas of computing too.

If your answers are: 1: math, 2: academia, 3: no, 4: all short, then MATLAB is fine. It is an excellent tool. I love it for doing something fast and not fussing with things; it's an industry standard in DSP and radar and other problems that rely very heavily on linear algebra. There is a lot of code written in it; it will be around for a long time to come.

Nevertheless, if those are not your answers, then between the two, I definitely suggest Python over MATLAB. You will become a much better programmer, and your job prospects will be much better, too. If your code gets long, beware that managing a large (many 1000-lines) MATLAB code is a nightmare IMO (namespaces, anyone?). If you find you have to use MATLAB at some later date, it will be easy. Conversely, if you learn MATLAB and later on find you need to learn Python + NumPy + numba, then you will probably find this very difficult.

This is all the more true if we are talking about optimization, which is at least tangential to machine learning (ML). MATLAB is trying to compete in this space, but if you look online for job ads, they are almost all asking for Python, not MATLAB. In fact, in ML, you will find (in my experience) more job ads asking for Julia than for MATLAB.

But again (and yes it is just my personal opinion), I would suggest you take a look at Julia. I have not been as excited about a language after trying Julia since.... well, since I started learning Python 18 years ago. I use all three at work, and Julia is my first choice most of the time; I think the language is truly going places. But, I'm a risk-taker. It's not the safe move. If you want safe, the safe move is Python.

I did my PhD on a topic involving numerical simulations of mechanical systems. I worked primarily in MATLAB, which I already had experience in and seemed to have some good 'out of the box' optimisation algorithms.

Looking back, I wish I did my work in Python. I work in Software Engineering now and Python is a lot more applicable to other languages and looks better on a CV. Also - as mentioned by others here - Python is open source. This means that your results would be more accessible by others and easier to reproduce (a big plus for research).

How can this be, aren't both using the same libraries under the hood?Yes, but clearly Matlab is much better at recognizing special forms of matrices and choosing optimized solvers for them. Once my graduate students have have put several weeks into optimizing their code, it is still much slower (seconds vs. tens of minutes). You may think my graduate students suck - but some of them are in a leading computational math graduate program.

One reason I am more likely to switch to Julia than Python is that many of the advertised advantages of Python, such as great string manipulation simply don't matter for scientific programming. Also the syntax is clumsy and verbose. Most people that know what they are doing are picking up Julia. For example the new Climate model ( ) is being developed in Julia, not Python. The same is true for various Astrophysical codes ( -studies/celeste/).

My first language I have learned was MATLAB. After learning C++ I realized MATLAB is bad for really learning about programming in such. I would recommend you also Python as language because it is easier to switch from Python to MATLAB then in the other direction.

Besides the downturns of commercial and closed software you should also keep in mind that MATLAB is a working universe in itself. For example you can really quick do a program for image pattern recognition without having (to much) knowledge about image processing and link your code into another toolbox for autonomous driving (just as an example, because that is something people like to forget about MATLAB).Python has a little bit steeper learning curve but for your case it is much worther I think.

Sounds like you are intending to build solvers. You will be better off in MATLAB. As you noted, matrix and vector operations are part of the language. You'll need matrix decompositions and maybe eigenvalue solvers to build your solvers, probably for dense and sparse matrices, and MATLAB has all of those. If you find that there's a library you need in some other language, you can call it from MATLAB.

Python is a general purpose programming language and is much more widely used in industry than MATLAB. In the industry you rarely find yourself just code optimization algorithms or models in your daily work. It is more like you write an entire pipeline from data acquisition, data cleaning, model development (both statistical predictive models and/or optimization models) to visualization. Python provides more flexibility in that sense and have more APIs developed to communicate with other tools or applications (e.g databases) used in the pipelines.

3a8082e126