Benchmarking

[Zenna Tavares]

Recently we published DReal.jl : a nonlinear constraint solver which supports optimization and has a MathProgBase interface.

I'm interested in comparing performance of different solvers over different benchmark problems.  I have been working on a benchmarking library (not yet ready for public consumption) to make this kind of thing easy, but the problem here is lack of benchmark problems themselves.   I wrote about 50 examples from a survey optimization paper into JuMP by hand before concluding there must be a better way.

Perhaps that better way is AMPL?  As others have pointed out there are hundreds if not thousands of models in AMPL format, which.we could use.  To be concrete, I would like to know:

- Does there exist a comprehensive benchmarking project for optimization in JuliaOpt already?

- If not are people interested in collaborating to build this?

- Does there exist already (perhaps in AMPL.jl) code to parse AMPL models into JuMP or MathProgBase

- Are AMPL benchmarks still challenging for modern solvers (some of these benchmark collections are decades old)? I am in particular interested in very hard nonlinear optimization problems.

Thanks

Zenna

 

[Joey Huchette]

Does the MathProgBase interface only support unconstrained optimization?

The [CUTEr](http://www.cuter.rl.ac.uk) and [CUTEst](http://ccpforge.cse.rl.ac.uk/gf/project/cutest/wiki) libraries are apparently the de facto test suite for nonlinear optimization, but the interface is pretty hairy. There are julia bindings in [CUTEst.jl](https://github.com/optimizers/CUTEst.jl), though I haven't personally used them. It might be worthwhile poking around [Hans Mittelmann's collection](http://plato.asu.edu/ftp/ampl-nlp.html) as well.

- Does there exist a comprehensive benchmarking project for optimization in JuliaOpt already?

Not yet, but it'd be great to have one.

 

- Does there exist already (perhaps in AMPL.jl) code to parse AMPL models into JuMP or MathProgBase

Not yet, but again it'd be a great thing to have. We get requests for this kind of thing fairly regularly.

 

[Tony Kelman]

I sat next to Hans Mittelmann on a flight yesterday and was chatting with him about a number of things. It sounds like one thing he'd like to do in the not-too-distant future is post his benchmark problems in .nl format, which would be easier to work with without needing the proprietary parts of AMPL. I don't think the AmplNLReader.jl package currently implements the MathProgBase expression tree API (the AMPL solver library itself might not have a C API for doing this), I'm not sure. The pure-Julia code that is currently in AmplNLWriter.jl could also potentially be expanded to reverse the translation from .nl into Julia expressions, but we'd have to ask Jack Dunn just how hard that would be. I've been meaning to do the same thing for reading from .osil in my CoinOptServices.jl package but haven't gotten around yet to implementing the reading direction.

Victor Zverovich from AMPL has also been compiling together a set of test problems in .mod format at https://github.com/ampl/global-optimization, including most if not all of the cute set. We may want to contact him and ask what he'd think about translating those into .nl, or using that as a set of inputs to try writing a .mod to JuMP translation. I think it would be best to avoid going through a Fortran intermediate translation layer, which is the architecture in CUTEst that I don't think is a particularly sustainable design long-term.

[Miles Lubin]

We will not be able to directly read AMPL's .mod files unless we reimplemement AMPL itself. I don't think that will happen anytime soon. What we can read is the .nl format which the ampl executable generates from .mod files if you have a valid AMPL license. The AmplNLReader.jl package can read these files and provide the MathProgBase *derivative-based NLP* interface, but not yet the expression graph interface. There are three ways you could pull out these expression graphs, none of which are particularly fun or trivial:

- AMPL's C API can be used to query these expression trees, afaik, so one could develop an extension to AmplNLReader.jl which exposes this API

- I believe the AMPL/MP project has a much more sensible .nl reader, but it's in C++

- You could go ahead and parse the .nl format yourself by using the documentation and AmplNLWriter.jl as references.

That said, it will still require less brain power to translate models by hand than than to interface with the AMPL C interface. A good benchmark solver for global solutions is BARON. If DReal is faster than BARON at least in some cases, I think that would draw interest from the (MI)NLP community. We have an experimental wrapper to call BARON from JuMP.

[Tony Kelman]

If we like the API of the more modern .nl reader from ampl/mp better than the classic asl, Victor would probably be willing to entertain pull requests to add extern "C" wrappers for whatever parts of it we'd need.

[Dominique Orban]

Almost the entire CUTEst collection has been available in AMPL format for a long time. Michael and I are maintaining the problems here: https://github.com/mpf/Optimization-Test-Problems

The original tranlation is due to Hande Benson, who was Bob Vanderbei's PhD student at Princeton. Bob's website has many more fascinating problems in AMPL format.

It's possible to expose AMPL's expression tree once you have the nl file. That's essentially what I did in DrAMPL. From there, I presume tranlating this tree to another format is doable.

What happens is that a large chunk of the coders in the optimization community comes from the Fortran school, or are direct descendants of the Fortran school. So CUTEst, though it's in Fortran and its interface appears hairy to Julians, is still regarded as a reliable problem collection. I've been toying with the idea of having an undergrad student translate the CUTEst models to JuMP in the fall, if I can find one.

Dominique

[John Hedengren]

I just added a Julia implementation of the Hock Schittkowski test suite here:

https://github.com/APMonitor/apm_julia

You won't be able to test your solver until I make a MathProgBase interface to APMonitor. There is support for APOPT, BPOPT, IPOPT, SNOPT, and MINOS currently. I'm still new to Julia-Opt group but it sounds appealing to have an interface where programming languages and solvers can be interchanged.

-John Hedengren