Load .sol file into instance object

[Johannes Bethcke]

Hello,

We're at a loss here. We want to generate <1000 .nl-files from slightly differing model and data files. For example: One instance with binary variable pumpOperating_25 relaxed and 4 time steps, a second instance with lower bounds on variables n_d and a particular high demand of something, and so on. This is the first job Pyomo. We use

instance = model.create_instance(data)
instance.write(path+"someName.nl", "nl", io_options={"symbolic_solver_labels": True})

This also generates a .row-file and a .col-file. Then we want to solve the <1000 .nl-files with a solver like SCIP.

Then we're planning to collect the .sol-files that SCIP produced, load them into the corresponding instance in Pyomo and evaluate (by plotting, accessing the optimal values, or the value of the pressure variables). This is the second Job of Pyomo. And here lies our Problem. We tried several things, for example:

instance.solutions.add_solution(path+"someName.sol", someSymbolicMap)

Would you know how to pass (or where to find) the symbolic map? We tried things like 

someSymbolicMap = (path+"someName.row", path+"someName.col")

But nothing worked.

Greetings from Frankfurt,

Johannes

[Gabriel Hackebeil]

The symbolic_solver_labels option is meant as a debugging tool. The row and col files are not meant to be used as a way for Pyomo to recover the symbol map (although they can be in many cases).

Is there any particular reason you are not just calling SCIP through our solver interface? Here is an example.

from pyomo.environ import *

from pyomo.opt import TerminationCondition

instance = model.create_instance(data)

with SolverFactory(‘scip’) as opt:

    result = opt.solve(instance)

    if results.solver.termination_condition == TerminationCondition.optimal:

        # do something (note that the solution is already loaded)

    else:

        # do something else

Currently our SCIP solver plugin only supports the NL input format that. That is you need to have scipampl compiled (but it looks like you are already using scipampl anyway).

Gabe

someSymbolicMap = (path+"bla.row", path+"bla.col")

But nothing worked.

Greetings from Frankfurt,

Johannes

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johannes Bethcke]

We want to carry out the calculations on a High Perfomance Computer (http://www.hhlr.tu-darmstadt.de/hhlr/index.en.jsp). One advantage is the availability of many (~750) cores. If we use let's say 16 cores for every instance we are able to solve many instances simultaneously. And we might be able to make use of parallelization.

So you would pass the server a script like

for i in range(100)
scip someMINLP(i).nl

You could of course also execute a python script. But we figured there wouldn't be a way to distribute the jobs meaningfully on the High Performance Computer. That is, Pyomo would either solve one instance after the other or try to solve all instances simultaneously. What do you think?

[Gabriel Hackebeil]

We have some Pyro-based tools that we use for similar purposes. Example usage would be (these commands are installed with Pyomo):

# launch the nameserver

$ pyomo_ns -n localhost &

# launch the dispatcher

$ dispatch_srvr -n localhost &

# launch the workers (you might have to use some mpirun magic to make sure these bind to separate cores?)

$ mpirun -np 30 pyro_mip_server

With the nameserver, dispatch_srvr, and pyro_mip_server (this is a bad name, it doesn’t have to solve MIPs) processes running, you can write a client script that hands jobs over to the dispatcher. We have a built-in solver manager to help do this. Here is some example code:

from pyomo.environ import *
from pyomo.opt import SolverManagerFactory
...
action_handles = dict()
with SolverManagerFactory(“pyro”, host='localhost’, port=0) as manager:

    with SolverFactory('scip') as opt:

        opt.options[‘threads’] = 16
        for i in range(100):
    instance = ...
            action_handles[manager.queue(instance, opt=opt)] = instance

while len(action_handles):
    ah = manager.wait_any(action_handles)
    instance = action_handles[ah]
    results = manager.get_results(ah)
    del action_handles[ah]

    … check results for termination condition, etc.

The downside with this approach is that it keeps all of the instances in memory.

Gabe

[David Woodruff]

Hi Johannes,

  Have you had a chance to look at the documentation in the scripting section called "Solving Multiple Instances in Parallel" in the documentation available at pyomo.org?

https://software.sandia.gov/downloads/pub/pyomo/PyomoOnlineDocs.html#_solving_multiple_instances_in_parallel

Dave

--

[Johannes Bethcke]

Thank you for your detailed answers. We tried this but are not really getting somewhere. One reason is that communication with the admins of the High Performance Compute is lengthy. We have virtually no background knowledge here.

For now this would leave us with the quick and dirty way I think. That would be, reading the .sol files into Pyomo and make use of its access to the many python libraries for plotting etc. What do you think?

Regards,

Johannes

PS: I'll be on vacation for two weeks:) Merry Christmas!

[Gabriel Hackebeil]

Johannes,

I’ve added an example to the PyomoGallery that probably does what you are looking for. In case it has not yet been merged onto the Wiki there, you can view the iPython notebook file directly on my GitHub fork at https://github.com/ghackebeil/PyomoGallery/blob/gabe/asl_io/asl_io.ipynb

Gabe

On Dec 16, 2015, at 8:49 AM, Johannes Bethcke <johanne...@hotmail.de> wrote:

Thank you for your detailed answers. We tried this but are not really getting somewhere. One reason is that communication with the admins of the High Performance Compute is lengthy. We have virtually no background knowledge here.

For now this would leave us with the quick and dirty way I think. That would be, reading the .sol files into Pyomo and make use of its access to the many python libraries for plotting etc. What do you think?

I attached an exemplary .nl-file and the corresponding .sol-file for clarification.

Regards,

Johannes

[Gabriel Hackebeil]

Updated link: https://github.com/ghackebeil/PyomoGallery/blob/asl_io_example/asl_io/asl_io.ipynb

[Johannes Bethcke]

Dear Gabe,

Your code did the trick! We are ready to go.

Thanks!

Greetings from Frankfurt,

Johannes

[Jose Lenin]

Thanks for this example, it works very nice. However, I can't retrieve the dual variables in this way, do you know an alternative procedure to do that?

[Gabe Hackebeil]

Do the dual variables get loaded if you solve the problem directly with a solver (not through the pyro solver manager)? If not, then you probably need to add a Suffix to your model with the name "dual", or the solver does not report duals for your problem class (e.g., it is a MIP). If duals do get loaded the other way, then it is a bug with the pyro solver manager. You might try updating to the latest version of Pyomo to make sure it's not something that has been fixed.

If it still doesn't work, please open an issue on our GitHub page.

Gabe

[Jose Lenin]

Hi Gabe, Thanks for answer.

Using the solver (without pyro) I get the dual without problem. However, the problem is recovering solutions from pyro,

Using your example,

while len (action_handles):
     ah = manager.wait_any (action_handles)

     instance = action_handles [ah]
     results = manager.get_results (ah)

at this point, I wanted to get the value of the objective function (it works)

     instance.solutions.load_from(results)
     sol_objective = instance.obj()

And here, previously added suffix, the dual is not found in the object

     d_object = getattr (instance, 'myConstraint')
     dual = instance.dual [d_object ['element']]

     del action_handles [ah]

Am I doing something wrong? I have noticed that the duals are just in the last ah.