constraints not recognized? upper/ lower bounds not recognized?

[vz101]

Hello group,

I'm struggling with getting my large, concrete model to run without the 'recursive' error. From what I've tried I'm seeing that there may be multiple issues that together may be producing the recursive error.

I've done the following:

1. I've re-structured the problem to be decision variables and constraints only, ie: no expressions. 

2. I've run it with diagnostics and get the attached error trace back referencing various pyomo scripts (for the 5 year run with 1825 time steps).

3. I'm running the model for less time steps (e.g., 2 or 365) to look for errors that might be causing the problem. The model solves properly (correct values/solution) but the output file does not list any constraints (see below). How is this possible? Am I interpreting the output file incorrectly?

4. In the same output, I notice the lower and upper bound of the problem are -inf to inf. I have defined all Var as Boolean, NonNegativeReals, PercentFraction, or positive values for the upper and lower bounds. I do not have any Param since concrete model. So why is pyomo thinking -inf to inf are possible for decision space?

Appreciate any tips, Viki

Here is the header of the output file for the 365 day model run:

Problem: 
- Lower bound: -inf
  Upper bound: inf
  Number of objectives: 1
  Number of constraints: 0
  Number of variables: 13646

  Sense: unknown

And here is the ending:

  Constraint: No values 

[Gabriel Hackebeil]

Viki,

3. I'm running the model for less time steps (e.g., 2 or 365) to look for errors that might be causing the problem. The model solves properly (correct values/solution) but the output file does not list any constraints (see below). How is this possible? Am I interpreting the output file incorrectly?

What you are printing is not the output file. This is the results object, which contains various information about the problem found in the results file returned by the solver. If you want to see the actual output file that Pyomo sends to the solver you can add keepfiles=True to the solve command (or --keepfiles when invoking the pyomo command). This should print a line such as:

Solver problem files: ('/var/folders/p5/4dj9d59x27v7_5_ddgn8r8zm0000gn/T/tmp_zRz5g.pyomo.lp’,)

You can inspect this file directly to see what is getting relayed to the solver. It might also help to add symbolic_solver_labels=True (--symbolic-solver-labels), so that the file contains human readable names that match with how you declare components on the model. Also, adding tee=True (--stream-solver with the pyomo command) will stream the solver output to the terminal so you can get a better idea of what is happening.

4. In the same output, I notice the lower and upper bound of the problem are -inf to inf. I have defined all Var as Boolean, NonNegativeReals, PercentFraction, or positive values for the upper and lower bounds. I do not have any Param since concrete model. So why is pyomo thinking -inf to inf are possible for decision space?

This all depends on the solver. Depending on the problem type and the solver interface, these values may not be reported with the solution. It might also be an indication that there was a solver failure. We have an issue right now where we sometimes aren’t reporting solver failures when they happen, so the code will just silently proceed as if everything is okay. It’s best to do something like the following when you are scripting:

from pyomo.opt import TerminationCondition

...

results = opt.solve(instance, load_solutions=False)

if results.solver.termination_condition != TerminationCondition.optimal:

    raise RuntimeError('Solver did not report optimality:\n%s’ % (results.solver))

instance.solutions.load_from(results)

And here is the ending:

  Constraint: No values 

This has to do with the solution. It means that there are no values in the solution associated with constraints. Examples of such values would be solutions to the dual problem (e.g., when solving an LP).

  Number of objectives: 1
  Number of constraints: 0
  Number of variables: 13646

This output does indicate, however, that your model might not have any constraints on it.

Gabe

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<RecursionTraceback_20160114.JPG>

[vz101]

Gabe,

Thank you for the ideas. Based on them and from a previous post of yours, I ran the model for only 2 time steps to get diagnostics with 

--report-timing --profile 10 --keepfiles --symbolic-solver-labels --stream-solver

Two interesting results

1. It seems that .nl file recognizes the constraints. And solution is returned. Yet constraints are said to be zero by results.yml. Perhaps error in generation of results.yml file?

2. none of the diagnostic files nor text streamed to command window (which is cut off because too many lines of code) show information about specific variables or constraints. They all reference pyomo code. I had seen a previous post where the user was able to get back information about which constraints may be the most time consuming to generate to help with the recursive issue. Am I not requesting that information properly?

Thanks again for your time and help, Viki

[Gabriel Hackebeil]

Yet constraints are said to be zero by results.yml. Perhaps error in generation of results.yml file?

It is possible that this is a bug with the solution parser. It would be helpful if you submitted a ticket about this (https://software.sandia.gov/trac/pyomo/newticket). If you attach your model (or a smaller example) and results file to that ticket, I’ll try to reproduce the issue and see if it’s something we can fix.

2. none of the diagnostic files nor text streamed to command window (which is cut off because too many lines of code) show information about specific variables or constraints.

There is probably a setting you can adjust that increases the buffer size for your terminal so that you can scroll back over more lines of output (assuming this is what you mean).

They all reference pyomo code. I had seen a previous post where the user was able to get back information about which constraints may be the most time consuming to generate to help with the recursive issue. Am I not requesting that information properly?

I’m almost certain that if you look closely at the stack trace there will be reference to a particular line in your model file, which you can map to a particular constraint. The recursion issue has to do with the Python interpreter (similar to a Python syntax error). There is not really anything we can do in Pyomo to help make the error message more helpful in this case. Someone correct me if I’m wrong about this.

It sounds like you are referring to the “report_timing=True” keyword that you can pass into model.create_instance(…) (--report-timing for the pyomo command). This prints timing information as components are constructed on your model. This might help diagnose your issue, but for the time being you have to start from an AbstractModel object for it to work.

Gabe

<results.yml><StreamedToWindow.JPG><tmp1v61g6r8.pyomo.col><tmp1v61g6r8.pyomo.nl><tmp1v61g6r8.pyomo.row><tmp1v61g6r8.pyomo.sol><tmpelu0ag54_bonmin.log><tmplkrhmg_z.profile>

[vz101]

Thanks Gabe. As you've seen, I've submitted a ticket for the output file anomaly.

I've run the model as an abstract model to get timing diagnostics in addition to the "profile" option. I've redirected the output to a file so I wasn't limited by windows command line. Attached is the output. I renamed the model Wmodel to help find where outputs refer to the specific model in question. 

1. For profile: I see many references to "Wmodel.py:2(<module>)" and a few "Wmodel.py:4621(cost)". The numbers are simply referencing the line of code in Wmodel where the abstract model is created and the objective function "cost" is defined. Am I supposed to get more information by looking at information before and after these references?

2. For timing: in attached file, there is 0 seconds associated with each component creation but in a previous run, I got 0.02 sec for two constraints. Does that mean those constraints might be reconfigured for better outcomes? If so, reconfigured how? 

3. traceback - attached is traceback for 5 year version that crashes. I've looked and searched for reference to Wmodel but do not see any. What am I missing?

I do wonder whether the recursion issue is due to the fact that i have mass balances for multiple watershed components for each time step and each time step's values affect the next time step. I don't know how I would get around this issue since it's not an artifact of the way I construct constraints but the way the physical watershed is that I am modeling. In that case, I wonder whether we can continue with Pyomo??

[Gabriel Hackebeil]

1. For profile: I see many references to "Wmodel.py:2(<module>)" and a few "Wmodel.py:4621(cost)". The numbers are simply referencing the line of code in Wmodel where the abstract model is created and the objective function "cost" is defined. Am I supposed to get more information by looking at information before and after these references?

These are usually more helpful when you define constraints over a set of indices. In your case, it looks like you define each individual constraint using its own Constraint declaration, so this isn’t very helpful at pointing you at groups of constraints that take the most time to build.

2. For timing: in attached file, there is 0 seconds associated with each component creation but in a previous run, I got 0.02 sec for two constraints. Does that mean those constraints might be reconfigured for better outcomes? If so, reconfigured how? 

There are various ways to write a constraint expression, some of which will generate intermediate copies of subexpressions before the final expression is created. There are ways to avoid this. For instance:

# slow

e = 0.0

for i in range(10000):

    e = e + model.x[i]

# fast

e = 0.0

for i in range(10000):

    e += model.x[i]

# also fast

e = sum(model.x[i] for i in range(10000))

3. traceback - attached is traceback for 5 year version that crashes. I've looked and searched for reference to Wmodel but do not see any. What am I missing?

You’re not missing anything. I was incorrect in assuming this would show something more helpful. It might help to just directly execute your model file with the python interpreter. It could be that the import method used by the pyomo command is masking a more helpful message. If you're comfortable sharing the 5 year model with me, I don’t mind taking a look to see if I can offer any help.

I do wonder whether the recursion issue is due to the fact that i have mass balances for multiple watershed components for each time step and each time step's values affect the next time step. I don't know how I would get around this issue since it's not an artifact of the way I construct constraints but the way the physical watershed is that I am modeling. In that case, I wonder whether we can continue with Pyomo??

I think the way around this is to add some intermediate variables and constraints. I think the recursion error is the result of something like the following:

constraint: x[1] + x[2] + … + x[N],

where N is so large that the expression tree generations hits this maximum recursion limit. The way around this would be:

constraint:                  x1_100 + x101_200 + … + xNm99_N

constraint1_100:        x1_100 = x[1] + x[2] + … + x[100]

constraint101_200:    x101_200 = x[101] + x[102] + … + x[200]

…

Obviously you would probably have some more logical grouping of subexpressions into intermediate variables and constraints that makes sense for your model.

Gabe

<cmdoutput_2steps_abstract.docx><cmdoutput_5yrs_abstract.docx>

[Siirola, John D]

I think the way around this is to add some intermediate variables and constraints. I think the recursion error is the result of something like the following:

 

constraint: x[1] + x[2] + … + x[N],

 

FYI: this shouldn’t create deep trees (the SumExpression is a n-ary expression, so the tree should be wide but shallow).

 

I will echo Gabe’s comment: if you can share a model (off-list is fine) that hits the recursion limit, then we can take a look and see (a) what is causing it and (b) if we can add more helpful error trapping in Pyomo to assist users in figuring out what is wrong and how to avoid it.

 

john

[vz101]

The solution was as Gabriel suggested offline. I needed to create an abstract model and use indexed variables and parameters. For me the index was time steps. This way I did not end up with lines that were 100,000s characters long (used sum function to sum over time steps) and made use of reading in data from .dat file for abstract model. After re-structuring the dreaded 'recursive error' message stopped and the solver initiated. Thank you for the help on this error that was so persistent!

To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<cmdoutput_2steps_abstract.docx><cmdoutput_5yrs_abstract.docx>

 

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

...