Ignore order of component definition during model initialization

[Matthias Fripp]

I have a large model which I would like to rewrite using Pyomo. I would like to group my code by topic (i.e., one subsystem of the model, then another), rather than sequencing the code based on which components depend on which. In the future, this would allow me to break the model into separate files for each subsystem, making it much easier to maintain and understand.

However, this is currently impossible because Pyomo initializes components in a single pass in the order they are defined. This means that each component must be defined after all the components that it depends on. So if two blocks of code (e.g., parts of my model, written in logical groupings) refer to components that are defined in each other, Pyomo will always raise errors during the initialization phase, when it discovers that one component depends on another one that hasn't been constructed yet.

The solution to this problem would be to make Pyomo indifferent to the order that components are defined in. This could be done in a fairly straightforward manner, but I wanted to ask the community (and particularly Pyomo's core developers) whether this idea raises any red flags.

Here is a tiny example of code that currently raises these errors:

============

from pyomo.environ import *

model = AbstractModel()

model.S = Set(initialize=[1, 2])

model.X_cons = Constraint(model.S, rule = lambda m, s: m.X[s] >= 2)

model.X = Var(model.S)

model.obj = Objective(expr = lambda m: summation(m.X), sense = minimize)

instance = model.create()

============

Note that X_cons is defined before X. Consequently, during the create() operation, Pyomo tries to access X while constructing X_cons, and discovers that X has not been constructed yet. It then raises ValueError("Error retrieving component X[1]: The component has not been constructed."). This example is fairly contrived, but it is parallel to real problems I am having with writing my model. If two blocks of code refer to components that are defined in each other, then there is no way to sequence them to avoid this kind of problem.

This problem would not occur if Pyomo were indifferent to the order that components are defined in. One way to achieve this would be to initialize the model data in multiple passes, repeating until all components are initialized successfully. This would be similar to the current initialization loop (the main loop in pyomo.core.base.PyomoModel._load_model_data()). However, if an error arises like the one above, it would be ignored and the loop would move on to the next component. After the loop finishes, if any of these errors have occurred, the whole initialization loop would be repeated. During subsequent passes, the loop will ignore components that were successfully initialized previously. This process would be repeated until the initialization loop completes without any errors, or until it completes without initializing any additional components (indicating circular dependencies).

I have added code to do this to my copy of PyomoModel.py (attached), and it seems to work fine. This behavior could be quite useful, since it frees modelers from having to worry about the order they in which they define components. I can't think of any downside, but I'd be interested to hear people's opinions. 

Thanks,

Matthias Fripp

[Watson, Jean-Paul]

You can define blocks of components within Pyomo – ConcreteModel and AbstractModel are glorified blocks. And you can nest blocks, which allows you to represent different subsystems. So you could do something like:

my_model = ConcreteModel()

my_model.subsystem_a = Block()

my_model.subsystem_a.some_var = Var() # keep adding block-specific components

my_model.subsystem_b = Block()

This gives you the nested structure that you’re looking for. Howevrer, you should not have components in one subsystem refer to components in another subsystem – at least within a block. Blocks should be self-contained. If there are constraints that relate variables in two subsystems, that constraint should be defined on the parent (my_model, in the above example) block. 

This isn’t precisely what you’re looking for, but is generally what we recommend when modeling hierarchical systems.

Now, regarding in-order construction of components – this is absolutely necessary, at a minimum for efficiency reasons. When constraints are constructed, they refer to individual data blocks of variables and parameters – if those variables and parameters are not yet constructed, then we would have to somehow dynamically infer that, and then delay construction. Computing the topological sort would also incur run-time overhead, which we try to avoid where possible. 

jpw

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Siirola, John D]

Matthias,

On the surface there is nothing wrong with your suggestion of recasting abstract model construction into a repeat-until-success model for advanced modelers. However there a couple gotchas that could arise - especially for novice modelers:

 - the first is that this scheme would only work for Abstract and not Concrete models. 

  - there are subtleties around what happens to partially-constructed components. For example, consider a ConstraintList where a rule successfully yielded a series of constraints before hitting an exception. This would be flagged as a failed construction and would be be-called in the next main pass. But, what should happen to the constraints that were successfully constructed?

  - this is a potentially big gotcha: there is nothing that prevents Pyomo rules from having side-effects (opening files, iterating through external data, setting / reading non-component (Python) variables, etc). As with the previous case, the final model could be very different from what the modeler intended, as the side-effects could be encoded more than once. 

  - this could invoke havoc with how we designed immutable Params with default values. I could easily see a situation where a param could provide its default value on the first pass and then a potentially different value the second pass. 

  - the scheme would probably have to "skip" any construction that generated any exception. This could mask strange errors (or potentially generate new ones - particularly for rules with side-effects). 

While I normally do not recommend diving into the (private, undocumented) bowels of the pyomo component model, if you look at the block class (which is the base class for models), you will find a list that defines the declaration order (_decl_order). If you change that order, you can change the order that the components will be iterated over. The easiest way to change the decl order is to first remove all the components from the model (block.del_component) and then re-add them in the order in which you want them constructed.

The "weirdness" plus the 

[Siirola, John D]

(Bumped send)

The weirdness that could happen - and the difficulty to make sure we can trap the edge cases - will make it unlikely that we would implement repeat-until-success in trunk. 

One can also make an argument that for decomposeability, you should think long and hard before having blocks that co-depend on each other (as JP hinted at). 

John

[Matthias Fripp]

John and JP, 

Thanks for these very thoughtful responses. I thought about the issue of partial construction after I sent my message, and you have pointed out several further issues that I didn't think of. For now, I think I will just follow a fixed order for my code sections, and ensure that any shared variables are declared in the first section that uses them (even if it's not necessarily the section where they "fit" best). 

I do still think there would be some value in giving AbstractModels the ability to ignore the order that components were defined in. But due to the issues of partial construction and side-effects, I think the repeat-until-success approach is probably not the best way. A cleaner option might be to recursively construct components whenever an unconstructed one is encountered during model initialization. This could work something like this: 

(1) PyomoModel.Model._load_model_data() sets a flag like self.being_constructed = True when it begins and self.being_constructed = False when it finishes.

(2) constraint.Constraint.construct() and similar methods set self.being_constructed = True when they begin and self.being_constructed = False when they finish.

(3) if sparse_indexed_component.SparseIndexedComponent.__getitem__() finds (not self._constructed), it responds in one of three ways: 

(i) If (not model.being_constructed) then it reports an access error, as now (e.g., due to accessing a component before model initialization).

(ii) if (model.being_constructed and not self.being_constructed) then it calls self.construct() (if available). This will cause each component to construct itself automatically if another component accesses it during the model construction phase.

(iii) If (model.being_constructed and self.being_constructed), then it would report a circular dependency and raise an exception. This will occur if one component accesses another one during the construction phase, causing recursive construction which eventually circles back to the first component.

I think this would address the concerns about performance, partial construction and side-effects. But I don't know whether calling component.construct() at a random point during the model initialization is enough to properly initialize a component. And I know this would interfere with the timing-reporting code in _load_model_data().

Anyway, I think I can work around the code-sequencing problems for now, but I may revisit this idea later if needed.

Thanks again,

Matthias

[Siirola, John D]

Matthias,

 

The recursive approach to Abstract model construction is actually intriguing.  I do not see an obvious argument against supporting it, apart from a couple trivially-manageable ones.  The biggest is that CPython supports only a rather shallow stack, so people who heavily rely on the recursion (think, writing your model in reverse order) could run across a stack overflow.  Similarly, unless you keep / manage the recursion stack yourself (or switch the Component constructed flag from a binary to a three state indicator), circular references would result in infinite recursion (until blowing the Python maximum stack depth).  You would also need some form of a global data portal object so that implicitly constructed components could find their data.

 

From a procedural programming standpoint, it still makes me uneasy (it flies in the face of most declarative programming models) – but I concede that this may come from spending too much time writing procedural code.

 

The best path forward from here is to file an enhancement ticket in the project Trac site.  I don’t think that we will be implementing it anytime soon (you will see that we have a large backlog of other things we want to do).  That said, we certainly welcome patches (especially when they come with unit tests!) and folks who want to become more involved with Pyomo development.  :)

 

john

[Matthias Fripp]

John,

Thanks for this encouraging response. I wasn't aware of the CPython stack depth problem. I doubt there are many models with more than 1000 components, but I guess you can't rule it out completely. I sketched out an idea for a de-facto three-state construction flag (by adding a flag called "being_constructed"). But I don't know enough about how the components are matched with data during the construction process to move ahead on this. If it ends up being very important for my model, I can work up some code (and unit tests) for my own use and then propose it as a patch.

Best,

Matthias

You received this message because you are subscribed to a topic in the Google Groups "Pyomo Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyomo-forum/dLbD2ly_hZo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyomo-forum...@googlegroups.com.

[Siirola, John D]

Matthias,

 

Final notes (mostly so that it is captured somewhere in the archives):

 

-          The stack depth is the function call depth.  When we construct a component, the component’s construct() generally calls a helper function that iterates over the indices calling the individual rules.  Within a rule, you can have a number of function calls related to operator overloading (this is how we build up the expression trees).  Finally within that, there will be a call to the Var/whatnot’s __getitem__ method.  It is at this point that we can realize that the Var is not constructed and call its constructor – likely with a helper function.  So, it is not unreasonable to assume you could get O(10) function calls on the stack each time you recurse back one component.

 

-          I was thinking to convert the “_constructed” flag to be a 3-state variable (maybe None, False, True) and not add another class attribute.  False might indicate ‘not constructed’, None could indicate ‘constructing’ and True would be ‘constructed’.  If you do it correctly, you can leave most of the existing tests for constructed the same (they should all use “if not self._constructed” – and False and None will both map to logical False).

[William Hart]

Matthias (et al):

I know that John and JP have commented, but I wanted to voice my concern about this approach.  Specifically, is this solving a problem that we need to solve?  This would clearly add significant complexity to the semantics of model construction, but to what end?

Software developers are generally faced with the reality of defining functions/classes/data before they are used.  So why should we expect modelers to be able to avoid this in their work?

And ... who would use this?  Only users who are working with AbstractModel objects, right?  My general sense is that new Pyomo users are more commonly using ConcreteModel objects.  So are we adding functionality that few users will employ?

--Bill

P.S.  If we really want to do this, I think we shouldn't do this as part of the construction logic.  Rather, we can run a preprocessing step to parse the construction rules, identify component dependencies, and reorder the components.  This segregates the reordering logic from the model construction logic, which seems like a saner thing to do.

--Bill