Reshaping a pyomo model - mutable indexes / rangesets

[Tom Kent]

Hi all,

I am currently working through the methods of taking my pyomo model, altering a mutable parameter value and resolving. I have this working well, and the resolve time is much smaller than creating a whole new instance - so I am very happy.

However I would like to go a little beyond that and actually change / reshape an indexed variable's size. 

So for example I have a parameter 'h' which is essentially an N by 3 vector of points which defines some polygon and I would like to be able to add and remove points from this vector and resolve accordingly. So for example i have:

model.N = Param(within=Integers, mutable = True) # index of the number of points

model.indexN = RangeSet(1, model.N) # Range over the points

# model.Id is a range set of {1,2,3} 

model.h = Param(model.indexN, model.Id, mutable = True) # N by 3 vector, mutable so I can alter it if I do not resize. 

This is then later used in a whole load of constraints (which I won't bore you with), and I assume I will need to alter them too.  

So, my question really is: Is there an efficient (or proper) way to change the size of h. 

That is change N, change indexN and then change h, then go through and update the later constraints accordingly.

I've attempted so far to use del_component and then just creating the Param again, and then setting the value of this param with getattr / setattr, but I feel that I need to alter a lot more indexes and attributes that are automatically created when data is 'properly' imported. 

I am currently using an abstract model which is initialised with a dict of data using the data=data_dict flag to avoid creating a intermediate .dat file. With this I have something similar working where I just alter this data_dict and then run create_instance on this updated data.

Does this sound at all doable - without having to run a whole create_instance, but by modifying the pyomo model directly?

I appreciate any input or words of wisdom.

Many thanks,

Tom

[Gabriel Hackebeil]

It’s hard to answer this question without knowing more about how how h is being used.

I am currently using an abstract model which is initialised with a dict of data using the data=data_dict flag to avoid creating a intermediate .dat file. With this I have something similar working where I just alter this data_dict and then run create_instance on this updated data.

In cases like this it is often more natural to work with a ConcreteModel. When you place a model definition inside of a function that takes

data as an argument and then populates and returns a ConcreteModel, you are essentially defining an abstract representation of that model. There is no need to duplicate your data inside Param components as you can work directly with your data (e.g., dicts, lists, objects) inside expressions.

You can even forgo the creation of Set() objects to index components and simply pass raw Python list or set object as arguments to indexed components. This is not recommend at this time as it causes Pyomo to implicitly create Set components that are added to the model under names like <component_name>_index (which can lead to unnecessary duplication of data and strange warnings in situations where you add and remove components). We are exploring the possibility of component containers that do not require external Set components, just as Python does not require an external set when you populate a dictionary or list.

That being said, you are using a feature of Param (mutable=True) that can not be duplicated using a raw Python data type. That is, expressions that reference the Param can be updated by updating the value of the Param. Note that Params were originally designed for the AbstractModel setting, with their main purpose being placeholders for external data (e.g., DAT files). Pyomo actually implements this idea using delayed construction rather than mutability. Thus, mutability of subexpressions is an orthogonal feature in Pyomo, and it has been generalized with a component called Expression. Expression components are like mutable Params except their value can be set to any other general expression (e.g., x + 5 rather than just 5). The drawback is that, unlike Params, Expressions can not be initialized from external data in the AbstractModel setting.

So, my question really is: Is there an efficient (or proper) way to change the size of h. 

The official answer is that unless you can find some way to avoid adding and removing indices of a component, you’re going to find yourself in the situation you are in, where you need to do record keeping over various components and their index sets lest you run into strange Pyomo errors (or you just rebuild the components entirely). There are components like ConstraintList that support add operations, but you’re limited to an index of the form 1…n and remove operations are not supported (unless you remove from the index set that gets implicitly created as well). Deactivating indices might be another option to explore rather than deleting them.

Unofficially, I can point you to some prototype components for Expression and Constraint that do not use index Sets, but you would need to be in the ConcreteModel setting to test them out. The general idea is that you explicitly populate them in a for-loop (like Python list and dict objects) rather than by defining a rule function that gets called for each index. This gives you the flexibility of later adding / overwriting / removing indices without updating an external index (e.g., model.c[5] = …, del model.c[‘a’, 2]).

Gabe

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Kent]

Hi Gabe,

Thanks for the detailed reply. I did create my model using the alternative concrete form for use with resolving after changing a few params - I used the whole del_components but it seemed like a great deal more work for what could be achieved with the abstract model. 

Since posting, I have made some headway (and got a little stuck in the inner workings of pyomo) - I am attempting to recreate the '_intialize_component' function in PyomoModel.py in my own code, specifically for 're-initializing' components. Although it is not yet working, I feel like this could be a reasonable way to go. In effect I am trying to mimic what Pyomo would be doing on a typical abstract initialization, but only for the bits that change, saving the overhead of recreating the entire instance again. 

I will update when/if I get it working - but do you think this sounds doable or just plain crazy? 

Thanks,

Tom 

[Matthias Fripp]

I know this is a late response, but it may be useful.

I have a model that repeatedly extends an indexing set and various components indexed over that set, and then re-solves. This is the approach I use, which works pretty well. 

from pyomo.environ import *

m = AbstractModel() # or m = ConcreteModel()

m.S = Set(initialize=[1, 2])

m.p = Param(m.S, initialize={1: 1, 2: 4}, mutable=True)

m.X = Var(m.S, within=NonNegativeReals)

m.C = Constraint(m.S, rule=lambda m, s: m.X[s] >= m.p[s])

m.E = Expression(m.S, rule=lambda m, s: m.X[s] + 5)

m.objE = Expression(rule=lambda m: sum(m.E[s] for s in m.S))

m.O = Objective(rule=lambda m: m.objE, sense=minimize)

# create a model instance (usually you wouldn't bother with if...:)

if isinstance(m, AbstractModel):    

    i = m.create_instance()

else:

    i = m

# Modify the instance

i.S.add(3)

i.p[3] = 9 

i.X.reconstruct()

i.C.reconstruct()

i.E.reconstruct()

i.objE.reconstruct()

# instance is ready for solving...

i.pprint()

All the reconstruct() calls are necessary because (a) this is the only documented way to extend the indexing of each component, and (b) once you reconstruct one component (e.g., m.X), the elements of any other objects that refer to elements of that component become outdated. e.g., m.E[1] refers to a vardata object which used to be m.X[1], but is no longer part of the model. Rebuilding m.E creates a new expression object m.E[1], which refers to the new vardata object at m.X[1], and also adds m.E[3] referring to the newly added m.X[3]. But then you have to rebuild m.objE to refer to these new expression objects. 

Note: you can only use reconstruct() this way for objects that have been defined via rules. If you define components (e.g. a constraint) directly via expressions, you may be able to add another term with something like i.C[3]=(m.X[3] >= m.p[3]) instead of i.C.reconstruct(), but then you will run into trouble with m.X. If you previously used m.X.reconstruct() to extend m.X, then i.C[1] will still refer to the old i.X[1] vardata object, which is no longer part of the model. So you'll have to re-create i.C[1] and i.C[2] somehow. Alternatively, you could try to find some way other than i.X.reconstruct() to extend i.X to include i.X[3] before you create i.C[3].

I'm not sure if my approach is the officially recommended way to build these kinds of models, but it seems to work pretty well. I don't use any reserved functions, and I may even have gotten advice on the reconstruct() calls from this list. But some changes in Pyomo 4.2 broke the reconstruct() function for Expressions and Objectives, so I don't think the Pyomo people consider this a mainstream technique. (Oh boy do I wish there were a way to automatically propagate changes like this through a model.)

By the way, you can work around the problem with Objective.reconstruct() by defining a single-valued Expression (as I did here) and then reconstructing that. This works because the Objective only refers to the Expression object, not any indexed sub-elements. The whole component retains its identity during reconstruct(), unlike sub-elements of indexed Expressions, which get discarded during reconstruct().

It's a little harder to work around the Expression.reconstruct() problem in Pyomo 4.2. You can either stick with 4.1, or use a patch like this sometime before you define your model:

from pyomo.environ import *

import pyomo.version

if pyomo.version.version_info >= (4, 2, 0, '', 0):

    # Pyomo 4.2 mistakenly discards the original expression or rule during 

    # Expression.construct(). This makes it impossible to reconstruct expressions

    # (e.g., for iterated models). So we patch it.

    # Test whether patch is still needed:

    test_model = ConcreteModel()

    test_model.e = Expression(rule=lambda m: 0)

    if hasattr(test_model.e, "_init_rule") and test_model.e._init_rule is None:

        print "Patching incompatible version of Pyomo."

        old_construct = pyomo.environ.Expression.construct

        def new_construct(self, *args, **kwargs):

            # save rule and expression, call the function, then restore them

            _init_rule = self._init_rule

            _init_expr = self._init_expr

            old_construct(self, *args, **kwargs)

            self._init_rule = _init_rule

            self._init_expr = _init_expr

        pyomo.environ.Expression.construct = new_construct

    else:

        print "NOTE: Pyomo no longer removes _init_rule during Expression.construct()."

        print "      The Pyomo patch for Expression.construct() is probably obsolete."

    del test_model

I hope this helps.

Matthias