Loading indexed sets from tabular data

[Vaaras]

Hi,

I ran to the issue of being unable to load indexed sets from csv files with the dataportal. The documentation mentions indexed sets but doesn't show any examples. Am i doing sometihng wrong or is it not possible to load indexed sets this easily.

A case similar to mine:

m = Pyo.abstractModel()

m.I = pyo.Set() # set of processes

m.Ji = pyo.Set(m.I) # set of materials associated with process i

...

data=pyo.DataPortal()

data.load(filename='../data/Itable.csv', set=m.I)

data.load(filename='../data/Jitable.csv', set=m.Ji)
instance = m.create_instance(data) 

Where

Itable.csv:

I

P1

Jitable.csv:

I,Ji

P1,A

P1,B

ERROR: Constructing component 'Ji' from data={None: [('P1', 'A')]} failed: KeyError: "Index 'None' is not valid for indexed component 'Ji'"
KeyError: "Index 'None' is not valid for indexed component 'Ji'"

What i want the Ji set to be defined as is {'P1':['A', 'B']}.

Should i move to implementing the data transfer some other way?

Thanks 

Santeri

[Ahmad Heidari]

Hi Santeri,

I'm not sure if I fully understood your question; however, when defining a set in Pyomo and leveraging spreadsheets, the order of indices must align with the set defined in Pyomo. 

Let me give you an example:

### Let's suppose we have defined a set as follows (we have five generating units):
model.Gen = Set(initialize=['g1', 'g2', 'g3', 'g4', 'g5'])

### Then, let's provide the import code (you should have pandas installed):
excel_file = 'Data_Gcode_3_1.xlsx'  # Replace with the path to your Excel file
df = pd.read_excel(excel_file, sheet_name='Sheet1', index_col=0)
df.index = df.index.str.strip()
data = {
    'a'   : df['a'].to_dict(),
    'b'   : df['b'].to_dict(),
    'c'   : df['c'].to_dict(),

    'd'   : df['d'].to_dict(),
}

model.a    = Param(model.Gen, initialize=data['a'])
model.b    = Param(model.Gen, initialize=data['b'])
model.c    = Param(model.Gen, initialize=data['c'])
model.d    = Param(model.Gen, initialize=data['d'])

### WHERE, the spreadsheet should look like the image attached:
 
Please let me know if you require any additional clarification.

[santer...@gmail.com]

Hi Ahmad,

 

Thank you for the reply. The reason I wanted to use the pyomo dataportal functionality was so that I can generalize my pyomo code and make modifications to the sets and parameters in the spreadsheet and automatically load them into the problem. This was so that I don’t need to manually make import functions for each set, and parameter in my project using e.g. pandas. So I specifically didn’t want to initialize any sets manually.

 

If I’m not completely mistaken this is easily possible in GAMS in the way I described and based on pyomo documentation it would seem to be possible using pyomo dataportal but fails when using indexed sets. Now reading the documentation again it might be that indexed sets are only supported with json and yaml files, but indexed parameters could be supported with tabular data.

 

Documentation I’m referring to: https://pyomo.readthedocs.io/en/6.9.3/howto/abstract_models/data/dataportals.html

 

Thanks

Santeri

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pyomo-forum/f5563efe-533f-45d2-b15e-29af8fca4ae0n%40googlegroups.com.

[Ahmad Heidari]

Hi Santeri, 

Yes, you are right. You can have the code below in Pyomo:

-----------------------------------------------------------------
from pyomo.environ import *

model = AbstractModel()

model.I = Set()

data = DataPortal()
data.load(filename="Gen.csv", set=model.I)

instance = model.create_instance(data)

print("I:", list(instance.I))

---------------------------------------------------------------------

The CSV file "Gen.csv" contains one-column data, as shown in the attached image.

Note that it could be any dimension based on your preferences. 

Best,

Ahmad