initialize a model Parameter from a pandas dataframe?

[David Alderson]

Sorry if this is a duplicate post, but I didn't find it anywhere.

I have csv data that is read into a pandas data frame.  I have a number of different model parameters that I would like to initialize from the data frame.  From what I can see, most folks tend to build intermediate dictionaries from the data frame first and use that to initialize any data.  But since the data frame itself has the index lookup like a dictionary, this seems unnecessary.  Is there a way to initialize a parameter directly from the data frame?

I guess another way to do it would be to skip the use of Parameters altogether and just access data directly from the data frame.  Any thoughts on this?

Thanks for any insights or pointers.

[Gabriel Hackebeil]

I’m going to assume you are starting from a ConcreteModel to make this explanation easier. Correct me if I am wrong.

If you already have your data organized an a form you like (such as a dictionary-like data structure), then there is no need to store it into a Param object unless you want to change the value of that parameter at a later time (which updates any expressions where it was used). To be able to do this, you have to declare the Param using the mutable=True keyword assignment. E.g.,

model.p = Param(model.index_set, mutable=True, initialize=…)

This will consume more memory than what is required to just use your data directly to build constraints. You can assign a single number to the initialize keyword or you can assign something that behaves like a dictionary (e.g., a DataFrame possibly).

Now an indexed constraint might look like

def c_rule(model, i, j):

    return model.x[i,j] >= model.d[i,j]

Assuming model.d represents data, it could be any number of things:

(1) model.d = Param(model.index1, model.index_2, …)

(2) model.d = dict() # assume this gets populated with entries for each index

(3) model.d = <something else> # that behaves like a dictionary

Also, you do not need to use dictionaries and you do not even need to assign them to your model. If d is global relative to c_rule’s scope, then you use it.

You can use the DataFrame object directly anywhere you’d like, and it is up to you to decide how you pull data out that DataFrame before using it in a constraint expression. You can extract it in bulk (perhaps store it into a Python dict), or you an extract a single element from everywhere you need something. My hunch is that DataFrames are implemented using numpy arrays, which means extraction in bulk once at the beginning will vastly outperform accessing scalar entries one at a time. To see this, compare the time of

for i in range(len(x)):

    x[i]

when x is a Python list and when x is a numpy array of the same size.

Gabe

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[KRISHAN GOPAL SHARMA]

Hi All,
Greeting of the day,

I want to access the data(i.e some specific time series) as the model input 
I have Created a data frame in pandas.
Before that, I was accessing the data from dat file

data = DataPortal()

data.load(filename='pp1.dat')

But now I have a larger set of data

so I want to import data from CSV->pd. data frame->????(how to use pd.data frame in the abstract model)

Cheers!

Thanks

Regards,
Krish