pysp - runph: how to retrieve the results per iteration?

[daniel lopez garcia]

We are interested in analyzing the convergence of the Progressive Hedging Algorithm. To do that, we would like to get a snapshot at every iteration step. 

For example, we aim to look at weight, rho, and the variables values at any iteration (We'd like to do a time-series plot, with time meaning iteration).

What would it be your recommendation in order to achieve this objective ?

We would like to get a JSON file containing all of the data per iteration (Or one JSON file per iteration, of course). Then we gather all of the data into a Python pandas dataframe to analyze it.

We currently propose a solution in the following terms:

We noticed about the possibility of retrieving PHA results in JSON format:

runph --model=models --instance=scenariodata --default-rho=1.0 --solver=ipopt --solution-writer=pyomo.pysp.plugins.jsonsolutionwriter

The command above executes the PHA (runph) and writes a file ph_solution.json with the results.

What we did is to load those data into a (Python) pandas DataFrame:

# python 2.7

import pandas as pd
import json

# load json file to a dict
runph_output_json = "ph_solution.json"
with open(runph_output_json) as f:
    ph_solution_dict = json.load(f)

# load _scenario solutions_ (only) to a (pandas) dataframe
scenario_soultions_list = [dict({"scenario": k}, **v) for k,v in ph_solution_dict["scenario solutions"].iteritems()]
scenario_soultions_df = pd.io.json.json_normalize(results_by_scen_list).set_index("scenario")

Thus, we obtained a dataframe (scenario_soultions_df) solution.

Now, we would like to do the same at any iteration. (Actually, our goal is to gather all data into a single dataframe indexed by iteration and scenario)

We are wondering whether post_iteration_k_solves from pyomo.pysp.plugins.examplephextension is a good way to do that. Our idea is to call  pyomo.pysp.plugins.jsonsolutionwriter inside post_iteration_k_solves.

Any suggestions?

Thanks in advance.

[Gabriel Hackebeil]

Try out (or take a look at) pyomo.pysp.plugins.phhistoryextension. It produces a file called ph_history.db, which is created using the shelve module. You can tell it to use JSON by exporting PHHISTORYEXTENSION_USE_JSON=1, but the JSON module tends to breakdown when this file gets large.

I believe it stores all values for variables, weights, and rho at each iteration. There is an example file on the GitHub master branch that shows how produce plots like what you are requesting. Here is how to use it

runph … --user-defined-extension=pyomo.pysp.plugins.phhistoryextension

python <pyomo-dir>/pyomo/pysp/tests/examples/test_model/plot_history.py ph_history.db

The script plot_history.py will generate a figure showing the per-scenario solution and weights for each non-anticaptive variable (so beware if you have many non-anticipative variables). It should be easy to adapt to show rho.

One last thing to note is that there is a little goofiness in the way we store rho in the history output file. Like xbar / xhat, for a given variable, rho should have a single value within a scenario tree node. Unlike xbar / xhat however, we store rho in duplicate on each scenario solution (rather than on a node solution). We will likely correct this in the future, but just keep it in mind when generating a plot.

Gabe

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.