Accessing external python code within APM

[Miles Abarr]

Hi,

I am exploring the idea of using APM for dynamically modeling various thermodynamic and energy storage systems, as well as 1D fluid dynamics. One thing I am curious about is if APM can call Python functions within a model. In particular, for my models currently written in Python, I use CoolProp and REFPROP to get fluid properties. I saw that APM has a thermo library, but I also need to be able to work with custom mixtures, which I am not sure if APM can do? In general, I think it would be nice to be able to access external Python code in APM models, so that I am not completely dependent on making sure everything is written in APM. Is this possible?

Thanks,

Miles

[John Hedengren]

Miles,

We're working on a version of APMonitor that will allow integration with custom models and solvers. In particular, it will be a DLL (Windows) or SO (Linux/MacOS) that will integrate nicely with other modeling languages or custom models such as those written in Python. One disadvantage of this interface will be that all custom models will need to have gradients. APMonitor normally provides the gradients (exact 1st and 2nd derivatives through automatic differentiation) to solvers such as APOPT, BPOPT, and IPOPT. The solvers APOPT and IPOPT can generate a BFGS update if 2nd derivatives are not available.

This will hopefully come out in the next months although I do have a prototype version available if you'd like to help us with the testing.

Best regards,

John Hedengren

--
--
APMonitor user's group e-mail list.
- To post a message, send email to apmo...@googlegroups.com
- To unsubscribe, send email to apmonitor+unsubscribe@googlegroups.com
- Visit this group at http://groups.google.com/group/apmonitor

---
You received this message because you are subscribed to the Google Groups "apmonitor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apmonitor+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

John Hedengren

APMonitor Modeling Language

jo...@apmonitor.com

[Miles Abarr]

John,

So by gradients you mean things like dh/drho and dp/dh for fluid states? With regards to fluids, can APM do fluid mixtures? I did find this link: http://www.apmonitor.com/thermo/, so I see that you do have a lot of fluids available, which is great.

I might be interested in testing, but let me get familiar with the APMonitor language first to make sure I can actually provide some help to you.

Miles

On Sunday, April 9, 2017 at 11:17:28 PM UTC-6, John Hedengren wrote:

Miles,

We're working on a version of APMonitor that will allow integration with custom models and solvers. In particular, it will be a DLL (Windows) or SO (Linux/MacOS) that will integrate nicely with other modeling languages or custom models such as those written in Python. One disadvantage of this interface will be that all custom models will need to have gradients. APMonitor normally provides the gradients (exact 1st and 2nd derivatives through automatic differentiation) to solvers such as APOPT, BPOPT, and IPOPT. The solvers APOPT and IPOPT can generate a BFGS update if 2nd derivatives are not available.

This will hopefully come out in the next months although I do have a prototype version available if you'd like to help us with the testing.

Best regards,

John Hedengren

On Sun, Apr 9, 2017 at 1:34 PM, Miles Abarr <mi...@brightes.com> wrote:

Hi,

I am exploring the idea of using APM for dynamically modeling various thermodynamic and energy storage systems, as well as 1D fluid dynamics. One thing I am curious about is if APM can call Python functions within a model. In particular, for my models currently written in Python, I use CoolProp and REFPROP to get fluid properties. I saw that APM has a thermo library, but I also need to be able to work with custom mixtures, which I am not sure if APM can do? In general, I think it would be nice to be able to access external Python code in APM models, so that I am not completely dependent on making sure everything is written in APM. Is this possible?

Thanks,

Miles

--
--
APMonitor user's group e-mail list.
- To post a message, send email to apmo...@googlegroups.com

- To unsubscribe, send email to apmonitor+...@googlegroups.com

- Visit this group at http://groups.google.com/group/apmonitor

---
You received this message because you are subscribed to the Google Groups "apmonitor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to apmonitor+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Hedengren]

Miles,

 

Yes, it is gradient to those variables but also more generally for any equation with respect to any variable that may be in your model. The solver needs gradient information for each equation and objective function. APMonitor provides this automatically when you write a model in the APM language because it compiles the model to byte code (fast like C++) and then proceeds to provide the solver (such as IPOPT) the gradients and function evaluations that it requests. You can use the built-in thermo or include your own equations for the thermo expressions. Most users prefer to provide their own equations because they can customize them and have more control and visibility versus using a hidden function call.

 

Keep us updated on your progress.

 

Best regards,

 

John Hedengren

[ThomasR]

Dear Dr. Hedengren,

being new to APMonitor/Gekko I'm interested in building process engineering models with it. Similar to the thread opener this will rather soon require the use of some external functions, e.g. for material properties or equilibrium calculations etc.

Although I understand that it is a difficult thing for gradient based solvers to handle such black box functions I wanted to ask your guidance on how to approach this with APM/Gekko (or would you advise not to use it for such modeling tasks...?). In the previous post you mentioned that this might be supported in the near future...

Moreover, thinking in terms of unit operations and in order to increase code re-usability, is there a way to load frequently used models and build a (nested) large model ("flowsheet") from several such sub-models (via connections?)?

Thanks,

Thomas

Am Montag, 10. April 2017 07:17:28 UTC+2 schrieb John Hedengren:

Miles,

We're working on a version of APMonitor that will allow integration with custom models and solvers. In particular, it will be a DLL (Windows) or SO (Linux/MacOS) that will integrate nicely with other modeling languages or custom models such as those written in Python. One disadvantage of this interface will be that all custom models will need to have gradients. APMonitor normally provides the gradients (exact 1st and 2nd derivatives through automatic differentiation) to solvers such as APOPT, BPOPT, and IPOPT. The solvers APOPT and IPOPT can generate a BFGS update if 2nd derivatives are not available.

This will hopefully come out in the next months although I do have a prototype version available if you'd like to help us with the testing.

Best regards,

John Hedengren

On Sun, Apr 9, 2017 at 1:34 PM, Miles Abarr <mi...@brightes.com> wrote:

Hi,

I am exploring the idea of using APM for dynamically modeling various thermodynamic and energy storage systems, as well as 1D fluid dynamics. One thing I am curious about is if APM can call Python functions within a model. In particular, for my models currently written in Python, I use CoolProp and REFPROP to get fluid properties. I saw that APM has a thermo library, but I also need to be able to work with custom mixtures, which I am not sure if APM can do? In general, I think it would be nice to be able to access external Python code in APM models, so that I am not completely dependent on making sure everything is written in APM. Is this possible?

Thanks,

Miles

--
--
APMonitor user's group e-mail list.
- To post a message, send email to apmo...@googlegroups.com

- To unsubscribe, send email to apmonitor+...@googlegroups.com

- Visit this group at http://groups.google.com/group/apmonitor

---
You received this message because you are subscribed to the Google Groups "apmonitor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to apmonitor+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Hedengren]

Thomas,

There is a way to build objects in APMonitor such as the objects created here: http://apmonitor.com/wiki/index.php/Main/Objects but we haven't opened the API publicly. I recommend that you try to build your objects with Python GEKKO.

-John Hedengren

- To unsubscribe, send email to apmonitor+unsubscribe@googlegroups.com

- Visit this group at http://groups.google.com/group/apmonitor

---
You received this message because you are subscribed to the Google Groups "apmonitor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to apmonitor+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best regards,

John Hedengren

APMonitor Optimization Suite

http://apmonitor.com

[ThomasR]

Thank you for your hint with the objects. This could be a way to implement unit operations ind connect them...  However, I'm not sure if I understand your answer well... this does not offer a way to call external (Python) functions within GEKKO equations/models, right?

Thanks,

Thomas

[John Hedengren]

Thomas,

The gradient-based optimizers (IPOPT, APOPT, etc) in GEKKO require information such as:

• Function (equality / inequality / differential) evaluations
• Objective evaluations
• Sparse structure of equations and objective(s)
  
• Equations (equality / inequality / differential) 1st and 2nd derivatives in sparse form
• Objective 1st and 2nd derivatives in sparse form

Part of the issue with external functions is that they rarely provide the derivatives that are required by gradient based optimizers and finite differences do not take advantage of the accuracy and speed of automatic differentiation. If you only have the function and objective evaluations then I recommend that you use a package such as Scipy.minimize.optimize (See http://apmonitor.com/che263/index.php/Main/PythonOptimization with Method #2) or fmincon for MATLAB (See http://apmonitor.com/che263/index.php/Main/MatlabOptimization Method #2). If you have an external function with at least the function and objective evaluations and 1st derivatives then I recommend that you use something like pyIPOPT or IPOPT (C++) directly. If you are able to provide the equations and objectives in GEKKO format then it provides the gradient information to the solver. GEKKO also transforms differential and algebraic equations systems into algebraic equations with the use of orthogonal collocation on finite elements. More details are given in the new GEKKO publication (open access journal Processes): http://www.mdpi.com/2227-9717/6/8/106

Below is a simplified example of object oriented programming with GEKKO. If your model can be translated into GEKKO form then you'll be able to take advantage of the efficient methods that GEKKO uses for large-scale optimization of DAE systems.

Best regards,

John Hedengren

from gekko import GEKKO

import numpy as np

 

z = GEKKO() # create GEKKO model

z.p = z.FV() # slope

z.q = z.FV() # intercept

# STATUS On

z.p.STATUS = 1

z.q.STATUS = 1

 

class MeasObj:

    def __init__(self,id):

        self.id = id

 

    def addPred(self,x):

        # y = Predicted value

        # x = Dependent value

        y = z.Var()

        z.Equation(y == z.p * x + z.q)

        return y

 

    def addPoint(self,x,ym):

        # x = Dependent value

        # ym = Measured value

        yp = self.addPred(x)

        z.Obj((yp-ym)**2)

        return

 

# first data set

d0 = MeasObj('set0')

n = 11

error = (np.random.randn(n)-0.5)*3.0

xm0 = np.linspace(0,10,n)

ym0 = 5.0 * xm0 + 3.0 + error

# fit data

for i in range(n):

    d0.addPoint(xm0[i],ym0[i])

 

# second data set

d1 = MeasObj('set1')

n = 30

error = (np.random.randn(n)-0.5)*2.0

xm1 = np.linspace(0,10,n)

ym1 = 4.0 * xm1 + 3.5 + error

# fit data

for i in range(n):

    d1.addPoint(xm1[i],ym1[i])

 

# solve

z.options.SOLVER = 1

z.solve()

 

# print parameter values

print('slope: ' + str(z.p.value))

print('intercept: ' + str(z.q.value))

 

# plot results

yp = z.p.value * xm0 + z.q.value

 

import matplotlib.pyplot as plt

plt.plot(xm0,ym0,'bo',label='Set 0 Data')

plt.plot(xm1,ym1,'rx',label='Set 1 Data')

plt.plot(xm0,yp,'k--',label='Regression')

plt.xlabel('x')

plt.ylabel('y')

plt.legend()

plt.show()

 

- To unsubscribe, send email to apmonitor+unsubscribe@googlegroups.com

- Visit this group at http://groups.google.com/group/apmonitor

---
You received this message because you are subscribed to the Google Groups "apmonitor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to apmonitor+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ThomasR]

John,

thank you for the further explanations and also the link to the new GEKKO paper (interesting read!). I still need a bit of time to experiment with GEKKO and gain more experience.

My main motivation (for the moment) is to use it as an algebraic or DAE solver (not yet optimization...). But in process engineering I don't really see a good way around using external "black box" functions (e.g. steam tables, thermodynamic equilibrium calcuations) without sacrificing versatility...

I saw that DAETools (I'm sure you know this project..) supports external scalar and vector functions (via automatic differentiation, I guess). However, the less restrictive equation and constraints formulation make APM/GEKKO really attractive! Maybe, user defined functions could (optionally?) provide gradient/Jacobian information to GEKKO and fall back to automatic differentiation if not provided!? Even if this comes with a performance hit, I imagine the increase in flexibility would be enormous! Do you see any chance that GEKKO will support that in the future?

Anyway, your recommendatio to look at pyIPOPT doesn't come as a surprise. However, at least under Windows this seems quite difficult to build. I understand thtat APM/GEKKO doesn't provide IPOPT support but maybe you can still provide a hint on how to get pyIPOPT runing under Windows?

Thanks again for your support and this great tool (APM/GEKKO)!

Thomas

[John Hedengren]

Thomas,

 

Thanks for the suggestions. I’ll look into the way that DAETools handles external functions. Another option is to link a solver that doesn’t require gradients but these are generally much slower for problems of industrial importance. A couple additional packages that may be of interest to you are CasADi and Pyomo (see APMonitor webinar links below):

 

CasADi: https://youtu.be/DvicSVRhVxQ

Pyomo: https://youtu.be/cjMkVHjhSBI

 

These packages are likely more flexible but as you mentioned, it comes as a performance hit. On one of our large-scale test problems, Pyomo was about 5 times slower than APM/GEKKO.

 

There is the ability (not yet released) to incorporate external functions in APMonitor but the model needs to be linked to the APMonitor executable. Another option is to load a DLL (Windows) or SO (Linux) library at run-time to allow external functions but we haven’t developed this yet. We also have APMonitor as a DLL / Shared Object (SO) so that it could be just a part of larger optimization problem. We haven’t found a good use case for this either because of the challenge of piecing together different software.

 

Some of the BYU PRISM students have tried pyIPOPT and they may be willing to chime in on the discussion. Your best option for support is to contact the developer if you run into problems that the documentation can’t fix. In general, however, I recommend a modeling language because this greatly simplifies the implementation and adaptability of the code for different problems.

 

-John Hedengren

 

From: apmo...@googlegroups.com <apmo...@googlegroups.com> On Behalf Of ThomasR
Sent: Tuesday, August 7, 2018 1:39 PM
To: apmonitor <apmo...@googlegroups.com>
Subject: Re: [APM] Accessing external python code within APM

 

John,

 

thank you for the further explanations and also the link to the new GEKKO paper (interesting read!). I still need a bit of time to experiment with GEKKO and gain more experience.

My main motivation (for the moment) is to use it as an algebraic or DAE solver (not yet optimization...). But in process engineering I don't really see a good way around using external "black box" functions (e.g. steam tables, thermodynamic equilibrium calcuations) without sacrificing versatility...

I saw that DAETools (I'm sure you know this project..) supports external scalar and vector functions (via automatic differentiation, I guess). However, the less restrictive equation and constraints formulation make APM/GEKKO really attractive! Maybe, user defined functions could (optionally?) provide gradient/Jacobian information to GEKKO and fall back to automatic differentiation if not provided!? Even if this comes with a performance hit, I imagine the increase in flexibility would be enormous! Do you see any chance that GEKKO will support that in the future?

 

Anyway, your recommendatio to look at pyIPOPT doesn't come as a surprise. However, at least under Windows this seems quite difficult to build. I understand thtat APM/GEKKO doesn't provide IPOPT support but maybe you can still provide a hint on how to get pyIPOPT runing under Windows?

 

Thanks again for your support and this great tool (APM/GEKKO)!

Thomas

Am Montag, 6. August 2018 13:57:08 UTC+2 schrieb John Hedengren:

Thomas,

 

The gradient-based optimizers (IPOPT, APOPT, etc) in GEKKO require information such as:

• Function (equality / inequality / differential) evaluations
• Objective evaluations
• Sparse structure of equations and objective(s)
• Equations (equality / inequality / differential) 1st and 2nd derivatives in sparse form
• Objective 1st and 2nd derivatives in sparse form

Part of the issue with external functions is that they rarely provide the derivatives that are required by gradient based optimizers and finite differences do not take advantage of the accuracy and speed of automatic differentiation. If you only have the function and objective evaluations then I recommend that you use a package such as Scipy.minimize.optimize (See http://apmonitor.com/che263/index.php/Main/PythonOptimization with Method #2) or fmincon for MATLAB (See http://apmonitor.com/che263/index.php/Main/MatlabOptimization Method #2). If you have an external function with at least the function and objective evaluations and 1st derivatives then I recommend that you use something like pyIPOPT or IPOPT (C++) directly. If you are able to provide the equations and objectives in GEKKO format then it provides the gradient information to the solver. GEKKO also transforms differential and algebraic equations systems into algebraic equations with the use of orthogonal collocation on finite elements. More details are given in the new GEKKO publication (open access journal Processes): http://www.mdpi.com/2227-9717/6/8/106

 

Below is a simplified example of object oriented programming with GEKKO. If your model can be translated into GEKKO form then you'll be able to take advantage of the efficient methods that GEKKO uses for large-scale optimization of DAE systems.

 

Best regards,

 

John Hedengren

 

--

--