SCIENCE (!!) and community awesomeness ...

[Mark Howells]

Dear Colleagues,

I get a little gushy when I see stuff like this. Please forgive the specific model focus (in this wonderful community of models and modelers) - but sharing in case helpful...

A while ago we wrote OSeMOSYS. It was the first generally-accessible / quickly-taken-up free & open from source-to-solver energy systems model. It got taken up by the UN.  Then a few awesome crazies in that community through that developing as a world first The Energy Model Base for Africa (TEMBA). This is used by the EC. And a community updated version just featured in a publication SCIENCE!

Angelo Carlino et al. ,Declining cost of renewables and climate change curb the need for African hydropower expansion.Science381,eadf5848(2023).DOI:10.1126/science.adf5848

Just awesome to see peeps building on stuff that (and I recall it well) was said to be a pipe dream when we started: A generally accessible, world class model, used by the EU and UN. A model of every African country. A community that could grow organically (and create a bigger flame with more amazing analysts than the spark and crew that started it), and a publication in science! (There are several Nature-Journal publications in the mix too (1, 2, and mentioned with Open-Mod 3) btw all by a large array of brilliant community members).

(Now - and as an Editor of a non-Science/Nature journal - I know that there are shortcomings associated with an exclusive focus on these 'mega publications' 🙂. And I need to point out that there are many other amazing groups on the bleeding edge (so no intention to unfairly single peeps out). But v. cool to see, non-the-less). 

So - If things feel a little far off. Or peeps think you and your groups ideas are a little bonkers - but you dare to breath the boundary - don't give up - push on!

Best, Mark 

[Robbie Morrison]

Hi Mark, all

For the history, there were other open models and projects around at the outset. In some kind of order,  Balmorel, GnuAE, and deeco predate OSeMOSYS.  TEMOA was contemporaneous with OSeMOSYS.  Some provided OSI‑approved software licenses, others tried hard to build communities, others not.  Two are still active.  I wrote up these early developments as best I could five years ago (doi:10.1016/j.esr.2017.12.010).

I think attempts to paint one project or another as somehow "first" — and necessarily in some qualified sense: suitable licensing, user engagement, technical scope, innovation, subsequent success, official interest, and so forth — distract somewhat from this whole journey.  And I would like to see newer projects — like CityLearn, to name just one —accorded space too.

Kudos to OSeMOSYS for its current profile and contribution. Clearly. Although I don't regard OSeMOSYS to be particularly unique at the outset. And OSeMOSYS was built on other open source tooling, including the GNU MathProg language and the GLPK solver.

Fast forward, and I do agree that we are on the cusp of open source becoming the default in much of our domain — although the big single‑institution in‑house models will hang on for as long as their funders and public sector clients allow. The populated models and supporting datasets necessarily following along well behind — and presenting at least as significant open challenge as the code. I won't mention names, but there are data visionaries out there who are yet to see much in the way of fruition or acknowledgment on that front.  In addition, only very recently have open energy system models had any resonance within the wider open source world.

Being "early" is sufficient qualification in my view.

For disclosure, I was involved with the deeco codebase, now archived on GitHub and Software Heritage.

with best wishes, Robbie

--
You received this message because you are subscribed to the Google Groups "openmod initiative" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openmod-initiat...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/DB9PR04MB8235C1D79823194C95267851F814A%40DB9PR04MB8235.eurprd04.prod.outlook.com.

-- 
Robbie Morrison
Address: Schillerstrasse 85, 10627 Berlin, Germany
Phone: +49.30.612-87617

[Mark Howells]

Dear Robbie.

I apologize if my previous statement that OSeMOSYS was the "first generally-accessible / quickly-taken-up free & open from source-to-solver energy systems model" came across as arrogant or overstated.

OSeMOSYS was certainly in development before Temoa, and a working version was shared at the 2010 IEW (1) with several applications - and a working pre-prototype released at the 2008 IEW (2).  At the very same 2010 IEW workshop the research agenda (not the code) for Temoa was released (1 & 3). There is a year or two of light between the two. Joe Decarolis, who brilliantly led the development of Temoa  is also a co-author on the peer reviewed OSeMOSYS paper of 2011 (4). The peer reviewed release of Temoa was released in 2013 (5). They are super complementary models but with different 'quite different in scope, approach, and focus' (2).  I am very humbled that they are mentioned in the same breath, but OSeMOSYS started in a relative vacuum.

As for the earlier models, Balmorel was not free-from-source-to-solver, although its 'coding' was accessible to non-coders. It used GAMS, which is an accessible modeling system that allows for faster formulation and easier for mainstream economists and engineers, for example to enter the field. Its focus (at the time) was electricity and heat. Not energy. GAMS is also not free. OSeMOSYS was all-energy from the outset, and was programmed in GNUMathprog, which is open source and free. So there was some distance between these. And, yes, there would be no OSeMOSYS without GNU!

The other models, of which DEECO clearly paved the way, were programmed in general languages such as C++. This means that they had a less easy to absorb codebase for non-coders and were therefore less generally accessible. (This is probably reflected in the publication count that followed. They were certainly the first open models but were less accessible - we only came to learn of DEECO's pathbreaking open sourcing in Open-Mod.)

I would be happy to provide any additional evidence to support the qualified claims. Or would be very happy to tone them down (!) with your guidance.  It was not the intention to cause offence.... Let me know if you want to pick this up offline.

...The point was that we were alone when getting OSeMOSYS started and ridiculed for the idea of doing so at the time - nothing like it was available (generally-accessible, free-from-source-to-solver, easy-to-take-up). People who feel they are in the same space - doing something new and feeling a little isolated - I want to encourage to push through, especially in the face of headwinds... And we do so certainly on the shoulders of giants. The OSeMOSYS team of which I was just one, were particularly inspired by Nordhaus and DICE, Makhorin and GNU MathProg, Manne and Hafele (who developed the basis of MESSAGE, MARKAL/TIMES and probably the inspiratoin DEECO too (6)). 

And just pretty cool to see others take that initial start-up make it better and publish in Science. Hopefully, many more from this amazing community will do the same and better - and should not be daunted to go for it! 'tis all. 

Best, Mark

(1) http://www.feem-project.net/planets/plastore/Intranet/IEW2010/IEW_programme.pdf 

(2) http://www.osemosys.org/about.html 

(3) https://temoacloud.com/wp-content/uploads/2019/12/DeCarolis_IEW2010_paper.pdf

(4) https://doi.org/10.1016/j.enpol.2011.06.033 

(5) https://doi.org/10.1016/j.eneco.2013.07.014
(6) https://pure.iiasa.ac.at/id/eprint/174/1/CP-74-003.pdf

---

From: openmod-i...@googlegroups.com <openmod-i...@googlegroups.com> on behalf of Robbie Morrison <robbie....@posteo.de>
Sent: 15 August 2023 15:32
To: openmod-i...@googlegroups.com <openmod-i...@googlegroups.com>
Subject: Re: [openmod-initiative] SCIENCE (!!) and community awesomeness ...

 

** THIS MESSAGE ORIGINATED OUTSIDE LOUGHBOROUGH UNIVERSITY **

** Be wary of links or attachments, especially if the email is unsolicited or you don't recognise the sender's email address. **

To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/3c8b4198-cb36-1852-4ac8-b213464543db%40posteo.de.

[Robbie Morrison]

Hi Mark, all

Thanks for the follow up and context. And hopefully this discussion might still be informative for some on this list who are relatively new to the domain.

I am going to talk about deeco (that stylization being how the project name was usually typeset). And perhaps in a somewhat lucid fashion.

My involvement with high‑resolution contiguous‑time energy systems modeling at a national level began in 1995. I had left my job at a small energy NGO, the Sustainable Energy Forum (SEF), frustrated with the "arm‑waving" analysis that dominated the policy debate on all sides. And enrolled in a new masters course on energy management run by the physics department at Otago University, Dunedin, New Zealand (my roommate instead modeled Bose‑Einstein condensation). I was given free rein and in January 1995 stumbled across a paper in Energy describing the NEMESS framework. I emailed lead author Helmuth Groscurth, Würzburg University, Germany, and asked for a copy of the source. The deeco research team conferred and decided that (1) being based in New Zealand I posed no risk and (2) I would not get the C++ source — which had been developed on an expensive HP workstation — to run on Intel hardware in any case. I was subsequently told that if I had been a German researcher, I would not have been given access.

As it happened, I did port the C++ to a commercial UNIX which provided the necessary compile‑time dependencies and set about applying the framework to New Zealand (but didn't get very far). deeco had been conceived for municipal systems and this was new territory. I was also later told that the idea of using deeco for national systems analysis had never been considered by the original deeco team.

In 2003, I convinced lead author Thomas Bruckner at TU Berlin to open source the project and duly did under Gnu GPL‑2.0. The deeco project ran on a dedicated SCO UnixWare server. Those developing remotely could edit files locally but would need to build and run on a SSH terminal measuring 80 chars × 24 lines (just stop and think about that for a moment!). By 2004, I had provided UNIX accounts about eight people from all over the world, but no one did anything remotely substantial. The decommissioned server is sitting behind me as a type and should, in theory, boot up. The website I wrote to support the project was abandoned by TU Berlin but still have the hand‑coded HTML somewhere. I later heard that the university was distinctly unhappy that the IP in the project had been thus squandered — but given no further details.

It should be remembered that few programmers back then wrote serious software in interpreted languages. MATLAB was admittedly used and also PERL for housekeeping tasks. Version control was almost non‑existent, bar some file level support in the emacs text editor.

Regarding GNU MathProg and GLPK, I contributed to that project for about four years.

MARKAL (TIMES came later) was not an inspiration for deeco.  Indeed, I recall crafting carefully worded criticisms of MARKAL not spanning contiguous time.  deeco in contrast modeled seasonal heat storage and related technologies quite well, for example — and was genuinely high‑resolution in all respects.  It also leveraged, for better or worse, the doctrine of object‑orientation (OOP), the dominant software development paradigm back then.

I was always interested in systems integration, so initiated this soft conference back in 2000. Just for fun, here is the flier I drew:

That original reference I mentioned (also worth flicking through as much the same paradigm is still widely used for frameworks like PyPSA):

• Groscurth, Helmuth-M, Thomas Bruckner, and Reiner Kümmel (1995). "Modeling of energy-services supply systems". Energy. 20 (9): 941–958. doi:10.1016/0360-5442(95)00067-Q.

Are there lessons in all this? That sharing source is good, I guess!

But in the context of public policy analysis, that was an idea that took a long time to gain traction — as Mark also alluded to. Helmuth and I meet the then New Zealand energy minister back in 1999, but that kind of analysis and planning was well out of style and neoliberalism was instead big.

And just as a marker, the Open Energy Modelling Initiative was formed in Berlin in 2014.

On the question of the timing of projects in general, I will comment on the recent Linux Foundation report on open source and sustainability, for which all history prior to the advent of GitHub is absent. And I also have a video presentation to upload to YouTube, so more on that very soon.

best wishes to all, Robbie

PS: @Mark: I don't think there's anything to follow up off‑list.  And my remarks about the timing of TEMOA were inaccurate as you rightly point out.  I also support Erica Thompson's comments about the need for diversity and controversy in modeling.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/DB9PR04MB8235AA44B2CA3C03B56A06B4F814A%40DB9PR04MB8235.eurprd04.prod.outlook.com.

[Robbie Morrison]

Hi again all

Two small follow‑ups. The deeco project did not use a dynamically‑linked optimization solver — free or otherwise — as is invariably the case today. Rather it used the circa 30 lines of C code given in the numerical recipes series, more‑or‑less directly pasted in:

• Press, William H, Brian P Flannery, Saul A Teukolsky, and William T Vetterling (30 October 1992). Numerical recipes in C: the art of scientific computing (2nd ed). Cambridge, United Kingdom: Cambridge University Press. ISBN 978-052143108-8. All code is copyright of the publisher.

This code implemented the simplex method that crawls over your necessarily convex solution space to find a not‑necessarily unique optimal.

That series of books (which included FORTRAN code) was widely used back then and researchers were generally oblivious to the notion that the code provided was not suitably licensed. Thomas even submitted a patch back to the authors to fix a corner case.

If you head to wikipedia, you will find a discussion on the licensing terms used by Numerical Recipes, their workability, and the fact that the non‑open nature of Numerical Recipes drove the development of the GNU Scientific Library.

While I am here, a small shout‑out for the HiGHS project, aimed a providing an open‑licensed high‑performance solver engine. Please trial these tools and submit bug reports when you encounter issues.

And the second follow‑up. I regard this paper from the deeco team (for which I had no part) as seminal. It shows that the process of high‑resolution optimization is essentially the process of uncovering latent synergies in the design and operation of the systems under investigation. Which I think is a nice way of framing this endeavor.

• Bruckner, Thomas, Helmuth-M Groscurth, and Reiner Kümmel (1997). "Competition and synergy between energy technologies in municipal energy systems". Energy. 22 (10): 1005–1014. ISSN 0360‑5442. doi:10.1016/S0360-5442(97)00037-6.

Sad that so much of this material remains paywalled. And a little known fact, but for German researchers, Elsevier publications have been off‑limits for five years because the associated Projekt DEAL negotiations failed. That means both access and authorship are Verboten. Indeed, German researchers are not supposed to submit to Elsevier titles. More on wikipedia.

with best wishes as always, kia ora, Robbie

To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/04f60059-d914-4844-3f5b-f3373670fd5c%40posteo.de.

[Mark Howells]

Dear Robbie - I thought this - but forgot to write it. This is a brilliant piece of history. Thank you for sharing. Best, Mark (and a whole lot of others in the list feel the same no doubt ! 🙂)

---

From: openmod-i...@googlegroups.com <openmod-i...@googlegroups.com> on behalf of Robbie Morrison <robbie....@posteo.de>

Sent: 16 August 2023 07:50

To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/d3f8427a-a7e3-f044-5fce-e51e6f8f52e1%40posteo.de.

[Ken Caldeira]

A little noted fact is that Brian Flannery, co-author of numerical recipes, appears to have participated in Exxon's effort downplay climate risks.

https://www.climatefiles.com/exxonmobil/esso/global-climate-change-esso-italiana/

To view this discussion on the web, visit https://groups.google.com/d/msgid/openmod-initiative/d3f8427a-a7e3-f044-5fce-e51e6f8f52e1%40posteo.de.