pyam release v0.12.0 - Algebraic operations on timeseries data

5 views
Skip to first unread message

Daniel Huppmann

unread,
Jun 14, 2021, 7:28:56 AMJun 14
to openmod initiative
Hi openmods,

I kind’a set myself a „don’t spam the openmod list about pyam more than once every six months“ rule - and the last email I sent was in October 2020, so here we go!

We just released v0.12 of the pyam package, a community Python package for scenario timeseries data processing, analysis & data-viz. As a quick reminder, I keep the first section from my email in October at the end of this mail…

TL;DR - it’s basically pandas-on-wheels, with a lot of convenience utility functions for macro-energy / energy systems / integrated assessment scenario ensembles!

Highlights of the new release

Thanks to an initiative by Patrick Jürgens (Fraunhofer ISE) and with some valuable input from core developers (hat tip to Zeb Nicholls), pyam can now perform algebraic operations (add, subtract, multiply, divide) directly on the timeseries data - and it will even handle units correctly for you! (via the pint package and the iam-units extension)

The screenshot below will give you a quick flavor of the power of this new feature - then check out this new tutorial for details!

PastedGraphic-1.png


More new features

  • For power users, there is now also an IamDataFrame.apply() function very similar to the pandas apply() function, so that you can execute custom functions on the timeseries data.
  • Drop negative weights by default when performing weighted regional aggregation to avoid very „odd“ results.
  • Allow recursive aggregation when (some) intermediate variables exist and perform validation of the existing intermediate variables.

More info

Take a look at the complete release notes on GitHub! Then join the mailing list...

Cheers,
Daniel

Aim

Develop a package similar in style and convenience to pandas, but geared towards energy systems timeseries analysis and data processing.

Meta-Aim

Develop a suite of tools that are sufficiently powerful to be used for scripted scenario-results processing and automated validation, but which are also helpful when playing around in a notebook, diving into your data, exploring scenario results, …

Features

 - A wide range of visualization and plotting features (check out the gallery)
 - Regional and temporal aggregation and downscaling
 - Validation and completeness checks
 - Unit conversion
 - Reading data directly from the World Bank Data Catalogue or any IIASA Scenario Explorer (think IPCC SR15 scenarios)
 - Data format conversion between xlsx, csv, frictionless datapackage

Of course, pyam supports yearly data, subannual resolution with representative timesteps, and continuous-time timeseries data (Python datetime).

Why bother...?

You may now think, well, I can program any of these features in 5 minutes myself in pandas or xarray or some other package, so why should I bother to take a look?

Well - because you can save 4 minutes and 30 seconds every time that you want to use one of these features, and you get the function with unit-testing, comprehensive documentation, and a set of explicit error messages when your input data doesn’t make sense...

Give it a try - check out the first-steps tutorial!



Dr. Daniel HUPPMANN
Research Scholar
Energy, Climate, and Environment (ECE) Program

webwww.iiasa.ac.at/staff/huppmann

International Institute for Applied Systems Analysis
Schlossplatz 1, A-2361 Laxenburg, Austria | www.iiasa.ac.at

Reply all
Reply to author
Forward
0 new messages