Parsing Green Button Data

[Ufuk Yılmaz]

https://groups.google.com/forum/?fromgroups=#!topic/energyos_espi/U2zWTvx4dkU

I am simply a novice developer and what I'm trying to do is parse green button sample data files on http://www.greenbuttondata.org/greendevelop.aspx and display it as a chart, like the one at the bottom of the page. On android platform, using Java.

What I couldn't figure out is, does a library exist like mentioned on the topic I linked above, or I have to figure out green button data format by myself by inspecting sample xml files and parse them using normal xml parsing methods?

Thanks in advance for any information.

[John Teeter]

Ufuk,

We have just began to re-kindle that thread within a libESPI project. I'll reach out to Peter and see if he's still interested.

The (parsing) library doesn't exist yet, but could be generated for the various language directly from the .xsd. 

John Teeter

--
--
You received this message because you are subscribed to the "energyos_espi" group.
To post to this group, send email to energy...@googlegroups.com
To unsubscribe from this group, send email to: energyos_esp...@googlegroups.com
 
http://www.energyos.org/espi/
 
---
You received this message because you are subscribed to the Google Groups "energyos_espi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to energyos_esp...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kevin Bourque]

JAXB does precisely this for Java. http://jaxb.java.net/

In .NET, it's also part of Visual Studio and available via the command line:

xsd espi.xsd /classes

This is how I've generated my base GreenButton.cs classes in C# and extend them as needed from there: http://stackoverflow.com/questions/5217665/generate-net-4-0-class-from-xsd

I think it's likely that most languages have similar libraries or built-in functionalities and this could be a good shortcut to re-kickoff the libESPI project (e.g. xsd2ruby for Ruby, generateDS for Python, etc...) These would form a series of vetted classes which contain desired Green Button properties. With these bases classes and a set of pre-defined unit tests, third party client developers should be well on their way to implementing their own Green Button apps.

It's possible that using the excellent test cases from the OpenSG/OpenADE committee, a set of Behavior Driven Development (BDD) files could be generated for language-specific unit testing. (see http://osgug.ucaiug.org/sgsystems/OpenADE/Shared%20Documents/Testing%20and%20Certification/GreenButtonTestPlan/GreenButtonTestCases.xlsm)

--
Kevin

[John Teeter]

thx Kevin. JAXB is the mechanism I used within the javaSpring code base as well. (and note that we expect to have an additional turn of that wheel with the espiDerived.xsd that should solidify this week). 

It would be interesting to explore your BDD comments more fully. The UML model may also support this more fully.

John

===== 

Phone: 208 875 2269
Skype: jateeter

[Marty Burns]

Ufuk,

 

The data format is described by a schema. The standard schema is available from http://www.naesb.org/ESPI_Standards.asp. It has some errors and there have been extensions based on lessons learned. There is a new proposed schema which I will provide  a draft of if you ask me individually.

 

Please look at https://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/GreenButtonSDK  which describes the tools for parsing and generating green button data files.

 

The example you saw on http://www.greenbuttondata.org/greendevelop.aspx was done using an xslt which is available – see the SDK user’s guide.

 

HTH,

Marty

 

Dr. Martin J. Burns, Hypertek Inc. for NIST

Technical Champion, Green Button

p-1(301)315-9101

c-1(301)257-9101

email: ma...@hypertek.us

[Salma Wahwah]

Hi Marty,

 

Please can you provide me with an answer for this question?

 

If the value output to the XML has been multiplied by 1000 should the PowerOfTen be 3 to indicate that we multiplied by 1000 or should it be -3 to indicate that the value should be divided by 1000 to return to the original value?

 

Thank you,

Salma

[John Teeter]

Marty may be traveling. I believe the PowerOfTen should indicate the multiplier to be applied to the reading value. This if I have a measured of .4kWh, I could have a value of 4, and a PowerOfTen of 2, yielding 400 Wh or .4kWh

jt

--

[Salma Wahwah]

Thank you

---

From: John Teeter <john....@energyos.org>

Sender: <energy...@googlegroups.com>

Date: Tue, 19 Mar 2013 16:36:03 -0600

To: energyos_espi<energy...@googlegroups.com>

ReplyTo: <energy...@googlegroups.com>

Subject: Re: [ESPI] Parsing Green Button Data

[Marty Burns]

Right you are on both counts, John!

Cheers, Salma!

Marty

Sent from Marty's iPhone

[Salma Wahwah]

 

K that doesn’t really match up with what the Green button XSLT is doing.  If for example we have a kWh value of 1.34235 we will multiply it by 1000 to convert it to Wh and get 1342.35 we then multiply it by another 1000 to get rid of decimals and get 1342350.  We then specify a multiplier of 3 in the XML file. 

 

When we then apply the XSLT to the generated XML it is displaying 1,342,350 kWh.

 

If we change the multiplier to -3, then applying the XSLT correctly converts the value to 1.34235 kWh.

 

Looking into the code of the XSLT we’ve found that when the kind is specified as electricity it will automatically multiply the value by 0.001 to convert to kWh it then looks at the multiplier and for a multiplier of 3 multiplies the value by another 1000 which results in going back to the original 1342350 that was specified in the XML.  Is there something wrong with the XSLT or are we doing something wrong?

 

Thank you

Salma

 

From: energy...@googlegroups.com [mailto:energy...@googlegroups.com] On Behalf Of Marty Burns
Sent: March-19-13 8:30 PM
To: energy...@googlegroups.com
Cc: energyos_espi
Subject: Re: [ESPI] Parsing Green Button Data

 

Right you are on both counts, John!

[Marty Burns]

Salma,

 

The Green Button data describes the measurement. So there is a number x 10^ powerOfTen. It is up to the xslt to present this to the consumer in a helpful way. There is no inherent notion of kWh in the data file. The UOM is Wh. If you want to get rid of decimals to provide more precision, you have to subtract from the exponent for every shift to the left of a decimal place.

 

So if we have a measurement that represents 1.34235 kWh, and if we don’t want to lose precision because “value” is an integer, and the units of measure (uom) for this measurement is Wh, then:

 

1.34235 kWh = 1342.35 Wh = 1342350 x 10^-3 Wh.

 

So this says that the powerOfTen multiplier should be -3. This is consistent with your analysis I think.

 

Note that in the sample xslt, there is a variable called gl_PresentationMultiplierElectricy:

            <xsl:variable name="gl_PresentationMultiplierElectricy" select="3"/>

 

The purpose of this constant is that we expect that we want to present to the viewer units of kWh. So the xslt takes the measurement as represented by the value and the powerOfTen multiplier and and scales it so that it is in kWh (this involves subtracting exponents).

 

Did I answer your question or did I miss something?

 

Cheers,

Marty

 

Dr. Martin J. Burns, Hypertek Inc. for NIST

Technical Champion, Green Button

p-1(301)315-9101

c-1(301)257-9101

email: ma...@hypertek.us

 

 

 

[Salma Wahwah]

FYI…

[Salma Wahwah]

Sorry sent it by mistake.

 

Thanks a lot Marty for your response.

[Salma Wahwah]

Actually, you answered my question perfectly.

 

Cheers,

[Ufuk Yılmaz]

Hi all again, I have a little question,

Do every greenbutton file contain exactly 6 entries like the ones generated by GreenButtonDataSetGenerator.xlsm in the order :

usage oints - local time parameters - meter readings - interval readings - reading type - power usage summary

If not, should a parser examine each entry and decide what information it contains.. I'm really new at this.

And how to process links, should I use an atom parser or a standart xml parser?

[Marty Burns]

Ufuk,

 

A Green Button data set can have any number of resources: UsagePoints, MeterReadings …. The order of entrys in the Atom feed is not fixed. The linking is the key to understanding what is associated with what. See https://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/GreenButtonSDK#Linking_of_Entries for description. The sample xslt presented in the SDK illustrates how to navigate the links to determine the associated resources.

 

As you can see from the xslt, the links are navigated through XPath statements. The tricky point is that for those resources that have a multiplicity of 0..* the links point to a “collection”. Thus you need to determine who points to the common collection which is not itself explicitly present in the feed.

 

HTH,

Marty

 

Dr. Martin J. Burns, Hypertek Inc. for NIST

Technical Champion, Green Button

p-1(301)315-9101

c-1(301)257-9101

email: ma...@hypertek.us

 

 

 

From: energy...@googlegroups.com [mailto:energy...@googlegroups.com] On Behalf Of Ufuk Yilmaz
Sent: Sunday, March 31, 2013 7:51 AM
To: energy...@googlegroups.com
Subject: Re: [ESPI] Parsing Green Button Data

 

Hi all again, I have a little question,

--

[Ufuk Yılmaz]

So it seems I have to implement my own logic to navigate between entries to define what information they contain, since Java and xslt are very different at processing xml.

[Marty Burns]

Yes, but note you will want to use a document object model in java for parsing the XML which will support XPath. Then it will be essentially the same XPath strings as in XSLT.

 

HTH,

Marty

[Ufuk Yılmaz]

Hi Marty,

I came up with a gimmicky way to easily parse greenbutton data, I want to hear your opinion about it before starting to implement:

Transform greenbutton data xml using provided greenbutton xslt, this creates a processed html string (usage data is clear and no need to compute local time parameters etc.) and than parse resulting html. Add id's and classes to html elements that I'm interested (by modifying sample xslt). How does this sound?

[Kevin Bourque]

Adding my two cents here:

Having the benefit of hindsight after myself implemented a client to read the data, the problem I foresee with your strategy is that you lose a layer of information by parsing just the HTML output. In particular, the link elements are one of the most important pieces of metadata in the Atom feed. The xslt leverages this via XPath and selectively outputs to HTML; there is nothing stopping you from making a custom xslt which outputs the link structure somewhere in the HTML output, but then you're back to square one where you'd have to parse this anyways. You'd be no further along and wouldn't have the benefit of XPath to navigate your HTML.

If I were to redo my application, I would:

1. Create a tree with all the link elements for easy navigation without having to parse the file each time. Usage points at the root.
   
2. Use the link tree to find the reading type, local time parameters, interval readings, etc... which are attached to the selected usage point, if any.
3. Do whatever you want with this data (i.e. aggregate, sum, average, graph, etc...)

This is actually similar to what the xslt does. You'll have to learn some sort of XML parsing language, but in my opinion this will be easier and more future-proof than parsing your HTML, where you'll likely end up having to learn something very similar. I used a built-in library in C# (.NET) which does much of the heavy lifting with XPath, but look around and I'm sure you can find something in Java (or whatever language you're using.) You seem to want to avoid to compute the time adjustments and admittedly these are non-trivial, but since you'll be using some form of XPath library, just re-use what's already in the existing xslt.

Regards,

Kevin

Kevin