Green Button Connect My Data
Marty Burns
All,
The California IOUs have agreed with their public utilities commission (the CPUC) to implement Green Button Connect My Data on an aggressive schedule this calendar year. They have been major participants in all our OpenADE calls and of course have also been first to implement Green Button Download My Data since fall of 2011 based on the United States Office of Science and Technology Policy (OSTP)’s initial call to action. SDG&E and PG&E have had a year of experience in implementing an initial version of Green Button Connect My Data and are implementing their first “standards-compliant” version at this time. Through their agreement with their public utility commission, they are obligated to deliver this capability later this calendar year. At the same time, we as a community endeavor to complete the specifications and software for certifying Green Button Connect My Data on the same schedule.
We observed that the progression of issue discussion and resolution was not occurring rapidly enough in our one hour weekly OpenADE sessions. Thus, the NIST Green Button team offered to sit down face to face with SDG&E, SCE, and PG&E in a two day tiger-team session aimed at making rapid progress on a consensus set of minimum implementation agreements of Green Button Connect My Data that all could support in their present activity. The goal was to ensure that the implementations deployed by these utilities is a proper subset of the Green Button Connect My Data function.
I would like to acknowledge the intensity of support of this activity by SDG&E, PG&E, and especially SCE (who graciously hosted) and all of their willingness to alter initial positions to achieve the consensus with the broadest possible scope. I would also like to thank NIST who sponsored myself and my teammates John Teeter our Presidential Innovation Fellow, Kin Lane API expert and evangelist who attended at the behest of the OSTP, and Don Coffin our OAuth evangelist.
This email details the results of this two day session and we are bringing it forward for broader review. Taken together the results of the meeting should provide clarification to all of us who are implementing or planning to implement Green Button Connect My Data by detailing some of the finer points that will be necessary to produce interoperable implementations across the ecosystem. No capabilities were removed or proposed to be dramatically altered in my view. As has been our practice in OpenADE we have once again benefited from the thinking and lessons learned of the early adopters and this was an intense instance of this capture method.
Next week we will discuss this information on the OpenESPI (for software related considerations) and OpenADE (for domain and testing related considerations) conference calls. Here are the logistics of these in case you have misplaced them:
EnergyOS OpenESPI Meetings: mondays at 12:00 EST
https://www1.gotomeeting.com/join/807543384, +1(213)493-0603;807543384#
UCAIug OpenADE Meetings: first tuesday of the month at 3:00 EST
https://www2.gotomeeting.com/join/844935738, +1(415)363-0070;844935738#
We have posted on the UCAIug website a zip file containing all the artifacts from the meeting. All artifacts from the meeting are in a single zip file (http://osgug.ucaiug.org/sgsystems/OpenADE/Shared%20Documents/Testing%20and%20Certification/GreenButtonTestPlan/OpenADE_2014_Notes/GreenButtonCMDWorkshop20140422.zip). I have attached the working agenda included in the power point and final resulting API spreadsheet that came out of the activity. Once we have reviewed these materials together next Monday and Tuesday, I will update the current working versions of these artifacts on the OpenADE task force GreenButtonTestPlan site. This will be a giant step forward!
Below, I will present the highlights of the meeting. The presentation is organized by the agenda topics discussed. I will point out that our goal was coverage of the agenda and all items were open discussions sometimes seeded with initial diagrams or tables and existing artifacts that were reviewed. Therefore we kept to the content and not necessarily the order of the agenda items. The attached PowerPoint contains the live edited diagrams and data with my post-meeting cleanup.
Thanks for reviewing and (in advance) commenting on this material. BTW – one final note; any errors in the interpretation of the agreements arrived at in the meeting are my own. We will correct them as discovered in our meetings next week at OpenESPI and OpenADE.
Cheers,
Marty
Dr. Martin J. Burns, Hypertek Inc. for NIST
Technical Champion, Green Button
email: ma...@hypertek.us
1 Architecture Guidelines
1.1 Each IOU Present Overview of their plans
Natalie Martinez, SCE: Provided input to program (3rd party to data retrieval); make system flexible to support future data initiatives. 3rd Parties will register on-line via a web portal (not directly on SCE.com) and test their systems with SCE. Customers will have to authorize release of their data on-line. At this time, the OAuth process occurs to assign a Unique ID to the Customer Account. Once the ID is created and all parties in agreement, the 3rd Party will receive 13 months of historical data and daily files with previous day’s data.
Marcela Hernandez, SDG&E: Still in discussions about scope of program and have not received estimates. There is currently a system process in place to provide 3rd parties interval usage data. They are looking to migrate existing 3rd parties and release in waves. There is a need to update front end web pieces. The biggest concern is migrating the existing 3rd parties from the old system to the new.
Jerry Yip, PG&E; Jerry Yip reviewed a Powerpoint presentation to the team. The 3rd parties will register on-line directly on the PG&E website. PG&E is looking to add additional data elements within 6 to 9 months. PG&E is allowing 3rd Parties to request data on-line if they did not pick the data up on time.
1.2 Key Definitions – SA, CISR, …
• Customer
– If residential it is individual (customer of record)
– If commercial it is an entity
• Customer Account
– Organization of the bill
– A customer may multiple customer accounts
• Service Account/Agreement
– Corresponds Customer to a meter
• Customer Information Service Request (CISR)
– A legal agreement that authorizes a California utility to provide customer information to a thirdparty – paper or online
• Service Point/Location
– Logical location representing the meter at the premises level
• ESPI
– RetailCustomer
• Owner of one or more UsagePoints (has right to authorize)
– Subscription
• Authorized access to a (sub) set of UsagePoints of a single RetailCustomer
– Bulk
• Authorized access to a collection of UsagePoints governed by individual authorizations to a third party
Topology of customer accounts from the DataCustodian perspective (special note from GreenButtonTestPlan.docx application profile 3.9 Web Customer Manages Multiple Customer Accounts):
2 Function Blocks
Note: there are other function blocks for data content that pertain to CMD that were defined for DMD previously. These are the additional FBs for Connect My Data. Those with a checkmark are the minimum required function blocks that all implementations of CMD should support:
|
FunctionBlocks for Green Button Connect My Data |
Description |
|
ü [FB_3] Core Green Button Connect My Data | Core Services |
|
ü [FB_13] Security and Privacy classes | HTTPS, SSL, TLS support |
|
ü [FB_14] Authorization and Authentication (OAuth) | Oauth 2.0 |
|
ü [FB_19] Partial update data | IntervalBlocks without full data sets (w/o UsagePoint,MeterReading, …) |
|
|
|
|
[FB_32] Resource Level REST |
Third Party Access to UsagePoints, MeterReading, … and collections |
[FB_33] Management REST Interfaces | GET PUT POST DELETE individual resources … |
|
[FB_34] SFTP for Bulk |
Optionally support the SFTP delivery of Bulk for Bulk request |
[FB_35] REST for Bulk | Support the REST request for Bulk |
|
[FB_36] Third Party (Client) Dynamic Registration |
Use Case 1 |
|
ü [FB_37] Query Parameters | published-max, published-min, updated-max, updated-min, max-results, start-index |
[FB_38] On Demand Requests | Without Notification |
|
ü [FB_39] PUSH model |
Notification followed by GET |
|
[FB_40] Offline Authorization to Complement OAuth | |
|
[FB_42] Third Party Core REST Services | |
|
[FB_43] Third Party Management REST Services | |
|
[FB_41] Manage ApplicationInformation |
Allows PUT and DELETE of ApplicationInformation |
|
[FB_44] Manage Authorization |
Allows PUT and DELETE of Authorization |
|
[FB_99] Future Proposed Features |
TBD |
3 Relationship Establishment and Management
3.1 Scope
In order to better satisfy the expressivity of scope parameters needed to convey the offerings of the IOUs on behalf of their customers to ThirdPartys the scope parameter was extended and otherwise “debugged” for use with Authorization. The following is the revised Scope EBNF and description (from Authorization.docx section 2.6 Encoding of OAuth 2.0 “scope” (note the change in history length is now seconds so that it is not dependent on the choice for BlockDuration; also several terms can now have multiple values to indicate offered options especially when used in expressing overall capabilities; Also note that all FBs now included with Connect My Data):
|
Term | Expansion |
Scope |
[ FBTerms ], [ ValueTerms ], [ ResourceTerms ]; |
|
FBTerms |
“FB=“, { [FBTerm], ”_”} , FBTerm, ScopeDelimiter ; |
|
FBTerm |
“4” | “5” | “6” | “7” | “8” | “9” | “10” | “11” | “12” | “15” | “16” | “17” | “18” | “19” | “27” | “28” | “29” | “31” | “32” | “33” | “34” | “35” | “36” | “37” | “38” | “39” | “40” | “41” | “43” |
ValueTerms | { ( "IntervalDuration=", namedOrNumber,{“_”, namedOrNumber}), |
| ( "BlockDuration=", namedOrNumber,{“_”, namedOrNumber}), | |
| ( "HistoryLength=", nonNegativeNumber), | |
| ( "SubscriptionFrequency=", nonNegativeNumber | namedFrequency), ScopeDelimiter }; | |
|
ResourceTerms |
{ (“AccountCollection=”, nonNegativeNumber) | “BR=”, brID), ScopeDelimiter} |
|
ScopeDelimiter |
“;” |
namedFrequency |
“billingPeriod” | “daily” | “monthly” | “seasonal” | “weekly” | |
|
namedOrNumber |
nonNegativeNumber | namedFrequency; |
|
brID |
Character, {Character}*; |
|
nonNegativeNumber |
digit, { digit }; |
|
Digit |
0 | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ; |
|
Character |
Digit | “-” | "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z" | "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" ; |
|
Where: | |
ResourceTerms |
If a Bulk resource is specified via the “BR” term, the value of the {bulkID} is provided after the equals sign (“=”). There could be one or more terms in this list that express the granularity of notifications about resource changes. If the Subscription has more than one UsagePoint, the AccountCollection term can indicate the number of UsagePoints included |
FBTerms | The function blocks supported |
|
ValueTerms |
These are parameterized terms |
|
IntervalDuration |
This is the minimum default length of an interval in seconds (e.g. 900 for 15 minutes, 3600 for one hour, …) |
|
BlockDuration |
This is the length of a block that contains the intervals (based on enumeration of MacroPeriodKind in ESPI above as namedFrequency) |
|
HistoryLength |
This is the length of history buffer seconds |
|
BulkAccountCollection |
Used where the DC wants to provide for the reporting of multiple UsagePoints in a single Subscription. The number of UsagePoints is represented by the value in the assignment statement – e.g. 4 UsagePoints would be BulkAccountCollection=4. |
Initial Scope Strings offered by the three utilities will be (of course subject to change but this was the initial take):
• Authorization Scope
– Sample DMD Scope
• ‘Scope =
“FB=4_5_12_15_16;IntervalDuration=900;
Ê BlockDuration=monthly;HistoryLength=34128000”
• ApplicationInformation
– SCE
• Scope = “FB=1_3_4_5_8_13_18_19_31_34_35_39
ÊIntervalDuration=900_3600;BlockDuration=Daily; HistoryLength= 34128000;SubscriptionFrequency=Daily;
Ê AccountCollection=5;BR=1;”
– PG&E
• Scope = “FB=1_3_4_5_7_8_13_14_15_18_19_31_32_34_35_37_38_39_40_42_43
ÊIntervalDuration=300_900_3600;BlockDuration=Daily_BillingPeriod_Weekly_Monthly; HistoryLength=63072000;SubscriptionFrequency=Daily;
Ê AccountCollection=5;BR=1;”
– SDG&E
• Scope = “FB=1_3_4_5_8_13_14_18_19_31_34_35_39_40
ÊIntervalDuration=300_900_3600;BlockDuration=Daily_BillingPeriod_Weekly_Monthly; HistoryLength=94608000;SubscriptionFrequency=Daily;
Ê AccountCollection=5;BR=1;”
3.2 Third Party Registration
See revised ApplicationInformation.docx which contains the final set of parameters that represents the agreement between ThirdParty and DataCustodian. These parameters can be negotiated via Use Case 1, or alternatively, through an external process that produces the data set.
3.3 Authorization
See revised AuthorizationXML.docx which summarizes the Authorization resourcecontents. Note that ThirdParty registration results in an instance of Authorization and individual customer authorizations result an Authorization resource.
4 Data Transfer
4.1 REST GETs
All three Ca IOUs are supporting the PUSH method that uses a Notification pushed to the ThirdParty followed by a GET request from the ThirdParty. See the slide deck for details.
4.2 SFTP
The OpenADE task force had previously converged on a function block that allowed retrieval of potentially large data sets via SFTP. Those agreed upon details are still present. However, one of the participants took away an action item to determine the best way to ensure that exchanges occur over TLS with the application layer provision of an access-token to ensure the right to make the request.
4.3 ServiceStatus
The activity clarified the syntax of the ServiceStatus message and response (See slides)
4.4 Chunking
The utilities desired a method to allow large data sets to be offered in pieces for easier retrieval by smaller devices. ESPI already provides the ability to add query parameters to GET requests to restrict the contents of a request. It was agreed that these same query parameters would be used in Notifications to allow the DataCustodian to offer the data in parts with the partial content identified via the query parameters.
4.4 deferred GET
PG&E introduced a message pattern where when an ad-hoc request is made by the ThirdParty to the DataCustodian, and, the data is not available due to cache limitations for example, the DataCustodian can return an HTTP response code of 202 (accepted - The request has been accepted for processing, but the processing has not been completed. The request may or may not eventually be acted upon, as it may be disallowed when processing actually takes place. there is no facility for status returns from asynchronous operations such as this). The DataCustodian may assemble the data at some arbitrary time interval (for example at the next batch processing of next day data), create the dataset, and produce a Notification to the ThirdParty with the same original URI so the results can be fetched.
John Teeter
--
--
You received this message because you are subscribed to the "energyos_espi" group.
To post to this group, send email to energy...@googlegroups.com
To unsubscribe from this group, send email to: energyos_esp...@googlegroups.com
http://www.energyos.org/espi/
---
You received this message because you are subscribed to the Google Groups "energyos_espi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to energyos_esp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.