Services and billing information - How can we retrieve service related usage? - Community Project?

[Julian Weber]

Hey guys,

in previous versions of the CC it was possible to collect billing event data (service create, service delete, service bind, service unbind) from the billing_events API endpoint. In the new version described in here ( http://apidocs.cfapps.io/ (App Usage Events, Events endpoint)) service information was completely stripped out of the events api.

What should be the new approach to gather billing data for services within a CF system? 

After asking Shannon for more information on this topic, he mentioned that it would be a great job for the community to implement a service event feature.

Are there other people on this mailing list that would be interested in researching and developing this functionality?

Please feel free to send me a message and we can talk about the details: jwe...@anynines.com

Regards,

Julian Weber

--

anynines.com

[Dr Nic Williams]

The apidocs aren't complete. Look in the code if you think an API endpoint is missing.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Dr Nic Williams]

Sorry, misread. You're looking for service billing events. 

On Wed, Jan 22, 2014 at 2:59 AM, Julian Weber <jwe...@anynines.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Athanasios Antoniou]

Hi Julian,

I would be willing to work towards creating service event logging. I would, however, like to confirm that such an approach would not clash with the new Pivotal and Cf-community vision, on how service billing data should be collected. 

Regards,

Nassos

[Shannon Coen]

I will discuss the current app_usage_events strategy internally and confirm with this list what the desired approach is for service events from a Pivotal perspective. 

Best,

Shannon 

[Matt H]

Hi,

just came across this thread, and yes, we are also looking into how to get billing events for services. We also think that this could be a good community project and are happy to contribute.

Regards,
Matt

On Wednesday, January 22, 2014 11:59:42 AM UTC+1, Julian Weber wrote:

[Julian Weber]

Shannon, Mark and I created a specification document. Have a look at the topic here: https://groups.google.com/a/cloudfoundry.org/forum/#!topic/vcap-dev/6gxkHbZjJ6w .

[Shannon Coen]

Anyone have any opinions about the design document? Many organizations are asking for the feature and we'd like to get started on the work. 

Shall we say we'll keep the doc open for comments until this Friday, Mar 14th? 

Julian, will you be leading the implementation? A few members of the community offered to help (Athanasios and Matt H); let them know if you would like to collaborate or if Anynines would prefer to do the implementation alone.

Thank you,

Shannon

[Shannon Coen]

Proposal for change to the design doc:

Can we leave out support for the inline-relations-depth parameter on new endpoints? This parameter has the potential for significant performance impact on the cloud controller, effectively supporting a DDOS vector the cloud controller. We'd like to move toward the cloud controller being an API for the platform, not an API for a database. 

Any objections?

[Shannon Coen]

inline-relations-depth has been removed from the design.

Julian tells us that Anynines won't have the bandwidth to implement this and submit and pull request.

Athanasios, Matt H, are you interested in contributing?

Thank you,

Shannon

[Kris Kobylinski]

The document proposes deployment of basic service life-cycle events like 'create' and 'delete' for the billing support. It seems to me that those events will be very limited in supporting billing for a variety of services. For some services with time based usage metrics those events may be sufficient like for example:

MySQL service has a tiered plans where fee structure is based on number of GBs per hour - a given plan will define the size of DB (GBs) and the events will provide time difference between "delete' and 'create' or "unbind' and 'bind'.

What about services which will use metrics of different nature? For example a "Push" service where charges will be defined per 1 Million of pushes ? The service life-cycle events will be quite irrelevant to capture usage of this service. What about any service where charges will be per number of API calls ? What about services with charges 'per-user' or 'per-device' ?

It seems to me that we should have an additional event type of which events could be generated by service instances and contain current usage information. 

Thank you,

Kris

[Shannon Coen]

Our intention with this proposal is parity with app_usage_events only. We understand that it only provides billing engines with subscription pricing only (charge the org every month until the instance is deleted). 

Support for arbitrary usage units is intended to be an enhancement; we haven't considered it MVP. It's more complicated than just a new field for arbitrary units and count. How does a billing engine discover how many API calls/users/devices the service has served since the last bill run? Cloud controller doesn't have this information because it doesn't ask for it on a regular basis. The only time the ccdb is updated with info about a service instance is when it is created, bound, unbound or deleted. What you're suggesting would require cloud controller to regularly poll every broker for usage about every service instance for usage data. That's more work, has performance implications, and requires some consideration. 

[Matt H]

Hi Shannon,

(somehow I didn't get the notification regarding updates on this thread, sorry for my late response!) But yes, we can look into this (earliest starting March 24th though). I also understand that in the current proposal limited information will be available - only create, bind, unbind, delete events, and that no modifications of the service / status updates of the services are captured currently. Nevertheless I think this would make sense to include it sometime. (Aristoteles comment regarding allowing the broker to publish events sounds reasonable to me though.)

Regards,
Matthias

[Shannon Coen]

Hello Matt,

We'd be grateful for your help on this; if you can start on it next week that would be fine. Please let us know if this is likely.

If we can limit the scope of the implementation to the design doc that would be ideal; in parallel we can start talking about an architecture for services to publish/push billing data and metrics to Cloud Foundry.

Thank you!

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

mobile: 415.640.0272

skype: shannoncoen

[Shannon Coen]

Hello all,

I haven't seen any activity on this topic in a while. I'm hoping someone from the community has enough interest in this feature that they could implement it and submit a pull request. Otherwise, I'll keep it in our icebox and prioritize it when possible.

Thank you,

Shannon

[Shannon Coen]

We have begun implementing service usage events here: https://www.pivotaltracker.com/epic/show/1084276

The design doc for this work can be found here: https://docs.google.com/a/gopivotal.com/document/d/1OTH9CGxaSmfDf3F1rkZmSIN3ylGxD6YltCebz14hLoQ/edit

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

mobile: 415.640.0272

skype: shannoncoen

[Matt H]

Hi Shannon,

sorry, I somehow missed to update this thread. I believe Brian Martin did talk to you about the internal work we are doing in regard to services and billing. This wasn't (and isn't) in a state to be published but he will connect with you on that when we'll have it ready.

Regards,
Matthias

[Shannon Coen]

We've just bumped the Cloud Controller API version to 2.6.0 in support of Service Usage Events. This feature should be available in the next final release of cf-release (173). Documentation can be found for the feature at http://apidocs.cloudfoundry.org.

Best,

Shannon 

[Julian Weber]

Hey Shannon,

thanks for implementing the endpoint. We are so busy going out of beta
right now that we couldn't find the time for a contribution here.

Cheers,
Julian Weber

Am 29.05.14 04:02, schrieb Shannon Coen:

> We've just bumped the Cloud Controller API version to 2.6.0 in support
> of Service Usage Events. This feature should be available in the next
> final release of cf-release (173). Documentation can be found for the
> feature at http://apidocs.cloudfoundry.org.
>
> Best,
> Shannon

--
Anynines.com

Avarteq GmbH
B.Sc. Computer Science
Julian Weber
Software Developer
Twitter: @anynines

----
Geschäftsführer: Alexander Faißt, Dipl.-Inf.(FH) Julian Fischer
Handelsregister: AG Saarbrücken HRB 17413, Ust-IdNr.: DE262633168
Sitz: Saarbrücken