ICE output in CUR format feasibility

[Aniket Bezalwar]

Hi ICE team,

We currently use ICE for its standard purpose. We recently have one more use case for which I was wondering, if we can let ICE processor output the data in CUR format itself without performing any aggregations based on tag group (

• Accounts
• Regions
• Services (e.g. EC2, S3, EBS)
• Usage types (e.g. EC2 - m1.xlarge)
• Cost and Usage Categories

) and also adding the custom tags (defined in ice.properties) on row level.

So output for each line would be in CUR format appended by all custom tags with names such as " resource_tags_{custom_tag_1}", " resource_tags_{custom_tag_2}", etc . Then we can use this data to create quicksight dashboards and perform some queries based on custom tags.I am new to ICE project but I see how much value it adds to organizations. Thanks for all the work on this project.

Best,
Aniket

[Nicolas Fonrose]

Hi Aniket,

We recently have one more use case for which I was wondering, if we can let ICE processor output the data in CUR format itself 

Ice currently only outputs its data in "Ice binary format" (which is then read by the Ice reader).

But it wouldn't be hard to make it generate CUR formatted data (only the manifest generation would be a little bit tedious).

In order to better answer your question, could you describe where the data would be coming from?
In the use case your are describingn, Ice looks like a step in a larger data processing chain (which ends in Quicksight).

What would be the first step of this chain? And what would be the initial data format?

Nicolas

--

Nicolas Fonrose | Teevity | Founder

+33.6.61.35.43.31

https://teevity.com - Cloud Costs Analytics for AWS, GCP and Azure powered by NetflixOSS Ice

twitter - @nfonrose / @teevity

--
You received this message because you are subscribed to the Google Groups "iceusers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iceusers+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iceusers/4cebf0c4-ed1c-4fbb-a00a-6e5b113eb6d1n%40googlegroups.com.

[Aniket Bezalwar]

Hi Nicolas, 

Thanks for your quick response. Actually this new use case is just an addition. We are not going to stop using current ICE dashboards. 

Currently there are QuickSight dashboards created based directly out of CUR data and then there is  ICE processor data aggregated based on tagGroup + custom tags and has its own output format. Currently, we cannot establish any relation between our quick sight dashboards and ICE dashboard which has custom user tags. What I am trying to achieve is let ICE output CUR format without aggregation with addition of custom tags. Then we can create quicksight dashboards on this data rather than AWS CUR data and make use of custom tags. 

As Ice, by default aggregates data based on taggroups, I am not sure about feasibility of removing that dependency. Once the costAndUsageData data structure is created in CUR format, I can write a serializer on my end to parse that.

[Aniket Bezalwar]

Hi Nicolas, 

I was wondering if we can append lineItemid to the tag group, it would help to remove the aggregation. Is my understanding correct ?

[Nicolas Fonrose]

Hi Aniket,

I was wondering if we can append lineItemid to the tag group, it would help to remove the aggregation. Is my understanding correct ?

Yes it would. But depending on the size of the billing data you are dealing with, that might result in super high usage of memory in the Ice instances (Processor and Reader).

Is the billing data for each month fairly large in your usecase?

Nicolas

 

--

Nicolas Fonrose | Teevity | Founder

+33.6.61.35.43.31

https://teevity.com - Cloud Costs Analytics for AWS, GCP and Azure powered by NetflixOSS Ice

twitter - @nfonrose / @teevity

To view this discussion on the web visit https://groups.google.com/d/msgid/iceusers/f5569342-3933-45c2-a647-09778d180a5bn%40googlegroups.com.

[Aniket Bezalwar]

Yes I was thinking on the same lines and yes the billing data is large.