Standard for De-Duplication and Compression

[Amir Yosha]

,Hi

I did not find any documentation for deduplication and compression in SNIA CDMI Version 1.1.0

 ? Are deduplication and compression supported by CDMI ? If yes which exactly standards are supported

 ? Do you have any idea also about OpenStack and Amazon’s S3 regarding these supported standards

,Thanks

Amir

[David Slik]

De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.

De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.

Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.

I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.

Thanks,

David Slik

Technical Director, Object Storage, NetApp, Inc.
Co-chair, SNIA Cloud Storage Technical Working Group

Cell: 604.363.2079 | www.netapp.com

--
You received this message because you are subscribed to the Google Groups "SNIA Cloud" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snia-cloud+...@googlegroups.com.
To post to this group, send email to snia-...@googlegroups.com.
Visit this group at http://groups.google.com/group/snia-cloud.
For more options, visit https://groups.google.com/d/optout.

[Amir Yosha]

Many thanks

Amir

 

בתאריך יום שישי, 31 באוקטובר 2014 04:10:05 UTC+2, מאת David Slik:

De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.

De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.

Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.

I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.

Thanks,

David Slik

Technical Director, Object Storage, NetApp, Inc.
Co-chair, SNIA Cloud Storage Technical Working Group

Cell: 604.363.2079 | www.netapp.com

[Amir Yosha]

Hi,

The same question regarding RAID.,

I don't find any CDMI API to RAID feature. Is it missing on purpose?  How can the user ask to implement  RAID 1/2/5/6... ?

Thanks,

Amir

 

בתאריך יום שישי, 31 באוקטובר 2014 בשעה 04:10:05 UTC+2, מאת David Slik:

De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.

 

De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.

 

Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.

 

I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.

Thanks,

 

David Slik

 

 

Technical Director, Object Storage, NetApp, Inc.

Co-chair, SNIA Cloud Storage Technical Working Group

 

 

Cell: 604.363.2079 | www.netapp.com

[Mark Carlson]

There is data system metadata to set the number of copies of the data, typically geographically separated. This will get you more than the protection of RAID without exposing actual RAID levels through the interface. Other metadata include RPO and RTO objectives as well.

Vendors are free to use RAID in their implementation but most don't. The market is moving away from this technique due to the increased time for rebuilds on larger and larger capacity drives.

-- mark 

[Gary Mazzaferro]

Hi

If you remember, this feature and similar capabilities were proposed in 2009 at a meeting in Colorado Springs.  Certain member(s) representing Netapp at the time were strongly opposed to the idea due to conflicts surrounding the division of responsibilities with SNIA's CIM based device management standards.  The degree of abstraction supported in the CDMI may be considered a double edged sword for some use cases. In one case it  protects the cloud storage consumer from underlying implementation details. In the other case, it creates and additional barrier for providers (including hybrid deployments) implementations where they're forced to address hardware management including RAID. There are distinctive configuration disconnects between the CDMI specification and mapping to hardware configurations, an area that seems ripe for vendor innovation.   

cheers,
gary

[Mark Carlson]

The Cloud Storage TWG doesn't want to exclude useful features from the standard. We have a very easy way to add such features, and that is by creating an extension:

http://www.snia.org/tech_activities/publicreview/cdmi

Really just a short document describing how to interoperably use the feature. Once that extension is implemented by multiple vendors it becomes part of the next version of CDMI.

We encourage folks to write up their ideas this way directly. No need to convince folks to add it, just convince them to adopt it.

-- mark

[Alan Yoder]

Gary, I would welcome input on this. I'm struggling with it in SSM (Simple Storage Management, over in SMI).

Thanks,

Alan