Standard for De-Duplication and Compression

54 views
Skip to first unread message

Amir Yosha

unread,
Oct 26, 2014, 12:58:37 AM10/26/14
to snia-...@googlegroups.com
,Hi
I did not find any documentation for deduplication and compression in SNIA CDMI Version 1.1.0
 ? Are deduplication and compression supported by CDMI ? If yes which exactly standards are supported
 ? Do you have any idea also about OpenStack and Amazon’s S3 regarding these supported standards
,Thanks

Amir

David Slik

unread,
Oct 30, 2014, 10:10:05 PM10/30/14
to snia-...@googlegroups.com, Amir Yosha
De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.

De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.

Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.

I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.

Thanks,

David Slik

Technical Director, Object Storage, NetApp, Inc.
Co-chair, SNIA Cloud Storage Technical Working Group


--
You received this message because you are subscribed to the Google Groups "SNIA Cloud" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snia-cloud+...@googlegroups.com.
To post to this group, send email to snia-...@googlegroups.com.
Visit this group at http://groups.google.com/group/snia-cloud.
For more options, visit https://groups.google.com/d/optout.

Amir Yosha

unread,
Oct 31, 2014, 6:49:56 PM10/31/14
to snia-...@googlegroups.com, amir...@gmail.com
Many thanks
Amir
 
בתאריך יום שישי, 31 באוקטובר 2014 04:10:05 UTC+2, מאת David Slik:
De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.

De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.

Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.

I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.
Thanks,

David Slik

Technical Director, Object Storage, NetApp, Inc.
Co-chair, SNIA Cloud Storage Technical Working Group

Amir Yosha

unread,
Mar 1, 2015, 12:21:29 PM3/1/15
to snia-...@googlegroups.com, amir...@gmail.com, yosha...@walla.com

Hi,

The same question regarding RAID.,

I don't find any CDMI API to RAID feature. Is it missing on purpose?  How can the user ask to implement  RAID 1/2/5/6... ?

Thanks,

Amir

 
בתאריך יום שישי, 31 באוקטובר 2014 בשעה 04:10:05 UTC+2, מאת David Slik:
De-duplication and compression, as implemented by the storage system, are transparent to the CDMI standard.
 
De-duplication for data transmission is not part of the CDMI standard. It could be added if there is industry/end-user/academic interest.
 
Compression for data transmission is supported via HTTP. For example, through the use of the gzip transfer encoding.
 
I am not aware of either Swift or S3 supporting these beyond the use of gzip transfer encoding.
Thanks,
 
David Slik
 
 
Technical Director, Object Storage, NetApp, Inc.
Co-chair, SNIA Cloud Storage Technical Working Group
 
 

Mark Carlson

unread,
Mar 1, 2015, 12:31:15 PM3/1/15
to snia-...@googlegroups.com, amir...@gmail.com, yosha...@walla.com
There is data system metadata to set the number of copies of the data, typically geographically separated. This will get you more than the protection of RAID without exposing actual RAID levels through the interface. Other metadata include RPO and RTO objectives as well.

Vendors are free to use RAID in their implementation but most don't. The market is moving away from this technique due to the increased time for rebuilds on larger and larger capacity drives.

-- mark 

Gary Mazzaferro

unread,
Mar 1, 2015, 7:23:25 PM3/1/15
to snia-...@googlegroups.com
Hi

If you remember, this feature and similar capabilities were proposed in 2009 at a meeting in Colorado Springs.  Certain member(s) representing Netapp at the time were strongly opposed to the idea due to conflicts surrounding the division of responsibilities with SNIA's CIM based device management standards.  The degree of abstraction supported in the CDMI may be considered a double edged sword for some use cases. In one case it  protects the cloud storage consumer from underlying implementation details. In the other case, it creates and additional barrier for providers (including hybrid deployments) implementations where they're forced to address hardware management including RAID. There are distinctive configuration disconnects between the CDMI specification and mapping to hardware configurations, an area that seems ripe for vendor innovation.   

cheers,
gary

Mark Carlson

unread,
Mar 1, 2015, 7:45:48 PM3/1/15
to snia-...@googlegroups.com
The Cloud Storage TWG doesn't want to exclude useful features from the standard. We have a very easy way to add such features, and that is by creating an extension:

Really just a short document describing how to interoperably use the feature. Once that extension is implemented by multiple vendors it becomes part of the next version of CDMI.

We encourage folks to write up their ideas this way directly. No need to convince folks to add it, just convince them to adopt it.

-- mark

Alan Yoder

unread,
Mar 29, 2015, 3:17:40 AM3/29/15
to snia-...@googlegroups.com
Gary, I would welcome input on this. I'm struggling with it in SSM (Simple Storage Management, over in SMI).

Thanks,

Alan
Reply all
Reply to author
Forward
0 new messages