MetaData store

189 views
Skip to first unread message

Nikhil

unread,
Feb 14, 2013, 5:02:35 AM2/14/13
to snia-...@googlegroups.com
Hi everyone,

I am new to CDMI. As a part of my project I need to implement Map/Reduce capabilities on CDMI. I wanted to know if the MetaData of the objects/containers is stored alongwith the objects or is metadata stripped off from the object and stored somewhere else.

Please let me know in case there is some reference material which provides greater insights on the same (trying to put MR on CDMI repositories)

Thanks & Regards,
Nikhil

Mark Carlson

unread,
Feb 14, 2013, 8:27:04 AM2/14/13
to snia-...@googlegroups.com
Hi Nikhil,

CDMI does not specify implementation details such as this.

From the user point of view the metadata and data are all part of each
object.

However, in order to efficiently process the CDMI query operations, most
implementations will likely cache some or all of the metadata that is
part of queries.

-- mark

Nikhil

unread,
Feb 15, 2013, 12:32:33 AM2/15/13
to snia-...@googlegroups.com
Hi Mark,

Thank you for your quick reply.

So what I understood is:

Since CDMI is open source, different users can implement this standard in their own ways (some may strip off metadata and keep on a different server, some may keep it with the object, etc.).

If my aim is to enable MapReduce functionality on CDMI repositories then which standard/implementation of CDMI should i use as reference?
Presently, I have downloaded the CDMI source code from http://www.snia.org/forums/csi/programs/CDMIportal

Also, is there any size limitation on an object/container? If the object becomes too large then is it stored in chunks in different servers?

Thanks & Regards,
Nikhil

Krishna Sankar

unread,
Feb 15, 2013, 1:34:41 AM2/15/13
to snia-...@googlegroups.com
Nikhil,
   A few points:
  1. First of all, the RI is open source but the different implementations are not. What one can rely on, are the interfaces, as prescribed by the standard.
  2. The internal mechanisms would definitely differ between implementations - including metadata management, the actual object store, chunking et al.
  3. The best practice would be to develop the MapReduce layer that works off of the standard; independent of the underlying implementation.   
HTH.
Cheers
<k/>


Thanks & Regards,
Nikhil


--
You received this message because you are subscribed to the Google Groups "SNIA Cloud" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snia-cloud+...@googlegroups.com.
To post to this group, send email to snia-...@googlegroups.com.
Visit this group at http://groups.google.com/group/snia-cloud?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.



Nikhil

unread,
Feb 17, 2013, 12:40:21 AM2/17/13
to snia-...@googlegroups.com
Hi Krishna,

Thank you for your reply. Can you please explain by "develop the MapReduce layer that works off of the standard; independent of the underlying implementation" in the sense of how to do it. 
I had thought that since Hadoop is already a well-implemented with MR, so is there any way that without "reinventing the wheel" we can use Hadoop to try to run MR on CDMI. 

Thanks & Regards,
Nikhil

Krishna Sankar

unread,
Feb 17, 2013, 1:41:32 AM2/17/13
to snia-...@googlegroups.com
Of course, one will implement the map & reduce as per the conventions and interfaces of the M framework like Hadoop. I was taling about CDMI ie works off of the CDMI standard interfaces, independent of the underlying CDMI implementation. Reason being, any optimization done at the CDMI interfaces based on internal assumptions could break across different systems. 
Cheers
<k/>  

Gary Mazz

unread,
Feb 17, 2013, 4:47:48 PM2/17/13
to snia-...@googlegroups.com

Btw,  I have proposals, with a partner company,  into the US fed using CDMI and OCCI as a primary interfaces to a reference architecture as well as an application architecture address to Big Data, Big Data/Cloud intersect. Hadoop is just one deployment model. 

Information and details will be made public as funding for open source permits.

cheers,
gary

Nikhil

unread,
Feb 21, 2013, 4:09:28 AM2/21/13
to snia-...@googlegroups.com
Hello Everyone,

Thanks a lot for helping me out by taking your time to reply to my queries. :)
I am trying to do MapReduce operation on CDMI servers. Can i get a free URL or account of a CDMI server, in order to test my implementation? I am not sure whether my approach would work or not but i wanted to try it out once.

Thanks & Regards,
Nikhil

Mark Carlson

unread,
Feb 21, 2013, 7:52:48 AM2/21/13
to snia-...@googlegroups.com
Nikhil,

During the cloudplugfest, there will be several servers of different designs available to work against.

The next one is March 19th and 20th:


Participation is free and can be remote.

-- mark

Nikhil

unread,
Feb 22, 2013, 2:00:09 AM2/22/13
to snia-...@googlegroups.com
Hi Mark,

Thank you for your quick replies and continued support!!

Since the plugfest is somewhat far, I thought of installing a CDMI server on my own system(Windows 7). I am running Hadoop client on a VM. I have downloaded the RI code and installed Maven 3.0.4. The RI documentation does not give any further guidance as to how to install/deploy the source code. Would this version of Maven work or should I change it to 2.0.9 ?

Also, can you refer me some documentation/link which explains in detail how to set-up a CDMI server. Basically, i want to install CDMI server in my windows, give it some space and then try to connect to CDMI server through Hadoop. 

Thanks a lot for reading this and wanting to help.

Regards,
Nikhil 

Nikhil

unread,
Feb 25, 2013, 2:24:49 AM2/25/13
to snia-...@googlegroups.com
Hi,

Can anyone help me out with installing  CDMI server....? When i try to run the source code of CDMI server it gives the error "Connection to http://localhost:8080 refused"

Regards,
Nikhil

Mark Carlson

unread,
Feb 25, 2013, 9:09:19 AM2/25/13
to snia-...@googlegroups.com
Nikhil,

Have you tried using Netbeans or Eclipse on WIndows?

Those tools make things much easier to set up the server if you have never started up Java servers in that environment before.

-- mark

Nikhil

unread,
Feb 26, 2013, 12:00:45 AM2/26/13
to snia-...@googlegroups.com
Hi Mark,

I could get the server running using eclipse. Thanks!! The documentation does not specify anything about using Glassfish, therefore i got stuck up but it is working now.

Regards,
Nikhil

Nikhil

unread,
Feb 26, 2013, 1:55:42 AM2/26/13
to snia-...@googlegroups.com
Hi,

Sorry!! I am not able to run it. I am able to build it and it shows BUILD SUCCESS. I have installed Glassfish server 3.1.1 and started it on localhost but i am not able to run the CDMI-server project in my eclipse on glassfish server. When i go to the Run Configuration --> Glassfish 3.1.1 the "Run" button does not get activated. 

Nikhil

unread,
Feb 26, 2013, 2:18:26 AM2/26/13
to snia-...@googlegroups.com
Hi,

Sorry for the trouble. by taking help of ORACLE documentation for Glassfish the CDMI-server is running now. 

Regards,
Nikhil 

Nikhil

unread,
Feb 26, 2013, 5:06:48 AM2/26/13
to snia-...@googlegroups.com
Hi,

My CDMI-server is running on localhost:8080 on glassfish server but when I run the CDMItest.java as JUnit Test it is able to create CDMI_capabilities but it fails to create CDMI container. the error being thrown is:

ETag: "Object Creation Error : java.lang.IllegalArgumentException: Cannot create container 'Test'"

If anyone has some idea regarding what needs to be modified in the code then please do share.

Thanks & Regards,
Nikhil

Nikhil

unread,
Feb 27, 2013, 6:20:12 AM2/27/13
to snia-...@googlegroups.com
Hi,

I read a post by David Slik (https://communities.netapp.com/community/netapp-blogs/context/blog/2012/11/26/cdmi-tutorial-2--storing-cdmi-data-objects-via-http) and tried to PUT objects in the CDMI server (which is running on my windows machine). But, i am not able to create an object/container. I was  not able to create container/objects by running the CDMItest.java (the error i had mentioned in previous post) therefore, I started trying it by using curl. I have attached the snapshot of my command prompt showing the details of error. If anyone can help me out with this then please do provide your valuable help. 

Also, are these objects (object/containers) created in JVM? Basically, how are they stored ?

Thanks & Regards,
Nikhil
Capture.PNG

Mark Carlson

unread,
Feb 27, 2013, 9:43:07 AM2/27/13
to snia-...@googlegroups.com
Nikhil,

It looks like you are trying to do what is called a non-CDMI content type operation. The reference implementation code you have does not yet support these operations but we are working on it.

For now, you would need to have the X-CDMI-Specification-Version header and encode your object in the "value" field of a JSON message body.

-- mark

Nikhil

unread,
Feb 28, 2013, 5:29:05 AM2/28/13
to snia-...@googlegroups.com

Hi Mark,

Than you very much for your continuous support !!!

I am trying to run the CDMItest.java which is wrtten by you only. I am not able to figure out how i am trying to do a non-CDMI content-type operation. I am just trying to run the testCapabilities() and testContainerCreate() functions.

It is successfully creating the capabilitiy object but fails to create container. Below is output of running the two functions:

Headers : 5
X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.1 Java/Oracle Corporation/1.7)
Server: GlassFish Server Open Source Edition 3.1.1
Date: Thu, 28 Feb 2013 10:17:59 GMT
Content-Type: application/cdmi-capability
Content-Length: 344
---------
HTTP/1.1
200
OK
HTTP/1.1 200 OK
---------
{"objectURI":"cdmi_capabilities\/","objectID":"0000000800183F5E33386433373632302D376162622D3462","parentURI":"\/","parentID":"0000000800183F5E33386433373632302D376162622D3462","capabilities":{"cdmi_metadata_maxitems":"1024","cdmi_metadata_maxsize":"4096","domains":"false","cdmi_export_occi_iscsi":"true"},"children":["container","dataobject"]}
Headers : 7
X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.1 Java/Oracle Corporation/1.7)
Server: GlassFish Server Open Source Edition 3.1.1
ETag: "Object Creation Error : java.lang.IllegalArgumentException: Cannot create container 'Test'"
Date: Thu, 28 Feb 2013 10:18:41 GMT
Content-Type: text/xml
Content-Length: 0
Connection: close
---------
HTTP/1.1
400

Regards,
Nikhil

Nikhil

unread,
Mar 12, 2013, 2:37:23 AM3/12/13
to snia-...@googlegroups.com
Hi,

I tried installing the cdmi-server Reference implementation in eclipse using glassfish 3.1.2. The server is getting deployed in glassfish but when i execute the CDMItest.java containing how to create capability, container, object i get the following error messages in eclipse console:
the name of my container is Test and name of object is first.txt

Headers : 7
X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2 Java/Oracle Corporation/1.7)
Server: GlassFish Server Open Source Edition 3.1.2
ETag: "Object PUT Error : org.codehaus.jackson.JsonParseException: Unexpected character ('o' (code 111)): was expecting comma to separate OBJECT entries  at [Source: [B@7774f1e3; line: 3, column: 42]"
Date: Tue, 12 Mar 2013 06:36:33 GMT
Content-Type: text/xml
Content-Length: 0
Connection: close
---------
HTTP/1.1
400


I anyone has any idea on how to proceed further then please do help.

Thanks & Regards,
Nikhil

Nikhil

unread,
Mar 12, 2013, 2:39:36 AM3/12/13
to snia-...@googlegroups.com
Also attached herein are the logs when i deploy the CDMI server on glassfish 3.1.2 in eclipse
logs.docx

Ancoron Luciferis

unread,
Mar 12, 2013, 3:00:07 AM3/12/13
to snia-...@googlegroups.com, Nikhil
Hi,

it looks like you didn't create the directory "C:\data", which is the
default for the CDMI server.

Either create it or point the server to a different path by modifying
the path inside the file "src/main/resources/applicationContext.xml" for
the Spring beans "containerDao" and "dataObjectDao":

<property name="baseDirectoryName" value="<your-dir>"/>

Cheers,

Ancoron
>> <https://communities.netapp.com/community/netapp-blogs/context/blog/2012/11/26/cdmi-tutorial-2--storing-cdmi-data-objects-via-http>)
>>>>> 1. First of
>>>>> all, the RI
>>>>> is open
>>>>> source but
>>>>> the
>>>>> different implementations are
>>>>> not. What
>>>>> one can
>>>>> rely on,
>>>>> are the
>>>>> interfaces,
>>>>> as
>>>>> prescribed
>>>>> by the
>>>>> standard.
>>>>> 2. The
>>>>> internal
>>>>> mechanisms
>>>>> would
>>>>> definitely
>>>>> differ
>>>>> between
>>>>> implementations
>>>>> - including
>>>>> metadata
>>>>> management,
>>>>> the actual
>>>>> object
>>>>> store,
>>>>> chunking et al.
>>>>> 3. The best
>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.
>>>>> For more
>>>>> options,
>>>>> visit
>>>>> https://groups.google.com/groups/opt_out
>>>>> <https://groups.google.com/groups/opt_out>.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this
>>>>> message because you
>>>>> are subscribed to
>>>>> the Google Groups
>>>>> "SNIA Cloud" group.
>>>>> To unsubscribe from
>>>>> this group and stop
>>>>> receiving emails
>>>>> from it, send an
>>>>> email to
>>>>> snia-cloud+...@googlegroups.com.
>>>>> To post to this
>>>>> group, send email
>>>>> to
>>>>> snia-...@googlegroups.com.
>>>>> Visit this group at
>>>>> http://groups.google.com/group/snia-cloud?hl=en
>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.
>>>>> For more options,
>>>>> visit
>>>>> https://groups.google.com/groups/opt_out
>>>>> <https://groups.google.com/groups/opt_out>.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this
>>>>> message because you are
>>>>> subscribed to the
>>>>> Google Groups "SNIA
>>>>> Cloud" group.
>>>>> To unsubscribe from
>>>>> this group and stop
>>>>> receiving emails from
>>>>> it, send an email to
>>>>> snia-cloud+...@googlegroups.com.
>>>>> To post to this group,
>>>>> send email to
>>>>> snia-...@googlegroups.com.
>>>>> Visit this group at
>>>>> http://groups.google.com/group/snia-cloud?hl=en
>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.
>>>>> For more options, visit
>>>>> https://groups.google.com/groups/opt_out
>>>>> <https://groups.google.com/groups/opt_out>.
>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message
>>>> because you are subscribed
>>>> to the Google Groups "SNIA
>>>> Cloud" group.
>>>> To unsubscribe from this
>>>> group and stop receiving
>>>> emails from it, send an
>>>> email to
>>>> snia-cloud+...@googlegroups.com.
>>>> To post to this group, send
>>>> email to
>>>> snia-...@googlegroups.com.
>>>> Visit this group at
>>>> http://groups.google.com/group/snia-cloud?hl=en
>>>> <http://groups.google.com/group/snia-cloud?hl=en>.
>>>> For more options, visit
>>>> https://groups.google.com/groups/opt_out
>>>> <https://groups.google.com/groups/opt_out>.
>>>>
>>>>
>>>
>>> --
>>> You received this message because you
>>> are subscribed to the Google Groups
>>> "SNIA Cloud" group.
>>> To unsubscribe from this group and
>>> stop receiving emails from it, send
>>> an email to
>>> snia-cloud+...@googlegroups.com.
>>> To post to this group, send email to
>>> snia-...@googlegroups.com.
>>> Visit this group at
>>> http://groups.google.com/group/snia-cloud?hl=en
>>> <http://groups.google.com/group/snia-cloud?hl=en>.
>>> For more options, visit
>>> https://groups.google.com/groups/opt_out
>>> <https://groups.google.com/groups/opt_out>.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to
>> the Google Groups "SNIA Cloud" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to snia-cloud+...@googlegroups.com.
>> To post to this group, send email to
>> snia-...@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/snia-cloud?hl=en
>> <http://groups.google.com/group/snia-cloud?hl=en>.
>> For more options, visit
>> https://groups.google.com/groups/opt_out
>> <https://groups.google.com/groups/opt_out>.

Nikhil

unread,
Mar 12, 2013, 3:29:37 AM3/12/13
to snia-...@googlegroups.com, Nikhil, ancoron....@googlemail.com
Hi Ancoron,

Thank you very much for your reply. It worked !! i am able to create containers now but the objects are still not being created.

I redeployed the CDMI-server and then ran the CDMItest.java.

Attached herein is the logs of glassfish server. 

Regards,
Nikhil
logs.docx

Nikhil

unread,
Mar 12, 2013, 3:45:20 AM3/12/13
to snia-...@googlegroups.com, Nikhil, ancoron....@googlemail.com
Hi,

I changed the value of my first.txt (name of my Object) and now it is giving a different kind of error:

INFO: org.codehaus.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value
 at [Source: [B@7e402d69; line: 3, column: 86]


does the contents of my object matter? i mean i should be able to place anything in my file(which i am putting as object in the container) ?

Please let me know if this error is actually caused by the contents of my object and how to take care of this.

Thanks & Regards,
Nikhil
>>> &
...

Nikhil

unread,
Mar 15, 2013, 6:20:53 AM3/15/13
to snia-...@googlegroups.com, Nikhil, ancoron....@googlemail.com
Hi,

I am able to create objects also. The error was a json parser error which i could resolve by adding few lines in the server code. 

Can anyone give me some pointers to doing http get operation on containers/objects like listing the contents of container/object?

Regards,
Nikhil
>>>>                                         <<a href="https://groups.google.com/groups/opt_out" target="_blank
...

Nikhil

unread,
Mar 15, 2013, 7:14:08 AM3/15/13
to snia-...@googlegroups.com, Nikhil, ancoron....@googlemail.com
Hi,

I was trying to read contents of an object and i used the following snippet to read the contents of objects. For small sized objects the code works fine but for very large size files the value of len is returned as -1 and it gives 

Content-Type: text/plain
Transfer-Encoding: chunked

The code snippet is:

 HttpEntity entity = response.getEntity();
            if (entity != null) {
                long len = entity.getContentLength();
                if (len != -1){
                    System.out.println(EntityUtils.toString(entity));
                }
                else
                System.out.println("cannot print:"+len);
             }

Regards,
Nikhil
...
Reply all
Reply to author
Forward
0 new messages