MetaData store

[Nikhil]

Hi everyone,

I am new to CDMI. As a part of my project I need to implement Map/Reduce capabilities on CDMI. I wanted to know if the MetaData of the objects/containers is stored alongwith the objects or is metadata stripped off from the object and stored somewhere else.

Please let me know in case there is some reference material which provides greater insights on the same (trying to put MR on CDMI repositories)

Thanks & Regards,
Nikhil

[Mark Carlson]

Hi Nikhil,

CDMI does not specify implementation details such as this.

From the user point of view the metadata and data are all part of each
object.

However, in order to efficiently process the CDMI query operations, most
implementations will likely cache some or all of the metadata that is
part of queries.

-- mark

[Nikhil]

Hi Mark,

Thank you for your quick reply.

So what I understood is:

Since CDMI is open source, different users can implement this standard in their own ways (some may strip off metadata and keep on a different server, some may keep it with the object, etc.).

If my aim is to enable MapReduce functionality on CDMI repositories then which standard/implementation of CDMI should i use as reference?
Presently, I have downloaded the CDMI source code from http://www.snia.org/forums/csi/programs/CDMIportal

Also, is there any size limitation on an object/container? If the object becomes too large then is it stored in chunks in different servers?

Thanks & Regards,
Nikhil

[Krishna Sankar]

Nikhil,

   A few points:

1. First of all, the RI is open source but the different implementations are not. What one can rely on, are the interfaces, as prescribed by the standard.
2. The internal mechanisms would definitely differ between implementations - including metadata management, the actual object store, chunking et al.
3. The best practice would be to develop the MapReduce layer that works off of the standard; independent of the underlying implementation.   

HTH.

Cheers

<k/>

Thanks & Regards,
Nikhil


--
You received this message because you are subscribed to the Google Groups "SNIA Cloud" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snia-cloud+...@googlegroups.com.
To post to this group, send email to snia-...@googlegroups.com.
Visit this group at http://groups.google.com/group/snia-cloud?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Nikhil]

Hi Krishna,

Thank you for your reply. Can you please explain by "develop the MapReduce layer that works off of the standard; independent of the underlying implementation" in the sense of how to do it. 

I had thought that since Hadoop is already a well-implemented with MR, so is there any way that without "reinventing the wheel" we can use Hadoop to try to run MR on CDMI. 

Thanks & Regards,

Nikhil

[Krishna Sankar]

Of course, one will implement the map & reduce as per the conventions and interfaces of the M framework like Hadoop. I was taling about CDMI ie works off of the CDMI standard interfaces, independent of the underlying CDMI implementation. Reason being, any optimization done at the CDMI interfaces based on internal assumptions could break across different systems. 

Cheers

<k/>  

[Gary Mazz]

Btw,  I have proposals, with a partner company,  into the US fed using CDMI and OCCI as a primary interfaces to a reference architecture as well as an application architecture address to Big Data, Big Data/Cloud intersect. Hadoop is just one deployment model. 

Information and details will be made public as funding for open source permits.

cheers,
gary

[Nikhil]

Hello Everyone,

Thanks a lot for helping me out by taking your time to reply to my queries. :)

I am trying to do MapReduce operation on CDMI servers. Can i get a free URL or account of a CDMI server, in order to test my implementation? I am not sure whether my approach would work or not but i wanted to try it out once.

Thanks & Regards,

Nikhil

[Mark Carlson]

Nikhil,

During the cloudplugfest, there will be several servers of different designs available to work against.

The next one is March 19th and 20th:

http://www.cloudplugfest.org/news/nextcloudplugfest-march2013

Participation is free and can be remote.

-- mark

[Nikhil]

Hi Mark,

Thank you for your quick replies and continued support!!

Since the plugfest is somewhat far, I thought of installing a CDMI server on my own system(Windows 7). I am running Hadoop client on a VM. I have downloaded the RI code and installed Maven 3.0.4. The RI documentation does not give any further guidance as to how to install/deploy the source code. Would this version of Maven work or should I change it to 2.0.9 ?

Also, can you refer me some documentation/link which explains in detail how to set-up a CDMI server. Basically, i want to install CDMI server in my windows, give it some space and then try to connect to CDMI server through Hadoop. 

Thanks a lot for reading this and wanting to help.

Regards,

Nikhil 

[Nikhil]

Hi,

Can anyone help me out with installing  CDMI server....? When i try to run the source code of CDMI server it gives the error "Connection to http://localhost:8080 refused"

Regards,

Nikhil

[Mark Carlson]

Nikhil,

Have you tried using Netbeans or Eclipse on WIndows?

Those tools make things much easier to set up the server if you have never started up Java servers in that environment before.

-- mark

[Nikhil]

Hi Mark,

I could get the server running using eclipse. Thanks!! The documentation does not specify anything about using Glassfish, therefore i got stuck up but it is working now.

Regards,

Nikhil

[Nikhil]

Hi,

Sorry!! I am not able to run it. I am able to build it and it shows BUILD SUCCESS. I have installed Glassfish server 3.1.1 and started it on localhost but i am not able to run the CDMI-server project in my eclipse on glassfish server. When i go to the Run Configuration --> Glassfish 3.1.1 the "Run" button does not get activated. 

[Nikhil]

Hi,

Sorry for the trouble. by taking help of ORACLE documentation for Glassfish the CDMI-server is running now. 

Regards,

Nikhil 

[Nikhil]

Hi,

My CDMI-server is running on localhost:8080 on glassfish server but when I run the CDMItest.java as JUnit Test it is able to create CDMI_capabilities but it fails to create CDMI container. the error being thrown is:

ETag: "Object Creation Error : java.lang.IllegalArgumentException: Cannot create container 'Test'"

If anyone has some idea regarding what needs to be modified in the code then please do share.

Thanks & Regards,

Nikhil

[Nikhil]

Hi,

I read a post by David Slik (https://communities.netapp.com/community/netapp-blogs/context/blog/2012/11/26/cdmi-tutorial-2--storing-cdmi-data-objects-via-http) and tried to PUT objects in the CDMI server (which is running on my windows machine). But, i am not able to create an object/container. I was  not able to create container/objects by running the CDMItest.java (the error i had mentioned in previous post) therefore, I started trying it by using curl. I have attached the snapshot of my command prompt showing the details of error. If anyone can help me out with this then please do provide your valuable help. 

Also, are these objects (object/containers) created in JVM? Basically, how are they stored ?

Thanks & Regards,

Nikhil

[Mark Carlson]

Nikhil,

It looks like you are trying to do what is called a non-CDMI content type operation. The reference implementation code you have does not yet support these operations but we are working on it.

For now, you would need to have the X-CDMI-Specification-Version header and encode your object in the "value" field of a JSON message body.

-- mark

[Nikhil]

Hi Mark,

Than you very much for your continuous support !!!

I am trying to run the CDMItest.java which is wrtten by you only. I am not able to figure out how i am trying to do a non-CDMI content-type operation. I am just trying to run the testCapabilities() and testContainerCreate() functions.

It is successfully creating the capabilitiy object but fails to create container. Below is output of running the two functions:

Headers : 5

X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.1 Java/Oracle Corporation/1.7)

Server: GlassFish Server Open Source Edition 3.1.1

Date: Thu, 28 Feb 2013 10:17:59 GMT

Content-Type: application/cdmi-capability

Content-Length: 344

---------

HTTP/1.1

200

OK

HTTP/1.1 200 OK

---------

{"objectURI":"cdmi_capabilities\/","objectID":"0000000800183F5E33386433373632302D376162622D3462","parentURI":"\/","parentID":"0000000800183F5E33386433373632302D376162622D3462","capabilities":{"cdmi_metadata_maxitems":"1024","cdmi_metadata_maxsize":"4096","domains":"false","cdmi_export_occi_iscsi":"true"},"children":["container","dataobject"]}

Headers : 7

X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.1 Java/Oracle Corporation/1.7)

Server: GlassFish Server Open Source Edition 3.1.1

ETag: "Object Creation Error : java.lang.IllegalArgumentException: Cannot create container 'Test'"

Date: Thu, 28 Feb 2013 10:18:41 GMT

Content-Type: text/xml

Content-Length: 0

Connection: close

---------

HTTP/1.1

400

Regards,

Nikhil

[Nikhil]

Hi,

I tried installing the cdmi-server Reference implementation in eclipse using glassfish 3.1.2. The server is getting deployed in glassfish but when i execute the CDMItest.java containing how to create capability, container, object i get the following error messages in eclipse console:

the name of my container is Test and name of object is first.txt

Headers : 7

X-Powered-By: Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2 Java/Oracle Corporation/1.7)

Server: GlassFish Server Open Source Edition 3.1.2

ETag: "Object PUT Error : org.codehaus.jackson.JsonParseException: Unexpected character ('o' (code 111)): was expecting comma to separate OBJECT entries  at [Source: [B@7774f1e3; line: 3, column: 42]"

Date: Tue, 12 Mar 2013 06:36:33 GMT

Content-Type: text/xml

Content-Length: 0

Connection: close

---------

HTTP/1.1

400

I anyone has any idea on how to proceed further then please do help.

Thanks & Regards,

Nikhil

[Nikhil]

Also attached herein are the logs when i deploy the CDMI server on glassfish 3.1.2 in eclipse

[Ancoron Luciferis]

Hi,

it looks like you didn't create the directory "C:\data", which is the
default for the CDMI server.

Either create it or point the server to a different path by modifying
the path inside the file "src/main/resources/applicationContext.xml" for
the Spring beans "containerDao" and "dataObjectDao":

<property name="baseDirectoryName" value="<your-dir>"/>

Cheers,

Ancoron

>> <https://communities.netapp.com/community/netapp-blogs/context/blog/2012/11/26/cdmi-tutorial-2--storing-cdmi-data-objects-via-http>)

>>>>> 1. First of

>>>>> all, the RI
>>>>> is open
>>>>> source but
>>>>> the
>>>>> different implementations are
>>>>> not. What
>>>>> one can
>>>>> rely on,
>>>>> are the
>>>>> interfaces,
>>>>> as
>>>>> prescribed
>>>>> by the
>>>>> standard.

>>>>> 2. The

>>>>> internal
>>>>> mechanisms
>>>>> would
>>>>> definitely
>>>>> differ
>>>>> between
>>>>> implementations
>>>>> - including
>>>>> metadata
>>>>> management,
>>>>> the actual
>>>>> object
>>>>> store,
>>>>> chunking et al.

>>>>> 3. The best

>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.

>>>>> For more
>>>>> options,
>>>>> visit
>>>>> https://groups.google.com/groups/opt_out

>>>>> <https://groups.google.com/groups/opt_out>.

>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this
>>>>> message because you
>>>>> are subscribed to
>>>>> the Google Groups
>>>>> "SNIA Cloud" group.
>>>>> To unsubscribe from
>>>>> this group and stop
>>>>> receiving emails
>>>>> from it, send an
>>>>> email to
>>>>> snia-cloud+...@googlegroups.com.
>>>>> To post to this
>>>>> group, send email
>>>>> to
>>>>> snia-...@googlegroups.com.
>>>>> Visit this group at
>>>>> http://groups.google.com/group/snia-cloud?hl=en

>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.

>>>>> For more options,
>>>>> visit
>>>>> https://groups.google.com/groups/opt_out

>>>>> <https://groups.google.com/groups/opt_out>.

>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this
>>>>> message because you are
>>>>> subscribed to the
>>>>> Google Groups "SNIA
>>>>> Cloud" group.
>>>>> To unsubscribe from
>>>>> this group and stop
>>>>> receiving emails from
>>>>> it, send an email to
>>>>> snia-cloud+...@googlegroups.com.
>>>>> To post to this group,
>>>>> send email to
>>>>> snia-...@googlegroups.com.
>>>>> Visit this group at
>>>>> http://groups.google.com/group/snia-cloud?hl=en

>>>>> <http://groups.google.com/group/snia-cloud?hl=en>.

>>>>> For more options, visit
>>>>> https://groups.google.com/groups/opt_out

>>>>> <https://groups.google.com/groups/opt_out>.

>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message
>>>> because you are subscribed
>>>> to the Google Groups "SNIA
>>>> Cloud" group.
>>>> To unsubscribe from this
>>>> group and stop receiving
>>>> emails from it, send an
>>>> email to
>>>> snia-cloud+...@googlegroups.com.
>>>> To post to this group, send
>>>> email to
>>>> snia-...@googlegroups.com.
>>>> Visit this group at
>>>> http://groups.google.com/group/snia-cloud?hl=en

>>>> <http://groups.google.com/group/snia-cloud?hl=en>.

>>>> For more options, visit
>>>> https://groups.google.com/groups/opt_out

>>>> <https://groups.google.com/groups/opt_out>.

>>>>
>>>>
>>>
>>> --
>>> You received this message because you
>>> are subscribed to the Google Groups
>>> "SNIA Cloud" group.
>>> To unsubscribe from this group and
>>> stop receiving emails from it, send
>>> an email to
>>> snia-cloud+...@googlegroups.com.
>>> To post to this group, send email to
>>> snia-...@googlegroups.com.
>>> Visit this group at
>>> http://groups.google.com/group/snia-cloud?hl=en

>>> <http://groups.google.com/group/snia-cloud?hl=en>.

>>> For more options, visit
>>> https://groups.google.com/groups/opt_out

>>> <https://groups.google.com/groups/opt_out>.

>>>
>>>
>>
>> --
>> You received this message because you are subscribed to
>> the Google Groups "SNIA Cloud" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to snia-cloud+...@googlegroups.com.
>> To post to this group, send email to
>> snia-...@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/snia-cloud?hl=en

>> <http://groups.google.com/group/snia-cloud?hl=en>.

>> For more options, visit
>> https://groups.google.com/groups/opt_out

>> <https://groups.google.com/groups/opt_out>.

[Nikhil]

Hi Ancoron,

Thank you very much for your reply. It worked !! i am able to create containers now but the objects are still not being created.

I redeployed the CDMI-server and then ran the CDMItest.java.

Attached herein is the logs of glassfish server. 

Regards,

Nikhil

[Nikhil]

Hi,

I changed the value of my first.txt (name of my Object) and now it is giving a different kind of error:

INFO: org.codehaus.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value

 at [Source: [B@7e402d69; line: 3, column: 86]

does the contents of my object matter? i mean i should be able to place anything in my file(which i am putting as object in the container) ?

Please let me know if this error is actually caused by the contents of my object and how to take care of this.

Thanks & Regards,

Nikhil

>>> &

...

[Nikhil]

Hi,

I am able to create objects also. The error was a json parser error which i could resolve by adding few lines in the server code. 

Can anyone give me some pointers to doing http get operation on containers/objects like listing the contents of container/object?

Regards,

Nikhil

>>>>                                         <<a href="https://groups.google.com/groups/opt_out" target="_blank

...

[Nikhil]

Hi,

I was trying to read contents of an object and i used the following snippet to read the contents of objects. For small sized objects the code works fine but for very large size files the value of len is returned as -1 and it gives 

Content-Type: text/plain

Transfer-Encoding: chunked

The code snippet is:

 HttpEntity entity = response.getEntity();

            if (entity != null) {

                long len = entity.getContentLength();

                if (len != -1){

                    System.out.println(EntityUtils.toString(entity));

                }

                else

                System.out.println("cannot print:"+len);

             }

Regards,

Nikhil

...