Complex data types

[Ben McCann]

Hi,

I'm curious whether and how virgil handles complex data types.  E.g. if I have a user object:

{

  'firstName': 'ben',

  'skills': ['java', 'javascript', 'html'],

  'education': {

    school: 'cmu',

    majors: ['computer science', 'business']

  }

}

Is this something that virgil can handle?  Will it map it to Cassandra using DynamicCompositeType?

Thanks,

Ben

[Brian O'Neill]

Ben,

We were looking at exactly the same thing.  We've been using JSON in our column values, but more recently started looking at composite types.

One of the challenges with composite types is locking in on a definition of composite type.  Presently, each of the clients (Hector, Astyanax, and Thrift) has a different implementation of composite types (and dynamic composite types).

Are you going to be connecting to Cassandra from any other client? (outside of Virgil)

-brian

---- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024

blog: http://weblogs.java.net/blog/boneill42/

blog: http://brianoneill.blogspot.com/

[Ben McCann]

Yes, I had exactly the same concern.  I would like something that is inter-operable between clients.  I was asking how to store complex data types on the Cassandra developer mailing list, which is how I learned of you project.  You may find the thread interesting.

http://www.mail-archive.com/d...@cassandra.apache.org/msg03902.html 

Thanks!

-Ben

[Brian O'Neill]

Ha, yep.  I saw the discussion and almost replied when you sent the original email. ;)

Its easy enough to add support for CompositeTypes to Virgil. We just have to lock-in on a format.

Interoperability is going to be the problem.

I'll chime in on the Cassandra list and maybe take it up with those guys to see if we can get some a standard definition for things like composite type, indexes, etc.

-brian

---- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024

blog: http://weblogs.java.net/blog/boneill42/

blog: http://brianoneill.blogspot.com/

[Ben McCann]

Haha.  Cool.  I'd be happy to help implement it somewhere whether that be virgil or astyanax or wherever.  I just want to store a list in a way we can all agree upon :-)

[Luke Philips]

Checking in on this topic.. I'm sure a year later is has been fully implemented? :)

[Brian O'Neill]

Unfortunately, we never got around to complex types.

We were waiting to see how Cassandra implemented collections.

http://brianoneill.blogspot.com/2013/01/native-support-for-collections-in.html

Since those collections aren't supported so well through Thrift, I've been contemplating a switch over to CQL.

-brian

--
You received this message because you are subscribed to the Google Groups "virgil-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virgil-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024

[Luke Philips]

Ah, those new collections look great. Though has it been discussed to simply take a json doc (or a subset) and just store that as a string/byte? Could be a cheap storage trick until moving over to the collections/alternative to Thrift.

[Brian O'Neill]

That's exactly what we are doing. =)

With virgil, nothing stops you from posting a JSON document as the body of the POST, which is then persisted as the value of a column (as a string).  

It might actually be a minor enhancement to make that compatible with the new CQL collections, because the thrift interface for collections is actually JSON.

See this ticket:

"Currently the result to a query is returned to thrift as JSON (so as a string in JSON format)"

https://issues.apache.org/jira/browse/CASSANDRA-3647

-brian

[Luke Philips]

Always a step ahead of me here.. hmm.. I'll have to give those another try or mess around with the POST body.. I think that idea kept failing because it was trying to parse the json doc that I wanted to store (likely not enough string-ification).

[Brian O'Neill]

Luke,

Here are some sample curl commands that insert JSON:

This inserts the json string:

curl -X PUT http://localhost:8080/virgil/data/dummy/book/collin/page99 -d "{\"jsonTest\":\"Like cold beverages.\"}"

Then this retrieves it:

curl -X GET http://localhost:8080/virgil/data/dummy/book/collin/page99 

{"jsonTest":"Like cold beverages."}

dummy is the keyspace.

book is the column family.

collin is the row.

page99 is the column.

-brian

---

Brian O'Neill

Lead Architect, Software Development

Health Market Science

The Science of Better Results

2700 Horizon Drive • King of Prussia, PA • 19406

M: 215.588.6024 • @boneill42  •  

healthmarketscience.com

This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited.

 

[hero...@gmail.com]

Hi Brian,

Do the curl commands listing above have been integrated in the latest Virgil?

I tried on a CompositeType CF (with single PRIMARY KEY and two UTF8 composite columns ) 

but got the result "Not enough bytes to read value of component 0"

Brian O'Neill於 2013年2月28日星期四UTC+8上午11時19分50秒寫道：

[Brian O'Neill]

Sorry, virgil does not support composite columns at this time.

That JSON is written as a string into a single column value.

[hero...@gmail.com]

Hi Brian,

Our group use Pycassa (C* python client) to access C*, 

finding it's difficult to integrated customized access control on it.

(maybe other C* clients also with the problem)

It seems that Virgil exposing with REST interface helps this a lot :)

After pulling from the submodule proj "cassandra-rest" I found that maybe add some features with java Thrift API can support CompositeType. I guess that the most important problem is how to set up the CompositeType format in REST api ? 
(In Pycassa, composite columns representing as 'tuple', but may not be suitable for others)

So I am curious would Virgil(cassandra-rest) be updated in the following days :)?

Brian O'Neill於 2013年3月2日星期六UTC+8上午12時35分12秒寫道：

[Luke Philips]

Brian,
If the column value itself was a json document, e.g.:
curl -X PUT http://localhost:8080/virgil/data/dummy/book/collin/page99 -d {"foo":"{"bar":"1","zoo":"33"}"}
Actually works fine if I properly 'parse' my json and escape the " in json like so

{"foo":"{\"bar\":\"1\",\"zoo\":\"33\"}"}
I'm happy with that. I'm just thinking aloud here.. if the virigl parsing of the body only 'parsed' the first level elements, then intentionally escaped all sub elements and turned them into strings. so if you don't 'flatten' your json, it'll be shoved into a column, if you do flatten you get all the columns.

Luke