Elasticsearch

[betepahos]

Is there 'some' implementation of cassandra triggers for Elasticsearch.. Thanks..

[Brian O'Neill]

We piloted use of ElasticSearch initially, but we didn't end up integrating it with Cassandra.

We plan to switch the triggers capability over to a synchronous model with a javascript capability.  With that, it should be fairly straightforward to integrate the two. (simply POST out to ES for each mutation)

We should have time in a few weeks to make that change in triggers.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

[betepahos]

Could you explain why you need to implement it  synchronously and why u need the javascript to implement it?... One more thing little of topic when ES index something from Cassandra (lets say profile or name) when the search performed to index (ES) will ES pull the record from Cassandra (hole point to "index" is pointing from wich document is the result) for further use when may be change (and change the record in cassandra not in the index).

[Brian O'Neill]

You don't need to do it synchronously, especially if you are willing to deal with an index that would be eventually consistent with the storage.  If you can accept that, then you could actually use the existing cassandra-triggers module to implement the bridge between Cassandra and ES.  

Implement the Triggers interface.  In your implementation, simply POST to ES to reflect the respective mutation.  Configure your trigger against the appropriate Column Family and you should be good-to-go.

I only suggested javascript, because implementing a Trigger right now is not as straightforward as we would like   since it requires implementing a Java interface and dropping the resulting jar onto the Cassandra classpath.  

If you do go ahead with the ES bridge, definitely let us know.

RE: the document retrieval

It depends how you implement the bridge between Cassandar and ES.  If you put the document in ES, then you could pull it directly from ES.  If you don't, then you'll first need to query ES for the docs, then perform the corresponding lookups in Cassandra.

Hope that helps.  Like I said, if we get to the javascript-based triggers, we might go ahead and implement the ES bridge because it will be dead simple.  

-brian

 

On Jun 21, 2012, at 7:37 PM, betepahos wrote:

Could you explain why you need to implement it  synchronously and why u need the javascript to implement it?... One more thing little of topic when ES index something from Cassandra (lets say profile or name) when the search performed to index (ES) will ES pull the record from Cassandra (hole point to "index" is pointing from wich document is the result) for further use when may be change (and change the record in cassandra not in the index).

-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/

[betepahos]

I can live with eventually consistent data. It`s not a realtime application. 

Retrieval. Is it slow down the time response big time when the app need to jump from ES index and get the data from Cassandra .

I think will be fit in the case with cassandra triggers. i like cassandra and ES (strangely when i first came across i did not, but rethinking the author ideas found for me that its good approach to common problems) and i am determent to use them together. Will let u know if go with the triggers.

[Brian O'Neill]

Definitely do.  And I'll let you know if we get to bridging the two as well.

-brian

[betepahos]

Ok thanks and good luck

[betepahos]

by meaning 'use of existing cassandra-triggers' meaning cassandra-triggers or cassandra-indexing-triggers wich is implemented for Solr... can it be adopted for elastic search or will simpler to use the cassandra-triggers..

[betepahos]

just to note somewhere : When installing virgil the Classpath folder is taken from current working directory *if you point to the directory within bin and try "sh virgil -host localhost" will give an error since virgil script is configured to pick the current directory (wich will be ~/virgil/bin and can`t navigate to /lib/aspectwaver-1.x.x.jar). You must cd to ~/virgil and bin/virgil -host. and evreything will work like charm :). Sorry if this feels intrusive.

[Brian O'Neill]

Thanks for pointing that out.  We'll update the shell script to fix it.

-brian

On Jun 25, 2012, at 11:01 AM, betepahos wrote:

just to note somewhere : When installing virgil the Classpath folder is taken from current working directory *if you point to the directory within bin and try "sh virgil -host localhost" will give an error since virgil script is configured to pick the current directory (wich will be ~/virgil/bin and can`t navigate to /lib/aspectwaver-1.x.x.jar). You must cd to ~/virgil and bin/virgil -host. and evreything will work like charm :). Sorry if this feels intrusive.

[Brian O'Neill]

You could use either.  It might be easiest to start with cassandra-indexing-triggers and adapt it to ES.

-brian

On Jun 24, 2012, at 4:56 PM, betepahos wrote:

by meaning 'use of existing cassandra-triggers' meaning cassandra-triggers or cassandra-indexing-triggers wich is implemented for Solr... can it be adopted for elastic search or will simpler to use the cassandra-triggers..

[Brian O'Neill]

Thanks again.  This is now fixed and will be included in the next release.

https://github.com/hmsonline/virgil/pull/19

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: betepahos <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Mon, 25 Jun 2012 08:01:03 -0700 (PDT)
To: <virgil...@googlegroups.com>

Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

No problem.. pleasure was mine to help.

[Dgiager]

After readed a few articles and posts, and asked a few dumb and several not so moron questions its looks like Java api for ES is our friend for the job. Particular Node.Client or Transport.Client. The implementation for the Solr requires to post on given endpoint (Solr host), but an implemtation for ES requires a comunication with ES Cluster (although it can be implemented like this case just to post at given url but Java client looks more natural i think) when the trigger is activated insted the data converted and posted to Solr host , be converted and handled to let say TransportClient (wich can sniff for new nodes and does not join the cluster) and let the client do its job handling the indexing, routing etc to ES cluster policy.This situation is somewhat desired because what if you want to (and shoud for faster searching) have one index for posts and one for people then the document will be forwarded accordingly to the proper node for indexing. I think it will be easier just handle the document to the client and let the cluster do the rest :) tell me what you think ..

[Brian O'Neill]

Great point.  Agreed.  I would use the Java API to avail of all the clustering facilities.

Do you plan on open-sourcing?  If so, let me know when you have something to share. I can help out.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Mon, 2 Jul 2012 12:23:43 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

yess of course if i have something i will notify you ... its fun to opensource ;)

[Brian O'Neill]

Excellent.  I look forward to taking a look and helping out.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Mon, 2 Jul 2012 15:59:33 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

it will be based on cassandra indexing triggers for the most part :)

[Dgiager]

my friend after 7 days of intentionally banging my head in the wall i finally got it working. will polish it (making the jsonobject deserialise in to separate rows etc etc.. will send it to you soon  for reviewing it (becouse im not sure it is coded good although its based mostly on your code for solr indexing triggers). Still need to figure out few things ,routing etc but its working :)  i need to sleep now ..

[Brian O'Neill]

Nice work!   Excellent!

I look forward to seeing it.

Sleep well. ;)

-brian

On Jul 11, 2012, at 12:36 AM, Dgiager wrote:

my friend after 7 days of intentionally banging my head in the wall i finally got it working. will polish it (making the jsonobject deserialise in to separate rows etc etc.. will send it to you soon  for reviewing it (becouse im not sure it is coded good although its based mostly on your code for solr indexing triggers). Still need to figure out few things ,routing etc but its working :)  i need to sleep now ..

[Dgiager]

where and how to send the code to work in parallel and take a peek. if there is something you can make please do .change it if it don`t look right (something about been not so skilled proggramer :( ). I now seek a way to get columz one at a time becouse when its serialised to JSON somehow its serialised twice i think its Due toJSONString().

[Dgiager]

i`ve created a github repository but i have to take a look its for the first time :)

[Brian O'Neill]

Sounds good. Let me know as soon as you have a github link to share.

We can take a look at things.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Wed, 11 Jul 2012 13:28:15 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

how about send you an email because  the jar contains elastic search classes and i dont have general idea for licences (where who when) if i send it on github u have to strip the es jar because the classlookups etc etc.. i will send just the indexing classes in the repo and will send you the full jar at the email you provided to just put the jar in the classpath and begin :) ok?

[Dgiager]

fwew that wans`t so hard  https://github.com/betepahos/ESCT  just the trigger classes u have to add elasticsearch jar strip the jar get elasticsearch.19.8.jar strip that too to include (common and org) to be available for class lookups) and the resulting jar will contains (com, org, common) or send me email will send the full jar ... but for now i have to sleep :)

[Dgiager]

or just download in the ESCT download section included licences. thats from me for now .

[Dgiager]

em.. i can`t make the trigger work with 'cql3' after thinking for a white (to figure out wich is wich for the cql3) i think if the solution includes making a column (i tried that ) com.hmsonline.cassandra.triggers.index.triggerindex will fail becouse cql3 doesn`t allow to make column with dots in the name "." either way i tried with the present way (cassandra-cli) and it did not send the columns to ES. Can we think about that (becouse cql3 is the next move for cassandra) and try inserint multiple values to see if it woking. But that after tell what u think for the trigger. looking for you response because i cql3 is the interface i want to work with. hope i do not get intrusive :)

[Brian O'Neill]

I haven't had a chance to play with CQL3 yet.  Let me take a look.  

We definitely want the triggers to work w/ CQL.  I'll try to take a look tonight.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Fri, 13 Jul 2012 00:53:00 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

ok :) i will look too to see if i can help.

[Dgiager]

any progress...

[Brian O'Neill]

Just started looking at it today.

You have a pull request waiting:

https://github.com/betepahos/ESCT/pull/1

I added support for Maven.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Wed, 25 Jul 2012 03:14:36 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

any progress...

[Dgiager]

ok merged :) any thought on cql3

[Brian O'Neill]

Sorry, didn't get a chance to look into cql3 yet.

I'll see if I can run a test this week.

-brian

-- 

Brian O'Neill
Lead Architect, Software Development
Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406
p: 215.588.6024
www.healthmarketscience.com

From: Dgiager <bete...@gmail.com>
Reply-To: <virgil...@googlegroups.com>
Date: Wed, 25 Jul 2012 11:16:33 -0700 (PDT)
To: <virgil...@googlegroups.com>
Subject: Re: [virgil-users] Elasticsearch

[Dgiager]

ok ...

[Dgiager]

any progress i need to know if this is going somewhere because it has more things to resolve to make it real thing ...