Neo4j Heroku Add On Issues

[James Thornton]

It would be helpful to have a Heroku Add On issue tracker/repository, since the Add On is becoming such a popular way to hook into Neo4j. 

In the meantime, here are a few items...

Some Gremlin scripts, such as custom Gremlin steps (https://github.com/tinkerpop/gremlin/wiki/Gremlin-Steps), must be pre-loaded before they can be used. It would be really useful if you could upload custom Gremlin libraries so that they are loaded on startup. 

I know Michael has been thinking about this issue. In the meantime, having a community-contributed library of Gremlin helpers might be a good first step.

For example, here's a custom Gremlin step that returns a tree, which would allow you to do things like build a threaded comment system for your site...

Gremlin Trees

https://gist.github.com/1197179

Gremlin Trees: Example Usage

gremlin> g.v(1).outTree('comment')

==>[v[1], [v[2], v[3], v[4], [v[5], v[3]]]]

gremlin> g.v(3).inTree() 

==>[v[3], [v[1], v[4], [v[1]], v[6]]]

And here is a transaction helper, which if pre-loaded would reduce the number of bits you have to transmit on each command request as well as the size of the script-engine class cache... 

Transaction Wrapper

https://gist.github.com/1654077

Transaction Wrapper: Example Usage

https://gist.github.com/1654092

Also, the Neo4j Heroku wiki page that is linked to from the blog posts is broken...

http://wiki.neo4j.org/content/Neo4j_Heroku_Addon

- James 

[Michael Hunger]

Thanks James,

we'll figure out where to put the issues for the add-on.

It would be great if the gremlin steps would be in a repository with versioning and deployment, why not make them part of the Gremlin-Script-Engine then they would be automatically available.

Michael

[Peter Neubauer]

Or,
If the scripts are on github.com, you can lose them directly off the Internet without any local storage?

Send from a device with crappy keyboard and autocorrection.

/peter

[Michael Hunger]

Forget it.

Michael

[Peter Neubauer]

Then possibly there could  a plugin watching a folder and loading scripts that then can be packaged in some way, or we do a script plugin template for wrapping custom scripts?

Send from a device with crappy keyboard and autocorrection.

/peter

[James Thornton]

On Sunday, January 22, 2012 2:28:54 AM UTC-6, Peter Neubauer wrote:

Then possibly there could  a plugin watching a folder and loading scripts that then can be packaged in some way, or we do a script plugin template for wrapping custom scripts?

The way the Gremlin-Groovy ScriptEngine works right now, if you execute a script that defines a method on one request and then you execute the method on the second request, the method is found and executes properly...

POST url:  http://localhost:7474/db/data/ext/GremlinPlugin/graphdb/execute_script  

POST body: {"params":null,"script":"def testme() { return 1234; } "} 

POST url:  http://localhost:7474/db/data/ext/GremlinPlugin/graphdb/execute_script  

POST body: {"params":null,"script":"return testme();"} 

1234

But obviously the method definition will be lost on a ScriptEngine restart.

The Java ScriptEngine class has an eval method that works just like the eval method used to execute individual requests, except that it takes a Reader object, which will allow you to load an entire file (http://docs.oracle.com/javase/6/docs/api/javax/script/ScriptEngine.html). 

Scoping should behave just like the example above regarding method definitions on individual requests, so it looks like this is simply a matter of creating a directory of Groovy script files to source each time the script engine is restarted.

- James

[Michael Hunger]

Please keep in mind that for hosted environments and production servers you _don't_ want to load arbitrary files or URLs. Really, you don't.

That's why I would suggest to include those useful scripts either into gremlin itself (talk to Marko) or make them part of the gremlin-script engine.

Michael

[Marko Rodriguez]

Hi,

> Please keep in mind that for hosted environments and production servers you _don't_ want to load arbitrary files or URLs. Really, you don't.

The way in which I use Gremlin in production is as follows:

1. I never use the WebAdmin unless I'm just browsin' around.
2. For simple queries (e.g. g.v(1).out('knows').in('knows').name), I use the GremlinPlugin via my client code.
3. For complex algorithms, I write a server extension in Groovy that have embedded Gremlin. E.g.

public MyServerExtension .... {
static {
Gremlin.load();
}

public List<String> myEndPoint(...) {
def y = g.v(x).out.collect{
// do some stuffs
}
complexFunction(y).into.another.gremlin.expression.iterate();
return someList;
}
}

In short, for a hardcoded algorithm that will be used over and over by a system, its written as an extension and written in native Pipes, Gremlin-Java, or in Gremlin-??? as seen fit.

HTH,
Marko.

http://markorodriguez.com

[Peter Neubauer]

Right,
That is my approach too. However, then we need to provide a way to load custom custom plugins. Michael, let's talk about that next week.

Thanks for the input Marko!

/peter

--

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

The Neo4j Heroku Challenge               - http://neo4j-challenge.herokuapp.com/
Hack Real Problems. Win Real Prizes - http://www.t-b-d.org

[James Thornton]

On Sunday, January 22, 2012 11:36:43 AM UTC-6, Peter Neubauer wrote:

That is my approach too. However, then we need to provide a way to load custom custom plugins. Michael, let's talk about that next week.

Being able to load custom Java plugins in the Heroku Add On will be great, and it is absolutely what you want for high-performance production sites. But from a marketing perspective, making that the only option will be a harder sell.

New Add On users will probably be Web developers accustomed to writing in dynamic languages. 

Saying you can write server-side scripts in a dynamic language like Groovy (or better yet JavaScript) is something you can sell, but telling them they need to understand and write Java plugins will be a turnoff.

From their perspective, it will be good to know there is a path to performance by way of plugins, but when you're in the stage of developing a prototype or MVP and the design is rapidly changing because you don't yet know what's needed and you're short on developers because you haven't reached the funding stage yet, the mindset is to use tools that are fast, light, and flexible. 

This is one of the reasons the flexibility of NoSQL is gaining market share over the rigidity of SQL. It's why Python and Ruby are the predominant Web development languages over Java. 

Is having script libraries loaded via the Gremlin.load() method the best way to make this work?

If so, would adding a configurable CustomLoader to Gremlin.groovy (https://github.com/tinkerpop/gremlin/blob/master/gremlin-groovy/src/main/groovy/com/tinkerpop/gremlin/groovy/Gremlin.groovy) be the way to go?

- James

 

[James Thornton]

I think there may be a way to load custom Gremlin-Groovy files at runtime by switching to the native GroovyScriptEngine and supplying a   ResourceConnector:

http://groovy.codehaus.org/api/groovy/util/GroovyScriptEngine.html
http://groovy.codehaus.org/api/groovy/util/ResourceConnector.html

Recall that Jochen Theodorou ("blackdrag", the Groovy Project lead) also recommended switching off the JSR223 ScriptEngine and onto the the native GroovyScriptEngine to fix the class cache PermGen issues (https://groups.google.com/d/msg/neo4j/5SvP5lkJESQ/g_GFKrIDrCcJ).

I posted a message to the Groovy Users group to find out if this is the right approach (https://groups.google.com/d/msg/groovy-user/ONkx_FB-t38/mcivWPGw7_sJ).

- James

[Matt Luongo]

I've got to agree strongly with James. And, in fact, I fit perfectly
in his unfunded startup category of users.

I think we're actually stuck in a similar position (James, you're
working on Bulbs, right?). I need to load some repeatable Gremlin-
Groovy server-side for neo4django. If neo4django requires server-side
plugins, it's going to punish people wanting to experiment by
requiring that they not only set up the database and download the
library- which is pretty standard- but also figure out how to install
my db plugin. What I really need is some way to just send a bunch of
Groovy (via REST) to the server for later use, and be confident it'll
stay available. I wasn't even aware function definitions would persist
between REST calls, so that's awesome and should help me a bit.

James, from a practical workaround perspective, do you know when/how
often the ScriptEngine restarts? In light of some definitions
persisting between calls to the Gremlin plugin, I could imagine a
situation where every bit of Groovy I send that relies on earlier
'library' definitions could just return a code if those definitions
aren't loaded, and the 'library' could be resent. Thoughts?

> If so, would adding a configurable CustomLoader to Gremlin.groovy (https://github.com/tinkerpop/gremlin/blob/master/gremlin-groovy/src/m...)

[Matt Luongo]

All that said earlier, I could still use the Gremlin server extension
approach for my company's work, just not neo4django.

Marko, I'm a Groovy novice- is there a decent example of a full Neo4j
server extension written in Groovy that you could point me to? I
thought I understood the Groovy/Java relationship but I have no idea
where you'd put the code you included, how Neo is supposed to pick it
up, etc. Sorry, again I know it's a simple question but I'm only so
familiar with Neo's inner workings.

Thanks,

--
Matt Luongo
Co-founder, Scholr.ly

[Peter Neubauer]

Matt,
I would actually suggest to maybe modify the gremlin-plugin (written
in Java, and executing gremlin code from the request) at
https://github.com/neo4j/gremlin-plugin/blob/master/src/main/java/org/neo4j/server/plugin/gremlin/GremlinPlugin.java
and extend it in a way that lets you cache, and or remove scripts.

This is not something we plan to support in hosted environments, but
with the hints James is investigating, we might come to a point where
it makes sense to allow it even in the hosted versions.

Let me know if you have questions on the plugins, would be happy to
help over Skype or so next week.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K

The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[James Thornton]

> James, from a practical workaround perspective, do you know when/how 
> often the ScriptEngine restarts? 

Neo4j Server is hardcoded to restart the script engine every 500 requests.

Rexster is hardcoded to restart the script engine every 1000 requests.

- James

[Matt Luongo]

Thanks for the help, guys!

Peter - because I need this for neo4django, I'm not sure if
customizing the Gremlin plugin will cut it- I think that might be a
barrier to people using the project. There's a lot of power in people
installing the db, then running `pip install neo4django` and then
being ready to go. I'll definitely keep that in mind, though.

[James Thornton]

Hey Matt -

Marko is looking into the possibility of switching to the native GroovyScriptEngine (https://groups.google.com/forum/#!topic/gremlin-users/-WCyydYILXo) so hopefully there will be a way soon.

Peter, the Heroku Add On page that is linked to from all the blog posts is broken (http://wiki.neo4j.org/content/Neo4j_Heroku_Addon).

- James

[Peter Neubauer]

Ok,
thatnks for pointing out, we will move the info!

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K

The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Peter Neubauer]

James,

we updated a blog post, do you remember where you found the broken links?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K

The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

On Sat, Jan 28, 2012 at 2:03 AM, James Thornton <james.t...@gmail.com> wrote:

[James Thornton]

Here's one...

http://www.readwriteweb.com/hack/2011/08/heroku-neo4j-addon-in-private.php

And there are more scattered throughout the Web and on the mailing list. Do any of the docs on heroku.com link back to that?

What about adding redirects from the old wiki to the new page?

- James