REST API love vs. embedded (was Re: [Neo4j] Re: Graph metadata?)

[Aseem Kishore]

Apologies if this is derailing the thread, but I notice a pattern where the REST API tends to get arguably less love than Neo4j embedded.

As a web developer, it's a bit disappointing -- I don't want to have to fire up a Java IDE, write Java and run my own custom Java app just to use Neo4j as a database.

(Imagine if MySQL required you to write C/C++ to customize this or that!)

I know it's not quite black or white, and I know there are philosophical differences at play; just tossing out an observation that's been building up from my end for a long time.

Aseem

On Fri, Mar 2, 2012 at 3:49 PM, Peter Neubauer <neubaue...@gmail.com> wrote:

Yes,
It's more like configuration and other potential metadata.

Send from a device with crappy keyboard and autocorrection.

/peter

On Feb 29, 2012 10:25 PM, "Josh Adell" <josh....@gmail.com> wrote:

So this isn't user data about the graph as a whole, but rather
configuration data the graph/server uses to manage itself. Is having
user data at graph-level (instead of just the node-/relationsip-level)
something that anyone would have a use for?

-- Josh

On Feb 29, 12:51 pm, Michael Hunger <michael.hun...@neotechnology.com>
wrote:
> I think it is not exposed by intent, as it is rather for "internal" config + metadata of the graph.
>
> Michael
>
> Am 29.02.2012 um 18:49 schrieb Josh Adell:
>
>
>
>
>
>
>
> > Is this exposed via the REST API? Could it be? Maybe doing a GET on "/
> > db/data/" gives it back as a JSON map called "data" along with all the
> > endpoints and server version.
>
> > -- Josh
>
> > On Feb 29, 11:59 am, Peter Neubauer <peter.neuba...@neotechnology.com>
> > wrote:
> >> Yes,
> >> same semantics as normal stuff, just without the requirement to be
> >> attached to a PropertyContainer.
>
> >> Cheers,
>
> >> /peter neubauer
>
> >> G:  neubauer.peter
> >> S:  peter.neubauer
> >> P:  +46 704 106975
> >> L:  http://www.linkedin.com/in/neubauer
> >> T:   @peterneubauer
>
> >> Neo4j 1.6 released                 - dzone.com/6S4K
> >> The Neo4j Heroku Challenge   -http://neo4j-challenge.herokuapp.com/
>
> >> On Wed, Feb 29, 2012 at 5:57 PM, Marko Rodriguez <okramma...@gmail.com> wrote:
> >>> Howdy,
>
> >>>> There are graph properties in since about early 1.6 which are intended for configuration and/or metadata. This is also transferred cross-cluster and in backups.
>
> >>> Just to be certain:
> >>>        1. Is it thread safe?
> >>>        2. Is it transactional? -- persisting is based on success/failure of a transaction.
>
> >>> Thank you,
> >>> Marko.
>
> >>>http://markorodriguez.com
>
> >>> On Feb 29, 2012, at 9:18 AM, Michael Hunger wrote:
>
> >>>> There are graph properties in since about early 1.6 which are intended for configuration and/or metadata. This is also transferred cross-cluster and in backups.
>
> >>>> AbstractGraphDatabase or GraphDatabaseSPI graph;
>
> >>>> PropertyContainer properties = graph.getKernelData().properties();
> >>>> properties.getProperty("key", "default");
>
> >>>> Michael
>
> >>>> Am 29.02.2012 um 16:29 schrieb Marko Rodriguez:
>
> >>>>> Hi,
>
> >>>>> Is there a way to store "metadata" with the graph. I don't want to make a node and add properties to it. However, I want to have (lets say) a Map that is simply serialized/transacted upon like other data in the graph, but not actually be graph data.
>
> >>>>> Thanks,
> >>>>> Marko.
>
> >>>>>http://markorodriguez.com

[Peter Neubauer]

Aseem,
while I agree that the REST API would need some love, this is not a
good example. The EmbeddedGraphDatabase.getKernelData etc is not part
of the official Java API either, so if you use it you are breaching
the contract of the API (Groovy just makes it look easy).

If you are embedded, you use internal classes if you can. If you use
REST, you write some Scripting language on the server side, or a
server plugin to provide additional REST endpoints not part of the
official API. Not a big issues IMHO.

The only way to bring the Java and non-native APIs on the same level
as I see it is via some language-independent declarative approach.
That is why we are working on mutating Cypher, amongst other reasons
(like much better remoting capabilities etc). This will make all
bindings first class citizens API (if stil not totally performance
wise), since ALL access to Neo4j then potentially could go via the
same (declarative) API. That is not feasible in the short term, but I
think you see the pattern.

Does that make sense?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[James Thornton]

On Friday, March 2, 2012 11:01:11 AM UTC-6, Aseem Kishore wrote:

Apologies if this is derailing the thread, but I notice a pattern where the REST API tends to get arguably less love than Neo4j embedded.

As a web developer, it's a bit disappointing -- I don't want to have to fire up a Java IDE, write Java and run my own custom Java app just to use Neo4j as a database.

(Imagine if MySQL required you to write C/C++ to customize this or that!)

I know it's not quite black or white, and I know there are philosophical differences at play; just tossing out an observation that's been building up from my end for a long time.

Hi Aseem -

I understand where you're coming from, and I have had the same concerns regarding the sellability of telling non-Java developers to write Java extensions (https://groups.google.com/d/msg/neo4j/FCt7f8OLo2A/ZeeipOolN9IJ).

Many JavaScript, Ruby, and Python devs aren't set up to write Java. This is especially true in the Web-dev world -- many don't have the Java environment set up, don't have the IDE/tools, and the thought of having to context switch and figure all that out is probably not appealing. 

This is where I think the server-side scripting option shines -- it doesn't require any special tooling, and it gives you access to the entire Java API.

Groovy syntax is not that different from other dynamic languages, and the upcoming sever-side JavaScript plugin will make things even easier -- esp for you since you're working on a Node.js library.

The key from a framework/library design perspective is to make it easy for developers to write and execute Gremlin scripts. Here's how I've approached it in Bulbs/Python:

To write Gremlin scripts in Bulbs 0.2, you had to put them in Python strings and do variable substitution. 

This was a PITA for multiple reasons -- you didn't get Groovy syntax highlighting/indention, it was messy, and hard to debug. A better way to go is to put the Groovy scripts in separate .groovy files and create a parser to load the scripts so the individual methods are easily accessible.

Ideally you would be able to store a custom Gremlin library on the server, but this isn't an option at the moment without some hackery. So unless you need pre-loaded scripts (such as a custom Gremlin step), the simplest way to go is to include the script method on each request.

The Bulbs 0.3 philosophy is to use Gremlin for anything that's not exposed via the REST API, and to use Gremlin anytime you would have to make multiple REST requests.

For example, when you want to create a node and index its properties, this would normally require one REST request to create the node, and a separate request for each property you want to index. 

Furthermore, doing multiple requests over REST is not transactional. You could use the batch API for this, but it's listed as "experimental," it's somewhat chatty, and not always flexible enough for day-to-day use.

The Neo4j Gremlin-Groovy script engine compiles the script the first time it sees it and uses the compiled version on subsequent requests. 

It's reasonably fast, and the more Java API methods you can pack into the script (rather than writing it out in raw Groovy), the faster it will be. It's definitely faster than issuing multiple HTTP requests and dealing with the HTTP overhead. 

Here are the pre-defined Gremlin scripts in Bulbs:

1. https://github.com/espeed/bulbs/blob/master/bulbs/gremlin.groovy

2. https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/gremlin.groovy

Bulbs supports multiple servers so the first file contains Gremlin methods that are generic and can be used on any Gremlin-enabled server. The second file contains methods specific to Neo4j and can override anything in the first file.  

As you can see, the second/Neo4j-specific file contains the the Neo4j Java metadata methods that aren't exposed via REST. I added those yesterday when I saw Michael's reply because I needed a place to store a "last_updated" var in Lightbulb (https://github.com/espeed/lightbulb/blob/master/lightbulb/engine.py#L251), to indicate when the last blog entry that was updated. 

When a Bulbs Graph object is created, it parses the Groovy files and stores each method in a Python dict, keyed by the Groovy method names.

Developers can create their own custom Gremlin library, and add it to the Graph object like this:

>>> from bulbs.neo4jserver import Graph
>>> g = Graph()
>>> g.scripts.update(scripts_file_path)

You can then execute your custom Groovy scripts from Python like this:

>>> script = g.scripts.get('your_custom_query')
>>> params = dict(_id=3)
>>> elements = g.gremlin.query(script, params)
>>> element = elements.next()

>>> script = g.scripts.get('your_custom_command')
>>> params = dict(_id=5)
>>> resp = g.gremlin.command(script, params)

This executes the script on Neo4j Server, and anything defined in params is passed to the Gremlin-Groovy script engine as bind variables. The query() method returns initialized elements, and the command() method returns the raw response object.

If you update the Groovy file while you are in the Python REPL, you can refresh the scripts in the Python Graph object like this:

>>> g.scripts.refresh()

Here's an example of how it's actually used in an app:

* https://github.com/espeed/lightbulb/blob/master/lightbulb/model.py#L52

* https://github.com/espeed/lightbulb/blob/master/lightbulb/gremlin.groovy

As you can see, the Entry Model _save() method passes the "save_blog_entry" Groovy script to the Python gremlin() method, which then executes the Groovy script on Neo4j Server via the Gremlin plugin.

If you were to create a blog entry using the standard REST API, it would take over 20 HTTP requests to create the entry node, three tags, index all the properties, and create the relationships between the entry, author, and tags. With Gremlin you can do it in one request, and it's transactional.

The Gremlin script is somewhat long and includes six closures. But the first five closures are generic, and since the methods are so frequently used, it may make sense to add them to the Java API. 

Then you could reduce the Gremlin script down to this:

def save_blog_entry(entry_bundle, author_id, topic_bundles) {

  def save_blog_entry = { 
    entry = create_or_update_vertex(entry_bundle, "docid");
    author = g.v(author_id)
    found = entry.out("author").filter{it == author}.count()
    if (!found) { g.addEdge(entry, author, "author"); }
    for (topic_bundle in topic_bundles)  {
      topic = get_or_create_vertex(topic_bundle, "name");
      found = entry.out("tagged").filter{it == topic}.count()
      if (!found) { g.addEdge(entry, topic, "tagged"); }
    }
    return entry;
  }
  
  return transaction(save_blog_entry);
}

Anyway, I have explored different approaches the client-side API design, and I think this is the pattern that presently has the most promise. The key is to make server-side scripting as simple as possible. 

What do you think?

- James

[Matt Luongo]

Aseem,

I take a somewhat different approach from James, but I do think that
Gremlin solves most of your problems.

MySQL doesn't ask you to write C/C++- it asks you to write SQL.
Similarly, Neo4j works best with Gremlin and Cypher.

- Matt

[Peter Neubauer]

+1 on that notion.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[Javier de la Rosa]

On Mon, Mar 5, 2012 at 11:33, Matt Luongo <mhlu...@gmail.com> wrote:
> MySQL doesn't ask you to write C/C++- it asks you to write SQL.
> Similarly, Neo4j works best with Gremlin and Cypher.

Is there any way to create, edit and remove nodes and relationships
using just Cypher or Gremlin?

--
Javier de la Rosa
http://versae.es

[Peter Neubauer]

Yes,
Gremlin is accessing the full Java API, see e.g.
http://docs.neo4j.org/chunked/snapshot/gremlin-plugin.html#rest-api-flow-algorithms-with-gremlin
for an example.

Cypher is in the process of adding full CRUD, stay tuned. See
https://docs.google.com/document/d/1fs8gM4KDIsJ9XNbbUtPi9Pyo25KxjeIovmMcyk08liw/edit
for what it could look like.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Neo4j 1.6 released                 - dzone.com/6S4K
The Neo4j Heroku Challenge   - http://neo4j-challenge.herokuapp.com/

[James Thornton]

Peter -

What do you think about adding these methods to the Java API?

These methods are used over and over to create/update nodes, and having them in the Java API would significantly reduce and simplify the amount of script code:

// Create a node and add it to the index for each indexKey

createIndexedNode(properties, indexName, indexKeys)

// Lookup a node by lookupKey, update it, and update the index for each indexKey

updateIndexedNode(properties, indexName, indexKeys, lookupKey)

// Lookup a node by lookupKey, create it if not found, or update it if found

createOrUpdateIndexedNode(properties, indexName, indexKeys, lookupKey)

// Lookup a node by lookupKey, and create it if not found, or return it

getOrCreateIndexedNode(properties, indexName, indexKeys, lookupKey)

If there you are mutating multiple nodes in a Gremlin script, passing all those params around can get cumbersome. 

So instead of passing them around individually, I bundle the properties, indexName and indexKeys in a Map  called "bundle" and pass that around (see https://github.com/espeed/lightbulb/blob/master/lightbulb/gremlin.groovy).  Having bundle-overloaded methods of the above would make things cleaner and easier to work with.

Also, a transaction wrapper that takes a function as could be used on almost every mutating request:

  def transaction = { final Closure closure ->
    g.setMaxBufferSize(0);
    g.startTransaction();
    try {
      results = closure();
      g.stopTransaction(TransactionalGraph.Conclusion.SUCCESS);
      return results; 
    } catch (e) {
      g.stopTransaction(TransactionalGraph.Conclusion.FAILURE);
      return e;
    }
  }

What do you think?

- James

[Marko Rodriguez]

I like the idea of transaction closures. It is a really clean model. Is it an issue in Gremlin GitHub?

Marko.

http://markorodriguez.com

[James Thornton]

On Monday, March 5, 2012 12:42:44 PM UTC-6, Marko Rodriguez wrote:

I like the idea of transaction closures. It is a really clean model. Is it an issue in Gremlin GitHub?

Yes, here it is (https://github.com/tinkerpop/gremlin/issues/277).  

- James