REST API responses can be too verbose

[Brian Levine]

Hello,

I ran into this while using the REST API to add properties of a node to a full-text index.  From the documentation (1.7), you can see that to add a property/value to a node index, one does:

POST http://localhost:7474/db/data/index/node/favorites 

with a JSON payload including the URI of the node, the key and the value.  On success, the response code is 201 and a Location header containing a URL of the form:

"http://localhost:7474/db/data/index/node/favorites/some-key/some%20value/106" 

This URL is also included in the "indexed" property in the JSON response which seems unnecessary to me (since it's already included in the Location header).

In my case, I'm using a full-text index and the value can be large which could result in a very large (and possible invalid) URL. I'm not sure what the caller would do with this location URL anyway. Can you do a GET on it?

I think this is a case where the REST API has erred on the side of returning too much information in an effort to support the RESTful best practice of discovery. In fact, I'd vote for an optional "non-verbose" mode in the REST API so that smaller payloads are returned. This is especially important for clients on mobile devices where your user might be paying for all that data.

-brian

[Peter Neubauer]

I think you could instead do a Cypher index lookup and then just return what you need, like

Start n=node:persons(name = 'test') return id(n)

?

[brian]

Thank you Peter.  I was actually referring to the REST response I receive when *adding* an index entry as opposed to querying the index.  But my post also had to do with how verbose REST responses are in general. I'd hate to think that I have to resort to issuing REST requests using some query language within the request just to work around the verbosity issue.  And in my specific case of adding an index entry, I'm wondering whether some of the information you're returning (e.g. in the Location header) is actually useful. What can you do with the URL returned in that Location header.

-b

On Monday, May 7, 2012 11:38:34 AM UTC-4, Peter Neubauer wrote:

I think you could instead do a Cypher index lookup and then just return what you need, like

Start n=node:persons(name = 'test') return id(n)

?

[Peter Neubauer]

Right.
We have been thinking of a more compact JSON format, as the default
one is really geared towards Hypermedia (so everything you get back is
actually discoverable and standalone, which makes it verbose).
However, it seems that more pragmatic formats are more popular.

Having the compact format is one thing, the other is the question if
the atomic REST API is the right way to express operations in the DB.
We now think that Cypher is a better way to do that. Indexing
operations are notably missing from that right now, since there are
plans to work on a much better indexing infrastructure in the
not-too-far future, so Cypher should wait for that.

So, I think using the REST indexing subset (maybe using streaming
Batch operations for better performance, see
http://docs.neo4j.org/chunked/snapshot/rest-api-batch-ops.html#rest-api-execute-multiple-operations-in-batch-streaming)
might be the best solution, or maybe Gremlin and server side groovy
access to the Java indexing API, see
http://docs.neo4j.org/chunked/snapshot/gremlin-plugin.html#rest-api-send-an-arbitrary-groovy-script---lucene-sorting
.

Does that make sense?

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

[brian]

This makes sense.  Although I think there's still a place for a "pure" RESTful API that addresses resources atomically. When that's too fine-grained or not performant, batching methods over REST can be used.  When even that's not sufficient either because it's not performant or not expressive enough, using a query language with mutation support can be used.  However, I start getting worried when the only way to use a REST API is to tunnel a query language (e.g. Cypher) or a programming language (e.g. Groovy) over REST.

-b

[Peter Neubauer]

Brian,
what is the rationale behind your worry? Would you like to access
MySQL via REST Tuple atomic operations rather than SQL? I think this
is a VERY important discussion, for all of us.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

[brian]

If you have to tunnel some other form within a REST API, it brings into question why you needed a REST API in the first place. But perhaps you're making the point that REST APIs are not well suited to database operations, which is certainly a very good point. 

My scenario:

I am writing a Ruby on Rails application.  I don't want to move to JRuby and so I can't make use of direct access to a Neo4j server.  My only option at this point is to use the REST API, perhaps with one of the available Ruby bindings such as Neography. If I were using some RDBMS rather than Neo4j, I'd be using whatever Ruby driver is available for that database to provide remote communication. If I were writing a Java application against an RDBMS, I'd be using JDBC.  So I no, I wouldn't expect to be using a REST API against the database itself.

So in short, I guess this a tension between whether REST is an appropriate remote communication mechanism for a database and the fact that it's the only thing available for Neo4j depending on the language in which your app is written. Maybe this is all leading to consideration of  some other remote communication mechanism such as Thrift? Or maybe you're right and, if the only option is REST (at least for now), the best we can do is tunnel query language inside of REST.

My original post was about some of the issues I ran into as I began to learn your REST API, but before I started considering some of the meta-issues mentioned above ;-)

-b

On Monday, May 7, 2012 12:54:22 PM UTC-4, Peter Neubauer wrote:

Brian,
what is the rationale behind your worry? Would you like to access
MySQL via REST Tuple atomic operations rather than SQL? I think this
is a VERY important discussion, for all of us.

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

[Nigel Small]

Comments inline...

On 7 May 2012 18:51, brian <blevi...@gmail.com> wrote:

If you have to tunnel some other form within a REST API, it brings into question why you needed a REST API in the first place. But perhaps you're making the point that REST APIs are not well suited to database operations, which is certainly a very good point. 

Being based on CRUD, the REST philosophy certainly has a large overlap with typical database operations so I'd argue that it is a good fit. The only place where this may not be true is when dealing with database transactions, due to their stateful nature. We also shouldn't forget that REST is merely an architectural approach and not a prescription for a solution - the current incarnation of the REST API is still evolving and if the consensus is that it's too verbose, it can easily evolve to become more terse, while remaining RESTful.

My scenario:

I am writing a Ruby on Rails application.  I don't want to move to JRuby and so I can't make use of direct access to a Neo4j server.  My only option at this point is to use the REST API, perhaps with one of the available Ruby bindings such as Neography. If I were using some RDBMS rather than Neo4j, I'd be using whatever Ruby driver is available for that database to provide remote communication. If I were writing a Java application against an RDBMS, I'd be using JDBC.  So I no, I wouldn't expect to be using a REST API against the database itself.

You also have the option to build a server plugin if you can determine a better method of communication for your use case. I'm slightly confused about what you are actually questioning though - REST, JSON or simply the absence of a binary protocol. The biggest issue we have found so far with the REST API has been that of performance and there is already a considerable amount of work going into looking at ways to improve this.

[brian]

To be honest, this thread started with my concern that certain responses would be too verbose especially when property values are included in the path of URLs in the Location header or in the JSON response.  So you can take that (FWIW) as specific feedback on the current state of the REST API. The rest (no pun intended) started because of Peter's suggestion that the work-around was to issue a Cypher query or send Groovy code over REST.  I didn't really intend to question the use of REST or JSON in Neo4J. Yes, I'm aware that I can extend or wrap the server with a protocol of my choosing.  It's just a small matter of some additional code ;-)

So leaving the discussion of some other remoting technology for some other thread:

- In general, I'd like to see the REST API support a terser representation.  This could still be JSON, but maybe leaving out the links.

- I think the fact that property/name values can appear in the path of a  URI (especially URIs that the API returns in Location headers) is a bug since those (the values at least) can become quite large.

-b

On Monday, May 7, 2012 4:55:12 PM UTC-4, Nigel Small wrote:

Comments inline...

[brian]

Just one clarification, I should have using the term key/values rather than property/values since we're talking about adding index entries here which don't have to be based on a property of a node or relationship.

-b

[Bert Fitié]

Just for information, a brief remark about my RailNet application that may be related to this thread. In this application ALL user interaction is done via the REST API wrapped in a few simple shell functions.

For the output part jsawk (https://github.com/micha/jsawk) is used to convert the verbose JSON responses in an awk-like fashion to nice lists or short to-the-point answers.

Further, to simplify user input, resty (https://github.com/micha/resty) is used in combination with a shell function to generate JSON input from user arguments.

-- Bert Fitié

[Michael Hunger]

Any chance to share it? Or at least blog about it? That would be awesome.

Michael

[Bert Fitié]

If you think there is interest in this, I will try to find time in the coming weeks to write it up and make a link available.

-- Bert Fitié

[Peter Neubauer]

That would be cool. Even Andreas Kollegger is playing around with a
very slimmed approach to interacting with the server, and Jake is
hacking on very fast HTTP without GC overhead, so there is a lot
boiling here. All approaches are welcome input to this discussion!

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

[Bert Fitié]

The following is the link to the promised write-up of RailNet: http://www.phynix.info/2012/railnet-and-the-neo4j-rest-api.md

-- Bert Fitié

[Peter Neubauer]

That is pretty cool.

Thanks for writing this up Bert, much appreciated!

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

If you can write, you can code - @coderdojomalmo
If you can sketch, you can use a graph database - @neo4j

[Jacob Hansson]

To address the three issues here:

+1 on Brian's point that putting the indexed value in the URI is a problem.. I'm not quite sure how to solve it, since the same entity can be indexed with the same key and multiple different values, and we need a way to address specific ones. Will put it into the backlog for consideration.

On the subject of limitations of rest, previous discussions have shown that 1) we can in fact properly introduce transactions in a RESTful way (see the fast-http prototype) 2) that the overhead of the verbose API is a much smaller part than the overhead induced by Jersey and jetty, in our internal prototypes we have successfully improved performance more than 100-fold without changing the data transferred over the wire

Jake

Sent from my phone, please excuse typos and brievety.

[nuba]

Hi folks,

Found this oldish thread while searching for other things, but this is important, so I'll chime in: +1 for keeping the links! :)

After educating myself better on REST and HATEOAS, I'm rewriting my Neo4j perl client module to receive only the API entry point from the user, and discover everything else from the links in the answers. The rationale: semantic ties will last outlast harcoded strings tied to API versions.

See also: http://martinfowler.com/articles/richardsonMaturityModel.html

Regards,

Nuba Princigalli

[Peter Neubauer]

Nuba,
cool! The Perl driver REST client will the first one to use these
links for discovery, it would be very
interesting to get your feedback on it!

Cheers,

/peter neubauer

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

Wanna learn something new? Come to @graphconnect.

[Lasse Westh-Nielsen]

+1!!

[Aseem Kishore]

node-neo4j also uses (mostly) only the links for discovery. =)

E.g. https://github.com/thingdom/node-neo4j/blob/develop/lib/GraphDatabase._coffee

In practice, quite a bit of knowledge is still required, e.g.:

- To get a node by ID, you have to know to append the ID:
"#{services.node}/#{id}"

- To get a relationship by ID, Neo4j doesn't convey the relationship endpoint, so we construct it manually:

services.node.replace('node', 'relationship')

- We don't make a separate fetch to the index endpoint just to get the template for the index URLs; we just do:

"#{services.node_index}/#{index}/#{key}/#{val}"

No great reason for this one btw. We just didn't see hardcoding a bunch of template strings as that much better, plus that wouldn't prevent having to know the POST params anyway, for adding to the index.

Just a few little things like that. Overall it's nice.

My one big feature request would be to have the returned URLs be (root-)relative instead of absolute. That would make it easier to maintain e.g. HTTP auth info in the URL, port forwarding, etc.

Aseem

[Aseem Kishore]

I should mention that returning relative URLs would also reduce the verbosity. Maybe not by a ton, but presumably still meaningful? (E.g. >10%)

[nuba]

What is the case for absolute URLs? I imagine it could make sense if one is partitioning the database in some way, spreading the graph in more servers (sharding) and/or having specialized servers (e.g. one for the graph, another for the indexes). But even then, you could just use absolute URLs when needed, and stick to relative, or root-relative ones, as the default. 

+1 for root-relative URLs :)

[Peter Neubauer]

You mean if a URL different than the current slef-URL of the server,
to the full length, otherwise relative?

Cheers,

/peter neubauer

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

Wanna learn something new? Come to @graphconnect.

[Aseem Kishore]

That may have been what he meant, yeah. But so far, is that ever the case?

[Mark Burns]

For the benefit of anyone here (who happens to be using Ruby and Neography) and finds the REST responses to be a little too verbose to be readable/usable:

```ruby

  def remove_hateoas_links(result)

    if result.is_a?(Hash) && result["data"].present?

      remove_hateoas_links(result["data"])

    elsif result.is_a?(Array)

      result.map{|a| remove_hateoas_links(a)}

    else

      result

    end

  end

```

This moves it back into the realms of being able to read and understand a response for me. E.g. in writing tests etc.

[Michael Hunger]

Thanks Mark

That's why the transactional http endpoint only uses a minimal payload

Neography has support for that too

Sent from mobile device

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Burns]

Really Michael? Would be good to find out how.

Well sounds like there's a chance to maybe improve the docs. I've been just executing cypher queries and trawling through the responses.

https://github.com/maxdemarzi/neography/wiki/Scripts-and-queries

I'll have a read through the source.

By the way, to anyone using/reading the method I wrote earlier, it will probably want tweaking. I was too eager to provide a solution and it skips out relationship types and empty responses. 

I'm not going to expand upon it here/correct it here, because it's not a great medium for updating code, and I'm hoping there's a better solution that negates the need for it.

--
You received this message because you are subscribed to a topic in the Google Groups "Neo4j" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/neo4j/vY4AbPYWUR0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to neo4j+un...@googlegroups.com.

[Michael Hunger]

Mark perhaps just issue a PR to neography for sanitizing responses with an optional param?

For the tx endpoint response format see

http://docs.neo4j.org/chunked/stable/rest-api-transactional.html

https://github.com/maxdemarzi/neography/wiki/Transactions

Sent from mobile device