parts of the REST API are not valid JSON

[Max De Marzi Jr.]

Ran into an ugly problem today.

In neography, I use the httparty gem which switched from crack to
multijson for json parsing.
Turns out multijson is strict and doesn't like some of the things the
neo4j rest api is returning:

For example:
http://docs.neo4j.org/chunked/snapshot/rest-api-relationships.html#rest-api-get-single-property-on-a-relationship

Example request

PUT http://localhost:7474/db/data/relationship/13/properties/cost
Accept: application/json
Content-Type: application/json
"deadly"


"deadly" is not valid json. It's valid 'text/plain'.
{"cost":"deadly"} would valid json.

You guys can change the content-type or the returned value, or I can
code a workaround... but others writing rest clients may run into this
issue too.

[Jim Webber]

Hi Max,

If you raise an issue against this, we'll get onto it. I like the idea of fixing it at source rather than having many workarounds in all the client libraries.

Jim

[Hendy Irawan]

Indeed Max is correct.

JSON root must be either an object or an array.

["deadly"] also works and the overhead is minimal (2 characters)

[Max De Marzi Jr.]

I made a bad copy and paste job. What I mean is that getting a single
node property does not return valid JSON.

[Max De Marzi Jr.]

Issue raised:

https://github.com/neo4j/community/issues/162

[James Thornton]

This issue is also mentioned here: https://github.com/neo4j/community/issues/119

- James

[Alistair Jones]

I'm happy to be corrected, but this is surprising to me. I've read
the page at http://json.org/ and I don't see anything that says object
and array are the only allowed parsing goals. In Chrome, I can do
JSON.parse("\"Hello\"") without error, and it appears to work for all
value types. I've just tried using Douglas Crockford's reference
parser implementation from here:
https://github.com/douglascrockford/JSON-js/blob/master/json_parse.js
This also allows values as valid JSON.

However, I can find other places where values are not considered valid
JSON, such as here:
http://json.parser.online.fr/
Clearly multijson is also in this camp.

Where can I find a standard which says that only objects and arrays
are allowed?

-Alistair

On Dec 31, 4:44 pm, "Max De Marzi Jr." <maxdema...@gmail.com> wrote:
> Ran into an ugly problem today.
>
> In neography, I use the httparty gem which switched from crack to
> multijson for json parsing.
> Turns out multijson is strict and doesn't like some of the things the
> neo4j rest api is returning:
>

> For example:http://docs.neo4j.org/chunked/snapshot/rest-api-relationships.html#re...
>
> Example request
>
> PUThttp://localhost:7474/db/data/relationship/13/properties/cost

[Tero Paananen]

> Where can I find a standard which says that only objects and arrays
> are allowed?

I'm not sure about this, but I'm going to comment on the structure of
the JSON package from a developer's perspective.

To me, it would always be better, if the JSON packages were "fully formed".

{ "propertyName" : "propertyValue" }

is inherently better to deal with than:

"propertyValue"

Two reasons that come quickly to my mind:

1. I can parse the response of this query with the same code that parses
responses for multiple properties, e.g. { "foo" : "bar", "yo" : "momma" }

2. Parsing the response isn't coupled to the request. In the latter version
the parsing code needs to know that I requested a particular property value.

-TPP

[Max De Marzi Jr.]

Alistair, I think the spec you want is http://www.ietf.org/rfc/rfc4627
, specifically:

2. JSON Grammar

A JSON text is a sequence of tokens. The set of tokens includes
six
structural characters, strings, numbers, and three literal names.

A JSON text is a serialized object or array.

JSON-text = object / array"

[Josh Adell]

Except that section 1 of http://www.ietf.org/rfc/rfc4627 seems to read
that valid JSON can be a simple scalar value:

1. Introduction
...
JSON can represent four primitive types (strings, numbers,
booleans,
and null) and two structured types (objects and arrays).
...

FWIW, the PHP built-in json_encode function treats scalars as valid
JSON.

-- Josh Adell


On Dec 31, 4:56 pm, "Max De Marzi Jr." <maxdema...@gmail.com> wrote:
> Alistair, I think the spec you want ishttp://www.ietf.org/rfc/rfc4627

[Josh Adell]

I just talked with one of the development leads for jQuery UI (not an
authoritative source, I know) and he says that scalar values are valid
JSON.

I also checked out ECMA-262, which defines how JSON is parsed in
Javascript, and it explicitly lists differences between Javascript's
JSON.parse/.stringify and RFC 4627.
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
at the bottom of page 201.

tl;dr: The RFC says the top level must be an object or array. ECMA
says it can be any valid JSON value (object, array or scalar).

So it seems that it depends on if you think ECMA or the RFC is the
standard, which probably accounts for why different libraries and
languages implement it differently.

My final thought on this is a memory of something a Javascript
developer told me about implementation differences between Javascript
parsers: "When in doubt, do what Crockford says."

Happy New Year, all!

-- Josh

On Dec 31, 5:33 pm, Josh Adell <josh.ad...@gmail.com> wrote:
> Except that section 1 ofhttp://www.ietf.org/rfc/rfc4627seems to read

[Peter Neubauer]

Sounds to me that a change away from scalars would satisfy everyone and be least friction.next issue is good versioning of the rest API.

Cheers

/peter

Sent from a device with crappy keyboard and autocorrect

[Alistair Jones]

Thanks Max, that's exactly what I was looking for - seems very clear.  My fault for looking at implementations rather than the spec.

cheers,

-Alistair

[Alistair Jones]

On 31 December 2011 23:33, Josh Adell <josh....@gmail.com> wrote:

I just talked with one of the development leads for jQuery UI (not an
authoritative source, I know) and he says that scalar values are valid
JSON.

I also checked out ECMA-262, which defines how JSON is parsed in
Javascript, and it explicitly lists differences between Javascript's
JSON.parse/.stringify and RFC 4627.
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
at the bottom of page 201.

This makes it all clear now - thank you.

tl;dr: The RFC says the top level must be an object or array. ECMA
says it can be any valid JSON value (object, array or scalar).

So it seems that it depends on if you think ECMA or the RFC is the
standard, which probably accounts for why different libraries and
languages implement it differently.

I think our starting point is that we advertise application/json mime type, and IANA refers to RFC4627, so that's what we should really conform to, not the behaviour of any JSON parser, such as the one defined by ECMAScript.

My final thought on this is a memory of something a Javascript
developer told me about implementation differences between Javascript
parsers: "When in doubt, do what Crockford says."

That's what I thought too :-) 

Happy New Year, all!

Happy New Year!

-Alistair

[Alistair Jones]

On 1 January 2012 00:22, Peter Neubauer <neubaue...@gmail.com> wrote:

Sounds to me that a change away from scalars would satisfy everyone and be least friction.next issue is good versioning of the rest API.

Yes, I agree now.  Sorry for the noise.

-Alistair

[Jacob Hansson]

This has been discussed before, and we decided against making this change.

I'm against it, for two reasons. One, it will break all current client implementations, and two, the solution using a json object is inelegant and will potentially cause confusion.

The argument that it breaks the spec is invalid, because the spec is, like Josh says, unclear on the subject. All JSON parsers I have ever used (among others, parsers in java, php, python, javascript, c and ruby) opt for following ECMA, rather than RFC.

The argument that the current solution forces me to write code that remembers what property it is fetching is true, but the exact same argument could be made for the object-based solution (the code needs to remember what node it is fetching the property from, and what database, and what server, ad infinitum), so I don't see how it's relevant. On the contrary, not remembering what property you are fetching means you have to iterate through the returned map to find the property key, as well as adding checks to make sure there is really only one property in the map and so on.

The only real argument for making the change is that there could be JSON parsers that do not support ECMA properly. So far multijson is the *only* parser we have encountered that does not implement ECMA JSON correctly.


So the actual decision is between these two options: break all current clients and introduce a less elegant API that is harder to use, or asking developers to not use this one uncompliant JSON parser.

/jake
 

-Alistair

--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins

[Jacob Hansson]

Does anybody disagree, or can I close the issue?

jake
 

 

-Alistair

--
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins

[Peter Neubauer]

ok,
please close it. Max, ok?

Cheers,

/peter neubauer

Google:    neubauer.peter
Skype:     peter.neubauer
Phone:     +46 704 106975
LinkedIn:  http://www.linkedin.com/in/neubauer
Twitter:    @peterneubauer
Tungle:    tungle.me/peterneubauer

brew install neo4j && neo4j start
heroku addons:add neo4j

On Thu, Jan 5, 2012 at 2:05 PM, Jacob Hansson

[Max De Marzi Jr.]

That's fine.
David Pitman sent me a pull request with a workaround this issue.

https://github.com/maxdemarzi/neography/pull/22

On Jan 5, 7:17 am, Peter Neubauer <peter.neuba...@neotechnology.com>
wrote:

> ok,
> please close it. Max, ok?
>
> Cheers,
>
> /peter neubauer
>
> Google:    neubauer.peter
> Skype:     peter.neubauer
> Phone:     +46 704 106975
> LinkedIn: http://www.linkedin.com/in/neubauer
> Twitter:    @peterneubauer
> Tungle:    tungle.me/peterneubauer
>
> brew install neo4j && neo4j start
> heroku addons:add neo4j
>
> On Thu, Jan 5, 2012 at 2:05 PM, Jacob Hansson
>
>
>
>
>
>
>
> <jacob.hans...@neotechnology.com> wrote:
> > On Mon, Jan 2, 2012 at 12:48 PM, Jacob Hansson
> > <jacob.hans...@neotechnology.com> wrote:
>
> >> On Sun, Jan 1, 2012 at 3:13 PM, Alistair Jones

> >> <alistair.jo...@neotechnology.com> wrote:

[James Thornton]

On Thursday, January 5, 2012 7:05:35 AM UTC-6, Jacob Hansson wrote:

Does anybody disagree, or can I close the issue?

Another issue is that clients have to maintain a slew of conditionals in the low-level serialization code to handle the different cases -- this is a hotspot for bugs. esp in a growing API. 

Ideally return data would always be in "data" and error messages would always be in a standard place. 

As it is right now, return data is usually in "data" but sometimes it's not. Sometimes just "null" is returned. Sometimes no data is returned (e.g. on updates). And error messages are not returned as JSON.

Consistency is key to avoiding exploding conditionals. 

- James