About X-JSON header and evil things... ;)

34 views
Skip to first unread message

Kjell Bublitz

unread,
Jan 19, 2007, 5:32:42 PM1/19/07
to rubyonrail...@googlegroups.com
While looking thru the brand new documentation i found myself
stumbling upon the X-JSON header "again" as being suggested for
automatic json evaluation within prototype.

I must say that this approach is bad. We should evaluate on content
type. It is simple to add and i wonder why no one else did it already.

The content-type "text/x-json" is the closest acceptable description
according to the RFCs and should be used by prototype to determine if
evaluation is in order. Just like text/javascript, which is
implemented already.

The whole reason behind this is, that a header line is limited to a
specific size. So if youre JSON data is to huge, the server will
return nothing (or refuse delivery.. i dont know exactly what happens
in the background - browser or server). I expirienced this in a
PHP/Apache enviroment.

We should stick to content types instead of "inventing" new headers
which are limited in size anyways..

This is what i added to make it work:

evalJSONResponse: function() {
try {
return eval('(' + this.transport.responseText + ')');
} catch (e) {}
},

respondToReadyState ...
[...]
if (this.header('Content-type') == 'text/x-json')
json = this.evalJSONResponse();


Opinions?

--
Kjell
-- www.m3nt0r.de

Ryan Gahl

unread,
Jan 19, 2007, 5:40:23 PM1/19/07
to rubyonrail...@googlegroups.com
This makes a lot of sense to me.

Peter Michaux

unread,
Jan 19, 2007, 6:40:20 PM1/19/07
to rubyonrail...@googlegroups.com
On 1/19/07, Kjell Bublitz <m3nt...@gmail.com> wrote:
>
> While looking thru the brand new documentation i found myself
> stumbling upon the X-JSON header "again" as being suggested for
> automatic json evaluation within prototype.
>
> I must say that this approach is bad. We should evaluate on content
> type. It is simple to add and i wonder why no one else did it already.
>
> The content-type "text/x-json" is the closest acceptable description
> according to the RFCs and should be used by prototype to determine if
> evaluation is in order. Just like text/javascript, which is
> implemented already.

Why "text/x-json" and not "application/json" which is the approved
JSON mime-type?

Peter
--
JavaScript for Rails: http://forkjavascript.org

Kjell Bublitz

unread,
Jan 19, 2007, 6:44:28 PM1/19/07
to rubyonrail...@googlegroups.com
Hi Peter

None of theme are approved, but according to the RFC, the text-tree is
closest one when it comes down to properly describing the content. You
put everything into "application" that is too specific or "does not
fit into any of the other categories"

Kjell


--
Kjell
-- www.m3nt0r.de

Kjell Bublitz

unread,
Jan 19, 2007, 6:47:27 PM1/19/07
to rubyonrail...@googlegroups.com
Just for reference towards discussion, here a quote from the RFC:

".4. The Application Content-Type

The "application" Content-Type is to be used for data which do not fit
in any of the other categories, and particularly for data to be
processed by mail-based uses of application programs. This is
information which must be processed by an application before it is
viewable or usable to a user. "

http://www.faqs.org/rfcs/rfc1521.html

Thats why i chose "text/x-json" over "application/x-json". It is
readable text, after all

Kjell


--
Kjell
-- www.m3nt0r.de

Peter Michaux

unread,
Jan 19, 2007, 7:48:42 PM1/19/07
to rubyonrail...@googlegroups.com
On 1/19/07, Kjell Bublitz <m3nt...@gmail.com> wrote:
>
> Hi Peter

[Re official JSON mime-types]

> None of theme are approved,

From Douglas Crockford himself

<URL: http://tech.groups.yahoo.com/group/json/message/337>

Colin Mollenhour

unread,
Jan 19, 2007, 7:56:08 PM1/19/07
to rubyonrail...@googlegroups.com
This is a *great* idea in my opinion. I myself have experienced the size limitations of the X-JSON header before.  However, I don't see why there can't be support for both so that backwards compatibility isn't broken. Programmers would just have to know that you can't use them both simultaneously. That said, I wouldn't mind seeing X-JSON replaced with Content-type but it would mean hours of going back and changing code.

Colin

Kjell Bublitz

unread,
Jan 19, 2007, 7:58:31 PM1/19/07
to rubyonrail...@googlegroups.com
Oooh .. application/json then... also i can't aggree on this decision,
as stated in my previous post. It is not binary, it is text.

But i dont work for IANA, so be it..

(back to topic)

Since this is a official mime, there is more need to leave the
"X-JSON" path and move towards application/json.. content-type should
be enough to negotiate eval.


On 1/20/07, Peter Michaux <peterm...@gmail.com> wrote:
>


--
Kjell
-- www.m3nt0r.de

David Zülke

unread,
Jan 20, 2007, 9:43:41 AM1/20/07
to rubyonrail...@googlegroups.com
You're missing the point of the X-JSON header (which doesn't have
anything to do with the content type). The idea is that the response
body returns data like an HTML fragment, and the X-JSON header
carries status information like "operation successful" or "operation
failed", or other simple code that contains rudimentary logic to go
with the actual content delivered in the response body.
If you want to pass back JSON only to the client, do so in the
response body, and eval by hand.

David

P.S.: http://www.ietf.org/rfc/rfc4627.txt?number=4627

Kjell Bublitz

unread,
Jan 20, 2007, 10:22:57 AM1/20/07
to rubyonrail...@googlegroups.com
I am about automatic evaluation support based on the mime-type. I know
that i can do everything i want by hand...of course.. but that ain't
automatic.

Secondly, there is no point for X-JSON to be a status message carrier
or "just for simple stuff". Drop it entirely. It's is unnecessary.

Most people will have an own error format / object notation which they
transport in the responseText as a replacement for the expected
content and act on it. (eg include a "status" key)

This two-way devlivery is just complicating things.

And lastly, the thing i am pointing at is, that prototype does not
automaticly evaluates responseText if X-JSON header was found, forcing
me to evaluate by hand..

A server should always deliver, regardless if error or data. With that
in mind, i don't need a X-JSON header to tell me that everything is
alright and that i could now go ahead and eval responseText.

Thats why i suggest automatic evaluation of responseText if the
content-type is application/json. Simple.. just like text/xml, just
like text/javascript ..

Regards


--
Kjell

Mislav

unread,
Jan 22, 2007, 7:42:24 AM1/22/07
to Ruby on Rails: Spinoffs
Will be done:
http://dev.rubyonrails.org/ticket/7295

In future, start discussions about core changes in the Core group.
Thanks!

--
Mislav

Dave Crane

unread,
Jan 22, 2007, 7:46:22 AM1/22/07
to rubyonrail...@googlegroups.com
On Saturday 20 January 2007 15:22, Kjell Bublitz wrote:
> Secondly, there is no point for X-JSON to be a status message carrier
> or "just for simple stuff". Drop it entirely. It's is unnecessary.
>
> Most people will have an own error format / object notation which they
> transport in the responseText as a replacement for the expected
> content and act on it. (eg include a "status" key)
>
My reading of the X-JSON header was that it's useful in cases where you want
to pass some secondary data back with a response. The practice of
'piggy-backing' data in this way isn't to everybody's taste, but it can be a
useful way of avoiding extra traffic.

Rails & Prototype both have a strong bent towards delivering fragments of HTML
rather than data in the response. In this case, If you're sending back
content, I don't see how to easily embed secondary data inside the main
response body, other than stuffing it into a hidden DIV, which loses several
hundred points for elegance. Sure, if you're delivering data anyway, X-JSON
isn't particularly useful.

On Monday 22 January 2007 12:42, Mislav wrote:
> Will be done:
> http://dev.rubyonrails.org/ticket/7295

Just saw this come in on the list - sounds like a good suggestion to me.

Just my $0.02

Dave
--
----------------------
Author
Ajax in Action http://manning.com/crane
Ajax in Practice http://manning.com/crane2
Prototype & Scriptaculous Quickly http://manning.com/crane3

Alexander Presber

unread,
Jan 22, 2007, 10:38:31 AM1/22/07
to rubyonrail...@googlegroups.com
Am 22.01.2007 um 13:46 schrieb Dave Crane:

>
> On Saturday 20 January 2007 15:22, Kjell Bublitz wrote:
>> Secondly, there is no point for X-JSON to be a status message carrier
>> or "just for simple stuff". Drop it entirely. It's is unnecessary.
>>
>> Most people will have an own error format / object notation which
>> they
>> transport in the responseText as a replacement for the expected
>> content and act on it. (eg include a "status" key)
>>
> My reading of the X-JSON header was that it's useful in cases where
> you want
> to pass some secondary data back with a response. The practice of
> 'piggy-backing' data in this way isn't to everybody's taste, but it
> can be a
> useful way of avoiding extra traffic.
>
> Rails & Prototype both have a strong bent towards delivering
> fragments of HTML
> rather than data in the response. In this case, If you're sending back
> content, I don't see how to easily embed secondary data inside the
> main
> response body, other than stuffing it into a hidden DIV, which
> loses several
> hundred points for elegance. Sure, if you're delivering data
> anyway, X-JSON
> isn't particularly useful.

Just want to support that opinion. We are using the X-JSON header
exactly for that.

Alex

Ed C.

unread,
Jan 22, 2007, 10:56:26 AM1/22/07
to rubyonrail...@googlegroups.com
This seems like almost a duplication of effort -- we already have access to the HTTP status codes, which could (should?) be used to determine success (example, 200 = OK, 500 = Error). The content body includes the results or error message(s).

Mislav Marohnić

unread,
Jan 22, 2007, 11:34:19 AM1/22/07
to rubyonrail...@googlegroups.com
Let us get something clear - the X-JSON header is a hack. Nothing serious, just a small hack which simplifies returning two types of data (HTML and application/json) at the same time. Of course we're using HTTP headers in a way we shouldn't, but has anyone seen how multipart-encoded message look like? Our header hack is way nicer and technically simpler than that kind of encoding.

The X-something headers are reserved for custom (proprietary) usage. We have the freedom to use them in any way we want to, and we have taken that liberty to do some highly conventional JSON magic.

For big loads of JSON data you'll want to use response body. The ticket I submitted (and for which I'll provide a patch for) will enable that.

Ed C.: yeah, HTTP status messages could be extracted and shown in this case, but X-JSON chunks are used for much more than that. I use them regularly to return small hashes representing certain records or other objects.

--
Mislav

Colin Mollenhour

unread,
Jan 22, 2007, 11:46:05 AM1/22/07
to rubyonrail...@googlegroups.com
This change, while I think is a good idea in general, will be somewhat difficult to make use of if the new JSON object replaces the old JSON object.
I, and apparently others, use X-JSON often for error checking, debug messages, status flags, etc..  When your application requires the occasional use of raw HTML and you don't want to encode it all in a JSON message, it is very useful. However, I still use JSON encoded data in the response body because of the size limitations of the X-JSON header.
Having X-JSON *replaced* by the evaluated responseText, will make it difficult for when I want to still use the X-JSON header for error/debug/status info, but send the main response body as either plain X-HTML or JSON due to the size limit.

To make it more clear, I would have to account for the following possible scenarios on the server side in order to make use of the new feature:
1) If response is HTML then error/debug is X-JSON
2) If response is JSON then error/debug is in response JSON

It would be nice if the body JSON didn't replace the X-JSON because then I could always use X-JSON for error/debug regardless of what is in the responseText.
This just seems like better separation of data and error/status handling to me.

My suggestion is perhaps rather than replacing X-JSON it should be placed inside the response object? e.g.  "xhr.json" similar to "xhr.responseText"

Thoughts?

Colin

----code examples-----
Hypothetical JS using my suggestion:
onComplete: function(xhr,json){
    console.debug(xhr.responseText); //the responseText
    console.debug(xhr.json); //the evaluated responseText (if "Content-type: application/x-json")
    console.debug(json); //the X-JSON evaluated header
    if(json && json.error){ alert(json.error); return; }
    if(json && json.login){ showLogin(); return; }
    //do something with xhr.json
}

Hypothetically on the server-side (PHP):
ob_start();
$status = Array();
$json = Array();
if(!$loggedIn){
    $status['login'] = 'true';
    header('X-JSON: ('.json_encode($status).')');
    return;
}

if( /*handling request using JSON */ ){
    if(/*error*/){ $status['error'] = 'Error message!'; }
    else{ $json['data'] = Array(/* data here */); }
}else if( /*handling request with HTML */){
    if(/*error*/){ $status['error'] = 'Error message!'; }
    else{ print '<b>HTML! Yay!</b>'; }
}
$content = ob_get_clean();

if($status){
    header('X-JSON: ('.json_encode($status).')');
}
if($json){
    header('Content-type: application/x-json');
    print '('.json_encode($json).')';
}else{
    print $content;

Mislav Marohnić

unread,
Jan 22, 2007, 12:16:55 PM1/22/07
to rubyonrail...@googlegroups.com
On 1/22/07, Colin Mollenhour <elite...@mollenhour.com> wrote:
This change, while I think is a good idea in general, will be somewhat difficult to make use of if the new JSON object replaces the old JSON object.

You have a point there. I didn't think people would care, but you have convinced me they could.

Your idea of writing to the "json" property of the xhr object is not bad. Alternatively, there could be a "getJson()" method that lazily evaluates "responseText".

Dave Crane

unread,
Jan 22, 2007, 9:33:18 AM1/22/07
to rubyonrail...@googlegroups.com
On Monday 22 January 2007 15:56, Ed C. wrote:
> This seems like almost a duplication of effort -- we already have access to
> the HTTP status codes, which could (should?) be used to determine success
> (example, 200 = OK, 500 = Error). The content body includes the results or
> error message(s).
>
HTTP 500 is generally used to signal an application error somewhere in the
system (the database is down, you tried to divide by zero, and so on).
Similarly, other status codes relate to the status of the attempted HTTP
request and response (e.g. 401 = your request requires authentication).

This is a different thing from an application error or status code e.g. 'the
object you are trying to update is a stale copy'. X-JSON can also be used for
status/update data such as 'you have three outstanding cases, here are their
names and ids', to automatically refresh a list in the background, every time
you request or update anything in your app (in which case the request body is
busy reporting something else).

Mislav - yes, X-JSON is a hack, and an acceptable one IMO, with a good
pedigree. As you point out, multipart MIME type responses are a hack on top
of HTTP - so are CGI parameters, and sessions, if you look at what HTTP was
originally designed to do.

Peter Michaux

unread,
Jan 22, 2007, 12:56:30 PM1/22/07
to rubyonrail...@googlegroups.com
On 1/19/07, Kjell Bublitz <m3nt...@gmail.com> wrote:
>
> Thats why i chose "text/x-json" over "application/x-json". It is
> readable text, after all

I puzzled over the choice of "application" first but can think of a
couple reasons "application" could be more natural.

JSON is expressed in JavaScript. JavaScript is distributed as text not
is some unreadable compiled binary. If it could be distributed in a
small compiled form it surely would and so "application" would be more
clearly appropriate.

JSON is expressed in JavaScript. JavaScript is a programming language
that describes behavior aspect of a web page. Behavior is in the
application realm.

Kjell Bublitz

unread,
Jan 22, 2007, 1:27:28 PM1/22/07
to rubyonrail...@googlegroups.com
If i query a XML service, i get a XML response (text/xml) - with error or data
If i query a JSON service, i get a JSON response (application/json) -
with error or data

Is there something to disagree?

Only because there is a hack which allows transport of HTML (or
whatever) along with a pure dataformat, i don't see why that should be
a keeper. I personally create such fragments from JSON data, or modify
the application behaviour. There is no text/html involved in that
action and i belief that this is the whole purpose of JSON. If my
application uses pre-generated HTML aside from JSON -in a row-, i
should ask myself why i use JSON at all. Just for a 'yay or nay'?? I
don't think so..

But.. it's a matter of application design after all. Generally: mixing
mime is bad.

- Kjell


--
Kjell
-- www.m3nt0r.de

William F. Attwood

unread,
Jan 22, 2007, 1:38:01 PM1/22/07
to rubyonrail...@googlegroups.com, rubyonrail...@googlegroups.com
Good Day -

While I read through these comments, I decided for the first time in awhile to throw in my own two cents.

JSON stands for: JavaScript Object Notation. If we take the first word in that title: JavaScript, we get our answer. JSON is, through it's basis, JavaScript -- so set the MIME type as so; application.

Just because you can read text and understand what it means and what it does is not a valid method to describe the data as simple text. I can read binary code (albeit slowly), but that does not mean it is text.

--Will

Mislav Marohnić

unread,
Jan 22, 2007, 2:06:57 PM1/22/07
to rubyonrail...@googlegroups.com
On 1/22/07, Kjell Bublitz <m3nt...@gmail.com> wrote:

But.. it's a matter of application design after all. Generally: mixing
mime is bad.

Ever uploaded a file through a browser, or received a HTML/txt e-mail? I'm sorry to break it to you like this, but you mixed mime when you did :)

William F.: we got over the MIME-type discussion. "application/json" is here to stay, "text/x-json" is invalid.

Kjell, if I understand correctly, you're suggesting to drop X-JSON header feature in favor of JSON in body?

If we did that, it would make a lot of developers sad :(

My reasons to keep it:
  1. it's a simple hack;
  2. it's easy to do;
  3. it's not bloat to the framework;
  4. HTML, text or even XML with a side-dish of JSON is yummy.
--
Mislav

Ryan Gahl

unread,
Jan 22, 2007, 2:15:25 PM1/22/07
to rubyonrail...@googlegroups.com
Are they necessarily mutually exclusive? Can X-JSON not stay for those who like it, and support for a _configurable_OR_standard_ content type be added? This seems like something that could be configurable at the application, page, or request level under various conditions or developer preference.
--
Ryan Gahl
Application Development Consultant
Athena Group, Inc.
Inquire: 1-920-955-1457
Blog: http://www.someElement.com

Tom Gregory

unread,
Jan 23, 2007, 4:05:12 PM1/23/07
to rubyonrail...@googlegroups.com
Pardon my ignorance, but huh? I thought this *was* the group for
Prototype/Script.aculo.us requests. Is there someplace else I should
be looking?

Or did I misunderstand--do you see this as a RoR change request, and
were suggesting it go there?


TAG

Kjell Bublitz

unread,
Jan 23, 2007, 4:15:16 PM1/23/07
to rubyonrail...@googlegroups.com
Probably misunderstood. This discussion is about a change/enhancement
in the prototype.js core.

You are in the group for Prototype/Script.aculo.us and RoR.
So.. this is the right place for your requests.


--
Kjell
-- www.m3nt0r.de

Tom Gregory

unread,
Jan 23, 2007, 4:18:42 PM1/23/07
to rubyonrail...@googlegroups.com
My bad. I found it list here: http://prototypejs.org/discuss

Full group address: http://groups-beta.google.com/group/prototype-core

I missed the announcement as I've not heavily perused the new
prototype site. Most of my interaction with Prototype/S.a.us comes
from this list.


TAG

Kjell Bublitz

unread,
Jan 23, 2007, 4:19:40 PM1/23/07
to rubyonrail...@googlegroups.com
Good call. I am not all against the support of X-JSON. I just would
like to see my suggestion as being added one way or another cause i
thought this was missing.

I personally like the object extension which was suggested earlier.
Maybe something like responseJSON ...


--
Kjell
-- www.m3nt0r.de

Kjell Bublitz

unread,
Jan 23, 2007, 4:26:30 PM1/23/07
to rubyonrail...@googlegroups.com
That group you linked is for prototype core discussions, yes.. but not
for RoR and S.a.us

The core group is new, so this discussion was initially created here.
As being requested by Mislav, future discussions of such core specific
topics will be taken there.

If you have question regarding RoR/S.a.us go ahead and create a topic
or search online if your question has already been answered in this
group.


PS: Please don't push this topic anymore.
If you have further questions create a new topic.


On 1/23/07, Tom Gregory <to...@byu.net> wrote:
>


--
Kjell
-- www.m3nt0r.de

Martin Bialasinski

unread,
Jan 23, 2007, 4:32:23 PM1/23/07
to rubyonrail...@googlegroups.com
On 1/23/07, Tom Gregory <to...@byu.net> wrote:
> My bad. I found it list here: http://prototypejs.org/discuss
>
> Full group address: http://groups-beta.google.com/group/prototype-core
>
> Most of my interaction with Prototype/S.a.us comes from this list.

I don't quite see the reason for the second list. Is there any problem
using this list for discussing changes and future developement?

Martin Bialasinski

unread,
Jan 23, 2007, 4:42:53 PM1/23/07
to rubyonrail...@googlegroups.com
On 1/23/07, Kjell Bublitz <m3nt...@gmail.com> wrote:
> Good call. I am not all against the support of X-JSON. I just would
> like to see my suggestion as being added one way or another cause i
> thought this was missing.

There is another thing that might speak towards application/json.
There is this fracking Norton Personal Firewall application that acts
as a web proxy and that injects its own javascript code into pages
fetched.

When I return JSON data in the return body, I set the content type to
application/json, as sending the data as text/html might seduce
Norton to mess it up. Although I can't test if it is really necessary
(I have no spare machine to infect with Norton). Did someone test a
similar setup?

Bye,
Martin

Christophe Porteneuve

unread,
Jan 23, 2007, 4:43:53 PM1/23/07
to rubyonrail...@googlegroups.com
Martin Bialasinski a écrit :

> I don't quite see the reason for the second list. Is there any problem
> using this list for discussing changes and future developement?

- Spinoffs is for assistance with Prototype and script.aculo.us both.
It's mostly a helpdesk.

- Prototype-Core is the list for discussion on the evolution of
Prototype, at all levels: global architecture, individual features, etc.

--
Christophe Porteneuve a.k.a. TDD
"[They] did not know it was impossible, so they did it." --Mark Twain
Email: t...@tddsworld.com

Martin Bialasinski

unread,
Jan 23, 2007, 5:03:26 PM1/23/07
to rubyonrail...@googlegroups.com
On 1/23/07, Christophe Porteneuve <t...@tddsworld.com> wrote:

> - Spinoffs is for assistance with Prototype and script.aculo.us both.
> It's mostly a helpdesk.
>
> - Prototype-Core is the list for discussion on the evolution of
> Prototype, at all levels: global architecture, individual features, etc.

These are the topics, but I was wondering why it was felt necessary to
divide the group.

Changes are mostly because of users' needs and uses. When the needs
and uses manifest themselves on the first list, and the discussion
about how and what to change happens on the second list, then I fear
the code created might not address the problems in the best way
because of missing input from divers users. Kind of an artificial
barrier between the two groups.

Time will tell, I wish you best luck.

I just don't feel the necessity to divide the two topics. They
pollinate each other, the message count is not overwhelming and the
different topics are easily manageable in any decent reader.

Mislav Marohnić

unread,
Jan 23, 2007, 5:27:34 PM1/23/07
to rubyonrail...@googlegroups.com
On 1/23/07, Martin Bialasinski <kling...@gmail.com> wrote:

These are the topics, but I was wondering why it was felt necessary to
divide the group.

Signal vs. noise ratio. If you didn't notice, the Prototype core team (except for Thomas) didn't participate much on this list. (TDD did, but he was not core back then.) I tried to catch up every now and then, but there was too much noise so I'm not subscribed to it anymore.

This is the place for practical questions where the community helps the community, and where we drop by every now and then to check what's new, what are the common topics, etc.

We needed a place where there would be less noise so we could concentrate on ideas for improvement, new features and similar things. Hence prototype-core group.

Rails did this also, and it turned out to be a great idea.

If anyone else feels he would add to the X-JSON discussion, go right ahead - no need to move the discussion to core group now. But, everything has been pretty much said by now, and we're not dropping the header feature.

-M

mryhanen

unread,
Jan 24, 2007, 11:17:53 AM1/24/07
to Ruby on Rails: Spinoffs
Glad I found this discussion and that auto evaluation of the response
body depending on Media Type is in the works since it's a good thing
and the X-JSON header does generate some confusion.

One suggestion though: Shouldn't the json text be verified before
evaluted, as described in the application/json RFC
(http://www.ietf.org/rfc/rfc4627.txt):

var my_JSON_object = !(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
text.replace(/"(\\.|[^"\\])*"/g, ''))) && eval('(' + text
+ ')');

Matti Ryhänen

Thomas Fuchs

unread,
Jan 24, 2007, 12:44:26 PM1/24/07
to rubyonrail...@googlegroups.com
As you can only generate yourself (as Ajax is not cross-domain),
that's probably not necessary, but it wouldn't hurt either.
One thing is that stuff like this can introduce very strange bugs
that can take very long to debug ("whoops, where's my JSON gone?").

Think about it, how about a Object.fromJSON(json) function for this?

Best,
Thomas

Thomas Fuchs

unread,
Jan 24, 2007, 1:36:46 PM1/24/07
to rubyonrail...@googlegroups.com
I meant, "Thinking about it, ...".

tobie

unread,
Jan 24, 2007, 9:59:13 PM1/24/07
to Ruby on Rails: Spinoffs
I've taken a go a it. see preliminary details here
http://dev.rubyonrails.org/ticket/7295 (last comments).

I'll post a patch tomorrow.

Best,

Tobie

tobie

unread,
Jan 24, 2007, 11:15:32 PM1/24/07
to Ruby on Rails: Spinoffs
Patch: http://dev.rubyonrails.org/ticket/7295

and live tests:
http://sandbox.tobielangel.com/prototype/rev_6028/trunk/test/unit/string.html
http://sandbox.tobielangel.com/prototype/rev_6028/trunk/test/unit/ajax.html

I'm having issues in IE (don't know if its the actual code or the
testing that is causing the problem).

The JSON sanitazing regex is taken from http://www.json.org/json.js
with permission from Douglas Crockford.

Would like to get as much feedback as possible so we can make it really
bulletproof.

Thanks,

Tobie

Matti Ryhanen

unread,
Jan 25, 2007, 7:23:41 AM1/25/07
to rubyonrail...@googlegroups.com
I think that the IE-issues are there regardless of your patch. At least the tests are failing for me without it too.

On IE7 there will be issues with the tests when using the XHR on a local file: URL, like the relative ones in the ajax.html tests, because the new native IE7 XHR-object will not allow request to local url:s.

The tests will run if the ActiveX XHR-object is loaded for all IE versions, but I don't think this is a good solution since there seem to be yet other issues with the ActiveX-object and IE7, and the Native IE7-object should be used.


The test for Ajax.Responders will fail even using the ActiveX object, and from what I can tell, the two failed test there happen because IE and firefox seem to fire readystatechanges for readyState 4 at different times. I'm not sure this is much of an issue though, and maybe the tests should be changed to reflect this?

Matti Ryhänen

Mislav Marohnić

unread,
Jan 25, 2007, 8:43:00 AM1/25/07
to rubyonrail...@googlegroups.com
Ajax.Responders test failing is a know issue due to behavior of the ActiveX XHR object.

All these (testing) issues will be addressed when Ajax is finally tested against a proper Webrick backend. That means no more testing Ajax without Ruby or over local "file:" scheme, but hey -- you couldn't even "compile" the framework without Ruby, either.

-M

Colin Mollenhour

unread,
Jan 25, 2007, 11:02:46 AM1/25/07
to rubyonrail...@googlegroups.com
Is the sanitize step necessary? What would the performance hit be like
on a large response, and is the added complexity worth the trouble
considering all responses come from a controlled environment? You don't
sanitize HTML or XML responses, I say just use eval inside a try/catch.

Thanks,
Colin

Michael Peters

unread,
Jan 25, 2007, 11:03:38 AM1/25/07
to rubyonrail...@googlegroups.com

Colin Mollenhour wrote:
> Is the sanitize step necessary? What would the performance hit be like
> on a large response, and is the added complexity worth the trouble
> considering all responses come from a controlled environment? You don't
> sanitize HTML or XML responses, I say just use eval inside a try/catch.

I agree. And in case the data can't be trusted then provide an optional
parameter signifying that it needs to be cleaned first.

--
Michael Peters
Developer
Plus Three, LP

tobie

unread,
Jan 25, 2007, 12:18:02 PM1/25/07
to Ruby on Rails: Spinoffs
Colin, Michael, sanitizeJSON is an option and is off by default.

If is data created by a user, you better sanitize it.

The performance hit isn't that bad really.

Tobie

On Jan 25, 11:02 am, Colin Mollenhour <eliteii...@mollenhour.com>
wrote:

tobie

unread,
Jan 25, 2007, 1:25:18 PM1/25/07
to Ruby on Rails: Spinoffs
Sorry for the sloppy typing. I obviously meant "If IT is data created

Martin Bialasinski

unread,
Jan 25, 2007, 2:56:45 PM1/25/07
to rubyonrail...@googlegroups.com
On 1/25/07, tobie <tobie....@gmail.com> wrote:
>
> Colin, Michael, sanitizeJSON is an option and is off by default.

Regarding your patch: Shouldn't the X-JSON processing also use evalJSON() ?

And WRT to options.sanitizeJSON, how about a AJAX.sanitizeJSON global
option for the default behaviour? Then a developer he want to sanitize
every response can set this to true once and he does not have to
specifiy this option on every call.

string.evalJSON() returns
Object if everything is OK
null if eval failed
undefined if sanitize was requested and failed

Shouldn't the last case also return null?

On the Ajax object, a seed for a discussion on how it should behave:

transport.responseJSON is always initialized with "undefined". Users
can compare with undefined to see if there was JSON data in the
response at all.

If content-type is application/json:
- a possible X-JSON header is disregarded.
- the responseText is eval'd via string.evalJSON()
transport.responseJSON is set to the returned value, thous it is
* an Object, if evalJSON() succeeds
* null, if evalJSON() fails because the eval() failed
* null, if sanitize was requested and the data did not pass

Question: should failing the sanitize process in evalJSON()
trigger onException? I believe yes.

If content-type is not application/json:
(only the X-JSON part)
- the X-JSON header is eval'd via string.evalJSON()
transport.responseJSON is set to the returned value, thous it is
* an Object, if evalJSON() succeeds
* null, if evalJSON() fails because the eval() failed
* null, if sanitize was requested and the data did not pass

Result:
transport.responseJSON always contains the JSON object returned (if
there was any), regardless how it was transported.

Question:
The callbacks get a parameter list of (transport, json). The json
parameter is not needed any more, if transport.responseJSON is used
consistently, but we cannot let it go for compatibility reasons.

If If content-type is not application/json, json gets its value from
the X-JSON header.
If If content-type is application/json, should it gets its value from
from the responseText?

I say yes. It should not matter what way (header / body) was used to
transfer the data.

> If is data created by a user, you better sanitize it.

Sure, but the server script creating the JSON response should create a
valid JSON representation of the user supplied data. If it cannot
assure this, then this is the component to fix first.

Bye,
Martin

Martin Bialasinski

unread,
Jan 25, 2007, 3:10:09 PM1/25/07
to rubyonrail...@googlegroups.com
On 1/25/07, Martin Bialasinski <kling...@gmail.com> wrote:

> Regarding your patch: Shouldn't the X-JSON processing also use evalJSON() ?

Oops. It does, of cause.

tobie

unread,
Jan 25, 2007, 3:45:08 PM1/25/07
to Ruby on Rails: Spinoffs
Hi, thanks for the feedback.

I'm working on a better handling of exceptions. Will post a patch asap.

Regarding x-json, I think we should leave it as it is for consitency
and backwards compatibility. It should also still be available if the
mime-type is set to application/json.

I had thaught about a global AJAX.sanitizeJSON option. It certainly
something we could implement at some point, but there are some
potential issues with that (especially if you rely on some third party
libs also using Prototype). Your best option for such cases is to
create a var holding your hash of options and reference it each time.

Thanks for the extended comment.

Cheers,

Tobie

Kjell Bublitz

unread,
Jan 25, 2007, 3:52:20 PM1/25/07
to rubyonrail...@googlegroups.com
I think sanitizeJSON should be optional.. {sanitize: true}

my 2 cents.

On 1/25/07, tobie <tobie....@gmail.com> wrote:
>


--
Kjell
-- www.m3nt0r.de

tobie

unread,
Jan 25, 2007, 4:12:47 PM1/25/07
to Ruby on Rails: Spinoffs
It is, see above

Colin Mollenhour

unread,
Jan 25, 2007, 4:49:28 PM1/25/07
to rubyonrail...@googlegroups.com
Martin, I disagree with your handling of parameters. As previously
stated, I believe the X-JSON header handling should be left intact,
as-is, both for backwards compatibility, and because it is still useful
as a separate feature. The Content-type header detected as
"application/json" should be an entirely new feature and be implemented
via a property set in the transport. (e.g. transport.responseJSON) See
my previous post for further reasoning and explanation. Several people
have agreed with this and I see no reason not to handle it this way.

Cheers,
Colin

Kjell Bublitz

unread,
Jan 25, 2007, 5:08:56 PM1/25/07
to rubyonrail...@googlegroups.com
responseText
responseXML
responseJSON

so self-evident ^^

my vote for responseJSON :)


--
Kjell
-- www.m3nt0r.de

Martin Bialasinski

unread,
Jan 25, 2007, 5:13:11 PM1/25/07
to rubyonrail...@googlegroups.com
On 1/25/07, tobie <tobie....@gmail.com> wrote:

> I had thaught about a global AJAX.sanitizeJSON option. It certainly
> something we could implement at some point, but there are some
> potential issues with that (especially if you rely on some third party
> libs also using Prototype).

That would be only a problem, if the third party module depends on
evalJSON() to not only return an object, but to also execute some
commands contained in the data, which makes the data not a JSON
structure in the first place and it is then an error to call evalJSON.

Bye,
Martin

tobie

unread,
Jan 25, 2007, 11:15:29 PM1/25/07
to Ruby on Rails: Spinoffs
> > I had thaught about a global AJAX.sanitizeJSON option. It certainly
> > something we could implement at some point, but there are some
> > potential issues with that (especially if you rely on some third party
> > libs also using Prototype).That would be only a problem, if the third party module depends on

> evalJSON() to not only return an object, but to also execute some
> commands contained in the data, which makes the data not a JSON
> structure in the first place and it is then an error to call evalJSON.

Hi Martin,

You're right. However, the only consistent way to do this in my
opinion, is to have ALL defaults options available as a property of the
Ajax object (Something similar to what script.aculo.us implemented for
Effect), like so:

Ajax.DEFAULT_OPTIONS = {
asynchronous: true,
sanitizeJSON: false,
etc...
}

As this might have further implications, I believe it should dealt as a
separate issue.

But I agree with you, it would be a great feature.

Best,

Tobie

tobie

unread,
Jan 26, 2007, 5:27:14 AM1/26/07
to Ruby on Rails: Spinoffs

Thomas Fuchs

unread,
Jan 26, 2007, 6:04:35 AM1/26/07
to rubyonrail...@googlegroups.com
Tobie:

See http://dev.rubyonrails.org/changeset/5986 for an example of using
the built-in HTTP server (WEBrick) test environment for
testing stuff like this.

You can add:

@server.mount_proc("/JSON") do |req, res| ...

Then just spit back stuff, depending on params.

Best,
Thomas

--
Thomas Fuchs
wollzelle

http://www.wollzelle.com

questentier on AIM
madrobby on irc.freenode.net

http://www.fluxiom.com :: online digital asset management
http://script.aculo.us :: Web 2.0 JavaScript
http://mir.aculo.us :: Where no web developer has gone before


tobie

unread,
Feb 12, 2007, 12:41:13 AM2/12/07
to Ruby on Rails: Spinoffs
Hi All,

Thanks for your patience. I've encountered annoying issues which I
hadn't foreseen, mainly that IE6 doesn't allow modifying or adding
properties to the response object.

I've dealt with that by cloning the transport when needed... and I'd
like as much input as possible on this proposition as I'm a bit
worried about the possible side effects.

You can find the patch here: http://dev.rubyonrails.org/ticket/7295

To summarize, this patch adds:
- automatic evaluation of JSON if the mime-type is set to application/
json.
- evaluation of JSON if the evalJSON option is set to true (regardless
of the mime-type).
- the possibility to sanitize your JSON by setting the sanitizeJSON
option to true (to guard against user-entered data for example).
- the evaluated JSON comes nicely wrapped up as the responseJSON
property of the transport object.
- the x-json header is kept as is (except it also gets sanitized if
the sanitizeJSON option is used).
- there's an added String.prototype.evalJSON method
- automatic Opera testing has been added to rake (would need a bit of
polishing on windows and Linux - if anybody volunteers).
- all the tests you need (they will only run if launched from rake -
you have been warned!)
- full support in Safari, Firefox, Opera 9, IE6 & 7

Thanks for your input and best regards,

Tobie

tobie

unread,
Feb 12, 2007, 12:46:14 AM2/12/07
to Ruby on Rails: Spinoffs
oh! and I completely forgot to mention the patch for JSON encoding
(which doesn't mess with Object.prototype): http://dev.rubyonrails.org/ticket/7427

Colin Mollenhour

unread,
Feb 12, 2007, 1:00:54 PM2/12/07
to rubyonrail...@googlegroups.com
Great work, I'm glad you were able to get it working despite IE6's
limitations. Looking forward to checking out that patch and giving it a try.

Thanks,
Colin

Colin Mollenhour

unread,
Feb 20, 2007, 3:32:16 AM2/20/07
to rubyonrail...@googlegroups.com
I just thought of a simple solution that would not require messing with the transport object. I posted a comment on Tobie's ticket (http://dev.rubyonrails.org/ticket/7295#comment:15) to the same effect, but I'll post it here for others to see as well.

Why not eval the X-JSON header, then, if Content-type is 'application/json', eval it as well and Object.extend the X-JSON eval'ed response! This is more similar to Tobie's original suggestion, only rather than replacing it you would extend it.

This solution has the following advantages:

  • Backwards compatibility with X-JSON response being the second argument to onXXX
  • Won't break cases where bind was used to add additional arguments to onXXX
  • Doesn't require writing to the transport object (or cloning)
  • Keeps the API simple by using one object argument to handle both cases
  • Users can migrate their code to Content-type without changing the JS code
  • No data is lost unless properties in the Content-type object override properties in the X-JSON object. This is the only danger, but I think it is acceptable.

So a response like:
---------------
X-JSON: ({status: "go"})
Content-type: application/json

({data: ['one','two','ten']})

----------------

Could be used like so:
----------------

Ajax.Responders.register({
    onSuccess: function(xhr,json){
        if(json.status && json.status != 'go'){ alert('Error!'); }
    }
});
new Ajax.Request(page,{
    onSuccess: function(xhr,json){
        $('count').update($A(json.data).map(function(item){ return '<li>'+item.camelize()+'</li>'; }).join(''));
    }
});

Thoughts? I personally prefer this over my original suggestion which was to add a new property to the transport, mainly for simplicity.

Colin

tobie

unread,
Feb 21, 2007, 12:07:09 AM2/21/07
to Ruby on Rails: Spinoffs
There you go:

http://dev.rubyonrails.org/ticket/7295

drops the ugly cloning in favor of a brand new Ajax.Response object.

Not ready for prime yet... as we still have to discuss how much we
want it to smoothen out browser discrepancies.

Comments welcomed.

NOTE: can only be tested through rake.

I've added Opera and WebKit to the lot.

This should probably not go into core like this as these browsers are
not officially supported.

Colin Mollenhour

unread,
Feb 21, 2007, 2:04:15 AM2/21/07
to rubyonrail...@googlegroups.com
Please explain the necessity of a wrapper for the httprequest object.. I
see it simply as more overhead and extraneous bulk. I definitely like
the string.prototype.evalJSON and the options.sanitizeJSON, but rather
than the Ajax.Response wrapper, consider this simple patch to the
current system:

+ var contentType = this.getHeader('Content-type');
if (state == 'Complete') {
try {
+ if(contentType &&
contentType.strip().startsWith('application/json'))
+ Object.extend(json || {},
transport.responseText.evalJSON(options.sanitizeJSON) || {});
this._complete = true;

Will these three lines not accomplish what this patch is all about,
making automatic JSON evaluation more usable? Keep it simple.

Thanks,
Colin

Colin Mollenhour

unread,
Feb 21, 2007, 2:21:36 AM2/21/07
to rubyonrail...@googlegroups.com
Well, that code has a problem or two, but you get the idea..
Reply all
Reply to author
Forward
0 new messages