application/json content-type

[Dan Larkin]

Hey guys,

I'm seeing behavior for which I don't have an explanation. When I set the content-type on a response to "application/json" the connection isn't closed, but for any other content-type it is. Further confusing the issue, when setting Content-Type (note the capitalization) to application/json, the connection does close.

Here's a little snippet:

(ns jsontest.core
(:use [aleph.http :only [start-http-server]]
[lamina.core :only [enqueue]]))

(defn app [channel request]
(enqueue channel {:status 200
:headers {"content-type" "application/json"}
:body "{}"}))

;; replace :headers with {"content-type" "text/plain"}
;; and {"Content-Type" "application/json"}

;; (start-http-server #'app {:port 6000})

Dan

[Zach Tellman]

That's weird, but you're probably running afoul of some Netty logic
which is case-sensitive for headers. There have already been some
issues with Ring where we get both "Content-Type" and "content-type"
headers in the response, so I need to just pre-capitalize any
responses before passing them through. I'll have a fix for that in
later today, and hopefully everything will go back to normal. If not,
I'll actually try to figure out why this is happening.

Zach

[Zach Tellman]

Actually, strike that, I think it may be my fault. When
"content-type" (case is significant) is equal to "application/json", I
assume it's not already encoded and try to turn the data structure
into JSON. I'm not sure why this would have any effect on whether the
connection closes, but there may be some double-encoding
content-length issues at play.

Is the JSON already encoded? I put in the auto-encoding feature
because it seemed useful, but thinking about it now I'm leaning
towards it being an option that's off by default.

Zach

[Dan Larkin]

Tricky! Yes, I'm sending back a string that is already-encoded json.

Yeah I'd lean towards less magic as well.

I look forward to the release of 0.15 :)

Dan

[Zach Tellman]

I'm not too happy with using uppercase Content-Type as a workaround in
0.1.4. I don't want to change the default behavior, but having a
:auto-transform? flag that's true by default in 0.1.4 and false by
default in 0.1.5 seems like a good way to make this explicit. This
should be minimally invasive, and will make me a happier library
maintainer.

Do you have any objections to this?

Zach

[Dan Larkin]

So you're thinking of issuing a 0.1.4.1 release? I'm not sure that a "feature" addition counts as a bug fix.

Shouldn't you just add the :auto-transform?, defaulting to false, to 0.1.5-SNAPSHOT?

Dan

[Zach Tellman]

Yeah, I'm a bit split on whether it's really a feature or not. I'll
be conservative and keep it in 0.1.5-SNAPSHOT, I guess.

Zach

[Wilson MacGyver]

auto encoding is fine as long as it's an option. most of the our JSON data
is also already encoded.

--
Omnem crede diem tibi diluxisse supremum.

[ck]

A stupid question: I am trying to follow the discussion by playing
around with the code.

The problem I run into is the browser (Safari) just spins when I use

(defn app [channel request]
(enqueue channel {:status 200
:headers {"content-type" "application/json"}

:body {"foo" "bar"}}))

Switching to "text/plain" and making the body a string works fine.
Fyi, I do use ':auto-transform true' during start-up

(defn start-server
[port]
(reset! server (start-http-server (wrap-ring-handler handler)
{:port port :auto-transform true})))

Any pointers are appreciated.

Thanks

-ck

[ztellman]

That's not a stupid question, that's a bug. I'm swamped today, but
I'll have it fixed this weekend.

Zach

[ninjudd]

It seems like :auto-encode is still not working in 0.1.5-SNAPSHOT. The following code:

(use 'aleph.http 'lamina.core)

(defn handler [ch request]

  (enqueue ch

    {:status 200

     :headers {"content-type" "application/json"}

     :body {"foo" 1 "bar" 2}}))

(start-http-server handler {:port 8080 :auto-transform true})

produces this error:

SEVERE: Error in handler, closing connection.

java.lang.Exception: Don't know how to write JSON of class org.jboss.netty.buffer.ByteBufferBackedChannelBuffer

at clojure.contrib.json$write_json_generic.invoke(json.clj:276)

at clojure.contrib.json$eval1619$fn__1620$G__1610__1627.invoke(json.clj:204)

at aleph.formats$data__GT_json__GT_channel_buffer.invoke(formats.clj:204)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at clojure.core$apply.invoke(core.clj:542)

at clojure.core$update_in.doInvoke(core.clj:4969)

at clojure.lang.RestFn.invoke(RestFn.java:445)

at aleph.http.core$encode_aleph_msg.invoke(core.clj:73)

at aleph.http.core$transform_aleph_message.invoke(core.clj:200)

at aleph.http.core$transform_aleph_response.invoke(core.clj:231)

at aleph.http.server$respond_with_channel_buffer.invoke(server.clj:137)

at aleph.http.server$respond.invoke(server.clj:160)

at aleph.http.server$non_pipelined_loop$fn__4723$fn__4727.invoke(server.clj:228)

I tried to track down the problem last night, but I don't know enough about the internals of Aleph.

Justin

[Zach Tellman]

I've pushed a fix for this. It was trying to double-encode the JSON,
and failing the second time since raw bytes aren't an acceptable
datatype. The tests were only looking at the streaming auto-transform
case; I've added in a simpler one which would have caught this issue.

Zach

[ninjudd]

Works now. Thanks for the quick fix!

I'm also wondering if you've considered using clj-json instead of clojure.contrib.json since it uses jackson and is supposed to be faster. I'm not certain about the efficiency since clj-json doesn't give you a way to generate data directly to a nio buffer. Anyway, if you're interested, here's some code in the style of aleph.formats that I used in a recent project.

(ns json.formats

  (:use [aleph.formats :only [string->channel-buffer channel-buffer->input-stream channel-buffer->string]])

  (:require [clj-json.core :as json])

  (:import (java.io InputStreamReader)))

(def string->json->data json/parse-string)

(def data->json->string json/generate-string)

(defn input-stream->json->data [stream]

  (-> stream InputStreamReader. json/parsed-seq first))

(defn data->json->channel-buffer [data]

  (-> data data->json->string string->channel-buffer))

(defn channel-buffer->json->data [buf]

  (-> buf channel-buffer->input-stream input-stream->json->data))

[Zach Tellman]

Yes, clj-json is significantly faster than the other libraries out
there (we did some benchmarks at Runa a while back), and I have been
intending to switch over. I'm not sure what the performance
implications of not going straight to bytes will be, but I imagine the
string->bytes transform isn't particularly expensive compared to the
parsing itself.

Zach