core.async | compojure ... odd error ... help!!

[mond]

My first core.async program ... all works outside of the web app but barfs once I put the functions inside a web container. I hope somebody in the group can point to my obvious mistake...

The idea is that the function 'respond-within-sla' will give back a result or a come back later message after N milliseconds. It is passed a number of ms, three functions and the arguments for the final function ... which is the one that should operate within the SLA.

(defn respond-within-sla [expected-result-milliseconds respond-ok respond-later data-fetcher & args]

  (let [data-channel (timeout expected-result-milliseconds)]

    (go (if-let [data (<!! data-channel)]

          ((async/close! data-channel)

           (respond-ok data))

          (respond-later)))

    (go

      (>! data-channel (apply data-fetcher args)))))

To keep the volume of code to parse to a minimum I have made a few toy functions that demonstrate the failure...

; test funcs

(defn ok [data]

  (prn (str "send HTTP 200 ... got data " data)))

(defn later []

  (prn (str "send HTTP 202 ... request received, come back later")))

(defn fetcher [arg1 arg2 arg3]

  (prn (str "fetching data with args " arg1 " " arg2 " " arg3))

  "response-data")

(defn failer [& args]

  (Thread/sleep 1000)

  (str "never gets here " args))

; test funcs

(defn ok [data]

  (prn (str "send HTTP 200 ... got data " data)))

(defn later []

  (prn (str "send HTTP 202 ... request received, come back later")))

(defn fetcher [arg1 arg2 arg3]

  (prn (str "fetching data with args " arg1 " " arg2 " " arg3))

  "response-data")

(defn failer [& args]

  (Thread/sleep 1000)

  (str "never gets here " args))

(defn generate-response [brand country resource]

  (let [sla (or (env :sla-milliseconds) 100)]

    (respond-within-sla sla ok later fetcher brand country resource)))

(defn generate-fail [brand country resource]

  (let [sla (or (env :sla-milliseconds) 100)]

    (respond-within-sla sla ok later failer brand country resource)))

From within the REPL it all works fine...

(generate-response "A" "B" "C")

=> #<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel@4b7ae3f7>

"fetching data with args A B C"

"send HTTP 200 ... got data response-data"

(generate-fail "A" "B" "C")

=> #<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel@4eb8b5a9>

"send HTTP 202 ... request received, come back later"

Here is the compojure route..

(defroutes app

           (GET "/:brand/:country/*" [brand country *]

                (generate-response brand country *))

           (ANY "*" []

                (route/not-found "You must use a REST style to specify brand and country keys in the URL")))

If I now start it up 'lein run' and try to exercise the functions from the web server...

$ curl -I http://localhost:5000/A/B/D.jpg

HTTP/1.1 500 Server Error

Date: Fri, 26 Sep 2014 23:02:03 GMT

Content-Length: 0

Connection: close

Server: Jetty(7.6.8.v20121106)

And on the server I see this:

$ lein run

Compiling redirector.web

2014-09-27 01:01:48.426:INFO:oejs.Server:jetty-7.6.8.v20121106

2014-09-27 01:01:48.458:INFO:oejs.AbstractConnector:Started SelectChann...@0.0.0.0:5000

2014-09-27 01:02:03.535:WARN:oejs.AbstractHttpConnection:/A/B/D.jpg

java.lang.IllegalArgumentException: No implementation of method: :render of protocol: #'compojure.response/Renderable found for class: clojure.core.async.impl.channels.ManyToManyChannel

at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:544)

at compojure.response$fn__213$G__208__220.invoke(response.clj:9)

at compojure.core$make_route$fn__332.invoke(core.clj:100)

at compojure.core$if_route$fn__320.invoke(core.clj:46)

at compojure.core$if_method$fn__313.invoke(core.clj:33)

at compojure.core$routing$fn__338.invoke(core.clj:113)

at clojure.core$some.invoke(core.clj:2515)

at compojure.core$routing.doInvoke(core.clj:113)

at clojure.lang.RestFn.applyTo(RestFn.java:139)

at clojure.core$apply.invoke(core.clj:626)

at compojure.core$routes$fn__342.invoke(core.clj:118)

at clojure.lang.Var.invoke(Var.java:379)

at ring.middleware.keyword_params$wrap_keyword_params$fn__534.invoke(keyword_params.clj:35)

at ring.middleware.nested_params$wrap_nested_params$fn__576.invoke(nested_params.clj:84)

at ring.middleware.params$wrap_params$fn__507.invoke(params.clj:64)

at ring.middleware.multipart_params$wrap_multipart_params$fn__612.invoke(multipart_params.clj:118)

at ring.middleware.flash$wrap_flash$fn__1286.invoke(flash.clj:35)

at ring.middleware.session$wrap_session$fn__1273.invoke(session.clj:98)

at ring.adapter.jetty$proxy_handler$fn__1426.invoke(jetty.clj:18)

at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

at org.eclipse.jetty.server.Server.handle(Server.java:363)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)

"fetching data with args A B D.jpg"

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

at java.lang.Thread.run(Thread.java:744)

"send HTTP 200 ... got data response-data"

FYI I have tested the functions prior to adding the core.async code and they all work fine from within Jetty and on Heroku.

It seems like somehow I am getting the wrong thing put into my response.

I am guessing that I have configured something wrong so here is my project.clj

(defproject redirector "1.0.0-SNAPSHOT"

  :description "Clojure HTTP redirector"

  :url "http://redirector.herokuapp.com"

  :license {:name "Eclipse Public License v1.0"

            :url "http://www.eclipse.org/legal/epl-v10.html"}

  :dependencies [[org.clojure/clojure "1.6.0"]

                 [compojure "1.1.9"]

                 [ring/ring-jetty-adapter "1.2.2"]

                 [com.novemberain/monger "2.0.0"]

                 [com.taoensso/carmine "2.7.0"]

                 [environ "0.5.0"]

                 [org.clojure/core.async "0.1.346.0-17112a-alpha"]]

  :min-lein-version "2.0.0"

  :plugins [[environ/environ.lein "0.2.1"]]

  :hooks [environ.leiningen.hooks]

  :main "redirector.web"

  :aot :all

  :uberjar-name "redirector-standalone.jar"

  :profiles {:production {:env {:production true}}})

Any ideas, help would be greatly appreciated.

Thanks

Ray

[James Reeves]

Hi Ray,

I don't entirely understand why you expected this to work. Channels aren't a valid Ring response body. The error message is essentially telling you that Compojure has no way of turning the channel object you've returned into a valid response.

The other problem you have is that the Ring Jetty adapter doesn't have any support for asynchronous operations.

Another small point. You're using "*" as an argument name, but this isn't really recommended. This only works by coincidence, and it may be removed in future versions. Instead use something like:

(GET ["/:brand/:country/:resource" :resource #".*"] [brand country resource] ...)

- James

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mond]

Hi James,

Er, nice tip on the routes - that was another thing that I didn't expect to work but was happy when it did ;-) I will of course adapt it to your recommendation.

Speaking to the main point, no I don't want to put a channel on to the response so that's a mistake that I see and would like to avoid. I would like to understand where I have gone wrong.

Maybe I need to use HTTPkit instead?

Thanks

Ray

[James Reeves]

On 27 September 2014 10:01, mond <r...@mcdermott.be> wrote:

Speaking to the main point, no I don't want to put a channel on to the response so that's a mistake that I see and would like to avoid. I would like to understand where I have gone wrong.

You're trying to use core.async with a library that has no understanding of core.async. It's important to understand that core.async isn't magic; it won't work with libraries that weren't designed for it.

 

Maybe I need to use HTTPkit instead?

HTTP Kit has good support for async, but uses its own lightweight protocol. Check out the examples on their website, and if you still want to connect a core.async channel up to HTTP Kit, you'll need something like:

    (async/take! async-ch (fn [msg] (httpkit/send! http-ch msg)))

This tells core.async to take values off the async channel and pass them onto HTTP Kit.

- James

[Ray McDermott]

Thanks for the tips. I wasn't expecting magic but otherwise your points are well taken

Ray

Sent from my iPhone

[Max Penet]

Jet supports what you were trying to do, it accepts a channel as return value to send the response to the client, and in the :body of the response (no matter the kind of response via a channel or map) to trigger chunked responses.

see here https://github.com/mpenet/jet#ring-async

[Ray McDermott]

Excellent - thanks. Nice to see core.async getting the ring middleware love :)

On 28 Sep 2014 at 10:48:57, Max Penet (m...@qbits.cc) wrote:

Jet supports what you were trying to do, it accepts a channel as return value to send the response to the client, and in the :body of the response (no matter the kind of response via a channel or map) to trigger chunked responses.

see here https://github.com/mpenet/jet#ring-async

On Saturday, September 27, 2014 8:08:48 PM UTC+2, mond wrote:

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/gOsd2Zzc1Vk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Mike Fikes]

Hi mond,

I've been using HTTPkit with Compojure and core.async (all fronted by Nginx so the entire stack is async FWIW).

To glue core.async with HTTPkit when handling inbound requests, I have a utility function

(defn handle-async! [handler req]

  (http-server/with-channel req channel

                            (take! (handler req) #(http-server/send! channel %))))

and I make use of that utility function when defining routes

(defroutes all-routes

           (GET "/covers/v1/:cover" [] (partial http/handle-async! fetch-cover!))

           (context "/books/v1" []

                    (GET "/search" [] (partial http/handle-async! search-handler!))

                    (GET "/:isbn" [] (partial http/handle-async! get-book-by-isbn!))))

The handler functions fetch-cover!, search-handler!, get-book-by-isbn!, all return channels (they either call go for stuff that needs to be async, or to-chan on collections read directly from memory.

- Mike